{"id":3711,"date":"2022-02-02T11:45:47","date_gmt":"2022-02-02T11:45:47","guid":{"rendered":"https:\/\/wiki.thomasandsofia.com\/?p=3711"},"modified":"2022-02-02T11:49:26","modified_gmt":"2022-02-02T11:49:26","slug":"bqfbd-overview","status":"publish","type":"post","link":"https:\/\/wiki.thomasandsofia.com\/?p=3711","title":{"rendered":"BQFBD &#8211; Overview"},"content":{"rendered":"<p><a href=\"\/bigquery-for-big-data\/\">Main Menu<\/a><\/p>\n<h1>Section 1: Intro to GCP and its services<\/h1>\n<p>&nbsp;<\/p>\n<h1>Section 2: Intro to BigQuery<\/h1>\n<h2>8. Conventional Data Warehouse Problems<\/h2>\n<h2>9. What is BigQuery<\/h2>\n<p>BigQuery is a fully managed, serverless, highly scalable and cost-effective cloud Data Warehouse designed for business agility.<\/p>\n<ul>\n<li>Both Batch and Streaming data ingestion\n<ul>\n<li>Can store 100,000 rows per second<\/li>\n<li>TB of batch data per second<\/li>\n<\/ul>\n<\/li>\n<li>Supports AI and ML\n<ul>\n<li>BigQuery ML<\/li>\n<li>Integration with the AI Platform\n<ul>\n<li>Prediction and TensorFlow<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>Full managed<\/li>\n<li>Scalability<\/li>\n<li>Pay as you go\n<ul>\n<li>Pay separately for storage and compute<\/li>\n<li>Pay for bytes that your query processes<\/li>\n<li>Results cached, so no need to pay for same query 2x<\/li>\n<\/ul>\n<\/li>\n<li>Automated data transfer\n<ul>\n<li>Fully managed data transfer<\/li>\n<li>Transfer from Teradata and S3 to BigQuery<\/li>\n<\/ul>\n<\/li>\n<li>Access control\n<ul>\n<li>Use IAM<\/li>\n<li>Assign read-write, running jobs, etc. per project.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>10. OOB Features<\/h2>\n<p>https:\/\/www.udemy.com\/course\/bigquery\/learn\/lecture\/22717593#overview<\/p>\n<ul>\n<li>BQ GIS\n<ul>\n<li>Geographic Information System<\/li>\n<li>Obtain insights from geographic data points using Long\/Lat<\/li>\n<\/ul>\n<\/li>\n<li>Auto Backup\n<ul>\n<li>7 days<\/li>\n<\/ul>\n<\/li>\n<li>Integration with other GCP\n<ul>\n<li>DataProc<\/li>\n<\/ul>\n<\/li>\n<li>Foundation for BI\n<ul>\n<li>Seamless integration, transformation, analysis, visualization<\/li>\n<\/ul>\n<\/li>\n<li>Programmatic Interaction\n<ul>\n<li>REST API<\/li>\n<li>Libraries in Java, Python, Node.js, c#, Go, Ruby and PHP<\/li>\n<\/ul>\n<\/li>\n<li>Security\n<ul>\n<li>At rest and transit<\/li>\n<li>Each data block encrypted with different keys<\/li>\n<\/ul>\n<\/li>\n<li>Logging, Monitoring and alerting\n<ul>\n<li>Cloud Audit Logs<\/li>\n<\/ul>\n<\/li>\n<li>Federated queries\n<ul>\n<li>Process data in Object Storage\n<ul>\n<li>Parquet, ORC, Open source<\/li>\n<\/ul>\n<\/li>\n<li>Process transactional databases\n<ul>\n<li>BigTable, Cloud SQL, spreadsheets in Drive\n<ul>\n<li>You can pull data directly from a CSV file&#8230;<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>Data Science Workloads\n<ul>\n<li>Spark, TensorFlow, scikit-learn<\/li>\n<li>No need to have multiple copies of the same data<\/li>\n<\/ul>\n<\/li>\n<li>Powerful data repository<\/li>\n<\/ul>\n<h2>11. Architecture of BigQuery<\/h2>\n<p>https:\/\/www.udemy.com\/course\/bigquery\/learn\/lecture\/22717627#overview<\/p>\n<ul>\n<li>Engine &#8211; Dremel\n<ul>\n<li>Combination of columnar data layouts and tree architecture<\/li>\n<\/ul>\n<\/li>\n<li>File system &#8211; Colossus\n<ul>\n<li>Columnar storage, Google&#8217;s distributed filesystem<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><a href=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2022\/02\/bq-arch.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-3712\" src=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2022\/02\/bq-arch.png\" alt=\"\" width=\"1115\" height=\"378\" srcset=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2022\/02\/bq-arch.png 1115w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2022\/02\/bq-arch-300x102.png 300w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2022\/02\/bq-arch-1024x347.png 1024w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2022\/02\/bq-arch-768x260.png 768w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2022\/02\/bq-arch-150x51.png 150w\" sizes=\"auto, (max-width: 1115px) 100vw, 1115px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Main Menu Section 1: Intro to GCP and its services &nbsp; Section 2: Intro to BigQuery 8. Conventional Data Warehouse Problems 9. What is BigQuery BigQuery is a fully managed, serverless, highly scalable and cost-effective cloud Data Warehouse designed for business agility. Both Batch and Streaming data ingestion Can store 100,000 rows per second TB ..<\/p>\n<div class=\"clear-fix\"><\/div>\n<p><a href=\"https:\/\/wiki.thomasandsofia.com\/?p=3711\" title=\"read more...\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[77,78],"tags":[],"class_list":["post-3711","post","type-post","status-publish","format-standard","hentry","category-bigquery","category-bigquery-for-big-data-engineers"],"_links":{"self":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts\/3711","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3711"}],"version-history":[{"count":3,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts\/3711\/revisions"}],"predecessor-version":[{"id":3717,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts\/3711\/revisions\/3717"}],"wp:attachment":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3711"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3711"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3711"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}