Introduction to Google BigQuery
Download as PPTX, PDF1 like1,009 views
This document provides an introduction to Google BigQuery, a cloud-based data warehouse that allows users to interactively query and analyze massive datasets. It begins with background on big data and technologies like Hadoop, Hive, and Spark. It then explains the differences between row-based and column-based data stores, with BigQuery using a columnar approach. The rest of the document demonstrates BigQuery through an example query on public datasets and provides pricing and resource information.
Introduction to Google BigQuery
- 1. Cloud Csaba Toth Presented By: Introduction to Google BigQuery
- 2. Our sponsors
- 3. Disclaimer
- 5. Goal âą Being able to issue queries âą Preferably in an SQL dialect âą Over Big Data âą As small response time as possible âą Preferably interactive web interface (thus no need to install anything)
- 6. Agenda âą Big Data âą Brief look at Hadoop, HIVE and Spark âą Row based data store vs. Column data store âą Google BigQuery âą Demo
- 7. Big Data Wikipedia: âcollection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applicationsâ Examples: (Wikibon - A Comprehensive List of Big Data Statistics) âą 100 Terabytes of data is uploaded to Facebook every day âą Facebook Stores, Processes, and Analyzes more than 30 Petabytes of user generated data âą Twitter generates 12 Terabytes of data every day âą LinkedIn processes and mines Petabytes of user data to power the "People You May Know" feature âą YouTube users upload 48 hours of new video content every minute of the day âą Decoding of the human genome used to take 10 years. Now it can be done in 7 days
- 8. Little Hadoop history âThe Google File Systemâ - October 2003 âą http://labs.google.com/papers/gfs.html â describes a scalable, distributed, fault-tolerant file system tailored for data-intensive applications, running on inexpensive commodity hardware, delivers high aggregate performance âMapReduce: Simplified Data Processing on Large Clustersâ - April 2004 âą http://queue.acm.org/detail.cfm?id=988408 â describes a programming model and an implementation for processing large data sets.
- 9. Hadoop âą Hadoop is an open-source software framework that supports data- intensive distributed applications âą A Hadoop cluster is composed of a single master node and multiple worker nodes
- 10. Hadoop Has two main services: 1. Storing large amounts of data: HDFS â Hadoop Distributed File System 2. Processing large amounts of data: implementing the MapReduce programming model
- 11. HDFS Name node Metadata Store Data node Data node Data node Node 1 Node 2 Block A Block B Block A Block B Node 3 Block A Block B
- 12. Job / task management Name node Heart beat signals and communication Jobtracker Data node Data node Data node Task- tracker Task- tracker Map 1 Reduce 1 Map 2 Reduce 2 Task- tracker Map 3 Reduce 3 Map-Reduce
- 13. Hadoop vs. RDBMS Hadoop / MapReduce RDBMS Size of data Petabytes Gigabytes Integrity of data Low High (referential, typed) Data schema Dynamic Static Access method Batch Interactive and Batch Scaling Linear Nonlinear (worse than linear) Data structure Unstructured Structured Normalization of data Not Required Required Query Response Time Has latency (due to batch processing) Can be near immediate
- 14. Apache Hive Log Data RDBMS Data Integration LayerFlume Sqoop Storage Layer (HDFS): row and columnar data, file data Computing Layer (MapReduce) Advanced Query Engines (Hive, Pig) Data Mining (Pegasus, Mahout) Index, Searches (Lucene) DB drivers (Hive driver) GUI (web interface, RESTful API, JavaScript) System management Distribution coordination (Zookeeper) JDBC ODBC JS
- 15. Apache Hive UI
- 16. Apache Hive UI
- 17. Beyond Apache Hive Goals: decrease latency âą YARN: the ânext generation Hadoopâ, improves performance in many respects (resource management and allocation, âŠ) âą Hadoop distribution specific solution: e.g. Cloudera Impala, MPP SQL Query engine, based on Hadoop
- 18. Apache Spark âą Cluster computing framework with multi- stage in-memory primitives âą Open Source, originates from Berkeley âą In contrast to Hadoopâs two-stage disk- based MapReduce paradigm, multi-stage in-memory primitives can provide up to 100x performance increase âą Requires YARN and HDFS
- 19. Spark and Hadoop
- 20. Spark and Hadoop
- 21. Storing data: row stores âą Traditional RDBMS and often the document stores are row oriented too âą The engine stores and retrieves rows from disk (unless indexes help) âą Row is a collection of column cell values together âą Rows are materialized on disk
- 22. Row stores Row cells are stored together on disk id scientist death_by movie_name 1 Reinhardt Maximillian The Black Hole 2 Tyrell Roy Batty Blade Runner 3 Hammond Dinosaur Jurassic Park 4 Soong Lore Star Trek: TNG 5 Morbius His mind Forbidden Planet 6 Dyson Skynet Terminator 2: Judgment Day
- 23. Row stores âą Not so great for wide rows âą If only a small subset of columns queried, reading the entire row wastes IO âą (Indexing strategies can help but I donât have time to cover them)
- 24. Row stores Bad case scenario: âą select sum(bigint_column) from table âą Million rows in table âą Average row length is 1 KiB The select reads one bigint column (8 bytes) âą Entire row must be read âą Reads ~1 GiB data for ~8MiB of column data
- 25. Column stores âą Data is organized by columns instead of rows âą Non material world: often not materialized during storage, exists only in memory âą Each row still has some sort of ârow idâ
- 26. Column stores âą A row is a collection of column values that are associated with one another âą Associated: every row has some type of ârow idâ âą Can still produce row output (assembling a row maybe complex though â under the hood)
- 27. Column stores Stores each COLUMN on disk id 1 2 3 4 5 6 title Mrs. Doubtfire The Big Lebowski The Fly Steel Magnolias The Birdcage Erin Brokovitch actor Robin Williams Jeff Bridges Jeff Goldblum Dolly Parton Nathan Lane Julia Roberts genre Comedy Comedy Horror Drama Comedy Drama row id = 1 row id = 6 Natural order may be unusual Each column has a file or segment on disk
- 28. Column stores ⹠Column compression can be way more efficient than row based compression (sometimes 10:1 to 30:1 ratio) ⹠Compression: RLE, Integer packing, dictionaries and lookup, other⊠⹠Reduces both storage and IO (thus response time)
- 29. Column stores Best case scenario: âą select sum(bigint_column) from table âą Million rows in table âą Average row length is 1 KiB The select reads one bigint column (8 bytes) âą Only single column read from disk âą Reads ~8 MiB of column data, even less with compression
- 30. Column stores Bad case scenario: select * from long_wide_table where order_line_id = 34653875; âą Accessing all columns doesnât save anything, could be even more expensive than row store âą Not ideal for tables with few columns
- 31. Column stores Updating and deleting rows is expensive âą Some column stores are append only âą Others just strongly discourage writes âą Some split storage into row and column areas
- 32. Column / Row stores âą RDBMS provide ACID capabilities âą Row stores mainly use tree style indexes âą B-tree derivative index structure provides very fast binary search as long as it fits into memory âą Very large datasets end up unmanageably big indexes âą Column stores: bitmap indexing Very expensive to update
- 33. BigQuery history âDremel: Interactive Analysis of Web-Scale Datasetsâ â 2010, describes a column store / retrieval system âą https://static.googleusercontent.com/media/research.goo gle.com/en//pubs/archive/36632.pdf Presentation with illustration about principles used in Dremel, from Google âą http://www.cs.berkeley.edu/~istoica/classes/cs294/11/not es/12-sameer-dremel.pdf
- 34. BigQuery âą A service that enables interactive analysis of massively large datasets âą Based on Dremel, a scalable, interactive ad hoc query system for analysis of read- only nested data âą Working in conjunction with Google Storage âą Has a RESTful web service interface
- 35. BigQuery âą You can issue SQL queries over big data âą Interactive web interface âą As small response time as possible âą Auto scales under the hood
- 36. BigQuery SaaS (/ PaaS) Interfacing: âą REST API âą Web console âą Command line tools âą Language libraries Insert only
- 37. Demo! Wikipedia public dataset Natalities public dataset Names (uploaded) Google Genomics https://cloud.google.com/genomics/ https://cloud.google.com/genomics/v1/public-data https://cloud.google.com/bigquery/web-ui-quickstart https://cloud.google.com/bigquery/query-reference
- 38. Future thoughts How to visualize data âą Possibly using Google Charts âą BigQuery alongside Google Maps Playing with genomics data â requires some bio-informatics knowledge
- 40. Our sponsors
- 41. Resources âą Slides: http://www.slideshare.net/tothc âą Contact: http://www.meetup.com/CCalJUG/ âą Csaba Toth: Introduction to Hadoop and MapReduce - http://www.slideshare.net/tothc/introduction-to-hadoop- and-map-reduce âą Justin Swanhart: Introduction to column stores - http://www.slideshare.net/MySQLGeek/intro-to-column- stores âą Jan Steemann: Column-oriented databases - http://www.slideshare.net/arangodb/introduction-to- column-oriented-databases
- 42. Resources âą https://anonymousbi.wordpress.com/2012/11/02/hadoo p-beginners-tutorial-on-ubuntu/ âą https://www.capgemini.com/blog/capping-it- off/2012/01/what-is-hadoop âą http://blog.iquestgroup.com/en/hadoop/#.Vgg2w2sRMeI âą https://www.cloudera.com/content/cloudera/en/docume ntation/core/latest/PDF/cloudera-impala.pdf âą https://www.keithrozario.com/2012/07/google-bigquery- wikipedia-dataset-malaysia-singapore.html âą https://cloud.google.com/bigquery/web-ui-quickstart âą https://cloud.google.com/bigquery/query-reference
- 43. Resources âą https://github.com/googlegenomics/getting-started-bigquery âą https://github.com/googlegenomics/bigquery-examples âą https://github.com/googlegenomics/readthedocs
- 44. Pricing https://cloud.google.com/bigquery/pricing Storage: $0.020 per GB/mo. Queries: $5 per TB processed Streaming inserts: $0.01 per 200 MiB (1 KiB rows) Columns are compressed but price is based on the uncompressed size
- 45. Tips âą Storage dominates the costs âą Plus you need to restrict queries, use as few columns as possible âą BigQuery scans the full columns which are involved in the query âą Do not select * âą LIMIT the result set âą Result caching