Executive Intro to BigQuery

Executive Intro to BigQuery
William M. Cohee
November 2017
Prepared using Apache OpenOffice 4.1.4

Presenter Bio
● 15+ years of Wall Street Technology
experience
● Expertise in front-office Fixed Income
Systems, Analytics, Pricing, Instrument,
& Entity Reference Data Management
● BA, Computer Science
● MS, Information Systems Engineering
● Certified Bloomberg Specialist
● Currently in the Chief Data Office
@ HSBC
● www.linkedin.com/in/billcohee

Topic
● Gaining insights from data becoming a strategic imperative
● Amazon, Microsoft, IBM, and Google aggressively competing for
your data and your $$
● Google's offering is the 'Google Cloud Platform' (GCP)
● Google considers BigQuery to be a game changing differentiator
● BigQuery is disruptive
➢ super scalable & super fast - runs at 'Google Speed'
➢ inexpensive – no charge to store. Pay for what you query
➢ easy to use – No MapReduce programming. Works with SQL
● BigQuery lets you and your data teams focus on analyzing Big Data

Agenda
● What is BigQuery [slides 5-7]
● How does BigQuery work [slides 8-10]
● BigQuery in Action: query from an R script [slides 11-14]
● Recap & where to learn more [slides 15-16]

BigQuery – What is it?
● In BigQuery, Google is monetizing core technology assets used
& perfected internally for over a decade
● A Cloud-powered, massively paralleled, interactive query service
● Provides extremely fast performance on extremely large datasets
● Can search billion row datasets in seconds using basic SQL
● BigQuery unleashes the processing power of Google on your Big Data

● Serverless, Fully Managed
➢ no provisioning, no capacity planning. No need for a DBA or an SA
➢ takes the costly Big Data support burden off of your IT organization
● Unlimited Storage – Google makes it cheap & easy to put data into it
● REST API – Google makes it easy to integrate and securely access
your data
➢ interfaces for R, Python, C#/.NET, Java, Google Sheets, Excel, etc
➢ integrate popular Visualization tools such as Tableau, Qlik, & Looker

● Data is automatically encrypted, distributed, & replicated in Google
Data Centers
➢ state-of-the-art Biometric access controls
● No indexes – every query is a Full Scan!
● With BigQuery, Full Scans are actually intended & inexpensive
● Google is able to achieve ludicrous performance by leveraging two
core technical innovations it relies on every day to run itself...

BigQuery – How does it work?
● The primary technologies behind BigQuery
➢ Colossus
➢ Dremel
● Colossus is Google's next generation Distributed File System
➢ lets Google treat a cluster of computers as one big virtual disk
➢ horizontally scalable with inexpensive, commodity hardware
➢ sequential I/O bottlenecks are avoided using parallelism
➢ seeks/read operations are optimized – makes searching very fast
➢ trivia: Colossus was the name of the World War II super-computer
that helped Cryptologists crack the German Enigma

● Dremel is an interactive query engine designed for interrogating
'web-scale' datasets using standard SQL syntax
➢ 'web-scale' pre-dates the term 'Big Data' – that's how mature
Dremel technology is
● Dremel uses Columnar Storage
➢ table data stored by columns instead of by rows
➢ optimal for reading large amounts of data
➢ affords better compression and more efficient use of disk
➢ drastically reduces read operations and memory required for
queries – only the columns requested need to be scanned

● Dremel employs a query tree to dispatch the fetching of data
➢ exploits Google's Cloud-infrastructure to execute queries
➢ the work of data retrieval is distributed with massive parallelism
➢ results are aggregated with blazing speed
➢ the tree architecture works very well on tabular data stored as
columns across a massively, parallel distributed file system
● The unique combination of Colossus, Dremel, and Google's Cloud
are what make awesome BigQuery performance possible

BigQuery in Action
● As a simple demonstration of BigQuery's power, we will run a small
R script to query the free, public Wikipedia dataset
● This sample dataset contains 313,797,035 rows
● Our query will look at all 2016 revisions to Wikipedia pages to find
the ID, title, and comment for all edits made to pages with 'HSBC' in
the title

BigQuery in Action [ lines 1-8 ]

BigQuery in Action [ lines 9-22 ]

BigQuery in Action [ script output ]
BigQuery scanned 313+ million rows and returned 5,936
results in less than 2.5 seconds

Recap & Resources
● BigQuery exposes the power of Colossus, Dremel, and Google's
massive computational infrastructure to everyone outside of Google
● BigQuery is super fast, cost-effective, & easy to use for analyzing
Big Data using standard SQL. An open REST API makes it
accessible from popular Data Science and Visualization platforms.
● True NoOps model lets teams stay focused on Big Data Analytics
● In the Amazon-Microsoft-IBM-Google cloud wars, Google may have
a big edge with BigQuery...

Recap & Resources
● Where to learn more...
● BigQuery homepage: https://cloud.google.com/bigquery/
● Getting Started with GCP: https://cloud.google.com/getting-started
● Jordan Tigani's BigQuery 101 presentation: https://youtu.be/kKBnFsNWwYM
● BigQuery Whitepaper: https://goo.gl/PBfXX6
● Hadley Wickham's bigrquery R package: https://goo.gl/EXQ3jd
● Coursera course: https://goo.gl/YDxjEv
● BigQuery Best Practices: https://goo.gl/iLYvkE
● HSBC Group CIO Darryl West presenting at Google Next Cloud:
https://youtu.be/esqArNcxaao
● Google Data Center Security: https://goo.gl/UVD2jW

Executive Intro to BigQuery

More Related Content

What's hot (20)

Similar to Executive Intro to BigQuery (20)

Recently uploaded (20)

Executive Intro to BigQuery