Big Data Analytics with Google BigQuery. GDG Summit Spain 2014
0 likes740 views
This document discusses using Google BigQuery for big data analytics. BigQuery is a serverless warehousing data analytics service that allows interactive analysis of petabytes of data. It uses columnar storage and a distributed execution tree to enable fast queries over large datasets. The document provides examples of loading and querying data from public datasets as well as tips for using BigQuery's SQL functions, views, and JSON fields.
![Things you always wanted to
try but were too scared to
select count(*) from
publicdata:samples.wikipedia
where REGEXP_MATCH(title, "[0-9]*")
AND wp_namespace = 0;
223,163,387
Query complete (5.6s elapsed, 9.13 GB processed, Cost: 32¢)
javier ramirez @supercoco9 https://teowaki.com](https://image.slidesharecdn.com/gdgsummitsantiago2014analyticsredisgooglebigquery-140929123245-phpapp01/85/Big-Data-Analytics-with-Google-BigQuery-GDG-Summit-Spain-2014-17-320.jpg)
![SELECT Year, Actor1Name, Actor2Name, Count FROM (
SELECT Actor1Name, Actor2Name, Year,
COUNT(*) Count, RANK() OVER(PARTITION BY YEAR ORDER BY
Count DESC) rank
FROM
(SELECT Actor1Name, Actor2Name, Year FROM
[gdelt-bq:full.events] WHERE Actor1Name < Actor2Name
and Actor1CountryCode != '' and Actor2CountryCode != ''
and Actor1CountryCode!=Actor2CountryCode),
(SELECT Actor2Name Actor1Name, Actor1Name Actor2Name,
Year FROM [gdelt-bq:full.events] WHERE
Actor1Name > Actor2Name and Actor1CountryCode != '' and
Actor2CountryCode != '' and
Actor1CountryCode!=Actor2CountryCode),
WHERE Actor1Name IS NOT null
AND Actor2Name IS NOT null
GROUP EACH BY 1, 2, 3
HAVING Count > 100
)
WHERE rank=1
ORDER BY Year](https://image.slidesharecdn.com/gdgsummitsantiago2014analyticsredisgooglebigquery-140929123245-phpapp01/85/Big-Data-Analytics-with-Google-BigQuery-GDG-Summit-Spain-2014-21-320.jpg)
Big Data Analytics with Google BigQuery. GDG Summit Spain 2014
- 1. Big Data Analytics with Google BigQuery javier ramirez @supercoco9
- 2. REST API + AngularJS web as an API client javier ramirez @supercoco9 https://teowaki.com nosqlmatters 2013
- 3. javier ramirez @supercoco9 https://teowaki.com
- 4. data that’s an order of magnitude greater than data you’re accustomed to javier ramirez @supercoco9 https://teowaki.com Doug Laney VP Research, Business Analytics and Performance Management at Gartner
- 5. data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the structures of your database architectures. Ed Dumbill program chair for the O’Reilly Strata Conference javier ramirez @supercoco9 https://teowaki.com
- 6. bigdata is doing a fullscan to 330MM rows, matching them against a regexp, and getting the result (223MM rows) in just 5 seconds javier ramirez @supercoco9 https://teowaki.com Javier Ramirez impresionable teowaki founder
- 7. bigdata is cool but... expensive cluster hard to set up and monitor not interactive enough
- 8. Our choice: Google BigQuery Data analysis as a service http://developers.google.com/bigquery javier ramirez @supercoco9 https://teowaki.com
- 9. Based on Dremel Specifically designed for interactive queries over petabytes of real-time data javier ramirez @supercoco9 https://teowaki.com
- 10. What Dremel is used for in Google • Analysis of crawled web documents. • Tracking install data for applications on Android Market. • Crash reporting for Google products. • OCR results from Google Books. • Spam analysis. • Debugging of map tiles on Google Maps. • Tablet migrations in managed Bigtable instances. • Results of tests run on Google’s distributed build system. • Disk I/O statistics for hundreds of thousands of disks. • Resource monitoring for jobs run in Google’s data centers. • Symbols and dependencies in Google’s codebase.
- 11. Columnar storage javier ramirez @supercoco9 https://teowaki.com
- 12. highly distributed execution using a tree javier ramirez @supercoco9 https://teowaki.com rubyc kiev 14
- 13. loading data You can feed flat CSV-like files or nested JSON objects javier ramirez @supercoco9 https://teowaki.com
- 14. bq cli bq load --nosynchronous_mode --encoding UTF-8 --field_delimiter 'tab' --max_bad_records 100 --source_format CSV api.stats 20131014T11-42-05Z.gz javier ramirez @supercoco9 https://teowaki.com
- 15. web console screenshot javier ramirez @supercoco9 https://teowaki.com
- 16. analytical SQL functions. correlations. window functions. views. JSON fields. timestamped tables. javier ramirez @supercoco9 https://teowaki.com
- 17. Things you always wanted to try but were too scared to select count(*) from publicdata:samples.wikipedia where REGEXP_MATCH(title, "[0-9]*") AND wp_namespace = 0; 223,163,387 Query complete (5.6s elapsed, 9.13 GB processed, Cost: 32¢) javier ramirez @supercoco9 https://teowaki.com
- 19. SELECT repository_name, repository_language, repository_description, COUNT(repository_name) as cnt, repository_url FROM github.timeline WHERE type="WatchEvent" AND PARSE_UTC_USEC(created_at) >= PARSE_UTC_USEC("#{yesterday} 20:00:00") AND repository_url IN ( SELECT repository_url FROM github.timeline WHERE type="CreateEvent" AND PARSE_UTC_USEC(repository_created_at) >= PARSE_UTC_USEC('#{yesterday} 20:00:00') AND repository_fork = "false" AND payload_ref_type = "repository" GROUP BY repository_url ) GROUP BY repository_name, repository_language, repository_description, repository_url HAVING cnt >= 5 ORDER BY cnt DESC LIMIT 25
- 20. Global Database of Events, Language and Tone quarter billion rows 30 years updated daily http://gdeltproject.org/data.html#googlebigquery
- 21. SELECT Year, Actor1Name, Actor2Name, Count FROM ( SELECT Actor1Name, Actor2Name, Year, COUNT(*) Count, RANK() OVER(PARTITION BY YEAR ORDER BY Count DESC) rank FROM (SELECT Actor1Name, Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name < Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode), (SELECT Actor2Name Actor1Name, Actor1Name Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name > Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode), WHERE Actor1Name IS NOT null AND Actor2Name IS NOT null GROUP EACH BY 1, 2, 3 HAVING Count > 100 ) WHERE rank=1 ORDER BY Year
- 23. our most active user javier ramirez @supercoco9 https://teowaki.com
- 24. 10 request we should be caching javier ramirez @supercoco9 https://teowaki.com
- 25. 5 most created resources select uri, count(*) total from stats where method = 'POST' group by URI; javier ramirez @supercoco9 http://teowaki.com
- 26. ...but /users/javier/shouts /users/rgo/shouts /teams/javier-community/links /teams/nosqlmatters-cgn/links javier ramirez @supercoco9 http://teowaki.com
- 27. 5 most created resources javier ramirez @supercoco9 http://teowaki.com
- 28. javier ramirez @supercoco9 https://teowaki.com
- 30. Automation with Apps Script Read from bigquery Create a spreadsheet on Drive E-mail it everyday as a PDF javier ramirez @supercoco9 https://teowaki.com
- 36. bigquery pricing $80 per stored TB 300000 rows => $0.007629392 / month $35 per processed TB 1 full scan = 84 MB 1 count = 0 MB 1 full scan over 1 column = 5.4 MB 10 GB => $0.35 / month *the 1st TB every month is free of charge javier ramirez @supercoco9 https://teowaki.com
- 37. Find related links at https://teowaki.com/teams/javier-community/link-categories/bigquery-talk Grazas Javier Ramírez @supercoco9