Big Data with BigQuery, presented at DevoxxUK 2014 by Javier Ramirez from teowaki
2 likes2,063 views
Javier Ramirez discusses Google BigQuery as a big data management solution, emphasizing its benefits for handling large datasets and real-time queries. He highlights the ease of loading data, executing SQL queries, and the cost structure for processing and storage. Additionally, he provides examples of query execution and analytics capabilities, including specific functionalities such as window functions and automating tasks with Apps Script.
![@supercoco9#DevoxxBigquery
Subselect and joins out of the box
SELECT Year, Actor1Name, Actor2Name, Count FROM (
SELECT Actor1Name, Actor2Name, Year, COUNT(*) Count, RANK() OVER(PARTITION BY YEAR ORDER BY
Count DESC) rank
FROM
(SELECT Actor1Name, Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name < Actor2Name
and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode),
(SELECT Actor2Name Actor1Name, Actor1Name Actor2Name, Year FROM [gdelt-bq:full.events] WHERE
Actor1Name > Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and
Actor1CountryCode!=Actor2CountryCode),
WHERE Actor1Name IS NOT null
AND Actor2Name IS NOT null
GROUP EACH BY 1, 2, 3
HAVING Count > 100
)
WHERE rank=1
ORDER BY Year
http://gdeltproject.org/data.html#googlebigquery](https://image.slidesharecdn.com/bigquerydevoxxgoogleteowaki-140613072254-phpapp02/85/Big-Data-with-BigQuery-presented-at-DevoxxUK-2014-by-Javier-Ramirez-from-teowaki-15-320.jpg)
![Things you always wanted to try but were too
scared to
select count(*) from
publicdata:samples.wikipedia
where REGEXP_MATCH(title, "[0-9]*")
AND wp_namespace = 0;
223,163,387 Query complete (5.6s elapsed, 9.13 GB processed, Cost: 32¢)](https://image.slidesharecdn.com/bigquerydevoxxgoogleteowaki-140613072254-phpapp02/85/Big-Data-with-BigQuery-presented-at-DevoxxUK-2014-by-Javier-Ramirez-from-teowaki-17-320.jpg)
![@supercoco9#DevoxxBigquery
Worldwide events in the last 36 years
SELECT Year, Actor1Name, Actor2Name, Count FROM (
SELECT Actor1Name, Actor2Name, Year, COUNT(*) Count, RANK() OVER(PARTITION BY YEAR ORDER BY
Count DESC) rank
FROM
(SELECT Actor1Name, Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name < Actor2Name
and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode),
(SELECT Actor2Name Actor1Name, Actor1Name Actor2Name, Year FROM [gdelt-bq:full.events] WHERE
Actor1Name > Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and
Actor1CountryCode!=Actor2CountryCode),
WHERE Actor1Name IS NOT null
AND Actor2Name IS NOT null
GROUP EACH BY 1, 2, 3
HAVING Count > 100
)
WHERE rank=1
ORDER BY Year
http://gdeltproject.org/data.html#googlebigquery](https://image.slidesharecdn.com/bigquerydevoxxgoogleteowaki-140613072254-phpapp02/85/Big-Data-with-BigQuery-presented-at-DevoxxUK-2014-by-Javier-Ramirez-from-teowaki-24-320.jpg)
Big Data with BigQuery, presented at DevoxxUK 2014 by Javier Ramirez from teowaki
- 1. @supercoco9#devoxxBigQuery Big Data with Google BigQuery Javier Ramirez @supercoco9 https://teowaki.com
- 2. @supercoco9#DevoxxBigquery Managing Big Data with BigQuery Javier Ramirez •Writing software since 1996 •Web dev. since 1999 (C++, JAVA, PHP, Ruby, JS...) •Founder of https://teowaki.com •Google Developer Expert on the Cloud Platform
- 7. bigdata is cool but... hard to set up and monitor expensive cluster not interactive enough
- 8. @supercoco9#DevoxxBigquery bigdata is doing a fullscan to 330MM rows, matching them against a regexp, and getting the result (223MM rows) in just 5 seconds
- 9. Google BigQuery Data analysis as a service http://developers.google.com/bigquery
- 10. Based on “Dremel” Specifically designed for interactive queries over petabytes of real-time data
- 11. @supercoco9#DevoxxBigquery Your only worries •Load data •Query the dataset
- 12. loading data. You just send the data in text (or JSON) format
- 13. up to 100K inserts per second in stream mode
- 14. It's just SQL select name from USERS order by date; select count(*) from users; select max(date) from USERS; select sum(total) from ORDERS group by user;
- 15. @supercoco9#DevoxxBigquery Subselect and joins out of the box SELECT Year, Actor1Name, Actor2Name, Count FROM ( SELECT Actor1Name, Actor2Name, Year, COUNT(*) Count, RANK() OVER(PARTITION BY YEAR ORDER BY Count DESC) rank FROM (SELECT Actor1Name, Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name < Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode), (SELECT Actor2Name Actor1Name, Actor1Name Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name > Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode), WHERE Actor1Name IS NOT null AND Actor2Name IS NOT null GROUP EACH BY 1, 2, 3 HAVING Count > 100 ) WHERE rank=1 ORDER BY Year http://gdeltproject.org/data.html#googlebigquery
- 16. @supercoco9#DevoxxBigquery specific extensions for analytics within flatten nest stddev top first last nth variance var_pop var_samp covar_pop covar_samp quantiles correlations
- 17. Things you always wanted to try but were too scared to select count(*) from publicdata:samples.wikipedia where REGEXP_MATCH(title, "[0-9]*") AND wp_namespace = 0; 223,163,387 Query complete (5.6s elapsed, 9.13 GB processed, Cost: 32¢)
- 19. highly distributed execution using a tree
- 22. @supercoco9#DevoxxBigqueryjavier ramirez @supercoco9 https://teowaki.com window functions
- 24. @supercoco9#DevoxxBigquery Worldwide events in the last 36 years SELECT Year, Actor1Name, Actor2Name, Count FROM ( SELECT Actor1Name, Actor2Name, Year, COUNT(*) Count, RANK() OVER(PARTITION BY YEAR ORDER BY Count DESC) rank FROM (SELECT Actor1Name, Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name < Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode), (SELECT Actor2Name Actor1Name, Actor1Name Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name > Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode), WHERE Actor1Name IS NOT null AND Actor2Name IS NOT null GROUP EACH BY 1, 2, 3 HAVING Count > 100 ) WHERE rank=1 ORDER BY Year http://gdeltproject.org/data.html#googlebigquery
- 27. SELECT repository_name, repository_language, repository_description, COUNT(repository_name) as cnt, repository_url FROM github.timeline WHERE type="WatchEvent" AND PARSE_UTC_USEC(created_at) >= PARSE_UTC_USEC("#{yesterday} 20:00:00") AND repository_url IN ( SELECT repository_url FROM github.timeline WHERE type="CreateEvent" AND PARSE_UTC_USEC(repository_created_at) >= PARSE_UTC_USEC('#{yesterday} 20:00:00') AND repository_fork = "false" AND payload_ref_type = "repository" GROUP BY repository_url ) GROUP BY repository_name, repository_language, repository_description, repository_url HAVING cnt >= 5 ORDER BY cnt DESC LIMIT 25
- 31. @supercoco9#DevoxxBigquery Automation with Apps Script ● Read from BigQuery ● Create a spreadsheet on Drive ● E-mail it everyday as a PDF https://developers.google.com/apps-script/
- 35. @supercoco9#DevoxxBigquery bigquery pricing $26 per stored TB 1000000 rows => $0.00416 / month £0.00243 / month $5 per processed TB 1 full scan = 160 MB 1 count = 0 MB 1 full scan over 1 column = 5.4 MB 100 GB => $0.05 / month £0.03 AppsScripts is for free
- 36. @supercoco9#DevoxxBigquery £0.054307 / month* per 1MM rows *the 1st 1TB every month is free of charge **assumming your rows have web server logs-like info price per month
- 38. @YourTwitterHandle#DVXFR14{session hashtag} @supercoco9#devoxxBigquery TH A N K S ! Javier Ramirez @supercoco9 https://teowaki.com Related links at: https://teowaki.com/teams/javier-community/link-categories/bigquery-talk
- 39. @supercoco9#DevoxxBigquery Thanks / Creative Commons •Presentation Template — Guillaume LaForge •The Queen — A prestigious heritage with some inspiration from The Sex Pistols and funny Devoxxians •Girl with a Balloon — Banksy •Tube — Michael Keen