How we are using BigQuery and Apps Scripts at teowaki
0 likes1,036 views
The document discusses the use of BigQuery and Apps Script for data analytics at Teowaki, highlighting features like non-intrusive metrics, cost-efficiency, and real-time processing of large datasets. It outlines the pricing structure for both BigQuery and cloud storage, which makes it accessible for handling extensive data. Additionally, various SQL query examples and automation strategies using Apps Script are provided.
How we are using BigQuery and Apps Scripts at teowaki
- 1. javier ramirez @supercoco9 How we are using BigQuery and Apps Scripts at teowaki
- 6. Set a distance. Set an expiration time. Bye bye noise.
- 12. Analytics flow
- 13. Analytics flow, by segment
- 14. Automatic Alerts
- 18. javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14 REST API (Ruby on Rails) + Web on top (AngularJS)
- 19. javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 20. data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the structures of your database architectures. Ed Dumbill program chair for the O’Reilly Strata Conference javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 21. 1. non intrusive metrics 2. keep the history 3. avoid vendor lock-in 4. interactive queries 5. cheap 6. extra ball: real time javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 22. javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 23. Cloud Storage: Cost-efficient storage of files javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 24. javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 25. Hadoop Cassandra Amazon Redshift ... javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14 tools we considered:
- 26. Our choice: Google BigQuery Data analysis as a service http://developers.google.com/bigquery javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 27. Based on “Dremel” Specifically designed for interactive queries over petabytes of real-time data javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 28. loading data You just send the data in text (or JSON) format javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 29. SQL javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14 select name from USERS order by date; select count(*) from users; select max(date) from USERS; select sum(total) from ORDERS group by user;
- 30. specific extensions for analytics javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14 within flatten nest stddev top first last nth variance var_pop var_samp covar_pop covar_samp quantiles
- 31. web console screenshot javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 32. javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14 our most active user
- 33. javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14 country segmented traffic
- 34. javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14 10 request we should be caching
- 35. javier ramirez @supercoco9 http://teowaki.com startup launch summit london 14 5 most created resources
- 36. new users per month
- 37. SELECT repository_name, repository_language, repository_description, COUNT(repository_name) as cnt, repository_url FROM github.timeline WHERE type="WatchEvent" AND PARSE_UTC_USEC(created_at) >= PARSE_UTC_USEC("#{yesterday} 20:00:00") AND repository_url IN ( SELECT repository_url FROM github.timeline WHERE type="CreateEvent" AND PARSE_UTC_USEC(repository_created_at) >= PARSE_UTC_USEC('#{yesterday} 20:00:00') AND repository_fork = "false" AND payload_ref_type = "repository" GROUP BY repository_url ) GROUP BY repository_name, repository_language, repository_description, repository_url HAVING cnt >= 5 ORDER BY cnt DESC LIMIT 25
- 41. Automation with Apps Script Read from bigquery Create a spreadsheet on Drive E-mail it everyday as a PDF javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 47. cloud storage pricing $0.032 per GB a gzipped 4.8 MB file stores 1MM rows $0.000092 / month per 1MM rows javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 48. bigquery pricing $26 per stored TB 1000000 rows => $0.00416 / month £0.00243 / month $5 per processed TB 1 full scan = 160 MB 1 count = 0 MB 1 full scan over 1 column = 5.4 MB 100 GB => $0.05 / month £0.03javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 49. £0.054307 / month* per 1MM rows *the 1st 100GB every month are free of charge javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 50. 1. non intrusive metrics 2. keep the history 3. avoid vendor lock-in 4. interactive queries 5. cheap 6. extra ball: real time javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
- 51. ig
- 52. Find related links at https://teowaki.com/teams/javier-community/link-categories/bigquery-talk Thanks! Javier Ramírez @supercoco9 startup launch summit london 14