Making advanced analytics accessible to more companies
0 likes159 views
The document discusses making advanced analytics more accessible to companies. It proposes using BigQuery, Google's serverless data warehouse, which allows storing and querying large datasets cost effectively without needing to manage infrastructure. Data is ingested via pipelines into BigQuery from various sources and then business analysts can easily build custom reports and dashboards in tools like Tableau without requiring developers. BigQuery reduces the time to insights from days to minutes by handling all the challenges of large-scale analytics.
Making advanced analytics accessible to more companies
- 1. Making advanced analytics accessible to more companies Márton Kodok / @martonkodok Google Developer Expert at REEA.net, Targu Mures, Romania 24 May 2017 Tirgu Mures, Romania Issue 59 - May 2017
- 2. ● Geek. Hiker. Do-er. ● Crafting Web/Mobile backends at REEA.net Targu Mures ● Among the TOP3 romanians on Stackoverflow.com ● Google Developer Expert on Cloud technologies ● BigQuery and database engine expert ● Active in mentoring Twitter: @martonkodok StackOverflow: pentium10 Slideshare: martonkodok GitHub: pentium10 Making advanced analytics accessible to more companies @martonkodok About me
- 3. Making advanced analytics accessible to more companies @martonkodok Agenda The Challenge Making advanced analytics accessible to more companies Architecture Overview Strategy & Tricks Winning Solution
- 4. Companies: ❏ must be able to identify, combine, and manage multiple sources of data ❏ should have the ability to obtain advanced analytics using concepts they are familiar with. ❏ have a deployment of the right technology architecture matching their capabilities. Making advanced analytics accessible to more companies @martonkodok 3 principles to get from small data to BigData
- 5. Making advanced analytics accessible to more companies @martonkodok Legacy Business Reporting System Web Mobile Web Server Database SQL Cached Platform Services CMS/Framework Report & Share Business Analysis Scheduled Tasks Batch Processing Compute Engine Multiple Instances
- 6. Making advanced analytics accessible to more companies @martonkodok Web Mobile Web Server Database SQL Cached Platform Services CMS/Framework Report & Share Business Analysis Scheduled Tasks Batch Processing Compute Engine Multiple Instances BehindtheScenes: DaysToInsights
- 7. Making advanced analytics accessible to more companies @martonkodok Legacy Business Reporting System Web Mobile Web Server Database SQL Cached Platform Services CMS/Framework Report & Share Business Analysis Scheduled Tasks Batch Processing Compute Engine Multiple Instances Minutes to kick in Hours to Run Batch Processing Hours to Clean and Aggregate DAYS TO INSIGHTS
- 8. ❏ Need backend/database to STORE, QUERY, EXTRACT data ❏ Deep analytics - large, multi-source, complex, unstructured ❏ Be real time ❏ Terabyte scale - Cost effective ❏ Run Ad-Hoc reports - Without Developer - interactive ❏ Minimal engineering efforts - no dedicated BigData team ❏ Simple Query language (prefered SQL / Javascript) Making advanced analytics accessible to more companies @martonkodok Desired system
- 9. Making advanced analytics accessible to more companies @martonkodok
- 10. ● Analytics-as-a-Service - Data Warehouse in the Cloud ● Fully-Managed by Google (US or EU zone) ● Scales into Petabytes ● Ridiculously fast ● SQL 2011 Standard + Javascript UDF (User Defined Functions) ● Familiar DB Structure (table, views, record, nested, JSON) ● Integrates with Tableau, Google Sheets + Cloud Storage + Pub/Sub connectors ● Decent pricing (queries $5/TB, storage: $20/TB cold: $10/TB) *May 2017 Making advanced analytics accessible to more companies @martonkodok What is BigQuery?
- 11. Making advanced analytics accessible to more companies @martonkodok Architecting for The Cloud BigQuery On-Premises Servers Pipelines ETL Engine Event Sourcing Frontend Platform Services Metrics / Logs/ Streaming
- 12. Making advanced analytics accessible to more companies @martonkodok Data Pipeline Integration Analytics Backend BigQuery On-Premises Servers Pipelines FluentD Event Sourcing Frontend Platform Services Metrics / Logs/ Streaming Development Team Data Analysts Report & Share Business Analysis Tools Tableau QlikView Data Studio Internal Dashboard Database SQL Application ServersServers Cloud Storage archive Load Export Replay Standard Devices HTTPS
- 13. Making advanced analytics accessible to more companies @martonkodok <filter frontend.user.*> @type record_transformer enable_ruby remove_keys host <record> bq {"insert_id":"${uid}","host":"${host}","created":"${time.to_i}"} </record> </filter> <match frontend.user.*> @type copy <store> @type forest subtype file <template> path /tank/storage/${tag}.*.log time_slice_format %Y%m%d time_slice_wait 10m </template> </store> <store> @type bigquery method insert ... </store> </match> ….bigquery section continued…. auth_method json_key json_key /etc/td-agent/keys/key-31da042be48c.json project project_id dataset dataset_name time_field timestamp time_slice_format %Y%m%d table user$%{time_slice} ignore_unknown_values schema_path /etc/td-agent/schema/user_login.json 1 2 3 4
- 14. ● On data that it is difficult to process/analyze using traditional databases ● On exploring unstructured data ● Not a replacement to traditional DBs, but it compliments the system ● Applying Javascript UDF on columnar storage to resolve complex tasks (eg: JS for natural language processing) ● On streams (form wizard ...) ● On IoT streams ● Major strength is handling Large datasets Making advanced analytics accessible to more companies @martonkodok Where to use BigQuery?
- 15. ● no manual sharding ● no capacity guessing ● no idle resources ● no manual scaling ● no provisioning/deploy/running out of resources ● run raw ad-hoc queries (either by analysts/sales) ● no more throwing away-, expiring-, aggregating old data. Making advanced analytics accessible to more companies @martonkodok BigQuery Benefits: Serverless Data Warehouse
- 16. Making advanced analytics accessible to more companies @martonkodok Easily Build Custom Reports and Dashboards
- 17. Thank you. Slides available on: slideshare.net/martonkodok Making advanced analytics accessible to more companies @martonkodok