Building a Self-Service Big Data Pipeline
13 likes2,617 views
Autodesk has built a large self-service big data pipeline to process large amounts of data from their various products and services on a daily basis. The pipeline ingests raw data, indexes it, aggregates and summarizes it over time, and makes it available to business users through various reporting and analytics tools. It processes over 2 billion transactions per day from many different data sources totaling over 800 terabytes of data.
Building a Self-Service Big Data Pipeline
- 1. © 2015 Autodesk Building a Self-Service Big Data Pipeline Charlie Crocker Business Analytics Program Lead Hadoop Summit, San Jose – June 2015
- 3. © 2015 Autodesk Multi-core & GPU Cloud Distributed Computing Reality Capture Model Sophistication Variations Data Compute
- 5. © 2015 Autodesk BIG DATA PIPELINE DETAILS
- 6. © 2015 Autodesk 0 1 0 1 1 0 1 1 0 0 1 0 0 0 1 0 1 0 1 1 1 0 1 1 1 0 0 0 1 0 1 1 0 1 1 0 0 1 0 0 0 1 0 1 0 1 1 1 0 1 1 1 0 0 © 2014 Autodesk CONSISTENT TRUSTED ACCESSIBLE INSTRUMENT COLLECT CONSUMEPROCESSORGANIZE
- 7. © 2015 Autodesk Production Big Data Pipeline Stats • Core Services • 360 Products/Services • Desktop Products • Operations Data • 2.1 billion transactions/day • 350 source types • 750-800 GB indexed daily • 165(+) active Users • 800 Terabytes total • 90 GB/day • 350 S3 Aggregations • 128 Tableau Desktop • 57 Tableau Server • 25 Datameer Users • 10 Qlikview Dashboards • 150 QV Users • >80 GBQ Tables Analytics & Reports Batch Oriented Business, Product & Customer Behavior Monitoring & Discovery Realtime Products, Platform & Infrastructure 1 month Indexed Data Data Gathering All analytics & debug data Raw service data 1 week Raw data Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Web Services 1 year (+) Aggregated & Summarized data Curated Data Product & Business Analysis Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Profile QlikView 1 year (+) Aggregated & summarized data ADSKDashboar d
- 8. © 2015 Autodesk Example: Specific Service Calls Over 60 million/day
- 10. © 2015 Autodesk Example: Desktop Analytics Managed Source: Trusted Consistent Accessible 3.1M Users/Wk
- 11. © 2015 Autodesk Production Big Data Pipeline Teams Engage Forward to Kafka Apply Log Schema Forward to Hadoop Define Cubes Deploy Cubes Publish Data & Explore Analytics & Reports Batch Oriented Business, Product & Customer Behavior Monitoring & Discovery Realtime Products, Platform & Infrastructure 1 month Indexed Data Data Gathering All analytics & debug data Raw service data 1 week Raw data Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Web Services 1 year (+) Aggregated & Summarized data Curated Data Product & Business Analysis Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Profile QlikView 1 year (+) Aggregated & summarized data ADSKDashboar d
- 12. © 2015 Autodesk Production Big Data Pipeline Teams Engage Forward to Kafka Apply Log Schema Forward to Hadoop Define Cubes Deploy Cubes Publish Data & Explore SLOW DOWN SLOW DOWN SLOW DOWN SLOW DOWN Analytics & Reports Batch Oriented Business, Product & Customer Behavior Monitoring & Discovery Realtime Products, Platform & Infrastructure 1 month Indexed Data Data Gathering All analytics & debug data Raw service data 1 week Raw data Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Web Services 1 year (+) Aggregated & Summarized data Curated Data Product & Business Analysis Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Profile QlikView 1 year (+) Aggregated & summarized data ADSKDashboar d
- 13. © 2015 Autodesk Production Big Data Pipeline Teams Engage Forward to Hadoop Define Cubes Deploy Cubes Publish Data & Explore SLOW DOWN SLOW DOWN Analytics & Reports Batch Oriented Business, Product & Customer Behavior Monitoring & Discovery Realtime Products, Platform & Infrastructure 1 month Indexed Data Data Gathering All analytics & debug data Raw service data 1 week Raw data Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Web Services 1 year (+) Aggregated & Summarized data Curated Data Product & Business Analysis Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Profile QlikView 1 year (+) Aggregated & summarized data ADSKDashboar d Forward to Kafka Apply Log Schema Onboard faster: Transition to Services
- 14. © 2015 Autodesk Production Big Data Pipeline Teams Engage Forward to Kafka Apply Log Schema Forward to Hadoop Define Cubes Deploy Cubes Publish Data & Explore Deliver value faster: Streamlined Access Onboard faster: Transition to Services Analytics & Reports Batch Oriented Business, Product & Customer Behavior Monitoring & Discovery Realtime Products, Platform & Infrastructure 1 month Indexed Data Data Gathering All analytics & debug data Raw service data 1 week Raw data Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Web Services 1 year (+) Aggregated & Summarized data Curated Data Product & Business Analysis Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Profile QlikView 1 year (+) Aggregated & summarized data ADSKDashboar d
- 15. © 2015 Autodesk TRANSITION TO SERVICES
- 16. © 2015 Autodesk Tools Fragmented Architecture Manual ingestion (Kafka) Dashboard POCs Production Scaling Services Architecture Alignment Managed Ingestion (CSE) ADSK Dashboard Framework
- 17. © 2015 Autodesk . highly available . secure . massively scalable . insanely high volume . cloud ops infrastructure Build Services
- 18. © 2015 Autodesk . easy to consume sdks . simple data contracts . self service onboarding . fault tolerant sdks Make Services Ridiculously Easy
- 19. © 2015 Autodesk Fast Access Layer Client SDKs Data Portal Analytics as a Service API Access Cross Service Eventing Metadata Management Analytics Tools Scoring Pipeline Dashboard Framework Other Services + Scaleable Compute Workflow Management Ingestion Injection
- 20. © 2015 Autodesk Platform Services Detail Desktop (Windows, Mac, Linux) Mobile (iOS, Android, Windows) Web (Chrome, Explorer, Safari, etc.) Client MPA Service Cloud Services Explore/Publish Datameer API Access Data Virtualization (EDW) Denodo Batch Processing (Hive Cluster) Fast Access Google BigQuery, Red Shift, Spark, QVD Reporting Tableau, Qlikview, Dashboards Core Services Traditional Data Warehouses Back Office (SAP, Siebel, etc.) Enterprise Data Lake: Storage (S3) Query Processing (Hive Cluster) CSE (Ingestion) Injector Govern Enterprise Data Lake: Metadata
- 21. © 2015 Autodesk STREAMLINED DATA ACCESS
- 22. © 2015 Autodesk Analytics Consumers Non-Technical Users 1000s 10s Business Analyst Data Analyst Data Scientists Analytics Ops © 2014 Autodesk • Excel like • Easy to access • Medium to small data set • Easy to display • Easy to aggregate • Handle large data • Data visualization • Integration with other tools• Connection with other data source • Handle unstructured data • Combine data from multiple sources
- 23. © 2015 Autodesk Self-Service Explore, Aggregation and Publish Non-technical users need to quickly explore, create, and publish aggregations from the data lake and visualize the results in their tool of choice. Analytics & Reports Batch Oriented Business, Product & Customer Behavior Monitoring & Discovery Realtime Products, Platform & Infrastructure 1 month Indexed Data Data Gathering All analytics & debug data Raw service data 1 week Raw data Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Web Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Unified Customer Profile QlikView Web Services 1 year (+) Aggregated & Summarized data Curated Data Product & Business Analysis Monitoring & Discovery Realtime Products, Platform & Infrastructure Data Gathering All analytics & debug data Raw service data Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 week Raw data 1 month Indexed data 1 year (or more) Aggregated & summarized data Services Infrastructure Hardware Pla) orms Network Security Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Kafka Profile QlikView 1 year (+) Aggregated & summarized data ADSKDashboar d
- 24. © 2015 Autodesk One Source, Multiple Access Points Daily push to S3 buckets and REST API Google Big Query or Redshift Access Tableau Server (GBQ) Qlikview (REST, QVDs) ADSK Dashboards (S3) Datameer (S3) Hive (EMR and S3) Data Products Early Warning System Syndicated Video Wall Executive Daily Reports Personalized Product Experiences Analytics & Reports Batch Oriented Business, Product & Customer Behavior Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 year (or more) Aggregated & summarized data Business & Transactional ODS SAP Subscrip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Unified Customer Profile QlikView 1 year (+) Aggregated & Summarized data Curated Data Product & Business Analysis Analytics & Reports Batch oriented Business, Product & Customer Behavior 1 year (or more) Aggregated & summarized data ess & actional ODS AP crip: on Product/Business Analysis Interactive & Focused Any amount needed Product Group Data Other... Metrics GA Data Cube Unified Customer Profile QlikView 1 year (+) Aggregated & summarized data ADSKDashboar d
- 25. © 2015 Autodesk From this
- 26. © 2015 Autodesk To this
- 27. © 2015 Autodesk Datameer: Big Data Analytics for Hadoop Wizard-led Data Integration No ETL 70+ Connectors + plug-in API Smart Sampling Point-and-click Analytics Spreadsheet UI 270+ pre-built functions Visual Data Profiling Drag-and-Drop Visualization 30+ Visualization Widgets HTML5 support View on any device
- 28. © 2015 Autodesk Datameer: Create Standard Aggregations Parse JSON from S3 Join to account data Process using EMR compute Output directly to S3 Output directly to Tableau Server Couple hours instead of 5 weeks waiting for engineering sprint
- 31. © 2015 Autodesk 0 1 0 1 1 0 1 1 0 0 1 0 0 0 1 0 1 0 1 1 1 0 1 1 1 0 0 0 1 0 1 1 0 1 1 0 0 1 0 0 0 1 0 1 0 1 1 1 0 1 1 1 0 0 © 2014 Autodesk CONSISTENT TRUSTED ACCESSIBLE INSTRUMENT COLLECT CONSUMEPROCESSORGANIZE
- 32. © 2015 Autodesk We’re Hiring! Data Geeks (Scientists?) Data Analysts Data Engineers Charlie.Crocker@autodesk.com
- 33. Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document. © 2015 Autodesk, Inc. All rights reserved.
Editor's Notes
- #8: Ingest 2.1 billion event/day 200 (+) data sources flowing into production Splunk 600-650 GB indexed daily Process 800 Terabytes of active data in Enterprise Data Lake (EDL) 90 GB/day entering the EDL 300 S3 Aggregations updating daily, with replicated in 50 GBQ Tables Consume >100 Tableau Desktop and 50 Tableau Server users Feeding 8 Qlikview dashboards with 150 active QV users Feeding the Early Warning System
- #11: Global Reach Fast ramp up Managed Data Source What does it mean to be managed Owner Pipeline Metadata Stays current
- #13: Interactive & Focused
- #14: Interactive & Focused
- #20: API access, workflow management (Oozie), support for data streaming and machine learning.