Big Data & Machine Learning Pipelines: A Tale of Lambdas, Kappas and Pancakes
6 likes1,412 views
Osama Khan, a big data engineer, presents an overview of the big data landscape, covering topics like the scale of big data, its ecosystem's evolution, and the architectures that support data pipelines and machine learning. The talk includes a discussion on Lambda and Kappa architectures, technology choices, and practical demonstrations of tools like Athena and EMR. Khan emphasizes the need for a clear understanding of big data and the importance of robust data management in today's technology-driven environment.
![This is BIG …
§ 390 Hyperscale Datacenters ( < 300, 2016)
§ Hyperscale == (5k servers, 10k sq.ft space)
§ > 400, 100M+ total servers
§ 56% web content in English
§ 8,000 languages spoken globally
§ Hello, friend.
§ 100M+ active users, 40M+ subscribers
§ 30M+ songs, 20K new per day, 2B+ playlists, 1B+ plays per day
§ 2,500 node Hadoop cluster, 100 PB+ Disk, 100TB+ RAM
§ 60TB+ per day log ingestion, 20k+ jobs per day
§ Listening History Query
§ user x track x [day/week/month/all time]
§ 300B elements
§ 800 workers, 32 core, 208 GB ram
§ 240TB in, 90TB out
§ Top Tracks in Vancouver (June 2017)
§ 30 date partitioned tables, 60TB data
§ 1 metadata table, 418GB
§ 94.2s, 4.82TB processed
§ Despacito – Remix (Luis Fonsi)
§ (2017)
§ 2.8 B+ US Tweets
§ Donald Trump (901.8 M)
§ Hillary Clinton (123.2 M)
§ Mike Pence (31.4 M)
§ 30x more than VP, 7x more than opponent
(2013)
§ 170M individual metrics (timeseries) per minute
§ 200M queries served/day, 47 charts/user](https://image.slidesharecdn.com/aws-user-group-lambdakappa-180119075741/85/Big-Data-Machine-Learning-Pipelines-A-Tale-of-Lambdas-Kappas-and-Pancakes-5-320.jpg)
![Pancake Stack
[Presto Arrow Nifi Cassandra Airflow Kafka ElasticSearch Spark Tensorflow AlgeBird CoreNLP Kibana]](https://image.slidesharecdn.com/aws-user-group-lambdakappa-180119075741/85/Big-Data-Machine-Learning-Pipelines-A-Tale-of-Lambdas-Kappas-and-Pancakes-30-320.jpg)
Big Data & Machine Learning Pipelines: A Tale of Lambdas, Kappas and Pancakes
- 1. A Tale of Lambdas, Kappas & Pancakes @osamakhn
- 2. Who am I? Osama Khan Big Data Engineer @ACLServices Grad Student @GTComputing AWS Big Data Specialist+ ! Vancouver, BC " : Java " : C# (met thru J#) # : Python $ : Golang, NodeJS % : Scala Previously: Robot Soccer, BI, Credit Rating, AML, O&G Portfolio, NLP/Governance, Doctor Triage, Energy Monitoring, Consulting, Private Equity Recently: Data/ML Pipeline, Tools & Platforms
- 3. What am I going to talk about today? The goal of this talk is to provide a high level overview of the big data landscape to help software engineers distinguish signal from noise 1) How big is BIG: Lets get our scales recalibrated to understand what we mean by BIG Data 2) Lineage: Evolution of the Big Data ecosystem; from EDW to Data Lakes 3) Lambda & Kappa Architectures: The foundation of data pipelines and machine learning systems 4) Technology Choices: SMACK that PANCAKE BUT I ❤ Serverless 5) Demos: Athena, EMR, Redshift, Quicksight, Sagemaker, ModelDB
- 4. How big is BIG? Big is BIG when Bieber breaks the Google Cloud (wat!?!) Lets get our scales recalibrated to understand what we mean by BIG Data
- 5. This is BIG … § 390 Hyperscale Datacenters ( < 300, 2016) § Hyperscale == (5k servers, 10k sq.ft space) § > 400, 100M+ total servers § 56% web content in English § 8,000 languages spoken globally § Hello, friend. § 100M+ active users, 40M+ subscribers § 30M+ songs, 20K new per day, 2B+ playlists, 1B+ plays per day § 2,500 node Hadoop cluster, 100 PB+ Disk, 100TB+ RAM § 60TB+ per day log ingestion, 20k+ jobs per day § Listening History Query § user x track x [day/week/month/all time] § 300B elements § 800 workers, 32 core, 208 GB ram § 240TB in, 90TB out § Top Tracks in Vancouver (June 2017) § 30 date partitioned tables, 60TB data § 1 metadata table, 418GB § 94.2s, 4.82TB processed § Despacito – Remix (Luis Fonsi) § (2017) § 2.8 B+ US Tweets § Donald Trump (901.8 M) § Hillary Clinton (123.2 M) § Mike Pence (31.4 M) § 30x more than VP, 7x more than opponent (2013) § 170M individual metrics (timeseries) per minute § 200M queries served/day, 47 charts/user
- 6. Lineage Evolution of the Big Data ecosystem; from EDW to Data Lakes
- 7. A journey from ETL to Distributed Transactions via the ELT alley… 2007 20172003 20142009 2011201020042000 FaunaDB, Aurora Lambda, Kappa Architectures UC Berkeley Spark Google File System LinkedIn Kafka FB Cassandra AWS DynamoDB Google Dremel 2012: Google Spanner IBM, Oracle, MSFT, Terradata, SAP CAP Theorem Google Map Reduce AWS, GCP, Azure, Hana 2006: Yahoo! Hadoop Occupy the Cloud: Distributed Computing for the 99%
- 8. Big Data Landscape Distributed systems rule the !
- 9. Yet Another Big Data Framework (YABDF) Doesn’t fit on a slide or two … and you thought you had library fatigue in the JS world ! http://mattturck.com/wp-content/uploads/2017/05/Matt-Turck-FirstMark-2017-Big-Data-Landscape.png
- 10. Lambda & Kappa Architectures The foundation of data pipelines for enterprise insights
- 11. Lambda Architecture: First Principles & Desired Properties Data special information from which everything else is derived Information processed data Data System query = function(ALL_DATA) 1. Robustness & Fault Tolerance 2. Low Latency Read & Update 3. Scalability 4. Generalization 5. Ad-hoc 6. Minimal Maintenance 7. Debuggability
- 12. The Lambda Architecture query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer ∑ ALL_DATA Δ NEW_DATA
- 13. The Lambda Architecture (in the enterprise) query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR
- 14. The Lambda Architecture (in the enterprise) query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR
- 15. The Lambda Architecture (in the enterprise) query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR
- 16. The Lambda Architecture (deep dive) Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer 1 2 3 4 5 > home of ‘master data’ > precomputed_batch_view = fn(ALL_DATA) > user_query = fn(precomputed_batch_view)
- 17. The Lambda Architecture (deep dive) Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer 1 2 3 4 5 > (distributed) db to store batch view data > produce fast results for known queries > allow (random) reads by users/systems
- 18. The Lambda Architecture (deep dive) Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer 1 2 3 4 5 > compensate for high latency of batch updates > run fast, incremental algorithms (probabilistic data structures, for the win) > realtime_view = fn(realtime_view, new_data) > user_query = fn(realtime_view) > user_query = fn(precomputed_batch_view)
- 19. The Lambda Architecture (deep dive) Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer 1 2 3 4 5 > user_query = fn( precomputed_batch_view, realtime_view)
- 20. The Lambda Architecture (ingest? speedlayer?) Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer 1 2 3 4 5
- 21. The Lambda Architecture (ingest? speedlayer?) Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer 1 2 3 4 5
- 22. The Kappa Architecture Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer 1 2 3 4 5
- 23. The Lambda Architecture (SMACK) Data Catalog Impact Analysis Data Lineage Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer 1 2 3 4 5
- 24. Big Picture of Metadata Management for Data Governance Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer lineage impact analysis semantic lineage Enterprise Vocabulary Semantic Mapping (metadata harvesting) (metadata stitching)
- 25. Big Picture of Metadata Management for Data Governance Access Control Secure Data Rest GDPR query query new data master dataset batch view batch view real-time view real-time view batch layer serving layer speed layer lineage impact analysis semantic lineage Enterprise Vocabulary Semantic Mapping
- 26. Machine Learning Pipelines BAML PIT
- 27. Machine Learning Pipelines BAML PITBAML PIT == $$100MM Blockchain based Adversarial Machine Learning Platform for IoT Testing
- 28. classic model
- 29. classic model
- 30. Pancake Stack [Presto Arrow Nifi Cassandra Airflow Kafka ElasticSearch Spark Tensorflow AlgeBird CoreNLP Kibana]
- 31. data science silo Data Source Data & Feature Engineering Adaptation of slide by Ben Lorica Model Building Deploy Monitor
- 33. what’s changing(-ed)? 1. Cloud (faas, serverless data pipelines, ml-as-a-service) 2. Consumer demand for ML features/products/applications 3. Targeted Models (we need to manage 20MM models for 10MM users maybe) 4. Localization (ASEAN facial recognition) 5. Security (Adv. ML, Side-channel attacks) 6. Transparency (Bias is a BUG) 7. Many toy sophisticated solutions but conventional, simpler techniques (regression) still deliver more business value! 8. Monitoring to ensure deployed models are making high quality predictions 9. Need practices to maintain (update or rebuild) models over time 10. and ….
- 34. feature engineering, wat? By @MLpuppy
- 35. rise of machine learning engineers
- 36. rise of machine learning engineers
- 37. Online Machine Learning Pipeline Model Inventory Model Output Monitoring Take Action ML Serving Layer Hyper-parameter Tuning