Zipline: Airbnb’s Machine Learning Data Management Platform with Nikhil Simha and Andrew Hoh
10 likes5,531 views
Zipline is Airbnb's machine learning data management framework. It handles feature engineering, discovering and accessing data sources, generating training sets, and monitoring data quality. Zipline includes a feature store, training set generation, and clients to access features and training data. It uses various data sources like Hive tables and streams data in and handles backfilling and mutations to training data. Zipline aims to make machine learning processes more scalable, robust, and transparent at Airbnb.
Zipline: Airbnb’s Machine Learning Data Management Platform with Nikhil Simha and Andrew Hoh
- 1. Nikhil Simha, Airbnb Andrew Hoh, Airbnb Zipline – Airbnb’s ML Data Management Framework #ML3SAIS
- 2. Feature Engineering 2#ML3SAIS Idea Discover Data Training Set Training & Evaluation Airbnb Zipline
- 3. Discovering data 3#ML3SAIS • Where is it? • Is it good? • Will it continue to be good? • Has it already been done before?
- 4. Training set 4#ML3SAIS • What processing engine to use? • Backfilling • Point-in-time correct windows
- 5. Production 5#ML3SAIS Training Set Generation Model Training Push model to prediction service Online Features Model monitoring Data quality monitoring
- 6. What is Zipline? • Part of Bighead – Airbnb’s e2e ml platform • Training Sets • Feature Store • Data quality 6#ML3SAIS
- 7. Concepts • Data Source • Feature-set • Training-set • Client 7#ML3SAIS
- 8. Feature-Set • Source queries – Hive tables – Event streams – Mutation streams • Primary Keys • Time Stamp 8#ML3SAIS
- 9. Feature-Set – Sources • Multiple sources – Backfills – Migrations 9#ML3SAIS
- 11. 11#ML3SAIS The picture can't be displayed. Data Quality
- 12. Training set • Driver query – primary keys – timestamps – Point-in-time correct 12#ML3SAIS
- 13. Training set – example 13#ML3SAIS
- 14. Training set – example 14#ML3SAIS
- 17. Online scoring • Backfills • Low latency • DB Mutations – Batch correction 17#ML3SAIS
- 18. Lambda • Batch – Spark • Streaming – Flink • GDPR 18#ML3SAIS User Conf Batch Streaming Zipline Client User App *Daily *Continuous KV Store
- 21. • Across batch and streaming • Fixed length sliding windows • Raw Events at the tail and head Aggregations 21#ML3SAIS 7 day sliding window
- 23. Client 23#ML3SAIS • Aggregation • Logic to handle availability • Java – The API is (List of Feature Names) => Map of feature values – as objects.
- 24. Mutations • Organizationally easy • Consistent • Not flexible • More complex – Inverse 24#ML3SAIS
- 25. Mutations - Aggregation 25#ML3SAIS • Monoids – Associative: (a + b) + c = a + (b + c) – Distribute aggregation – Sum, Avg, Count etc.. • Group – Invertible – Min, Max, Median, Ntile – Not possible without memory – Compromise
- 26. Mutations - Aggregation 26#ML3SAIS • Deletes = Inverse(before) • Updates = Inverse(before) + after
- 28. When? • Plan to open source in Q3 2018. • Will be part of Bighead – Talk tomorrow. 28#ML3SAIS