Scaling Production Machine Learning Pipelines with Databricks
1 like483 views
Condé Nast uses machine learning across various applications like recommendations, personalization, advertising, and content understanding. They developed an MLOps platform called Spire to manage the machine learning lifecycle including datasets, modeling, inference, and tracking. Spire includes components like Aleph for feature engineering, Kalos for model training and deployment, and utilizes technologies like Astronomer, Databricks, and APIs to execute workflows and integrate with other systems. Condé Nast aims to improve their platform over time by enhancing models and features as well as consolidating libraries.
Scaling Production Machine Learning Pipelines with Databricks
- 1. Machine Learning Infrastructure at Condé Nast James Evers & Max Cantor
- 5. 110M+ Monthly Unique Visitors 800M+ Monthly Page Views 100K+ Monthly New Or Revised Pieces of Content Digital Scale at Condé Nast
- 6. Landscape of MLOps - Datasets
- 7. Landscape of MLOps - Datasets - Modeling
- 8. Landscape of MLOps - Datasets - Modeling - Inference
- 9. Landscape of MLOpsLandscape of MLOps - Datasets - Modeling - Inference - Tracking
- 10. MLOps Across the Industry - Michelangelo & Ludwig (Uber) - FBLearner (Facebook) - Noether (Spotify) - Bighead (Airbnb) - Metaflow (Netflix)
- 12. Motivations - Recommendations & Personalization - Advertisement & User Segmentation - Content Understanding - Audience Forecasting - Search Optimization
- 13. Challenges ● Facts of the media industry ○ low on-site times (engagement and bounce rate) ○ one and done’s (cold start) ○ Dependence on social media ● Business vs Engineering interests ● Legacy infrastructure requirements ● Staffing & overall technical resources
- 14. Features Labeled Data Dataprep Spire Workflow Stages Model Train Scores Predict Thresholds
- 15. Aleph - User-level behavioral feature space - Content-derived feature generation - Vetted feature pipelines for models Features Labeled Data
- 16. Aleph
- 17. - Model interface standardization - Model serialization - Model versioning & tracking - Hyperparameter tuning Model Kalos
- 18. Kalos
- 20. Execution Environments - Astronomer / Airflow - Databricks - Local Development
- 21. Spire Command-Line Interface - Development - User Queries - Models - Execution
- 22. API Wrappers - Bridges Airflow Operators and Spark Jobs - Close integration with Databricks APIs - Development iteration, speed, and stability in processes
- 23. - Spire versioning - Spire releases - GitHub/repository - CI/CD Release Management
- 24. Future - Model reduction - Model architecture enhancement - Extend abstraction flexibility to feature sets - In-house library consolidation
- 25. Thank you!