ML Production Pipelines: A Classification Model
0 likes369 views
Data Insights is a Munich-based startup providing agile data consultancy, focusing on big data, data science, and cloud project implementation. The document outlines their journey from proof of concept to production, detailing their development process and the role of Databricks and MLflow in establishing a robust machine learning architecture. Key achievements include creating a shared production environment, implementing security measures, and ensuring quality checks in data processing.
ML Production Pipelines: A Classification Model
- 2. © Data Insights, 2020 2 ML PRODUCTION PIPELINES: A CLASSIFICATION MODEL Python, MLflow, and Git for ML in Production
- 3. © Data Insights, 2020 Agenda 1. About Adri & Data Insights 2. How everything started 3. Our Approach 4. How databricks helped 5. Challenges and improvements
- 4. © Data Insights, 2020 4 About me Adriana Menegozzo Senior Data Scientist Data Insights GmbH Tumblingerstr. 12 D-80337 München E-Mail: adriana@datainsights.de
- 5. © Data Insights, 2020 5 DATA INSIGHTS: WHAT WE DO 01 – DATA INSIGHTS We cooperate with premium partners, but also offer technology-independent consulting to individually address a wide range of requirements. Tailored Solutions Data Insights is an agile, dynamic and fast-growing start-up company offering IT data consultancy based in Munich. Our experts advise and support you comprehensively in the implementation of Big Data, Data Science & Cloud projects. Our range of services covers the entire life cycle. We support our customers - from the initial idea to the implementation and operation. From Idea to ProductionStrategy & TechnologyAgile
- 6. 6© Data Insights, 2020 WHEN EVERYTHING STARTED….. 01
- 7. © Data Insights, 2020 7 Classification model § Predictive Maintenance § Churn Prediction § Product Recommendation WHEN EVERYTHING STARTED…..
- 8. © Data Insights, 2020 8 TO GO BAD ….
- 9. © Data Insights, 2020 9 Proof of Concept • Single Jupyter notebook • Fairly small Dataset • Local environment HOW EVERYTHING STARTED
- 10. © Data Insights, 2020 10 From PoC to Production what do we need? • constant development • Software development standards • Shared environment • Monitoring system • Model Versioning • Security standards HOW EVERYTHING STARTED
- 11. 11© Data Insights, 2020 OUR APPROACH.. 02
- 12. © Data Insights, 2020 12 From notebook to library HOW EVERYTHING STARTED Configurations via YAML ML code structure Library structure
- 13. © Data Insights, 2020 13 Great achievement! Structure code into Functions Add requirements Versioning Testable Runnable scripts HOW DATABRICKS HELPED …
- 14. © Data Insights, 2020 14 Something is still missing… • Shared environment • Security layer for production data • production-ready architecture • CI/CD HOW EVERYTHING STARTED Wait a minute!
- 15. 15© Data Insights, 2020 HOW DATABRICKS HELPED … 03
- 16. © Data Insights, 2020 16 HOW DATABRICKS HELPED … Our Architecture
- 17. © Data Insights, 2020 17 HOW DATABRICKS HELPED … Connect Git to Databricks
- 18. © Data Insights, 2020 18 Our Pipelines HOW DATABRICKS HELPED …
- 19. © Data Insights, 2020 19 Great achievement! Shared environment Security layer for production data Constant quality checks on data CI/CD HOW DATABRICKS HELPED …
- 20. © Data Insights, 2020 20 Something is still missing… • Tracking of ML experiments • Model versioning • Model storage • Model serving HOW DATABRICKS HELPED … Wait another minute!
- 21. 21© Data Insights, 2020 MLFLOW AND MODEL SERVING 03
- 22. © Data Insights, 2020 22 Model registry MLFLOW AND MODEL SERVING
- 23. © Data Insights, 2020 23 What we achieved! • Read the production model from S3 • Create features and force input schema • Have an up to date cluster • Using simple pipelines Next steps: • Alerting system • Orchestration • Use the new one node cluster MLFLOW AND MODEL SERVING That’s cool!
- 24. Data Insights GmbH ׀ Tumblingerstr. 12 ׀ 80337 München ׀ Germany ׀ +49 (0) 89 24 21 74 44 ׀ info@datainsights.de Thank You! Adriana Menegozzo 24
- 25. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.