Why more than half of ML models don't make it to production
0 likes82 views
The document outlines the challenges faced by data science teams in deploying machine learning models into production, highlighting that 87% of projects never reach this stage due to issues like team friction, miscommunication, and lack of executive buy-in. It emphasizes the need for streamlined processes between data science and engineering teams, as well as the importance of starting with a minimum viable product and iterating for improvement. The text also points out that a low time to production can significantly impact business outcomes, leading to increased revenue.
Why more than half of ML models don't make it to production
- 2. whoami ● Solutions Architect @ cnvrg.io ● = built by data scientists, for data scientists to help teams: ○ Get from data to models to production in the most efficient and fast way ○ Bridge science and engineering ○ Automate MLOps ○ Help teams streamline every element of their pipelines Aaron Schneider aaron@cnvrg.io LinkedIn: azschneider
- 3. def agenda(30 mins): ● Understanding the problems and symptoms ● Discussing the root causes ● Best practices for closing the gap ● Tools to streamline the production process
- 4. Deployment ML.overview() Data Processing Training MonitorDeployment Deployment Deployment
- 5. production.statistics() https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/ Algorithmia: 2020 state of enterprise machine learning 87% of ML projects don’t make it to production 55% of companies haven’t deployed a single model
- 6. production.statistics() https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/ Algorithmia: 2020 state of enterprise machine learning 50% of models take 8-90 days before production 13% take 91-365 days Investment in ML is increasing
- 7. Model != in_production ● Good work being done by good data scientists ● Functional and accurate models being made ● But not being used ● If being used, often take a long time before implemented ● Undermines the whole process as ML not being used or no longer accurate ● Enterprise problem impacting business results ● Need to accelerate time from research into production
- 8. production_issue(team_friction) ● Multiple teams: ○ Data Science Team (Development) ○ Engineering Team (DevOps) ● Data Science can build the model but engineering can’t (won’t?) put it into production ● Causes: ○ Miscommunication ○ Developed in Python but implement in C/Java ○ Mismanagement
- 9. production_issue(iteration) ● You can’t reach desired accuracy through one grid search. Have to continually improve the model ● Can get stuck in the grind towards the perfect model ● Management wants higher accuracy before production ● Like any software/agile method: Start with MVP ● Build a model and get it to production, THEN monitor and iterate ● Already puts you leaps and bounds ahead of competitors Accuracy = 62% Version 2 Accuracy = 60% Version 1 Accuracy = 64% Version 3
- 10. production_issue(containerizing) ● How do we even implement a model? What technology do we use? ● Model developed separately from product, how do we align? ● Containerization is a logical answer ● Tough to implement properly ● Links back to first issue we discussed
- 11. production_issue(scaling) ● Demand isn’t static ● Need complicated infrastructure to support the service you are delivering ● Must be able to scale without interruption to service ● Kubernetes is great, but hard ● Need whole different set of skills to build the architecture for autoscaling with k8
- 12. production_issue(executive_buy_in) ● Might have the best DS team in the world ● But if higher-ups push back, all a waste of time ● Need the management and executive on board with the vision ● Might be excited by the flashiness of ML/AI/Big Data but need to take risks and believe in the DS teams capability
- 13. cnvrg.demo()
- 14. webinar.summary() ● There are many issues that keep models from production ● Very rarely are they DS issues, usually team friction and engineering complexities ● It is the responsibility of the DevOps and Management teams to build a structure that can minimize time to production ● This is a real metric that can have real business consequences, both positively and negatively ● Low time to production = $$$ ● High time to production = -$$$