“Houston, we have a model...” Introduction to MLOps
1 like352 views
The document introduces MLOps (Machine Learning Operations) and the need to operationalize machine learning models beyond just model deployment. It discusses challenges like data and model drift, retraining models, software dependencies, monitoring models in production, and the need for automation, testing, and reproducibility across the full machine learning lifecycle from data to deployment. An example MLOps workflow is shown using GitHub and Azure ML to enable experiment tracking, automation, and continuous integration and delivery of models.
“Houston, we have a model...” Introduction to MLOps
- 1. Rui Quintino AI & Analytics Strategy Advisor DevScope 20210219 “Houston, we have a model...” Introduction to MLOps
- 2. devscope.net
- 3. We have a model... It’s great! YOU ARE HERE
- 4. Now what?
- 5. We just deploy it!
- 7. What about the data Pipeline/preproc?
- 8. Or Data Drift -> causing Model Performance degradation? https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/work-with-data/datadrift-tutorial/datadrift-tutorial.ipynb
- 9. Plus... • How/who can we retrain if needed (ex: data drift)? • What are the software requirements? • How can we later update lib/os versions? (ex: security) • How can we monitor model performance in production? • How do we ensure data checks in production? • How was the model trained? What was the performance? • How to scale? And Reproduce? • ...
- 10. Model & ML code just tip of the iceberg...
- 11. Model & ML code just tip of the iceberg... https://cloud.google.com/solutions/machine-learning/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
- 12. Introducing MLOps – Machine Learning & Operations
- 13. DevOps MLOps Code testing Code reproducibility App deployment Model retraining Model validation Model reproducibility Model deployment
- 14. Build app Collaborate Test app Deploy app Monitor app App developer
- 15. Model reproducibility Model retraining Model deployment Model validation Train model Validate model Deploy model Monitor model Retrain model Build app Collaborate Test app Release app Monitor app Audit /interpretability App developer Data scientist
- 16. GitHub & Azure ML MLOps Demo
- 22. Lets try a new hyper param configuration...
- 27. Compute, Experiment, Model, Deployment in Azure ML
- 43. Tracking critical for MLOps, but also for Experimenting
- 44. With GitHub & CML • Automation for Experimentation • Ex: Smoke Test Locally (Fast) • Queue Reproducible in Github • Using notebooks Experimentation
- 45. GitHub Workflows to support Experiment Queues Check Continuous ML (CML) framework in Github
- 47. Branch for new experiment, change code, queue in GitHub
- 48. Commit & Push branch to queue workflow in GitHub
- 50. Notebook execution using papermill on Github agent
- 51. Train performance results on commits (also PRs)
- 52. MLOps Recap •From Models to Pipelines •Execution of Pipelines produce models •Train for Deployment is fully automated/reproducible •Pipelines enable auto model retraining •Automation, Automation, Automation... •Monitoring for Data Drift, Concept Drift, Model Drift •Tracking, Metadata & Artifact Stores
- 53. Mature MLOps Example (Level 1) https://cloud.google.com/solutions/machine-learning/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
- 54. Not best fit for interative/model development
- 55. Final notes •MLOps doesn’t guarantee ML success •MLOps should improve feedback and learning (not block DataScientists from business/data understanding) •Dont reinvent the wheel... •Still a young field Dont believe everything you read in docs (and ppt...) •When/If to introduce MLOps (it depends...)
- 56. MLOps References
- 58. 58
- 60. Also • https://azure.microsoft.com/en-us/services/machine- learning/mlops/ • https://cloud.google.com/solutions/machine-learning/mlops- continuous-delivery-and-automation-pipelines-in-machine- learning • https://github.com/microsoft/MLOps • https://databricks.com/blog/2020/10/13/using-mlops-with- mlflow-and-azure.html • https://github.com/Azure/MachineLearningNotebooks
- 62. © 2016 DevScope. All rights reserved. Rua Passos Manuel Nº 223 – 4º Andar 4000-385 Porto T. +351 223 751 350/51 F. +351 223 751 352 Av. Sidónio Pais, Nº 2 – 3º Andar 1050-214 5 Lisboa info@devscope.net www.devscope.net