How to monitor your ML models in production with Kubernetes
0 likes125 views
The document outlines a workshop focused on monitoring and managing the health of machine learning models in production, emphasizing the use of Kubernetes and various open-source tools like Grafana, Kibana, and Elasticsearch for monitoring metrics and logs. It discusses the importance of tracking model performance, preventing drift, and ensuring timely retraining and updates. The workshop also highlights a live demo using cnvrg.io for deploying and monitoring models efficiently.
How to monitor your ML models in production with Kubernetes
- 1. WORKSHOP Monitor ML models in production How to monitor and manage the health of your ml models Leah Kolben, CTO @leah4kosh leah@cnvrg.io
- 2. whoami • Developer/Data scientist => CTO • cnvrg.io = built by data scientists, for data scientists to help teams: • Get from data to models to production in the most efficient and fast way • bridge science and engineering
- 3. agenda • Introduction – recap previous webinars • Kubernetes overview • Why should we monitor our models • Monitor tools • LIVE workshop • Summary
- 4. Introduction • Previous webinars: • Train your ML models on Kubernetes • Run ETL jobs using spark on Kubernetes • Deploy your ML model in production • Today: “My model is in production, now what?” • Use Grafana to monitor your model metrics – CPU & Memory • Use Kibana to monitor your ML model logs • Use elasicsearch to index your model and create alerts
- 5. Kubernetes - recap • Provides a runtime environment for Docker containers • Provides an abstraction layer for containers to run on • Deploy as micro services • All services are natively load balanced • Can scale up and down dynamically • Monitor the health of the containers • Schedule runs and cronjobs • Use the same API across EVERY cloud provider and bare metal!
- 6. • Goals: • Quickly deploy your ML model as a service • Reduce costs and man power with auto scaling • Load balanced the traffic • Natively monitored by Kubernetes • Update your model continuously: canary deployments, blue/green deployments ML in production - recap
- 7. Why should we monitor our models? • Able to track you model performance • Prevent model drift • Monitor auto scaling and high load of traffic • Know when to retrain your model and updat it • Monitor updates – validate new version is better the old one • Keep you production models updated and relevant!
- 8. Monitor tools • Use the EKS stack on kubernetes • Use open source tools that can be quickly installed on kuberentes using helm • Use grafana to monitor the resources you rmodel uses • Use Kibana to show and track you input/output • Use elasticsearch to index the logs • Use elasalert for tagging and creating alerts regarding the health of your models
- 9. Grafana • Open source analytics and monitoring for a variety of databases • Natively connects to the EKS stack • Use custom dashboards to track your model usage
- 10. Grafana
- 11. Grafana – custom dashboard
- 12. Kibana • Part of the EKS stack • Log all input and output • Log internal logs • Search and find prediction outputs • Visualize and understand the data
- 13. Kibana
- 14. Elasticsearch & elasalert • Store and index all input/output of you models • Get sense of you data • Query your data • Create custom rules on the data to monitor the health of your model • Trigger custom webhooks upon alerts – retrain CI/CD pipelines
- 15. Let’s do it!
- 16. Using cnvrg to deploy & monitor your models • Your only responsibility is to write the predict function • end-2-end pipelines • EKS stack is deployed automatically • Dedicated Grafana for each endpoint • Dedicated kibana for each endpoint • Alerts and triggers out of the box • Continual learning support
- 17. DEMO
- 18. Summary • Kubernetes is the becoming the standard way to deploy models • Overview Kubernetes • Overview of what’s ML in production • Monitor – why and how • Live demo • Deploy, monitor and retrain models using cnvrg