CD in Machine Learning Systems
0 likes1,159 views
The document discusses continuous deployment (CD) in machine learning, emphasizing the importance of rapid deployment of new features, improved quality through smaller changes, and the need for structured testing and monitoring. It highlights the critical role of data management, model performance evaluation, and the implications of employing machine learning systems. Best practices, including feature toggles, test pyramids, and monitoring for errors, are outlined to ensure effective model validation and operationalization in production environments.
CD in Machine Learning Systems
- 1. CD in Machine Learning systems Juan LĂłpez @juaneto
- 3. Continuous deployment What it is and why everybody wants it Idea Develop Deploy in prod
- 4. Continuous deployment What it is and why everybody wants it Idea Develop Deploy in prod â New features on the fly.
- 5. Continuous deployment What it is and why everybody wants it Idea Develop Deploy in prod â New features on the fly. â Quality goes up (smaller changes).
- 6. Continuous deployment What it is and why everybody wants it Idea Develop Deploy in prod â New features on the fly. â Quality goes up (smaller changes). â Faster development.
- 7. Continuous deployment What it is and why everybody wants it Idea Develop Deploy in prod â New features on the fly. â Quality goes up (smaller changes). â Faster development. â Experimentation.
- 8. Continuous deployment Idea Develop Deploy in prod What it is and why everybody wants it â New features on the fly. â Quality goes up (smaller changes). â Faster development. â Experimentation. â Innovation.
- 9. So⊠we want to reduce the gap between a new idea and when this idea is in production.
- 10. Machine learning Where do we use it? Not only hype
- 11. Machine learning Where do we use it? Not only hype â Image recognition â Recommendations â Predictions â etc.
- 12. Machine learning What is it?
- 13. Machine learning What is it? â Subset of artificial intelligence.
- 14. Machine learning What is it? â Subset of artificial intelligence. â Statistical models that systems use to effectively perform a specific task.
- 15. Machine learning What is it? â Subset of artificial intelligence. â Statistical models that systems use to effectively perform a specific task. â It doesnÂŽt use explicit instructions, relying on patterns and inference instead.
- 16. So⊠we want to reduce the gap between a new idea and when this idea is in production.
- 17. How do we achieve CD?
- 18. 2017 The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, D. Sculley Google, Inc. How do we achieve CD?
- 19. How do we achieve CD?
- 20. How do we achieve CD?
- 24. Code Apply the best practices for writing your code. Code is always code
- 25. Code Apply the best practices for writing your code. Code is always code â Not only model. Complex systems.
- 26. Code Apply the best practices for writing your code. Code is always code â Not only model. Complex systems. â Extreme programming.
- 27. Code Apply the best practices for writing your code. Code is always code â Not only model. Complex systems. â Extreme programming. â Quality gates.
- 28. Code Apply the best practices for writing your code. Code is always code â Not only model. Complex systems. â Extreme programming. â Quality gates. â Feature toggles.
- 29. Code Apply the best practices for writing your code. Code is always code â Not only model. Complex systems. â Extreme programming. â Quality gates. â Feature toggles. â Test Pyramid. Manual session based testing Automated GUI tests Automated unit tests Automated integration tests Automated API tests Automated component tests * Vishal Naik (Thoughtworks insights)
- 30. Builds Test Continuous integration Acceptance test Deploy to staging Continuous delivery Deploy to pro Smoke test Continuous deployment Code pipeline
- 31. Unlike in traditional software systems, the šbehavior of ML systems is not specified directly in code but is learned from dataš.
- 32. Unlike in traditional software systems, the šbehavior of ML systems is not specified directly in code but is learned from dataš. So our tests depend on the sets of data for training models.
- 34. Data pipeline
- 35. Ingest
- 37. Ingest â Data lake â Know your sources. Data Catalog.
- 38. Ingest â Data lake â Know your sources. Data Catalog. â Have a schema. Governance your data.
- 39. Ingest â Data lake â Know your sources. Data Catalog. â Have a schema. Governance your data. â Watch for silent failures.
- 41. Data wrangling/mungling â Datamart (not data warehouse).
- 42. Data wrangling/mungling â Datamart (not data warehouse). â Be careful with data cooking: if your features are bad, everything is bad.
- 43. Data wrangling/mungling â Datamart (not data warehouse). â Be careful with data cooking: if your features are bad, everything is bad. â Data cleaning
- 45. Get training data â data scientist. Make their life easier.
- 46. Get training data â data scientist. Make their life easier. â Big data. Importance-weight sampled.
- 47. Get training data â data scientist. Make their life easier. â Big data. Importance-weight sampled. â Data security.
- 48. Get training data â data scientist. Make their life easier. â Big data. Importance-weight sampled. â Data security. â Versioning data.
- 49. â data scientist. Make their life easier. â Big data. Importance-weight sampled. â Data security. â Versioning data. â Training/Serving Skew. Get training data
- 50. âAll models are wrongâ. Common aphorism in Statistics.
- 51. âAll models are wrongâ. Common aphorism in Statistics. âAll models are wrong, some are usefulâ. George Box.
- 52. âAll models are wrongâ. Common aphorism in Statistics. âAll models are wrong, some are usefulâ. George Box. âAll models are wrong, some are useful for a short period of timeâ. TensorflowÂŽs team.
- 54. First of all
- 55. First of all â Design & evaluate the reward function.
- 56. First of all â Design & evaluate the reward function. â Define errors & failure.
- 57. First of all â Design & evaluate the reward function. â Define errors & failure. â Ensure mechanisms for user feedback.
- 58. First of all â Design & evaluate the reward function. â Define errors & failure. â Ensure mechanisms for user feedback. â Try to tie model changes to a clear metric of the subjective user experience.
- 59. â Design & evaluate the reward function. â Define errors & failure. â Ensure mechanisms for user feedback. â Try to tie model changes to a clear metric of the subjective user experience. â Objective vs many metrics. First of all
- 60. Model pipeline
- 61. Code new model candidate
- 62. â Code is code. Code new model candidate
- 63. â Code is code. â Run test in your pipeline. Code new model candidate
- 64. â Code is code. â Run test in your pipeline. â New version of the model. Code new model candidate
- 65. Training model
- 66. Training model â Feature engineering. (Unbalancing data, unknown unknowns, etc).
- 67. Training model â Feature engineering. (Unbalancing data, unknown unknowns, etc). â Be critical with your features: data dependencies cost more than code dependencies.
- 68. Training model â Feature engineering. (Unbalancing data, unknown unknowns, etc). â Be critical with your features: data dependencies cost more than code dependencies. â Training/serving Skew.
- 69. Training model â Feature engineering. (Unbalancing data, unknown unknowns, etc). â Be critical with your features: data dependencies cost more than code dependencies. â Training/serving Skew. â Deterministic training dramatically simplifies.
- 70. Training model â Feature engineering. (Unbalancing data, unknown unknowns, etc). â Be critical with your features: data dependencies cost more than code dependencies. â Training/serving Skew. â Deterministic training dramatically simplifies. â Tune hyperparameters.
- 71. Model competition PRODUCTION Model in PRO Model 1 Model 2 Model n Model in PRO
- 73. Model performance â Test performance with production data.
- 74. Model performance â Test performance with production data. â Check your reward functions and failures. E.g: ROC curve.
- 75. Model performance â Test performance with production data. â Check your reward functions and failures. E.g: ROC curve. â Be careful. Satisfy a baseline of quality in all data slices.
- 76. Model performance â Test performance with production data. â Check your reward functions and failures. E.g: ROC curve. â Be careful. Satisfy a baseline of quality in all data slices. â Baseline of accuracy.
- 77. Model performance â Test performance with production data. â Check your reward functions and failures. E.g: ROC curve. â Be careful. Satisfy a baseline of quality in all data slices. â Baseline of accuracy. â Feedback loop.
- 78. Model champion PRODUCTION Model in PRO Model 2 Model n Model in PRO Model 1
- 80. Deploy champion model â Shadow traffic.
- 81. Deploy champion model â Shadow traffic. â Test the models with real data.
- 82. Deploy champion model â Shadow traffic. â Test the models with real data. â Canary releases.
- 83. Deploy champion model â Shadow traffic. â Test the models with real data. â Canary releases. â Tests A/B.
- 84. Deploy champion model â Shadow traffic. â Test the models with real data. â Canary releases. â Tests A/B. â Rollbacks.
- 87. Monitoring
- 88. Monitoring â Create a dashboard with clear and useful information.
- 89. Monitoring â Create a dashboard with clear and useful information. â Schema changes.
- 90. Monitoring â Create a dashboard with clear and useful information. â Schema changes. â Infra monitoring (training speed, serving latency, RAM usage, etc).
- 92. Monitoring â User feedback. â Stale models.
- 93. Monitoring â User feedback. â Stale models. â Feedback loop.
- 94. Monitoring â User feedback. â Stale models. â Feedback loop. â Errors (model, apis, etc).
- 95. Monitoring â User feedback. â Stale models. â Feedback loop. â Errors (model, apis, etc). â Silent failures.
- 96. Conclusions
- 97. â Code is always code â Objective driven modeling â Know your data â Clear metrics for complex systems