Version Control in Machine Learning + AI (Stanford)
7 likes512 views
The document outlines a workshop on version control for machine learning and AI, focusing on tools like Datmo that improve upon traditional version control systems for quantitative-oriented developers (QODs). It explains how Datmo manages model version control, environments, and reproduces results using snapshots, addressing the unique challenges of MLOps. The workshop includes practical demonstrations using the Iris dataset to illustrate Datmo's capabilities in streamlining machine learning workflows.
![@anandsampat
Reproducing the Model
https://datmo.com/signup
Ensure you are signed up on Datmo:
$ [sudo] datmo setup
One time initial setup:
https://datmo.com/settings/integration
Connect Github:](https://image.slidesharecdn.com/stanford-versioncontrolinmachinelearning-180111215631/85/Version-Control-in-Machine-Learning-AI-Stanford-42-320.jpg)
![@anandsampat
Fetch your model from Datmo
$ datmo clone [YOUR-USERNAME]/workshop-iris-classification
Clone the Datmo Model:
$ cd workshop-iris-classification
Jump into this directory:](https://image.slidesharecdn.com/stanford-versioncontrolinmachinelearning-180111215631/85/Version-Control-in-Machine-Learning-AI-Stanford-44-320.jpg)
Version Control in Machine Learning + AI (Stanford)
- 1. @anandsampat Version Control for Machine Learning + AI Workshop Stanford
- 2. @anandsampat Before we begin: datmo.com/get-started Datmo installation:* Install VirtualBox and follow along instead https://docs.datmo.com/guides/using-datmo-on-virtualbox.html Having Trouble?
- 4. @anandsampat Workshop Outline 1. Conventional version control 2. The curious case of QoD’s 3. How Datmo bridges the gap 4. Iris dataset model example 5. Reproduce + use the model
- 5. @anandsampat What is Version Control? The management of changes to documents, computer programs, large web sites, and other collections of information. *AKA `Source Control` “
- 8. @anandsampat You’ve probably heard of Git. Git is a version control system for tracking changes in computer files and coordinating work on those files among multiple people. It is primarily used for source code management in software development, but it can be used to keep track of changes in any set of files.
- 9. @anandsampat So, GitHub, right? (Yes, and no.)
- 11. @anandsampat GitHub = SCM + Hosting + Much More
- 12. @anandsampat For developers: For enterprises: • Self-managed SCM servers became a thing of the past • Developers could leverage industry best practices for their own personal work • Community of knowledge built around a known standard • Collaboration on Open Source Software • Advent of continuous integration / deployment • Removed need for external code issue tracking tool • Consolidation of code storage and versioning tool • Pull Requests, code review, documentation through ReadMe
- 13. @anandsampat Workshop Outline 1. Conventional version control 2. The curious case of QoD’s 3. How Datmo bridges the gap 4. Iris dataset model example 5. Reproduce + use the model
- 14. @anandsampat QoD’s == Quantitative Oriented Developers Artificial IntelligenceData Science Machine Learning
- 17. @anandsampat It’s time to talk about MLOps https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning- systems.pdf
- 18. @anandsampat MLOps: The Elephant in the Room https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning- systems.pdf
- 19. @anandsampat ML systems have a special capacity for incurring technical debt, because they have all of the maintenance problems of traditional code plus an additional set of ML-specific issues. This debt may be difficult to detect because it exists at the system level. “ — Google (Sculley et. al, 2015)
- 20. @anandsampat Typical methods for paying down code level technical debt are not sufficient to address ML-specific technical debt at the system level. “ — Google (Sculley et. al, 2015)
- 21. @anandsampat http://eng.uber.com/wp-content/uploads/2017/09/image8.png Here’s where traditional tools fall short
- 22. @anandsampat http://eng.uber.com/wp-content/uploads/2017/09/image8.png Here’s where traditional tools fall short
- 23. @anandsampat
- 25. @anandsampat As for everyone else?
- 26. @anandsampat Workshop Outline 1. Conventional version control 2. The curious case of QoD’s 3. How Datmo bridges the gap 4. Iris dataset model example 5. Reproduce + use the model
- 27. @anandsampat What is Datmo? Datmo is a workflow tool for ML, AI, and Data Science developers. It helps with managing model version control, easy environment handling, and reproducing results through the power of snapshots.
- 28. @anandsampat What are Datmo Snapshots? Code Environment Configuration Files* Metrics
- 29. @anandsampat Why are they important? Environment Configuration Metrics Datmo Snapshots Git Commits Code Files*
- 30. @anandsampat How will it help? Datmo leverages containers to quickly spin up perfectly reproducible developer environments. It tracks this environment, along with model metadata inside of snapshots.
- 31. @anandsampat From a broad perspective: Make ML Ops and workflows manageable and simple, not completely abstracted away. Reduce the amount of glue code so that people can have more robust pipelines.
- 32. @anandsampat From a broad perspective: Make ML Ops and workflows manageable and simple, not completely abstracted away. Reduce the amount of glue code so that people can have more robust pipelines.
- 33. @anandsampat GitHub = SCM + Hosting + More Datmo = Model Versioning + Environments + Deployment + More
- 34. @anandsampat Workshop Outline 1. Conventional version control 2. The curious case of QoD’s 3. How Datmo bridges the gap 4. Iris dataset model example 5. Reproduce + use the model
- 35. @anandsampat Datmo in today’s example We’re going to use Datmo to show how we can quickly iterate on our model and streamline our workflow. We’ll go through using snapshots for A/B testing, saving our tasks, and enabling you all to reproduce my results/make your own changes to the model.
- 36. @anandsampat Problem: Multiple Classification of Flower Species
- 37. @anandsampat Dataset: Fisher’s Iris Flower http://archive.ics.uci.edu/ml/datasets/Iris
- 38. @anandsampat At a glance: - 4 Features - 3 Classes - 150 Rows (50 per class)
- 41. @anandsampat Workshop Outline 1. Conventional version control 2. The curious case of QoD’s 3. How Datmo bridges the gap 4. Iris dataset model example 5. Reproduce + use the model
- 42. @anandsampat Reproducing the Model https://datmo.com/signup Ensure you are signed up on Datmo: $ [sudo] datmo setup One time initial setup: https://datmo.com/settings/integration Connect Github:
- 43. @anandsampat Fork the model Fork from Web Platform GUI (top right corner): https://datmo.com/anands/workshop-iris-classification
- 44. @anandsampat Fetch your model from Datmo $ datmo clone [YOUR-USERNAME]/workshop-iris-classification Clone the Datmo Model: $ cd workshop-iris-classification Jump into this directory:
- 45. @anandsampat Checkout an existing snapshot
- 46. @anandsampat View all model snapshots $ datmo snapshot ls
- 47. @anandsampat Checkout to a particular snapshot $ datmo snapshot checkout --id ______
- 48. @anandsampat Create your own snapshot
- 50. @anandsampat Run the Task $ datmo task run “python3 classifier.py”
- 51. @anandsampat Run the Task $ datmo task run “python3 classifier.py” We want our Python file to be run inside of the container. Why?
- 52. @anandsampat Create a Snapshot from Task output $ datmo snapshot task --id _________
- 53. @anandsampat What just happened? • Datmo cloned the model from the platform, bringing all of the necessary resources to local. • Datmo set your current code to the state of the desired snapshot. • Datmo built the environment inside of a container. • Datmo executed the task inside of the container, and logged the results. • Datmo combines the task output files, environment, code, configs, and metrics into a snapshot datmo clone datmo snapshot checkout Command Result datmo task run datmo snapshot task
- 54. @anandsampat 1. Traditional Source Control isn’t enough for QoD (Data Science, ML, and AI) Key Takeaways 2. Think about ML Ops before you’re “in too deep” 3. In the same way GitHub revolutionized Software Engineering, Datmo does the same for QoD’s
- 56. @anandsampat Full Slides Available at: https://bit.ly/stanford-version-control
- 58. @anandsampat 2. Learn more about ML and browse more content at our blog: https://blog.datmo.com Next Steps 3. Interested in updates? You’ll be signed up for our weekly newsletter if you signed up today. 4. Stay tuned for our open source library this month. It’ll be at https://github.com/datmo/datmo 1. Check out example workflows in our docs to create your own Datmo project here
- 61. @anandsampat Nuts and Bolts of Source Control: http://ericsink.com/scm/source_control.html
- 62. @anandsampat 2015 NIPS Paper from Google https://papers.nips.cc/paper/5656-hidden- technical-debt-in-machine-learning-systems.pdf