ModelDB: A System to Manage Machine Learning Models: Spark Summit East talk by Manasi Vartak
2 likes435 views
ModelDB is a management system designed to handle machine learning models, providing features such as experiment tracking, versioning, and reproducibility. It aims to address challenges faced in model development, including the lack of historical records and difficulties in collaboration. The system supports diverse languages and offers a user-friendly interface for querying and managing various modeling artifacts.
ModelDB: A System to Manage Machine Learning Models: Spark Summit East talk by Manasi Vartak
- 1. ModelDB: A system to manage machine learning models Manasi Vartak PhD Student, MIT DB Group
- 2. People Manasi Vartak PhD student, MIT Srinidhi Viswanathan MEng, MIT Samuel Madden Faculty, MIT Matei Zaharia Faculty, Stanford Harihar Subramanyam MEng, MIT Wei-En Lee MEng student, MIT
- 3. Building a credit recommendation algorithm Profession Credit History Risk of Default Politician Reasonable 0.3 Struggling artist Poor 0.7 Investor Has more money than our company 0.0 … … … … Barack Obama Lindsay Lohan Warren Buffet
- 4. Model 1
- 6. Accuracy: 58% Model 2 RandomForestClassifier val udf1: (Int => Int) = (delayed..) df.withColumn(“timesDelayed”, udf1)
- 7. Accuracy: 63% RandomForestClassifier df.withColumn(“timesDelayed”, udf1) .withColumn(“percentPaid”, udf2) val lrGrid = new ParamGridBuilder() .addGrid(rf.maxDepth, Array(5, 10, 15)) .addGrid(rf.numTrees, Array(50, 100)) Model 5 credit-default-clean.csv
- 8. df.withColumn(“timesDelayed”, udf1) .withColumn(“percentPaid”, udf2) .withColumn(“creditUsed”, udf3) … val lrGrid = new ParamGridBuilder() .addGrid(lr.elasticNetParam, Array(0.01, 0.1, 0.5, 0.7)) val scaler = new StandardScaler() .setInputCol(“features”) … val labelIndexer1 = new LabelIndexer() val labelIndexer2 = new LabelIndexer() … Model 50 val udf1: (Int => Int) = (delayed..) val udf2: (String, Int) = … credit-default-clean.csv
- 9. Why is this a problem?
- 10. Why is this a problem? • No record of model history Did my colleague do that already?
- 11. Why is this a problem? • No record of model history • Insights lost along the way Did my colleague do that already? How did normalization affect my ROC?
- 12. Why is this a problem? • No record of model history • Insights lost along the way • Difficult to reproduce results Did my colleague do that already? How did normalization affect my ROC? What params did I use?
- 13. Why is this a problem? • No record of model history • Insights lost along the way • Difficult to reproduce results • Cannot search for or query models Did my colleague do that already? How did normalization affect my ROC? Where’s the LR model I tried last week with featureX? What params did I use?
- 14. Why is this a problem? • No record of model history • Insights lost along the way • Difficult to reproduce results • Cannot search for or query models • Difficult to collaborate Did my colleague do that already? How did normalization affect my ROC? How does someone review your model? Where’s the LR model I tried last week with featureX? What params did I use?
- 15. Requirements from model management tool
- 16. Requirements from model management tool • Experiment tracking
- 17. Requirements from model management tool • Experiment tracking • Versioning
- 18. Requirements from model management tool • Experiment tracking • Versioning • Reproducibility
- 19. Requirements from model management tool • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search
- 20. Requirements from model management tool • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration
- 21. Requirements from model management tool • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration *With minimal effort
- 22. ModelDB: a system to manage machine learning models https://github.com/mitdbg/modeldb http://modeldb.csail.mit.edu
- 24. ModelDB: model management system Ingest models, metadata
- 25. ModelDB: model management system Model, Pipeline Storage Versioning Ingest models, metadata
- 26. ModelDB: model management system Model, Pipeline Storage Versioning Query Ingest models, metadata
- 27. ModelDB: model management system Model, Pipeline Storage Versioning Query Ingest models, metadata Collaboration, Reproducibility
- 28. User quotes “I should have had this in my self-driving cars class; it would have made things so much easier” “…it can really help with reproducibility … and collaboration in multi-person teams…” “I used it to track models for a research project; it was so simple”
- 29. Demo
- 30. ModelDB Architecture & Design Decisions
- 31. ModelDB Architecture & Design Decisions
- 32. ModelDB Architecture & Design Decisions 1. Support for diverse languages and environments
- 33. ModelDB Architecture & Design Decisions 1. Support for diverse languages and environments 2. Minimal changes to existing workflows
- 34. ModelDB Architecture & Design Decisions 1. Support for diverse languages and environments 2. Minimal changes to existing workflows 3. Rich visual interface
- 35. ModelDB Architecture & Design Decisions 1. Support for diverse languages and environments 2. Minimal changes to existing workflows 3. Rich visual interface 4. Support for complex queries
- 36. “Oh, but why not git?” • All code treated equal • Some elements are special: data sources, parameters, metrics, models • Difficult to tease that out • No semantics, so can’t run interesting queries
- 37. Requirements from model management tool
- 38. Requirements from model management tool • Experiment tracking
- 39. Requirements from model management tool • Experiment tracking • Versioning
- 40. Requirements from model management tool • Experiment tracking • Versioning • Reproducibility
- 41. Requirements from model management tool • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search
- 42. Requirements from model management tool • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration
- 43. Requirements from model management tool • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration *With minimal effort
- 44. ModelDB Features (currently available) • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration
- 45. ModelDB Features (currently available) • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration Log models, params, pipelines etc. via ModelDB API
- 46. ModelDB Features (currently available) • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration Log models, params, pipelines etc. via ModelDB API Every modeling run = version
- 47. ModelDB Features (currently available) • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration Log models, params, pipelines etc. via ModelDB API All pipeline details, params logged Every modeling run = version
- 48. ModelDB Features (currently available) • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration Log models, params, pipelines etc. via ModelDB API Model search, query, comparison via frontend All pipeline details, params logged Every modeling run = version
- 49. ModelDB Features (currently available) • Experiment tracking • Versioning • Reproducibility • Comparisons, queries, search • Collaboration Log models, params, pipelines etc. via ModelDB API Model search, query, comparison via frontend Central repository of models Review models, annotate All pipeline details, params logged Every modeling run = version
- 51. ModelDB Features (ongoing) • Unified Querying of Modeling Artifacts
- 52. ModelDB Features (ongoing) • Unified Querying of Modeling Artifacts Base data, intermediates, models, predictions, metadata
- 53. ModelDB Features (ongoing) • Unified Querying of Modeling Artifacts Base data, intermediates, models, predictions, metadata “How did the GBDTs do on married customers who are interested in gardening?”
- 54. ModelDB Features (ongoing) • Unified Querying of Modeling Artifacts Base data, intermediates, models, predictions, metadata “How did the GBDTs do on married customers who are interested in gardening?” Base Data is_married=T
- 55. ModelDB Features (ongoing) • Unified Querying of Modeling Artifacts Base data, intermediates, models, predictions, metadata “How did the GBDTs do on married customers who are interested in gardening?” Base Data is_married=T Intermediates gardening=T
- 56. ModelDB Features (ongoing) • Unified Querying of Modeling Artifacts Base data, intermediates, models, predictions, metadata “How did the GBDTs do on married customers who are interested in gardening?” Base Data is_married=T Intermediates gardening=T Metadata type= GBDT Models ids={..}
- 57. ModelDB Features (ongoing) • Unified Querying of Modeling Artifacts Base data, intermediates, models, predictions, metadata “How did the GBDTs do on married customers who are interested in gardening?” Base Data is_married=T Intermediates gardening=T Predictions accuracy(…) Metadata type= GBDT Models ids={..}
- 58. ModelDB Features (ongoing) • Unified Querying of Modeling Artifacts Base data, intermediates, models, predictions, metadata “How did the GBDTs do on married customers who are interested in gardening?” Base Data is_married=T Intermediates gardening=T Predictions accuracy(…) Metadata type= GBDT Models ids={..} What query language? How to persist data?
- 60. ModelDB Features (ongoing) • Mining data in ModelDB
- 61. ModelDB Features (ongoing) • Mining data in ModelDB Model Features Params Metric M13 X3,X9... l1=0.3 0.63 M22 X1,X4,X7 l2=0.7 0.8 M34 X11,X13 l1=0.7 0.55 … … … …
- 62. ModelDB Features (ongoing) • Mining data in ModelDB Given model history, what should we try next? Bayesian Modeling/AutoML Model Features Params Metric M13 X3,X9... l1=0.3 0.63 M22 X1,X4,X7 l2=0.7 0.8 M34 X11,X13 l1=0.7 0.55 … … … …
- 63. ModelDB Features (ongoing) • Mining data in ModelDB • Full model lifecycle management Given model history, what should we try next? Bayesian Modeling/AutoML Model Features Params Metric M13 X3,X9... l1=0.3 0.63 M22 X1,X4,X7 l2=0.7 0.8 M34 X11,X13 l1=0.7 0.55 … … … …
- 64. ModelDB Features (ongoing) • Mining data in ModelDB • Full model lifecycle management Given model history, what should we try next? Bayesian Modeling/AutoML Model Features Params Metric M13 X3,X9... l1=0.3 0.63 M22 X1,X4,X7 l2=0.7 0.8 M34 X11,X13 l1=0.7 0.55 … … … …
- 65. ModelDB Features (ongoing) • Mining data in ModelDB • Full model lifecycle management Given model history, what should we try next? Bayesian Modeling/AutoML Model performance degrades Retrain model over time Model Features Params Metric M13 X3,X9... l1=0.3 0.63 M22 X1,X4,X7 l2=0.7 0.8 M34 X11,X13 l1=0.7 0.55 … … … …
- 68. ModelDB available now! • Download, try it out!
- 69. ModelDB available now! • Download, try it out!
- 70. ModelDB available now! • Download, try it out! • Tell us what you think; what can we do better?
- 71. ModelDB available now! • Download, try it out! • Tell us what you think; what can we do better? • Contribute! (see Issues on repo for some ideas)
- 72. ModelDB: a system to manage machine learning models mvartak@csail.mit.edu | @DataCereal http://modeldb.csail.mit.edu *Icons from FlatIcon