CI/CD Templates: Continuous Delivery of ML-Enabled Data Pipelines on Databricks
0 likes487 views
The document discusses the challenges faced by ML teams in implementing CI/CD for machine learning pipelines, particularly the difficulties in integrating traditional CI/CD tools with Databricks notebooks. It introduces Databricks Labs CI/CD templates as a solution that simplifies the creation and management of scalable CI/CD pipelines, allowing seamless integration with GitHub actions for testing and deployment. The template provides a structured approach, enabling teams to efficiently deploy ML models to production and improve overall business value from their machine learning initiatives.
CI/CD Templates: Continuous Delivery of ML-Enabled Data Pipelines on Databricks
- 1. CI/CD Templates: Continuous Delivery of ML-Enabled Data Pipelines on Databricks Michael Shtelma, Sr. Solutions Architect Ivan Trusov, Solutions Architect
- 2. Agenda The Challenges of implementing CI/CD for ML pipelines The CI/CD challenges forcing ML teams to choose between Databricks notebooks or local IDEs Introducing DatabricksLabs CI/CD Templates How CI/CD Templates solves ML team production challenges Demo and Next Steps
- 3. Problem: Organisations are struggling to get Business to start using their models to drive additional revenue Cause: Due to complexity of ML lifecycle only few models end up in production and drive additional revenue for business. Most of them are either delayed or discontinued during different ML Project stages It is challenging for organizations to gain value from ML due to complexity of the ML lifecycle
- 4. What challenges do ML teams face when then try to implement CD4ML?
- 5. ML teams struggle to combine traditional CI/CD tools with Databricks notebooks 1. Benefits to Databricks notebooks Easy to use Scalable Provides access to ML tools such as mlflow for model logging and serving 2. Challenges Non-trivial to hook into traditional software development tools such as CI tools or local IDEs. 3. Result Teams find themselves choosing between using traditional IDE based workflows but struggling to test and deploy at scale or using Databricks notebooks or other cloud notebooks but then struggling to ensure testing and deployment reliability via CI/CD pipelines.
- 7. CI/CD Templates gives you the benefits of traditional CICD workflows and the scale of databricks clusters CI/CD Templates allows you to ● create a production pipeline via template in a few steps ● that automatically hooks to github actions and ● runs tests and deployments on databricks upon git commit or whatever trigger you define and ● gives you a test success status directly in github so you know if your commit broke the build
- 8. A scalable CI/CD pipeline in 5 easy steps 1. Install and customize with a single command 2. Create a new github repo containing your databricks host and token secrets 3. Initialize git in your repo and commit the code. 4. Push your new cicd templates project to the repo. Your tests will start running automatically on Databricks. Upon your tests’ success or failure you will get a green checkmark or red x next to your commit status. 5. You’re done! You now have a fully scalable CICD pipeline. 1 2 3 4 5
- 9. Project structure 1. Python package where the logic of the project will be developed. Your models and pipelines will be developed here. 2. Configuration where you can configure define Databricks jobs which can run pipelines developed in python package 3. Tests directory where local unit tests and integration tests will be developed 1 2 3
- 10. CI/CD Templates execute tests and deployments directly on databricks while storing packages, model logging and other artifacts in Mlflow
- 11. CI/CD Templates - now powered by dbx With dbx you can: ● customize project structure and specify it during deployments ● use new CI tools easily (PRs are welcome!) ● run custom data pipelines pipelines directly from IDE on interactive clusters
- 12. Push Flow
- 13. Release Flow
- 15. Summary The Challenges of implementing CD4ML The CI/CD challenges forcing ML teams to choose between Databricks notebooks or local IDEs Introducing DatabricksLabs CI/CD Templates How CI/CD Templates solves ML team production challenges Next Steps Search DatabricksLabs cicd-templates or go directly to https://github.com/databrickslabs/cicd-templates to get started michael.shtelma@databricks.com ivan.trusov@databricks.com
- 16. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.