PASS 2024 - Best Practices for Development on Azure Databricks
0 likes259 views
The document outlines best practices for development on Azure Databricks, emphasizing the importance of version control, writing clean code, and automating testing and deployment workflows. Key components discussed include the developer lifecycle, various development options, and the utilization of Databricks features like Git integration and asset bundles. The presentation aims to enhance collaboration, reliability, and the overall efficiency of data projects on Databricks.
PASS 2024 - Best Practices for Development on Azure Databricks
- 1. Best Practices for Development on Azure Databricks Dustin Vannoy Kimberly Mahoney
- 2. #PASSDataSummit Guiding Best Practices Through the Full Developer Lifecycle on Databricks • Version Control • Writing Code • Testing • Getting to production Agenda
- 9. #PASSDataSummit The Developer Lifecycle Inner Dev Loop ● Develop ● Test ● Debug Outer Dev Loop ● Test ● Integrate ● Deploy
- 10. #PASSDataSummit Why do we need best practices Deploying 1. Improve reliability 2. Enhance collaboration and team efficiency 3. More flexible to changes 4. Faster releases
- 12. ETL & Real-time Analytics Orchestration Data Warehousing Data Science & AI Mosaic AI Delta Live Tables Workflows Databricks SQL Unified security, governance, and cataloging Unity Catalog Databricks Data Intelligence Platform Unified data storage for reliability and sharing Delta Lake UniForm An AI powered data intelligence engine to understand the semantics of your data DatabricksIQ Open Data Lake All Raw Data (Logs, Texts, Audio, Video, Images)
- 13. #PASSDataSummit
- 14. Version Control
- 15. #PASSDataSummit Why: • Track from the beginning • Backup & Recovery • Quicker and cleaner iteration • Improved collaboration Best practice: Use Version Control
- 17. #PASSDataSummit With Version Control checkout c1 c2 c3 peer review checkout c1 peer review
- 18. #PASSDataSummit Benefits of Version Control • Track incremental changes • Avoid conflicting edits • Maintain a complete project history • Easily revert to earlier versions
- 19. #PASSDataSummit Databricks Git Folders Integrate git repositories into workspace filesystem Inspect, commit, push and pull code changes. Create and work with different branches. New - Resolve merge conflicts. Git Folders support for git lets users collaborate on the same code without interfering with each others’ work. Protected branches and pull requests provide guardrails for what code moves to production AWS CodeCommit
- 20. #PASSDataSummit
- 21. Demo
- 22. Writing Code
- 23. #PASSDataSummit Development Options (Code Editors) IDE Support Databricks-native ● Databricks Files and Notebooks ● Multi-language ○ R, Python, SQL, Scala ● Collaborative editing ● AI-Assisted authoring ● Databricks VS Code Extension ● Pycharm plugin by JetBrains
- 24. #PASSDataSummit Best practice: Write clean code Clean Code - readable, modular, structured, simple The why • Improves readability and maintainability • Simplifies Debugging and Testing • Enhances Coding Assistant Suggestions
- 25. #PASSDataSummit Best practice: Write clean code Clean Code - readable, modular, structured, simple The how • Adopt a style guide • Follow language conventions • Start small • Use meaningful names • Break up long blocks of logic into smaller chunks
- 26. Clean and Readable Code: The Foundation of Good Software Poor naming Hardcoded values No comments No reusability
- 27. Clean and Readable Code: The Foundation of Good Software Meaningful names No hardcoded values Comments & documentation Reusable modules
- 28. #PASSDataSummit Improving code quality Syntax Highlighting Automatically highlight potential issues in real-time. Databricks Assistant Provides an interactive assistant to guide you through fixing identified errors Modularize Code Import R and Python modules stored in workspace files alongside your Databricks notebooks. Protect against out of order variable references in notebooks
- 29. #PASSDataSummit And more.. • Debugger • Code Formatter • Autocomplete • Variable Inspection • Go to definition
- 30. Seamless Workflow Integrates error detection, correction, and testing into a single process.
- 31. Testing
- 32. #PASSDataSummit 5 reasons for automated tests 1. Reduce repetition 2. Ensure quality early 3. Avoid fear of code changes 4. Clarify purpose of functions 5. Speed up development
- 33. The test pyramid - Test full pipeline (source to target) - Test one step in pipeline - Single function
- 34. #PASSDataSummit Difference in test types Unit Testing Integration Testing End-to-end Code / Resource Python functions Notebook / Python script Databricks Workflow + external services Framework pytest / unittest Workflow + validation code Workflow + run job task + validation code Test trigger Developer CI pipeline or Developer Release pipeline
- 35. Demo
- 37. #PASSDataSummit Best practice: Automate and Orchestrate Workflows • Connect dependent tasks • Automate Error Handling and Retries • Monitor performance • Trigger with time and event-based triggers
- 38. #PASSDataSummit Unified orchestration for your data projects • Orchestrate many different task types • Flexible triggers and scheduling • Built in alerting and monitoring • Support for parameters Enter Databricks Workflows
- 40. #PASSDataSummit Workflow Task Types Many task types to choose from, all focused around running on Databricks. If you can do it on Databricks, you can do it with a workflow.
- 41. Is that it?
- 42. #PASSDataSummit What about these other best practices.. • Version Workflow Configurations • Track configurations to maintain consistency. • Automate Workflow Deployments • Streamline updates and reduce manual effort. • Run Tests in the Outer Dev Loop • Validate workflows in production-like conditions. • Separate Environments for Deployments • Use dev, staging, and prod to ensure stability.
- 43. #PASSDataSummit MAPPING TO DATABRICKS ● Databricks Notebooks ● Libraries (e.g. Python Wheels) ● SQL files ● JARs ● … Code “I swear this cell was working yesterday… Did someone change it?”
- 44. #PASSDataSummit MAPPING TO DATABRICKS ● Databricks Workflows ● Delta Live Table Pipelines ● Model Serving Endpoints ● Lakehouse Monitoring ● … Resource configuration “The refresh job failed this morning… Was it updated to use the most recent set of expected parameters?”
- 45. #PASSDataSummit MAPPING TO DATABRICKS ● Databricks Workspaces ● Unity Catalog (e.g. catalogs) ● Execution identity ○ User or service principal Environment separation “We need a larger cluster in prod, who has access to update it?”
- 46. #PASSDataSummit MAPPING TO DATABRICKS ● Databricks Workspaces ● Unity Catalog (e.g. catalogs) ● Execution identity ○ User or service principal Environment separation “We need a larger cluster in prod, who has access to update it?” <catalog>.<schema>.<table > dev prod
- 47. FROM DEVELOPMENT TO PRODUCTION Development Dev Catalog Staging Staging Catalog Production Prod Catalog Version control deploy deploy deploy main my-feature release pull request release
- 48. #PASSDataSummit PROPERTIES WE’RE LOOKING FOR ● Single source of truth for code and resource configuration ● Define resources using an easy to understand format (e.g., YAML) ● Specialize resources based for target environment (dev/staging/prod) ● Ensure deployment isolation ● Easily automate deployment (CI/CD)
- 49. Enter Databricks Asset Bundles
- 50. #PASSDataSummit Databricks Asset Bundles What? YAML files that specify the artifacts, resources, and configurations of a Databricks project - databricks.yml
- 51. #PASSDataSummit Databricks Asset Bundles How? The bundle subcommand of the Databricks CLI lets you interact with this definition; to validate it, to deploy the bundle, or to run resources defined in the bundle.
- 52. #PASSDataSummit Databricks Asset Bundles Where? These commands are executed client-side, so really anywhere. You can do development on your laptop and/or run automated production deployments from CI/CD systems such as GitHub Actions or Azure DevOps.
- 53. Demo
- 54. #PASSDataSummit Summary • Use version control + peer review • Write clean, readable code • Test Your Code Thoroughly with Automated Tests • Automate Workflows & Deployments
- 55. #PASSDataSummit Summary • Databricks Git Folders • Rich native editor and support for IDE’s • Build out testing and deployment pipelines with Databricks Asset Bundles
- 56. #PASSDataSummit Try it out yourself
- 57. #PASSDataSummit Evaluate this session at: www.PASSDataCommunitySummit.com/evaluation Your feedback is important to us
- 58. #PASSDataSummit Thank you ● youtube.com/DustinVannoy Develop and Deploy Code Easily With IDEs ● How to Get the Most Out of Databricks Notebooks ● Databricks Asset Bundles: A Unifying Tool for Deployment on Databricks ● Best Practices for Unit Testing PySpark To learn more… Resources + Repo at dustinvannoy.com