![Component III: Shared cloud compute & storage resources
DSVM: Convenient cloud compute resource for Data Scientists [Optional]
18
• The Microsoft DSVM is an Azure virtual
machine (VM) image pre-installed and
configured with several popular DS
tools
• Microsoft R Server Developer Edition
• Anaconda Python distribution
• Jupyter notebook (with R, Python kernels)
• Visual Studio Community Edition
• Power BI desktop
• SQL Server 2016 Developer Edition
• Machine learning and Data Analytics tools
• Deep Learning Toolkits](https://image.slidesharecdn.com/guhathakurta-5th-gbc-santaclara-aug29-31-2017-170830235001/85/Team-Data-Science-Process-Presentation-TDSP-Aug-29-2017-18-320.jpg)
Team Data Science Process Presentation (TDSP), Aug 29, 2017
- 2. How To Implement a Standardized Process for Execution and Delivery Of Data Science Solutions: Microsoft’s Team Data Science Process (TDSP) Debraj GuhaThakurta Microsoft AI & Research Algorithms and Data Science debraj.guhathakurta@microsoft.com Date: Aug 29, 2017
- 3. • Standardized process: A challenge in Data Science • Microsoft Team Data Science Process (TDSP) o Principle and Objective o Components o Adoption • Ongoing & future work • Summary • Resources Note: All relevant links are provided in Resources section 3
- 4. The opportunity and challenge of data science in enterprises • Opportunity: 17% … had a well-developed Predictive/Prescriptive Analytics program in place, while 80% … planned on implementing such a program within five years – Dataversity 2015 Survey • Challenge: Only 27% of the big data projects are regarded as successful – CapGenimi 2014 4 While tools & platforms have matured, there is still a major gap in executing on the potential
- 6. Diversity & growth creates process challenges in enterprise DS teams 6 Organization Collaboration Quality Knowledge Accumulation Agility Global Teams •Five Geographic Locations Team Growth •Onboard New Members Rapidly Varied Use Cases •Industries and Use Cases Diverse DS Backgrounds •DS have diverse backgrounds, experiences with tools, languages
- 7. Well-known DS processes are usually high-level descriptions 7 Cross-Industry Standard Process for Data Mining (CRISP-DM) Knowledge Discovery in Databases (KDD)
- 8. A process to make enterprise DS teams more efficient
- 9. First - why a process? A process specifies a detailed sequence of activities necessary to perform specific business tasks It is used to standardize procedures and establish best practices 9 Technology and tools are changing rapidly. A standardized process can provide continuity and stability of work-flow. - Based on discussions with Luis Morinigo, Dir. IoT, NewSignature
- 10. Data Science can borrow processes from Software Development Life Cycle (SDLC) 10 Data Science != Software Development But, we can learn, especially on standardized processes SDLC has had much more to evolve, standardize and build generalized best practices
- 11. Guiding principle and objective of TDSP 11 Objective: Provide a process to improve the efficiency of development & delivery of DS solutions Principle: Integrate key practices of software Development with that of a data-science work-flow → Improve productivity, collaboration and quality
- 12. 4 Key components of TDSP 12 Standard DS Lifecycle Project Structure, Templates & Roles Shared, Distributed Data Platforms and Servers (incl. version control server) Productivity Tools, Shared Utilities
- 13. Component I: Standardized DS lifecycle 13 • 4 major stages:
- 14. DS lifecycle stages can be integrated with specific deliverables & checkpoints 14 Business Understanding • Project Objective • Data, Target & Feature Definition • Data Dictionary Data acquisition and understanding Modeling Deployment
- 15. Component II: Project structure and templates 15 Business understanding & problem scope definition
- 16. Component II: Project roles (examples) 16 • Governance and Project Management • Data Science and Engineering
- 17. Component III: Shared and distributed resources Can be cloud or on–premises • Virtual machines (VMs), or clusters are disposable compute, added to projects as needed • Many-to-many relationship between data scientists, VMs and projects possible • Data typically stored in cloud stores, such as blob or database • Project artifacts & code permanently stored in central Git (version control) repositories. 17 Project execution needs compute + data + process
- 18. Component III: Shared cloud compute & storage resources DSVM: Convenient cloud compute resource for Data Scientists [Optional] 18 • The Microsoft DSVM is an Azure virtual machine (VM) image pre-installed and configured with several popular DS tools • Microsoft R Server Developer Edition • Anaconda Python distribution • Jupyter notebook (with R, Python kernels) • Visual Studio Community Edition • Power BI desktop • SQL Server 2016 Developer Edition • Machine learning and Data Analytics tools • Deep Learning Toolkits
- 19. Component III: Shared Git repository for distributed development and versioning 19 • Git is a Version Control System • Each repo contains the full change history • Used in a distributed way with a single remote repo and several local repos (on local machine or a VM) Remo te Local LocalLocal Local TDSP Git Template
- 20. Component IV: Productivity tools and utilities - Analytics IDEAR: Interactive data exploration and reporting (R and Python) 20 o Data quality assessment o Getting business insights from the data o Association between variables o Generating data reports automatically Clustering Correlations Distribution assessment
- 21. Component IV: Productivity tools and utilities - Analytics AMAR: Automated modeling and reporting (R) 21 o Building baseline models quickly o Evaluating model accuracy o Generating standardized model reports o Model explainability (feature importance) Predicted vs. Actual (diff. algorithms) Feature importance (diff. algorithms)
- 22. Component IV: Productivity tools - Agile planning Agile-like work planning and execution template DS 22 • Use VSTS to create data science agile process template • Use VSTS to track data science activities • Use KANBAN to track the PBI/Stories Visual Studio Team Services Helps you customize Agile-derived work-templates
- 23. TDSP in action on Azure: E2E process templates 23 • Azure ML • Spark • Hadoop/MR https://azure.microsoft.com/en-us/documentation/learning-paths/data-science-process/ • SQL-server with R and Python • Azure Data Lake
- 24. Adoption: TDSP can be adopted in phases in a DS organization 24 Level 1 Level 2 Level 3 Level 4 • Git repository per data science project • Standard directory structure in each repo • Use provided document templates like charter, exit reports • Track work items • Level 1 + • Customize and standardize key document templates to be used by the team • Develop shared DS utilities like IDEAR and AMAR • Level 2 + • Utility repository to share scripts as utilities • Level 3 + • Link git branch with work items • Code review • Manage all models and data/features (model management) • Testing
- 25. Adoption: Organizations may choose customize Preserving the DS lifecycle and 4 key components 25 • TDSP lifecycle and components provide a process, but with flexibility • Customization possible as per team/org needs • Specific DS tools, algos etc. • Heavier or lighter document templates for deliverables • E.g. Consulting orgs’ delivery models, long-term engagements, PoCs projects • Templates for different business verticals
- 26. Adoption: Internal and external organizations that have adopted TDSP 26 • Microsoft internal • Microsoft consulting services (MCS) • AI & R Cloud Platform: Algorithm and data sciences team • Windows Devices DS team • Technology Evangelism and Delivery DS team • External partners • New Signature • BlueGranite
- 27. Ongoing work • Enabling better project scoping, specification of RoI, criteria for success at each stage of project • Further incorporate SDLC features into DS process • Continuous testing, release • Instantiation from various data products • Evolving process for deep learning • Relevant worked-out templates for vision, NLP and others • Providing appropriate data platforms guidelines
- 28. TDSP Summary: Integrate SDLC + DS workflows Making DS teams more efficient 28 SWD Practices Data Science Workflow and reports
- 29. TDSP components aim to solve DS process challenges 29 Organization Organization, Collaboration Collaboration, quality Collaboration, knowledge accumulation Standard DS Lifecycle Project Structure, Templates & Roles Shared, Distributed Data Platforms and Servers (incl. version control server) Productivity Tools, Shared Utilities Improved DS project execution & delivery
- 31. Resources • TDSP on GitHub: https://github.com/Azure/Microsoft- TDSP
- 32. Public resources, blogs, announcements by Microsoft 32 • Process Overview and Lifecycle: • https://docs.microsoft.com/en-us/azure/machine-learning/data-science-process-overview • Links to TDSP execution on Azure using Azure ML and other data platforms: • AzureML: https://azure.microsoft.com/en-us/documentation/learning-paths/data-science-process/ • Other data platforms (Spark, SQL-server): https://docs.microsoft.com/en-us/azure/machine-learning/data-science-process- walkthroughs • Blogs by Microsoft (releases and announcements): • https://blogs.technet.microsoft.com/machinelearning/2016/10/11/introducing-the-team-data-science-process-from-microsoft/ • https://blogs.technet.microsoft.com/machinelearning/2017/04/05/latest-rev-of-utilities-for-microsoft-team-data-science- process-tdsp-now-available/
- 33. Blogs and announcements by Microsoft Partners and Community 33 • Blogs by Microsoft Partners (not exhaustive): • https://newsignature.com/articles/new-signature-team-data-science-process/ • https://www.blue-granite.com/blog/getting-more-from-your-data-science-teams-organization-and-process- considerations • Blogs and announcements by community (not exhaustive): • https://www.onmsft.com/news/team-data-science-process-hopes-to-help-improve-project-productivity-through-team- learning • https://mspoweruser.com/microsoft-announces-team-data-science-process-agile-methodology-improve-collaboration/ • http://news.thewindowsclub.com/microsoft-data-science-utilities-88002/ • https://www.mgicomputers.com/tech-news/team-data-science-process-hopes-to-help-improve-project-productivity- through-team-learning
- 34. Confidential under NDA Thank you! Contact information: debraj.guhathakurta@microsoft.com