Data Con LA 2018 - Towards Data Science Engineering Principles by Joerg Schad
2 likes238 views
This document discusses the need for data science engineering principles to standardize the machine learning development process. It outlines several challenges in machine learning projects including data quality issues, requirements engineering, continuous integration, testing, debugging, profiling, monitoring, and model serving. Standardizing these aspects into a repeatable pipeline can help address challenges like building reliable models, deploying models with zero downtime, and ensuring model quality during both training and serving. The document advocates for applying software engineering best practices to machine learning in order to more easily build and scale impactful AI technologies.
Data Con LA 2018 - Towards Data Science Engineering Principles by Joerg Schad
- 1. Towards Data Science Principles @joerg_schad
- 2. tl;dr (Why) Do we need Data Science Engineering Principles? 2 Software Engineering The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software IEEE Standard Glossary of Software Engineering Terminology
- 3. Jörg Schad Technical Lead/Engineer Deep Learning ● Core Mesos developer at Mesosphere ● Twitter: @joerg_schad
- 4. Why is machine learning taking off? 4
- 5. 5
- 6. CONFIDENTIAL
- 9. What you want to be doing 9 Data (clean) Write awesome ML code Train (once) Deploy (not me)
- 10. © 2018 Mesosphere, Inc. All Rights Reserved. Deep Learning: The Challenges 10
- 11. 1. Data Preparation & Model Engineering 2. Model Training 3. Monitoring 4. Debugging 5. Model Serving
- 12. 1. Data Preparation using Spark 7. Streaming of requests ... Public Cloud Pipeline 1. Data Preparation & Model Engineering 2. Model Training 3. Monitoring 4. Debugging 5. Model Serving
- 13. 1. Data Preparation using Spark 7. Kafka stream of requests DIY Open Source Pipeline 1. Data Preparation & Model Engineering 2. Model Training 3. Monitoring 4. Debugging 5. Model Serving
- 14. 14 Sculley, D., Holt, G., Golovin, D. et al. Hidden Technical Debt in Machine Learning Systems What you’re actually doing
- 15. 15
- 17. 17 • Regulatory Requirements • Dependencies • CI/CD • Git • MLFlow Challenge: Repeatable Builds Step 1: Training (In Data Center - Over Hours/Days/Weeks) Dog Input: Lots of Labeled Data Output: Trained Model Deep neural network model
- 18. 18
- 19. 19Data Science CI/CD Train Model(s) Optimize Model(s) Test Model(s) Build Serving Container(s) Deploy Model Engineering Continuous Integration
- 20. 20 • What dataset(s)? • What target/serving environment? • What model architecture? • Pre-trained model available? • How many training resources? Challenge: Requirements Engineering
- 21. 21 Challenge: Data Quality Challenges • Data is typically not ready to be consumed by ML job* – Data Cleaning • Missing/incorrect labels – Data Preparation • Same Format • Same Distribution * Demo datasets are a fortunate exception :)
- 22. 22 Challenge: Data Quality • Data is typically not ready to be consumed by ML job* – Data Cleaning • Missing/incorrect labels – Data Preparation • Same Format • Same Distribution * Demo datasets are a fortunate exception :) Don’t forget about the serving environment!!
- 23. 23 “Libraries” • Different Frameworks • Existing architectures • Pretrained models
- 24. © 2017 Mesosphere, Inc. All Rights Reserved. TensorFlow Hub 24 https://www.tensorflow.org/hub/
- 25. 25 Writing Distributed Model Functions
- 26. 26 Testing • Training/Test/Validation Datasets • Unit Tests? • Different factors – Accuracy – Serving performance – …. • A/B Testing with live Data
- 29. 29 Monitoring Challenges • Understand {...} • Debug • Model Quality – Accuracy – Training Time – … • Overall Architecture – Availability – Latencies – ... Solutions • TensorBoard • Traditional Cluster Monitoring Tool
- 30. 30 Serving Challenges • How to Deploy Models? – Zero Downtime – Canary Solutions • TensorFlow Serving
- 31. 31 Serving Challenges • Small/Fast model without losing too much performance • 500 KB models….
- 32. Data Science Pipeline Continuous Integration Monitoring & Operations Distributed Data Storage and Streaming Data Preparation and Analysis Storage of trained Models and Metadata Use trained Model for Inference Distributed Training using Machine Learning Frameworks Data & Streaming Model Engineering Model Management Model Serving Model Training Management
- 33. © 2018 Mesosphere, Inc. All Rights Reserved. 33 THANK YOU! ANY QUESTIONS? @dcos users@dcos.io /groups/8295652 /dcos /dcos/examples /dcos/demos chat.dcos.io https://mesosphere.com/resources/building-data-science-platform/
- 34. CONFIDENTIAL Make it insanely easy to build and scale world-changing technology