Continuous Deployment for Deep Learning

Continuous Deployment
for Deep Learning
—
Nick Pentreath
Principal Engineer
@MLnick

About
IBM Developer / © 2019 IBM Corporation
– @MLnick on Twitter & Github
– Principal Engineer, IBM CODAIT (Center for
Open-Source Data & AI Technologies)
– Machine Learning & AI
– Apache Spark committer & PMC
– Author of Machine Learning with Spark
– Various conferences & meetups
2

CODAIT
Improving the Enterprise AI Lifecycle in Open Source
Center for Open Source
Data & AI Technologies
IBM Developer / © 2019 IBM Corporation 3
CODAIT aims to make AI solutions dramatically
easier to create, deploy, and manage in the
enterprise.
We contribute to and advocate for the open-source
technologies that are foundational to IBM’s AI
offerings.
30+ open-source developers!

Agenda
– Overview of Continuous Integration &
Deployment
– The Machine Learning Workflow
– How is CI/CD for ML Different & Challenges
– Model Asset Exchange
– Conclusion
4

Continuous Integration
& Deployment

Testing
App
Data Store
Public Front-end Services Back-end Services
Service
Remote
Service
Remote
Service

Testing
App
Data Store
Service
Remote
Service
Remote
Service
Unit testing

Testing
App
Data Store
Service
Remote
Service
Remote
Service
Integration testing

Testing
App
Data Store
Service
Remote
Service
Remote
Service
Post-deploy / Acceptance testing

Development Process
Build Test Merge Deploy
Dependencies

Starting Point
Build Test Merge
Dependencies
Manual
Automated
Deploy

Build Test Merge
Dependencies
Manual
Automated
Deploy

Continuous Deployment
Build Test Merge
Dependencies
Manual
Automated
Deploy

Source of changes
Dependencies
Changes come from:
• Our code
• Internal dependencies
• 3rd party dependencies

The Machine Learning
Workflow

Perception

In reality the
workflow spans teams …

… and tools …

… and is a small (but critical!)
piece of the puzzle
*Source: Hidden Technical Debt in Machine Learning Systems

What is a “model”?

Deep learning pipeline
beagle: 0.82
basset: 0.09
bluetick: 0.07
...
Input image Image pre-processing Prediction
Decode image
Resize
Normalization
Convert types / format
Inference Post-processing
[0.2, 0.3, … ]
(label, prob)
Sort
Label map
PIL, OpenCV, tf.image,
…
Custom
Python
* Logos trademarks of their respective projects

Pipelines, not Models
– Deploying (and testing) just the model
part of the workflow is not enough
– Entire pipeline must be taken into
account
• Data transforms
• Feature extraction & pre-processing
• DL / ML model
• Prediction transformation
– Even ETL is part of the pipeline!
– Pipelines in frameworks
• scikit-learn
• Spark ML pipelines
• TensorFlow Transform
• pipeliner (R)

for Machine Learning

Source of changes
Dependencies
Changes come from:
• Our code
• Internal dependencies
• 3rd party dependencies
• Data
• Model
• Time
Data Model

Data
– Data types
• Images, video, audio, raw text,
unstructured
– Size
– Schemas
25

Models
– Size of models
– Resource requirements
– Hardware
• CPU, GPU, TPU, Mobile, Edge
– Need to manage and bridge many
different languages, frameworks
– Formats
– State of the art is changing very
rapidly
26

Monitoring & Feedback
over Time

Monitoring
Performance Business
Monitoring
Software
Traditional software monitoring
Latency, throughput, resource usage, etc
Model performance metrics
Traditional ML evaluation measures
(accuracy, prediction error, AUC etc)
Business metrics
Impact of predictions on business
outcomes
• Additional revenue - e.g. uplift from recommender
• Cost savings – e.g. value of fraud prevented
• Metrics implicitly influencing these – e.g. user
engagement

Feedback
Data
Transform
TrainDeploy
Feedback
Adapt
An intelligent system must automatically
learn from & adapt to the world around it
Continual learning
Retraining, online learning,
reinforcement learning
Feedback loops
Explicit: models create or directly
influence their own training data
Implicit: predictions influence behavior
in longer-term or indirect ways
Humans in the loop

AI Fairness,
Transparency &
Security
31IBM Developer / © 2019 IBM Corporation
https://github.com/IBM/AIX360/
https://github.com/IBM/AIF360
https://github.com/IBM/adversarial-robustness-toolbox

32
Model
Asset
eXchange
(MAX)
ibm.biz/model-exchange

What is
MAX?
33
- One place for state-of-art open
source deep learning models
- Wide variety of domains
- Tested code and IP
- Free and open source
- Both trainable and trained versions

Deployable asset on MAX
Data ExpertiseComputeModel
REST API specificationPre Trained ModelI/O processing
Deployable Asset on Model Asset
Exchange
MetadataInferenceSwagger
Specification

MetadataInferenceSwagger
Specification

Hosted resources
Model
Service
Web App
Kubernetes

Trainable asset on MAX
Exchange

Trainable asset on MAX
Exchange
Standardized scripts & wrappers

MAX and CI/CD
– TravisCI for GitHub repositories
– Jenkins for more complex workflows
• Deploying long-running instances to
Kubernetes
• Periodic jobs
• Trainable model testing (using Watson
Machine Learning service)

Conclusion
– ML Pipelines, not just models
– Need to take into account
• Data
• Models
• Variation over time
• Fairness, explainability,
transparency
– Containers / Kube?
• KubeFlow, MLFlow
• Tekton

Thank you
Sign up for IBM Cloud and try Watson Studio: https://ibm.biz/BdznGk
codait.org
twitter.com/MLnick
github.com/MLnick
developer.ibm.com
42
Model Asset Exchange:
https://ibm.biz/model-exchange

Continuous Deployment for Deep Learning

More Related Content

What's hot (20)

Similar to Continuous Deployment for Deep Learning (20)

More from Databricks (20)

Recently uploaded (20)

Continuous Deployment for Deep Learning