"Deployment for free": removing the need to write model deployment code at Stitch Fix

“Deployment for free”:
removing the need to write model
deployment code at Stitch Fix
mlconf April 2021
Stefan Krawczyk
#mlconf #MLOps #machinelearning
@stefkrawczyk
linkedin.com/in/skrawczyk
Try out Stitch Fix → goo.gl/Q3tCQ3

> Stitch Fix
“Deployment for free”
Model Envelope & envelope mechanics
Impact of being on-call
Summary & Future Work

Stitch Fix is a personal styling service
Key points:
1. Very algorithmically driven company
2. Single DS Department: Algorithms (145+)
3. “Full Stack Data Science”
a. No reimplementation handoff
b. End to end ownership
c. Built on top of data platform tools & abstractions.
For more information: https://algorithms-tour.stitchﬁx.com/ & https://cultivating-algos.stitchﬁx.com/
3

Where do I ﬁt in?
Stefan Krawczyk
Mgr. Data Platform - Model Lifecycle
4
Pre-covid look

Stitch Fix
> “Deployment for free”

Typical Model Deployment Process
6
● Many ways to approach.
● Heavily impacts MLOps.

Model Deployment at Stitch Fix
7
Once a model is in
an envelope...
This comes for free!

Who owns what?
8
DS Concerns Platform Concerns DS Concerns

Deployments are “triggered”
9
Guess who is on-call?

Reality: two steps to get a model to production
10
Self-service: takes <1 hour
No code is written!
Can be a terminal point.
Step 1 Step 2

Step 1. save a model via Model Envelope API
11
etl.py
import model_envelope as me
from sklearn import linear_model
df_X, df_y = load_data_somehow()
model = linear_model.LogisticRegression(multi_class='auto')
model.fit(df_X, df_y)
my_envelope = me.save_model(instance_name='my_model_instance_name',
instance_description='my_model_instance_description',
model=model,
query_function='predict',
api_input=df_X, api_output=df_y,
tags={'canonical_name':'foo-bar'})
Note: no deployment trigger in ETL code.

Step 2a. deploy model as a microservice
12
Go to Model Envelope Registry UI:
1) Create deployment conﬁguration.
2) Create Rule for auto deployment.
a) Else query for model & hit deploy.
3) Done.
Result:
● Web service with API endpoints
○ Comes with a Swagger UI & schema →
● Model in production < 1 hour.

Step 2b. deploy model as a batch task
13
Create workflow configuration:
1) Create batch inference task in workflow.
a) Specify Rule & inputs + outputs.
2) Deploy workflow.
3) Done.
Result:
● Spark or Python task that creates a table.
● We keep an inference log.
● Model in production < 1 hour.

Stitch Fix
> Model Envelope & envelope mechanics

Q: What is the Model Envelope? A: It’s a container.
15
Enables treating the
model as a “black box”
Enables thinking about models as a “black box”.

🤔 Wait this feels familiar?
16
You: “MLFlow much?”
Me: Yes & No.
This is all internal code -- nothing from open source.
In terms of functionality we’re closer to a mix of:
● MLFlow
● ModelDB
● TFX
But this talk is too short to cover everything...

Typical Model Envelope use
1. call save_model() right after model creation in an ETL.
2. also have APIs to save metrics & hyperparameters, and retrieve envelopes.
3. once in an ✉ information is immutable except:
a. tags -- for curative purposes.
b. metrics -- can add/adjust metrics.
#mlconf #MLOps #machinelearning 17

What does save_model() do?
1
2
3
4

What does save_model() do?
Let’s dive deeper into these.
1
2
3
4

How do we infer a Model API Schema?
Goal: infer from code rather than explicit speciﬁcation.
Require either fully annotated functions with only python/typing standard types:
def good_predict_function(self, x: float, y: List[int]) -> List[float]:
def predict_needs_examples_function(self, x: pd.Dataframe, y):
model=model,
Or, example inputs that are inspected to get a schema from:
required for DF inputs →

Model API Schema - Under the hood
● One of the most complex parts of the code base (90%+ test coverage!)
● We make heavy use of the typing_inspect module & isinstance().
○ We create a schema similar to TFX.
● Key component to enable exercising models in different contexts.
○ Enables code creation and input/output validation.
● Current limitations: no default values, one function per envelope.
21

How do we capture python dependencies?
import model_envelope as me
from sklearn import linear_model
df_X, df_y = load_data_somehow()
model = linear_model.LogisticRegression(multi_class='auto')
model.fit(df_X, df_y)
model=model,
22
Point: no explicit passing of scikit-learn to save_model().

How do we capture python dependencies?
23
Assumption:
We all run on the same* base linux environment in training & production.
Store the following in the Model Envelope:
● Result of import sys; sys.version_info
● Results of > pip freeze
● Results of > conda list --export
Local python modules (not installable):
● Add modules as part of save_model() call.
● We store them with the model bytes.

How do we build the python deployment env.?
24
Filter:
● hard coded list of dependencies to ﬁlter. E.g. jupyterhub.
● upkeep cheap; add/update every few months.

Stitch Fix
> Impact of being on-call

Remember this split:
26
My team is on-call for

27
Two truths:
● No one wants to be paged.
● No one wants to be paged for a model they didn’t write!
But, this incentivizes Platform to build out MLOps capabilities:
● Capture bad models before they’re deployed!
● Enable observability, monitoring, and alerting to speed up debugging.
Luckily we have autonomy and freedom to do so!
a pager

What can we change?
28
API
Automatic capture == license to change:
● Model API schema
● Dependency capture
● Environment info: git, job, etc.
Incentives for DS to additionally provide:
● Datasets for analysis
● Metrics
● Tags
Deployment
MLOps approaches to:
● Model validation
● Model deployment & rollback
● Model deployment vehicle:
○ From logging, monitoring, alerting
○ To architecture: microservice, or Ray, or?
● Dashboarding/UIs

Overarching beneﬁt
29
1. Data Scientists get to focus more on modeling.
a. more business wins.
2. Platform focuses on MLOps:
a. can be a rising tide that raises all boats!

Stitch Fix
> Summary & Future Work

We enable deployment for free by:
● Capturing a comprehensive model artifact we call the Model Envelope.
● The Model Envelope facilitates code & environment generation for model deployment.
● Platform owns the Model Envelope and is on-call for generated services & tasks.
Business wins:
● Data Scientists get to focus more on modeling.
● Platform is incentivized to improve and iterate on MLOps practices.
Summary - “Deployment for free”

Future Work
● Better MLOps features:
○ Observability, scalable data capture, & alerting.
○ Model Validation & CD patterns.
● “Models on Rails”:
○ Target speciﬁc SLA requirements.
● Conﬁguration driven model creation:
○ Abstract away glue code required to train & save models.

Thank you! We’re hiring! Questions?
Try out Stitch Fix → goo.gl/Q3tCQ3
@stefkrawczyk
linkedin.com/in/skrawczyk

"Deployment for free": removing the need to write model deployment code at Stitch Fix

More Related Content

What's hot (20)

Similar to "Deployment for free": removing the need to write model deployment code at Stitch Fix (20)

Recently uploaded (20)

"Deployment for free": removing the need to write model deployment code at Stitch Fix