Rsqrd AI: Exploring Machine Learning Model Predictions

Age(years)
Explaining Machine Learning
Model Predictions
Scott Lundberg

Why do we care so much about explainability
in machine learning?

model 55%
chance John will have
repayment problems
John, a bank customer
No loan
Why?!
Why?! AI magic!

Interpretable Accurate
Complex model ✘ ✔
Simple model ✔ ✘
Interpretable or accurate: choose one.
😀 ⚖ 💰
3

Complex models are
inherently complex!
But a single prediction involves only a
small piece of that complexity.
Input value
Output value
5

6
How did we get here?
Base rate Prediction for John
20% 55%

7
Base rate
Work experience = 1 yr
20% 35%
Day trader Open accounts = 1
70%
Capital gains
90%55%

8
The order matters!
Work experience = 1 yr
Day trader
Nobel Prize in 2012
Lloyd Shapley

9
SHapley Additive exPlanation (SHAP) values
Age = 20
Day trader
Shapley values result from averaging over all N! possible orderings.
(NP-hard)

Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Mortality risk model

Reveal rare high-magnitude mortality effects
Mortality model
(F/M)
Conflates the
prevalence of an effect
with the
magnitude of an effect
Mortality risk model

Mortality model
(F/M)

Mortality model
(F/M)
Rare high magnitude effects

Mortality model
(F/M)
Lots of ways to die young…Not many ways to live longer…

Dependence plots reveal the increased
danger of early onset high blood pressure (C)
SHAPvalueforsystolicbloodpressure
withouttheageinteraction
(logrelativeriskofmortality)
Sys
(B)
(logrelativeriskofmortality)
Systolic blood pressure (mmHg)
Age(years)
Kidney model
=
Vertical dispersion is
driven by interaction effects
21

The varying risk of sex over a lifetime
22

Model monitoring
Time
Training performance Test performance
Can you find where we introduced the bug?
24

Model monitoring
Now can you find where we introduced the bug?
25
False True

Model monitoring
Time
Transient electronic medical record
Time
26
False True

Model monitoring
Time
Gradual change in atrial fibrillation
ablation procedure durations
Time
27
False True

Don’t take my word for it, try it yourself J
github.com/slundberg/shap

github.com/slundberg/shap
…
Don’t take my word for it, try it yourself J

30
Important questions to ask when using SHAP:
1. If you are using a model agnostic explainer, have you drawn
enough samples?
2. What background population are you using to estimate the
effect of a feature being “missing”?
3. What model output are you explaining? (log-odds,
probability, rank-order, etc.)
4. Are you perturbing a model in ways that don’t make sense?
(dealing with tightly correlated features)

Rsqrd AI: Exploring Machine Learning Model Predictions

More Related Content

Recently uploaded (20)

Featured (20)

Rsqrd AI: Exploring Machine Learning Model Predictions