Interpretability in ML & Sparse Linear Regression

Department of Mathematics unchitta@ucla.edu
Interpretability in ML
& Sparse Linear Models
Unchitta Kanjanasaratool
Mentor: Denali Molitor

Several components: model transparency, holistic model interpretability,
modular-level interpretability, local interpretability for a single prediction or a
group of predictions.
Proposed levels of evaluating interpretability (Doshi-Velez & Kim, 2017):
1. Application level
2. Human level
3. Function level
Definition (non-mathematical): Interpretability is the degree to which a
human can consistently explain or interpret why a model makes certain
decisions.
Interpretability in ML: Why It’s Important

We may not always need interpretability e.g. in low-risk or extensively studied
problems, but knowing the why can help us learn more about why a model
might fail.
Transparency & Ethics: The EU for example mandates that automated decisions
must be explainable and should respect fundamental rights.
Interpretability also satisfies human curiosity and learning.
Interpretability helps explain why a model makes certain predictions.
Interpretability in ML: Why It’s Important

Assume a linear relationship between the input and response variables, for example
Y = β0
+ β1
X1
+ … + βp
Xp
+ ϵ ,
we can predict
ŷ = + X1
+ Xp
,
where the ’s are the coefficients in the residual sum of squares (RSS) minimization
problem, namely
β^
= arg minβ0…βp
𝚺i=0,i=p
(yi
- ŷi
)2
.
The linear coefficients (the β’s) make this model easy to interpret on a modular
level, and help us see the level of influence each variable has on the prediction.
Interpretable Models: Linear Regression

In practice however, interpreting linear models can still be hard if there are too many
variables. There are certain variations of linear regression that can help with this
problem, such as the sparse models. An example would be the “least absolute
shrinkage and selection operator,” or lasso regression which minimizes
RSS subject to 𝚺j=0,j=p
|βj
| ≤ s.
This is equivalent to minimizing, with regularization, the quantity
RSS + ƛ 𝚺j=0,j=p
|βj
|,
where the second term in the quantity is some constant lambda times the L1 norm of
the coefficients. (Lambda normally chosen using cross-validation to yield the minimal
β’s).
Sparse Models & Lasso Regression

Intuitively, lasso penalizes large models and selects small coefficients. Unintuitively
(but we will see why), it also yields a model where a number of the coefficients are 0
or essentially close to 0. This is an example of a sparse regression model, which
penalizes large models (i.e. lots of features) and performs variable selection.
The penalization is controlled via lambda: the larger it is, the bigger the sparsity. If
lambda is small enough, lasso will yield the same results as the least square
estimates (standard linear regression).
Sparse Models & Lasso Regression

Having fewer variables often mean better interpretability.
Your model is explained by only a number of significant features, which reduces
complexity and increases explainability. This is especially important when your data
has hundreds or thousands of features; the complexity may be beyond human
comprehension and there may not be enough observations.
Other methods for introducing sparsity to linear models include feature selection
processes, subset selection and step-wise procedures e.g. forward and backward
selection, sparse PCA.
Sparsity & Interpretability

Sparse Property of Lasso
The variable selection property of lasso regression comes from the L1 regularizer.
Consider the following figure in 2D problems. In higher dimensions the constraint
region becomes polytopes (shapes with flat sides/sharp edges).
Courtesy of Introduction to Statistical Learning, G. James, et al.
The red ellipses represent the
contours of the RSS of lasso (left)
and ridge regression (right). The solid
green areas are the corresponding
constraint functions,
and .

Example: UCI Bike Sharing Dataset
Here we are trying to predict the
number of bike rentals as a function of
11 variables. As lambda increases the
response variable depends on smaller
number of predictors.

Example: UCI Bike Sharing Dataset
Likely the most
significant
features

Introduction to Statistical Learning with
Applications in R, Gareth James, et al.
Resources
Interpretable Machine Learning: A Guide for
Making Black Box Models Explainable,
Christopher Molnar

Interpretability in ML & Sparse Linear Regression

More Related Content

Similar to Interpretability in ML & Sparse Linear Regression (20)

Recently uploaded (20)

Interpretability in ML & Sparse Linear Regression