SlideShare a Scribd company logo
Explainable AI – Making ML and DL models more interpretable
Explainable AI – Making ML and DL models more interpretable
About Me
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
2
Aditya
Bhattacharya
I am currently working as the Lead AI/ML Engineer at West Pharmaceutical
Services with the responsibility of leading and managing a global AI team and
creating AI products and platforms at West. I am well seasoned in Data Science,
Machine Learning, IoT and Software Development. and has established the AI
Centre of Excellence and worked towards democratizing AI practice for West
Pharmaceuticals and Microsoft. In the Data Science domain, Computer Vision,
Time-Series Analysis, Natural Language Processing and Speech analysis are my
forte.
Apart from my day job, I am an AI Researcher at an NGO called MUST Research,
and I am one of the faculty members for the MUST Research Academy :
https://must.co.in/acad
Website : https://aditya-bhattacharya.net/
LinkedIn: https://www.linkedin.com/in/aditya-bhattacharya-b59155b6/
Key Topics 1. Necessity and Principles of Explainable AI
2. Model Agnostic XAI for ML models
3. Model Agnostic XAI for DL models.
4. Popular frameworks for XAI
5. Research Questions to consider
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 3
Necessity and Principles of
Explainable AI
5
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
6
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
7
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
8
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
XAI
Trace model
prediction from
logic of math to
nature of data
Understand the
reasoning behind
each model
predictions
Understand the
model using which
AI decision
making is based
Traceable
AI
Reasonable
AI
Understand
able
AI
9
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
1 0
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Model
Agnostic
Results
Visualizations
Influence
Methods
Example
Based
Methods
Knowledge
Extractions
1 1
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Using Surrogate models like linear
models or decision trees to explain
complex models
Estimates the
importance or relevant
features.
Extracting statistical
information from input
and the output
Select instances of the datasets that
explains the behaviour of the model
1 2
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Problem, Data,
Audience
Post Hoc
Analysis
Model
Predictive
Accuracy
Descriptive
Accuracy
Iterative
Explainability
Model Agnostic XAI for ML models
1 4
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
1 5
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
1 6
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Explainer
Surrogate Models
Predictions
Blackbox ML Model
1 7
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 1 8
Prediction: Deny Loan
Loan Application
Suggestion: Increase your salary by 50K & pay your credit card bills on time for next 3 months
Predictive
Model
Loan Applicant
Counterfactual Generation
Algorithm
Model Agnostic XAI for DL models
1 9
2 0
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Understanding flow of information through gradient flow between the
layers of Deep Neural Network model using the following approaches:
1. Saliency Maps
2. Guide Backpropagation
3. Gradient Class Activation Methods
• Layer GRAD CAM
• Layer Conductance using GRAD CAM
• Layer Activation using GRAD CAM
Saliency Maps Guided Backprop GRAD CAM Layer Conductance Layer Activation
Can such explainability
methods be applied for
complex models?
2 1
Explain ab l e AI: Making ML and DL models more interpr et a b l e
2 2
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Image Captioning
using Attention based
Encoder-Decoder
Architecture
2 3
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
[Kim et. al., 2018]
Zebra
(0.97)
How important is the notion of “stripes” for this prediction?
Testing with Concept Activation Vectors (TCAV) is an interpretability method to understand what signals
your neural networks models uses for prediction.
https://github.com/tensorflow/tcav
Pattern representation plays
a key role in decision making
from both images and text.
2 4
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
[Tan et. al., 2019]
Model
Predictions
Label 1
Label 1
.
Label 2
.
v1, v2.
v11,
v12
.
Data
Explainer
Interpretable Mimic Learning – Compressing information from Deep Networks to Shallow Network
2 5
Explainable AI: Making ML and DL models more interpretable
What features need to be changed and by how much to flip a model’s prediction?
[Goyal et. al., 2019]
Popular frameworks for XAI
Explain ab l e AI: Making ML and DL models more interpr et a b l e
Popular frameworks for XAI
2 7
LIME
Local Interpretable
Model-agnostic
Explanations is
interpretability
framework that
works on
structured data,
text and image
classifiers.
SHAP
SHAP (SHapley
Additive
exPlanations) is a
game theoretic
approach to
explain the output
of any machine
learning model.
ELI5
Explain like I am 5
is another popular
framework that
helps to debug
machine learning
classifiers and
explain their
predictions.
SKATER
Skater is a unified
framework for XAI
for all forms of
models both
globally(inference
on the basis of a
complete data set)
and
locally(inference
about an individual
prediction).
TCAV
Testing with
Concept Activation
Vectors (TCAV) is a
new interpretability
method to
understand what
signals your neural
networks models
uses for prediction.
Explainable AI: Making ML and DL models more interpretable
2 8
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
• Behind the workings of LIME lies the assumption that every complex model is linear on a local scale. LIME tries
to fit a simple model around a single observation that will mimic how the global model behaves at that
locality.
• Create the perturbed data and predict the output on the perturbed data
• Create discretized features and find the Euclidean distance of perturbed data to the original observation
• Convert distance to similarity score and select the top n features for the model
• Create a linear model and explain the prediction
2 9
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
The lime package is on PyPI. `pip install lime`
3 0
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
There is a high-speed exact algorithm for tree ensemble methods (Tree SHAP arXiv paper). Fast C++
implementations are supported for XGBoost, LightGBM, CatBoost, and scikit-learn tree models!
• SHAP assigns each feature an importance
value for a particular prediction.
• Its novel components include: the
identification of a new class of additive
feature importance measures, and theoretical
results showing there is a unique solution in
this class with a set of desirable properties.
• Typically, SHAP values try to explain the
output of a model (function) as a sum of the
effects of each feature being introduced into
a conditional expectation. Importantly, for
non-linear functions the order in which
features are introduced matters.
SHAP can be installed from PyPI
3 1
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
The following figure from the KDD 18 paper, Consistent Individualized Feature
Attribution for Tree Ensembles summarizes this in a nice way!
SHAP Summary Plot
SHAP Dependence Plots
3 2
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Available from pypi. pip install eli5
Check docs for more.
3 3
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
SKATER provides an unified framework for both Global and Local Interpretation.
Feature Importance Partial Dependency Plots
LIME integration for explanability
Project Link:
3 4
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Testing with Concept Activation Vectors (TCAV)
is a new interpretability method to understand
what signals your neural networks models uses
for prediction.
What's special about TCAV compared to
other methods?
TCAV instead shows importance of high
level concepts (e.g., color, gender, race)
for a prediction class - this is how humans
communicate!
TCAV gives an explanation that is generally true for a class of interest, beyond one image (global
explanation).
For example, for a given class, we can show how much race or gender was important for classifications in
InceptionV3. Even though neither race nor gender labels were part of the training input!
pip install tcav https://github.com/tensorflow/tcav
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
3 5
The Concept Activation Vectors (CAVs) provide an interpretation of a neural net’s internal
state in terms of human-friendly concepts. TCAV uses directional derivatives to quantify the
degree to which a user-defined idea is vital to a classification result–for example, how sensitive
a prediction of “zebra” is to the presence of stripes.
TCAV essentially learns ‘concepts’ from examples. For instance, TCAV needs a couple of
examples of ‘female’, and something ‘not female’ to learn a “gender” concept. The goal of
TCAV is to determine how much a concept (e.g., gender, race) was necessary for a prediction
in a trained model even if the concept was not part of the training.
Research question to consider …
All these frameworks are great
and can bring interpretability
to a great extent, but can non-
expert consumers of AI
models interpret these
interpretability methods?
3 7
Explain ab l e AI: Making ML and DL models more interpr et a b l e
Summary
• Why is Explainable AI (XAI) important?
• Commonly used Model Agnostic XAI for ML models
• Commonly used Model Agnostic XAI for DL models.
• Popular frameworks for XAI
• Can we evolve XAI and extend explainability to non-expert
users?
3 8
Explain ab l e AI: Making ML and DL models more interpr et a b l e
Thank you
Aditya Bhattacharya
https://aditya-bhattacharya.net/
aditya.bhattacharya2016@gmail.com
https://www.linkedin.com/in/aditya-bhattacharya-b59155b6/
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 3 9

More Related Content

PDF
Explainability and bias in AI
PDF
DC02. Interpretation of predictions
PPTX
Explainable AI
PPTX
Explainable AI in Industry (KDD 2019 Tutorial)
PPTX
Explainable AI in Industry (WWW 2020 Tutorial)
PDF
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
PDF
Explainable AI (XAI) - A Perspective
PPTX
Explainable AI
Explainability and bias in AI
DC02. Interpretation of predictions
Explainable AI
Explainable AI in Industry (KDD 2019 Tutorial)
Explainable AI in Industry (WWW 2020 Tutorial)
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Explainable AI (XAI) - A Perspective
Explainable AI

What's hot (20)

PPTX
Explainable AI in Industry (AAAI 2020 Tutorial)
PDF
Interpretable machine learning : Methods for understanding complex models
PDF
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
PDF
Artificial Intelligence Machine Learning Deep Learning PPT PowerPoint Present...
PPTX
PDF
Interpretable Machine Learning
PDF
Unsupervised Anomaly Detection with Isolation Forest - Elena Sharova
PPTX
Explainable Machine Learning (Explainable ML)
PPTX
Explainable AI in Industry (FAT* 2020 Tutorial)
PPTX
Interpretable machine learning
PDF
Explainable AI
PPTX
An Introduction to XAI! Towards Trusting Your ML Models!
PDF
Big Data & Artificial Intelligence
PDF
Machine Learning Explanations: LIME framework
PPTX
Interpretable Machine Learning
PDF
Explainable AI (XAI)
PDF
Machine Learning
PDF
The 7 steps of Machine Learning
PDF
Data Science Tutorial | Introduction To Data Science | Data Science Training ...
PDF
Intro to LLMs
Explainable AI in Industry (AAAI 2020 Tutorial)
Interpretable machine learning : Methods for understanding complex models
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
Artificial Intelligence Machine Learning Deep Learning PPT PowerPoint Present...
Interpretable Machine Learning
Unsupervised Anomaly Detection with Isolation Forest - Elena Sharova
Explainable Machine Learning (Explainable ML)
Explainable AI in Industry (FAT* 2020 Tutorial)
Interpretable machine learning
Explainable AI
An Introduction to XAI! Towards Trusting Your ML Models!
Big Data & Artificial Intelligence
Machine Learning Explanations: LIME framework
Interpretable Machine Learning
Explainable AI (XAI)
Machine Learning
The 7 steps of Machine Learning
Data Science Tutorial | Introduction To Data Science | Data Science Training ...
Intro to LLMs
Ad

Similar to Explainable AI - making ML and DL models more interpretable (20)

PPTX
ODSC APAC 2022 - Explainable AI
PDF
PPTX
Responsible AI in Industry: Practical Challenges and Lessons Learned
PDF
​​Explainability in AI and Recommender systems: let’s make it interactive!
PDF
Model Evaluation in the land of Deep Learning
PDF
DSDT meetup July 2021
PDF
Practical Explainable AI: How to build trustworthy, transparent and unbiased ...
PDF
Reds interpretability report
PDF
Evaluating text classification with explainable artificial intelligence
PPTX
Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...
PDF
Towards Human-Centered Machine Learning
PPTX
Machine Learning Interpretability - Mateusz Dymczyk - H2O AI World London 2018
PPTX
Explainable AI (XAI) for college seminar.pptx
PPTX
Explainable-Artificial-Intelligence-XAI-A-Deep-Dive (1).pptx
PDF
Human-Centered Interpretable Machine Learning
PDF
GDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for Everyone
PPTX
Introduction to Interpretable Machine Learning
PDF
Can Machine Learning Models be Trusted? Explaining Decisions of ML Models
PDF
Explainable Ai For Practitioners Designing And Implementing Explainable Ml So...
PPTX
UNIT-5 Advanced Topic in Data Science.pptx
ODSC APAC 2022 - Explainable AI
Responsible AI in Industry: Practical Challenges and Lessons Learned
​​Explainability in AI and Recommender systems: let’s make it interactive!
Model Evaluation in the land of Deep Learning
DSDT meetup July 2021
Practical Explainable AI: How to build trustworthy, transparent and unbiased ...
Reds interpretability report
Evaluating text classification with explainable artificial intelligence
Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...
Towards Human-Centered Machine Learning
Machine Learning Interpretability - Mateusz Dymczyk - H2O AI World London 2018
Explainable AI (XAI) for college seminar.pptx
Explainable-Artificial-Intelligence-XAI-A-Deep-Dive (1).pptx
Human-Centered Interpretable Machine Learning
GDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for Everyone
Introduction to Interpretable Machine Learning
Can Machine Learning Models be Trusted? Explaining Decisions of ML Models
Explainable Ai For Practitioners Designing And Implementing Explainable Ml So...
UNIT-5 Advanced Topic in Data Science.pptx
Ad

More from Aditya Bhattacharya (9)

PPTX
Directive Explanations for Monitoring the Risk of Diabetes Onset - ACM IUI 2023
PPTX
Accelerating Data Science and Machine Learning Workflow with Azure Machine Le...
PPTX
Machine learning and Deep learning on edge devices using TensorFlow
PPTX
Time series Segmentation & Anomaly Detection
PPTX
Application of Masked RCNN for segmentation of brain haemorrhage from Compute...
PPTX
Aditya Bhattacharya - Enterprise DL - Accelerating Deep Learning Solutions to...
PPTX
Aditya Bhattacharya Chest XRay Image Analysis Using Deep Learning
PPTX
Computer vision-must-nit-silchar-ml-hackathon-2019
PPTX
Computer vision-nit-silchar-hackathon
Directive Explanations for Monitoring the Risk of Diabetes Onset - ACM IUI 2023
Accelerating Data Science and Machine Learning Workflow with Azure Machine Le...
Machine learning and Deep learning on edge devices using TensorFlow
Time series Segmentation & Anomaly Detection
Application of Masked RCNN for segmentation of brain haemorrhage from Compute...
Aditya Bhattacharya - Enterprise DL - Accelerating Deep Learning Solutions to...
Aditya Bhattacharya Chest XRay Image Analysis Using Deep Learning
Computer vision-must-nit-silchar-ml-hackathon-2019
Computer vision-nit-silchar-hackathon

Recently uploaded (20)

PPTX
Major-Components-ofNKJNNKNKNKNKronment.pptx
PPTX
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
PDF
.pdf is not working space design for the following data for the following dat...
PDF
Clinical guidelines as a resource for EBP(1).pdf
PPTX
Acceptance and paychological effects of mandatory extra coach I classes.pptx
PDF
Galatica Smart Energy Infrastructure Startup Pitch Deck
PPTX
Introduction to Knowledge Engineering Part 1
PPTX
Introduction to machine learning and Linear Models
PPT
Miokarditis (Inflamasi pada Otot Jantung)
PPTX
Supervised vs unsupervised machine learning algorithms
PPTX
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
PPTX
Business Ppt On Nestle.pptx huunnnhhgfvu
PPT
Reliability_Chapter_ presentation 1221.5784
PPTX
IB Computer Science - Internal Assessment.pptx
PPTX
Data_Analytics_and_PowerBI_Presentation.pptx
PPTX
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
PPTX
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
PDF
Foundation of Data Science unit number two notes
PDF
Fluorescence-microscope_Botany_detailed content
PPTX
ALIMENTARY AND BILIARY CONDITIONS 3-1.pptx
Major-Components-ofNKJNNKNKNKNKronment.pptx
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
.pdf is not working space design for the following data for the following dat...
Clinical guidelines as a resource for EBP(1).pdf
Acceptance and paychological effects of mandatory extra coach I classes.pptx
Galatica Smart Energy Infrastructure Startup Pitch Deck
Introduction to Knowledge Engineering Part 1
Introduction to machine learning and Linear Models
Miokarditis (Inflamasi pada Otot Jantung)
Supervised vs unsupervised machine learning algorithms
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
Business Ppt On Nestle.pptx huunnnhhgfvu
Reliability_Chapter_ presentation 1221.5784
IB Computer Science - Internal Assessment.pptx
Data_Analytics_and_PowerBI_Presentation.pptx
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
Foundation of Data Science unit number two notes
Fluorescence-microscope_Botany_detailed content
ALIMENTARY AND BILIARY CONDITIONS 3-1.pptx

Explainable AI - making ML and DL models more interpretable

  • 1. Explainable AI – Making ML and DL models more interpretable Explainable AI – Making ML and DL models more interpretable
  • 2. About Me E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 2 Aditya Bhattacharya I am currently working as the Lead AI/ML Engineer at West Pharmaceutical Services with the responsibility of leading and managing a global AI team and creating AI products and platforms at West. I am well seasoned in Data Science, Machine Learning, IoT and Software Development. and has established the AI Centre of Excellence and worked towards democratizing AI practice for West Pharmaceuticals and Microsoft. In the Data Science domain, Computer Vision, Time-Series Analysis, Natural Language Processing and Speech analysis are my forte. Apart from my day job, I am an AI Researcher at an NGO called MUST Research, and I am one of the faculty members for the MUST Research Academy : https://must.co.in/acad Website : https://aditya-bhattacharya.net/ LinkedIn: https://www.linkedin.com/in/aditya-bhattacharya-b59155b6/
  • 3. Key Topics 1. Necessity and Principles of Explainable AI 2. Model Agnostic XAI for ML models 3. Model Agnostic XAI for DL models. 4. Popular frameworks for XAI 5. Research Questions to consider E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 3
  • 4. Necessity and Principles of Explainable AI
  • 5. 5 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
  • 6. 6 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
  • 7. 7 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
  • 8. 8 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e XAI Trace model prediction from logic of math to nature of data Understand the reasoning behind each model predictions Understand the model using which AI decision making is based Traceable AI Reasonable AI Understand able AI
  • 9. 9 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
  • 10. 1 0 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
  • 11. Model Agnostic Results Visualizations Influence Methods Example Based Methods Knowledge Extractions 1 1 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Using Surrogate models like linear models or decision trees to explain complex models Estimates the importance or relevant features. Extracting statistical information from input and the output Select instances of the datasets that explains the behaviour of the model
  • 12. 1 2 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Problem, Data, Audience Post Hoc Analysis Model Predictive Accuracy Descriptive Accuracy Iterative Explainability
  • 13. Model Agnostic XAI for ML models
  • 14. 1 4 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
  • 15. 1 5 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
  • 16. 1 6 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Explainer Surrogate Models Predictions Blackbox ML Model
  • 17. 1 7 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
  • 18. E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 1 8 Prediction: Deny Loan Loan Application Suggestion: Increase your salary by 50K & pay your credit card bills on time for next 3 months Predictive Model Loan Applicant Counterfactual Generation Algorithm
  • 19. Model Agnostic XAI for DL models 1 9
  • 20. 2 0 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Understanding flow of information through gradient flow between the layers of Deep Neural Network model using the following approaches: 1. Saliency Maps 2. Guide Backpropagation 3. Gradient Class Activation Methods • Layer GRAD CAM • Layer Conductance using GRAD CAM • Layer Activation using GRAD CAM Saliency Maps Guided Backprop GRAD CAM Layer Conductance Layer Activation
  • 21. Can such explainability methods be applied for complex models? 2 1 Explain ab l e AI: Making ML and DL models more interpr et a b l e
  • 22. 2 2 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Image Captioning using Attention based Encoder-Decoder Architecture
  • 23. 2 3 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e [Kim et. al., 2018] Zebra (0.97) How important is the notion of “stripes” for this prediction? Testing with Concept Activation Vectors (TCAV) is an interpretability method to understand what signals your neural networks models uses for prediction. https://github.com/tensorflow/tcav Pattern representation plays a key role in decision making from both images and text.
  • 24. 2 4 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e [Tan et. al., 2019] Model Predictions Label 1 Label 1 . Label 2 . v1, v2. v11, v12 . Data Explainer Interpretable Mimic Learning – Compressing information from Deep Networks to Shallow Network
  • 25. 2 5 Explainable AI: Making ML and DL models more interpretable What features need to be changed and by how much to flip a model’s prediction? [Goyal et. al., 2019]
  • 26. Popular frameworks for XAI Explain ab l e AI: Making ML and DL models more interpr et a b l e
  • 27. Popular frameworks for XAI 2 7 LIME Local Interpretable Model-agnostic Explanations is interpretability framework that works on structured data, text and image classifiers. SHAP SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the output of any machine learning model. ELI5 Explain like I am 5 is another popular framework that helps to debug machine learning classifiers and explain their predictions. SKATER Skater is a unified framework for XAI for all forms of models both globally(inference on the basis of a complete data set) and locally(inference about an individual prediction). TCAV Testing with Concept Activation Vectors (TCAV) is a new interpretability method to understand what signals your neural networks models uses for prediction. Explainable AI: Making ML and DL models more interpretable
  • 28. 2 8 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e • Behind the workings of LIME lies the assumption that every complex model is linear on a local scale. LIME tries to fit a simple model around a single observation that will mimic how the global model behaves at that locality. • Create the perturbed data and predict the output on the perturbed data • Create discretized features and find the Euclidean distance of perturbed data to the original observation • Convert distance to similarity score and select the top n features for the model • Create a linear model and explain the prediction
  • 29. 2 9 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e The lime package is on PyPI. `pip install lime`
  • 30. 3 0 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e There is a high-speed exact algorithm for tree ensemble methods (Tree SHAP arXiv paper). Fast C++ implementations are supported for XGBoost, LightGBM, CatBoost, and scikit-learn tree models! • SHAP assigns each feature an importance value for a particular prediction. • Its novel components include: the identification of a new class of additive feature importance measures, and theoretical results showing there is a unique solution in this class with a set of desirable properties. • Typically, SHAP values try to explain the output of a model (function) as a sum of the effects of each feature being introduced into a conditional expectation. Importantly, for non-linear functions the order in which features are introduced matters. SHAP can be installed from PyPI
  • 31. 3 1 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e The following figure from the KDD 18 paper, Consistent Individualized Feature Attribution for Tree Ensembles summarizes this in a nice way! SHAP Summary Plot SHAP Dependence Plots
  • 32. 3 2 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Available from pypi. pip install eli5 Check docs for more.
  • 33. 3 3 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e SKATER provides an unified framework for both Global and Local Interpretation. Feature Importance Partial Dependency Plots LIME integration for explanability Project Link:
  • 34. 3 4 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Testing with Concept Activation Vectors (TCAV) is a new interpretability method to understand what signals your neural networks models uses for prediction. What's special about TCAV compared to other methods? TCAV instead shows importance of high level concepts (e.g., color, gender, race) for a prediction class - this is how humans communicate! TCAV gives an explanation that is generally true for a class of interest, beyond one image (global explanation). For example, for a given class, we can show how much race or gender was important for classifications in InceptionV3. Even though neither race nor gender labels were part of the training input! pip install tcav https://github.com/tensorflow/tcav
  • 35. E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 3 5 The Concept Activation Vectors (CAVs) provide an interpretation of a neural net’s internal state in terms of human-friendly concepts. TCAV uses directional derivatives to quantify the degree to which a user-defined idea is vital to a classification result–for example, how sensitive a prediction of “zebra” is to the presence of stripes. TCAV essentially learns ‘concepts’ from examples. For instance, TCAV needs a couple of examples of ‘female’, and something ‘not female’ to learn a “gender” concept. The goal of TCAV is to determine how much a concept (e.g., gender, race) was necessary for a prediction in a trained model even if the concept was not part of the training.
  • 36. Research question to consider …
  • 37. All these frameworks are great and can bring interpretability to a great extent, but can non- expert consumers of AI models interpret these interpretability methods? 3 7 Explain ab l e AI: Making ML and DL models more interpr et a b l e
  • 38. Summary • Why is Explainable AI (XAI) important? • Commonly used Model Agnostic XAI for ML models • Commonly used Model Agnostic XAI for DL models. • Popular frameworks for XAI • Can we evolve XAI and extend explainability to non-expert users? 3 8 Explain ab l e AI: Making ML and DL models more interpr et a b l e