DNN Model Interpretability
Download as PPTX, PDF1 like344 views
This document discusses various techniques for interpreting and explaining deep neural network models. It begins by motivating the need for interpretability and distinguishing between interpretation and explanation. It then covers several techniques for interpreting models, including activation maximization, sensitivity analysis, and simple Taylor decomposition. For explaining individual predictions, it discusses gradient-based and decomposition-based approaches like layer-wise relevance propagation and guided backpropagation. It also evaluates explanation quality based on continuity and selectivity. Finally, it discusses applications like model validation and analyzing scientific data domains.
![Why Simple Taylor doesn’t work?
1. Root point is hard to find or too far from 𝑥 (losses the context of 𝑥)
2. Gradient shattering the gradient of deep nets has low informative value [Balduzzi ‘17]](https://image.slidesharecdn.com/dnnmodelinterpretability-181116193250/85/DNN-Model-Interpretability-32-320.jpg)
![LRP propagation rules
• LRP - 𝛼𝛽 rule [Landecker`13, Bach`15, Zhang`16, Montavon`17]
• General rule:
1
0
-1
1
0
-1
1
0
-1](https://image.slidesharecdn.com/dnnmodelinterpretability-181116193250/85/DNN-Model-Interpretability-35-320.jpg)
![LRP and deep Taylor decomposition
• LRP - 𝛼1 𝛽0 rule deep Taylor decomposition [Montavon`17]
Assumption:](https://image.slidesharecdn.com/dnnmodelinterpretability-181116193250/85/DNN-Model-Interpretability-37-320.jpg)
DNN Model Interpretability
- 1. Interpretation and Explanation Methods For DNN Models Presented by: • Subhashis Hazarika, The Ohio State University CSE 6559: Seminar Presentation TopicsCovered Activation Maximization Sensitivity Analysis Simple Taylor Decomposition Layer-wise Relevance Propagation Guided Backpropagation LRP and deep Taylor Explanation Evaluation Techniques Applications Concept Activation Vectors (CAV) Testing with Concept Activation Vectors (TCAV) TCAV Applications
- 2. Methods for Interpreting and Understanding Deep Neural Networks Authors: • Gregoire Montavon, TU Berlin • Wojciech Samek, Fraunhofer Heinrich Hertz Institute, Berlin • Klaus-Robert Muller, TU Berlin Presented by: • Subhashis Hazarika, The Ohio State University CSE 6559: Seminar Presentation
- 3. Outline • Motivation • Interpretation vs Explanation • Interpretation • Activation Maximization • Explanation • Gradient vs Decomposition • Relevance Propagation (deep Taylor decomposition) • Evaluating Explanations • Applications
- 4. Motivation 1. Verify that the classifier works as expected
- 6. Motivation 3. Learn from the learning machine
- 7. Motivation 4. Interpretability in the sciences
- 8. Motivation 5. Compliance to legislation
- 9. Understanding Deep Nets : Two Views
- 10. Understanding Deep Nets : Two Views
- 11. Model Analysis Interpretation
- 12. Decision Analysis Explanation
- 14. Interpreting a DNN Model • Build a prototype in the input domain of the DNN (e.g image or text) that is interpretable and representative of the abstract learned concept • Activation Maximization (AM): An analysis framework that searches for an input pattern that produces a maximum model response for a quantity of interest.
- 15. Improving AM
- 16. Improving AM
- 17. Improving AM • Simple AM : • Improving AM with an expert: • Perform AM in code space: • These techniques require an unsupervised model of the data, either a density model p(x) or a generator model g(z).
- 18. Comparison of AM variants
- 19. Enhanced AM on Natural Images
- 20. Limitation of Global Interpretations
- 21. From Global Interpretations to Individual Explanations Explanability
- 22. Explaining DNN decisions • “why” does the model arrive at a certain prediction? • “why” is the image below classified as a shark?
- 23. Explainability: Basic Techniques • Sensitivity Analysis (Gradient Approach) • Simple Taylor Decomposition (Function-decomposition Approach) • Backward Propagation Techniques • Layer-wise Relevance Propagation (decomposition) • Guided Backpropagation (gradient)
- 25. What does Sensitivity Analysis decompose?
- 26. What does Sensitivity Analysis decompose?
- 27. Decomposing the Correct Quantity 𝒇(𝒙) = Taylor Series:
- 28. Simple Taylor Decomposition • Achievable for linear models and deep ReLU 𝑓 𝑡𝑥 = 𝑡𝑥 → 𝑓 0 = 0 • We can choose 𝑥 = lim 𝑡→0 𝑡. 𝑥 • Final Relevance; 𝑅𝑖 = 𝜕𝑓 𝜕𝑥 𝑖 𝑥=0 . 𝑥𝑖 • i.e, gradient x input
- 29. Simple Taylor Decomposition • Achievable for linear models and deep ReLU 𝑓 𝑡𝑥 = 𝑡𝑥 → 𝑓 0 = 0 • We can choose 𝑥 = lim 𝑡→0 𝑡. 𝑥 • Final Relevance; 𝑅𝑖 = 𝜕𝑓 𝜕𝑥 𝑖 𝑥=0 . 𝑥𝑖 • i.e, gradient x input
- 32. Why Simple Taylor doesn’t work? 1. Root point is hard to find or too far from 𝑥 (losses the context of 𝑥) 2. Gradient shattering the gradient of deep nets has low informative value [Balduzzi ‘17]
- 33. Backward Propagation Techniques • Make explicit use of the graph structure • Layer-wise Relevance Propagation (Conserving) • Guided Backpropagation (Non conserving) • BPT were shown empirically to scale better to complex DNN models • Facilitates filtering (breaks the explanation into multiple subtask)
- 34. Layer-wise Relevance Propagation • Each neuron receives a share of the network output, and redistributes it to its predecessors in equal amount, until the input variables are reached Conservation Property: LRP - 𝛼1 𝛽0 rule:
- 35. LRP propagation rules • LRP - 𝛼𝛽 rule [Landecker`13, Bach`15, Zhang`16, Montavon`17] • General rule: 1 0 -1 1 0 -1 1 0 -1
- 36. LRP vs Guided Backpropagation
- 37. LRP and deep Taylor decomposition • LRP - 𝛼1 𝛽0 rule deep Taylor decomposition [Montavon`17] Assumption:
- 38. LRP and deep Taylor decomposition 1. Build the Relevance Neuron
- 39. LRP and deep Taylor decomposition 2. Expand the Relevance Neuron
- 40. LRP and deep Taylor decomposition 3. Decompose Relevance
- 41. LRP and deep Taylor decomposition 4. Pooling relevance over all outgoing neurons
- 42. Evaluating Explanation Quality • Explanation Continuity • Explanation Selectivity
- 43. Explanation Continuity • “If two data points are nearly equivalent, then the explanations of their predictions should also be nearly equivalent.” • Explanation continuity (or lack of it) can be quantified by looking for the strongest variation of the explanation R(x) in the input domain
- 44. Explanation Continuity 2-layer ReLU network:
- 46. Explanation Selectivity • Can be quantified by measuring how fast f(x) goes down when removing features with the highest relevance scores • “Pixel flipping” test for image data
- 48. Applications • Model Validation Procedure • Analysis of scientific data
- 49. Model Validation • Traditional validation use validation set (subset of the initial training data) • Human interpretable results can add to basic validation procedures
- 50. Model Validation • Traditional validation use validation set (subset of the initial training data) • Human interpretable results can add to basic validation procedures
- 51. Analysis of Scientific Data • The context of interpretability and explanability can extended to domain specific knowledge, useful for scientific inferences • Atomistic simulation:
- 52. Analysis of Scientific Data • The context of interpretability and explanability can extended to domain specific knowledge, useful for scientific inferences • Human Brain Pattern:
- 53. Analysis of Scientific Data • The context of interpretability and explanability can extended to domain specific knowledge, useful for scientific inferences • DNA sequence:
- 54. Analysis of Scientific Data • The context of interpretability and explanability can extended to domain specific knowledge, useful for scientific inferences • Human face analysis:
- 55. Summary so far • ML model transparency: • Interpreting the concepts learned by a model by building prototypes • Explaining the model’s decisions by identifying the relevant input variables • Crucial distinction between sensitivity analysis and decomposition approaches • Evaluating Explanations • Continuity • Selectivity • Applications
- 56. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) Authors: • Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres • Affiliation: Google Brain CSE 6559: Seminar Presentation Presented by: • Subhashis Hazarika, The Ohio State University
- 57. Goal of TCAV
- 58. Goal of TCAV
- 59. Goal of TCAV
- 60. Overview
- 61. Testing with CAV
- 62. Other Analysis • TCAV extension: Relative TCAV • Validating the learned CAVs:
- 63. TCAV for medical application • Diabetic Retinopathy (DR) (level 0- 4) • Validating the trained model with known concepts from the doctors
- 64. Thank you!