![EXPL AINING WITH
INTEGRATED GRADIENTS
from captum.attr import IntegratedGradients
attr_algo = IntegratedGradients(model)
input = torch.rand(1, 3)
attributions = attr_algo.attribute(input, target=0)
FEATURE 0
FEATURE 1
FEATURE 2
TARGET 0
TARGET 1
OUTPUT
attributions: tensor([[-0.41, 0.54, 0.88]])](https://image.slidesharecdn.com/buildinginterpretablesecureaiusingpytorch-200606110801/85/Building-Interpretable-Secure-AI-Systems-using-PyTorch-12-320.jpg)
![EXPL AINING WITH
INTEGRATED GRADIENTS
from captum.attr import IntegratedGradients
attr_algo = IntegratedGradients(model)
input = torch.rand(1, 3)
attributions, delta = attr_algo.attribute(input,
target=0,
return_convergence_delta=True)
FEATURE 0
FEATURE 1
FEATURE 2
TARGET 0
TARGET 1
OUTPUT
attributions: tensor([[-0.41, 0.54, 0.88]])
delta: 0.0190](https://image.slidesharecdn.com/buildinginterpretablesecureaiusingpytorch-200606110801/85/Building-Interpretable-Secure-AI-Systems-using-PyTorch-13-320.jpg)
![EXPL AINING WITH
INTEGRATED GRADIENTS
from captum.attr import IntegratedGradients
attr_algo = IntegratedGradients(model)
input = torch.rand(1, 3)
baseline = torch.rand(1, 3)
attributions, delta = attr_algo.attribute(input,
target=0,
return_convergence_delta=True,
n_steps=5000,
baselines=baselines)
FEATURE 0
FEATURE 1
FEATURE 2
TARGET 0
TARGET 1
OUTPUT
attributions: tensor([[0.0, 0.88, -2.45]])
convergence delta: 1.5497e-06](https://image.slidesharecdn.com/buildinginterpretablesecureaiusingpytorch-200606110801/85/Building-Interpretable-Secure-AI-Systems-using-PyTorch-14-320.jpg)
![EXPL AINING BERT MODELS FOR QUESTION ANSWERING
text = 'It is important to us to include, empower and support humans of all kinds.'
question = 'What is important to us?'
[CLS]
tokens
what [SEP]to isimportant ?is us it important to us
to include em, and support humans of all kinds .##power
P(Start Position) = 0.72 P(End Position) = 0.73
[SEP]](https://image.slidesharecdn.com/buildinginterpretablesecureaiusingpytorch-200606110801/85/Building-Interpretable-Secure-AI-Systems-using-PyTorch-21-320.jpg)
![# explaining layers
for i in range(model.config.num_hidden_layers):
lc = LayerConductance(squad_pos_forward_func,
model.bert.encoder.layer[i])
layer_attributions_start = lc.attribute(
input_embed, baselines=ref_emb, ..., 0))
layer_attributions_end = lc.attribute(
input_embed, baselines=ref_emb, ..., 1))
EXPL AINING BERT MODELS FOR
QUESTION ANSWERING](https://image.slidesharecdn.com/buildinginterpretablesecureaiusingpytorch-200606110801/85/Building-Interpretable-Secure-AI-Systems-using-PyTorch-22-320.jpg)
![CRYPTEN import crypten
import torch
crypten.init() # sets up communication
x = torch.tensor([1.0, 2.0, 3.0])
x_enc = crypten.cryptensor(x) # encrypts tensor
x_dec = x_enc.get_plain_text() # decrypts tensor
assert torch.all_close(x_dec, x) # this passes!
y_enc = crypten.cryptensor([2.0, 3.0, 4.0])
xy_enc = x_enc + y_enc # adds encrypted tensors
xy_dec = xy_enc.get_plain_text()
assert torch.all_close(xy_dec, x + y) # this passes!
z = torch.tensor([4.0, 5.0, 6.0])
xz_enc = x_enc + z # adds FloatTensor to CrypTensor
xz_dec = xz_enc.get_plain_text()
assert torch.all_close(xz_dec, x + z) # this passes!
K E Y F E AT U R E S :
• Tensors and CrypTensors coexist and can be mixed
and matched
• Uses standard eager execution — No compilers! Easy
debugging and learning
• Support for Secure multi-party computation (MPC)
A platform for research in machine learning using
secure-computation techniques](https://image.slidesharecdn.com/buildinginterpretablesecureaiusingpytorch-200606110801/85/Building-Interpretable-Secure-AI-Systems-using-PyTorch-35-320.jpg)
![1. Create a PyTorch or ONNX model.
2. Import model into CrypTen.
3. All computations are now encrypted.
PY TORCH / ONNX
INTEGRATION
import torchvision.datasets as datasets
import torchvision.models as models
# download and set up ImageNet dataset:
transform = transforms.ToTensor()
dataset = datasets.ImageNet(
imagenet_folder,
transform=transform,
)
# download pre-trained ResNet-18 model and encrypt it:
model = models.resnet18(pretrained=True)
encrypted_model = crypten.nn.from_pytorch(
model, dataset[0],
)
# do inference on encrypted images with encrypted model:
encrypted_image = crypten.cryptensor(dataset[1])
encrypted_output = encrypted_model(encrypted_image)
output = encrypted_output.get_plain_text() # this works](https://image.slidesharecdn.com/buildinginterpretablesecureaiusingpytorch-200606110801/85/Building-Interpretable-Secure-AI-Systems-using-PyTorch-40-320.jpg)
Building Interpretable & Secure AI Systems using PyTorch
- 1. BUILDING INTERPRETABLE & SECURE AI SYSTEMS USING PYTORCH GEETA CHAUHAN AI PARTNER ENGINEERING FACEBOOK AI
- 2. AGENDA 01 INTERPRETABLE AI 02 SECURE & PRIVACY PRESERVING AI 03 REFERENCES
- 3. I N T E R P R E T A B I L I T Y
- 4. WHAT IS MODEL INTERPRETABILIT Y ? “THE ABILIT Y TO DESCRIBE AI MODEL INTERNALS AND PREDICTIONS IN HUMAN UNDERSTANDABLE TERMS*” * LH Gilpin, et. al., Explaining explanations: An overview of interpretability of machine learning in IEEE 5th International Conference on data science and advanced analytics (DSAA), 2018
- 5. I N C R E A S E D T R A N S P A R E N C Y D E B U G G I N GB E T T E R U N D E R S TA N D I N G MODEL INTERPRETABILIT Y Screenshot of the tool Attributing to dog Attribution Magnitudes
- 6. MODEL INTERPRETABILIT Y LIBRARY FOR PY TORCH M U LT I M O D A L E A S Y T O U S EE X T E N S I B L E class MyAttribution(Attribution): def attribute(self, input, ...): attributions = self._compute_attrs(input, ... ) # <Add any logic necessary for attribution> return attributions visualize_image_attr(attr_algo.attribute(input), ...) captum.ai
- 7. GradientSHAP DeepLiftSHAP SHAP Methods Integrated Gradients Saliency GuidedGradCam Attribute model output (or internal neurons) to input features LayerGradientSHAP LayerDeepLiftSHAP SHAP Methods LayerConductance InternalInfluence GradCam Attribute model output to the layers of the model DeepLift NoiseTunnel (Smoothgrad, Vargrad, Smoothgrad Square) LayerActivation LayerGradientXActivationLayerDeepLiftFeatureAblation / FeaturePermutation GuidedBackprop / Deconvolution AT TRIBUTION ALGORITHMS Input * Gradient LayerFeatureAblation LayerIntegratedGradients Occlusion Shapely Value Sampling Gradient Perturbation Other
- 8. attributions = Attribution(forward_func, ...).attribute(inputs, ...)* * Check out our Getting Started docs and API: https://github.com/pytorch/captum https://captum.ai/api/ GradientAttribution PerturbationAttribution IntegratedGradients FeatureAblation ... ...
- 9. attributions = Attribution(forward_func, ...).attribute(inputs, ...)* * Check out our Getting Started docs and API: https://github.com/pytorch/captum https://captum.ai/api/ the importance of inputs to forward_func model's forward function or any modification of it
- 10. EXPL AINING WITH INTEGRATED GRADIENTS FEATURE 0 FEATURE 1 FEATURE 2 TARGET 0 TARGET 1 from captum.attr import IntegratedGradients attr_algo = IntegratedGradients(model) input = torch.rand(1, 3) attributions = attr_algo.attribute(input, target=0)
- 11. EXPL AINING WITH INTEGRATED GRADIENTS from captum.attr import IntegratedGradients attr_algo = IntegratedGradients(model) input = torch.rand(1, 3) attributions = attr_algo.attribute(input, target=0) FEATURE 0 FEATURE 1 FEATURE 2 TARGET 0 TARGET 1
- 12. EXPL AINING WITH INTEGRATED GRADIENTS from captum.attr import IntegratedGradients attr_algo = IntegratedGradients(model) input = torch.rand(1, 3) attributions = attr_algo.attribute(input, target=0) FEATURE 0 FEATURE 1 FEATURE 2 TARGET 0 TARGET 1 OUTPUT attributions: tensor([[-0.41, 0.54, 0.88]])
- 13. EXPL AINING WITH INTEGRATED GRADIENTS from captum.attr import IntegratedGradients attr_algo = IntegratedGradients(model) input = torch.rand(1, 3) attributions, delta = attr_algo.attribute(input, target=0, return_convergence_delta=True) FEATURE 0 FEATURE 1 FEATURE 2 TARGET 0 TARGET 1 OUTPUT attributions: tensor([[-0.41, 0.54, 0.88]]) delta: 0.0190
- 14. EXPL AINING WITH INTEGRATED GRADIENTS from captum.attr import IntegratedGradients attr_algo = IntegratedGradients(model) input = torch.rand(1, 3) baseline = torch.rand(1, 3) attributions, delta = attr_algo.attribute(input, target=0, return_convergence_delta=True, n_steps=5000, baselines=baselines) FEATURE 0 FEATURE 1 FEATURE 2 TARGET 0 TARGET 1 OUTPUT attributions: tensor([[0.0, 0.88, -2.45]]) convergence delta: 1.5497e-06
- 15. ORIGINAL IMAGE ATTRIBUTING* TO DOG ATTRIBUTING* TO CAT * MATTHEW D ZEILER, ROB FERGUS, OCCLUSION: VISUALIZING AND UNDERSTANDING CONVOLUTIONAL NETWORKS, IN SPRINGER INTERNATIONAL PUBLISHING SWITZERLAND, 2014 VISUALIZATIONS USING RESNET152 MODEL
- 16. VISUALIZING EXPL ANATIONS OF A TEXT CL ASSIFICATION MODEL USING IMDB DATASET WITH CAPTUM INSIGHTS
- 17. VISUALIZING EXPL ANATIONS OF MULTIMODAL VQA MODELS WITH CAPTUM INSIGHTS
- 18. VISUALIZING EXPL ANATIONS OF A 3-L AYER MLP MODEL USING TITANIC DATASET WITH CAPTUM INSIGHTS
- 19. CASE STUDY FOR BERT MODELS
- 20. EXPL AINING BERT MODELS + Fine-tuning BERT model for Question Answering on SQUAD dataset + Evaluating on Dev Set Exact Match: 78% F1-Score: 86% + Understanding the importance of different types of word tokens, layers and neurons + Already existing research in understanding and visualizing attention heads + What Does BERT Look At? An Analysis of BERT's Attention, Clark, et. al. 2019, BlackBoxNLP@ACL + ExBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models, Hoover, et. al., 2019,
- 21. EXPL AINING BERT MODELS FOR QUESTION ANSWERING text = 'It is important to us to include, empower and support humans of all kinds.' question = 'What is important to us?' [CLS] tokens what [SEP]to isimportant ?is us it important to us to include em, and support humans of all kinds .##power P(Start Position) = 0.72 P(End Position) = 0.73 [SEP]
- 22. # explaining layers for i in range(model.config.num_hidden_layers): lc = LayerConductance(squad_pos_forward_func, model.bert.encoder.layer[i]) layer_attributions_start = lc.attribute( input_embed, baselines=ref_emb, ..., 0)) layer_attributions_end = lc.attribute( input_embed, baselines=ref_emb, ..., 1)) EXPL AINING BERT MODELS FOR QUESTION ANSWERING
- 23. AT TRIBUTION HEAT MAP OF ALL TOKENS ACROSS ALL 12 BERT L AYERS FOR START POSITION PREDICTION
- 24. AT TRIBUTION HEAT MAP OF ALL TOKENS ACROSS ALL 12 BERT L AYERS FOR END POSITION PREDICTION
- 25. THE LIMITATIONS OF AT TRIBUTIONS + Attributions do not capture feature correlations and interactions + Finding good baselines is challenging + They are difficult to evaluate + Attributions do not explain the model globally
- 26. FUTURE DIRECTIONS + captum.robust + adversarial robustness and attacks + studying the connections between model robustness and interpretability + captum.metrics + model interpretability, sensitivity, trust, infidelity and robustness related metrics + captum.benchmarks + benchmarks for different datasets and methodologies + sanity checks + captum.optim + optimization-based visualizations ...
- 27. S E C U R E & P R I V A C Y P R E S E R V I N G A I
- 28. IS IT POSSIBLE TO: answer questions using data we cannot see?
- 29. What do handwritten digits look like? ◆ Step 1: Download data ◆ Step 2: Download SOTA training script ◆ Step 3: Run script.
- 30. Source: Wikipedia Commons What do tumors look like in humans? ◆ Step -1: Persuade a VC. ◆ Step 0: Buy a dataset from a hospital. ◆ Step 1: Download millions of tumor images.
- 31. Getting access to private data is HARD!
- 32. We SOLVE tasks which are accessible: ✓ ImageNet ✓ MNIST ✓ CIFAR-10 ✓ Librispeech ✓ WikiText-103 ✓ WMT ◆ Cancer ◆ Alzheimers ◆ Dementia ◆ Depression ◆ Anxiety ◆ … Covid-19 Cure? … but what about?
- 33. TOOLS + Remote Execution + OpenMined PySyft + Search and Example Data + OpenMined PyGrid + Differential Privacy + OpenMined PyDP + Secure Multi-Party Communication + CrypTen.ai
- 34. INTRODUCING
- 35. CRYPTEN import crypten import torch crypten.init() # sets up communication x = torch.tensor([1.0, 2.0, 3.0]) x_enc = crypten.cryptensor(x) # encrypts tensor x_dec = x_enc.get_plain_text() # decrypts tensor assert torch.all_close(x_dec, x) # this passes! y_enc = crypten.cryptensor([2.0, 3.0, 4.0]) xy_enc = x_enc + y_enc # adds encrypted tensors xy_dec = xy_enc.get_plain_text() assert torch.all_close(xy_dec, x + y) # this passes! z = torch.tensor([4.0, 5.0, 6.0]) xz_enc = x_enc + z # adds FloatTensor to CrypTensor xz_dec = xz_enc.get_plain_text() assert torch.all_close(xz_dec, x + z) # this passes! K E Y F E AT U R E S : • Tensors and CrypTensors coexist and can be mixed and matched • Uses standard eager execution — No compilers! Easy debugging and learning • Support for Secure multi-party computation (MPC) A platform for research in machine learning using secure-computation techniques
- 36. B
- 37. HELLO CRYPTENSOR 1. CrypTensor wraps an implementation that does: 1. Arithmetic secret sharing. 2. XOR secret sharing. 3. Conversions between both secret sharings. 4. A large number of operations. 2. CrypTensor exposes these via a PyTorch-like API. PyTorch LongTensor Binary (XOR) Sharing CrypTensor Arithmetic Sharing Trusted Party Numerical Library Secure Computation Primitives Secure Computation Protocol Protocol-Independent Layer uses abstracts uses AutogradCrypTensor MPCTensor B2A/A2B Conversion uses Automatic di erentiation User-level code Neural networks, etc. uses Parties
- 38. 1. Make a CrypTen Model. 2. Encrypt Data 3. Train! ENCRYPTED TRAINING import crypten crypten.init() # sets up communication class LogisticRegression(crypten.nn.Module): def __init__(self): super().__init__() self.linear = crypten.nn.Linear(28 * 28, 10) def forward(self, x): return self.linear(x) model = LogisticRegression().encrypt() # encrypts tensor
- 39. 1. Join Encrypted Data 2. Encrypt Model 3. Train! Training Across Par ties import crypten crypten.init() # sets up communication alice_images_enc = crypten.load("/tmp/data/alice_images.pth", src=ALICE) bob_labels_enc = crypten.load("/tmp/data/bob_labels.pth", src=BOB) model = LogisticRegression().encrypt() train_model(model, alice_images_enc, bob_labels_enc)
- 40. 1. Create a PyTorch or ONNX model. 2. Import model into CrypTen. 3. All computations are now encrypted. PY TORCH / ONNX INTEGRATION import torchvision.datasets as datasets import torchvision.models as models # download and set up ImageNet dataset: transform = transforms.ToTensor() dataset = datasets.ImageNet( imagenet_folder, transform=transform, ) # download pre-trained ResNet-18 model and encrypt it: model = models.resnet18(pretrained=True) encrypted_model = crypten.nn.from_pytorch( model, dataset[0], ) # do inference on encrypted images with encrypted model: encrypted_image = crypten.cryptensor(dataset[1]) encrypted_output = encrypted_model(encrypted_image) output = encrypted_output.get_plain_text() # this works
- 41. USE CASES + COVID-19 Sols + Cancer Research + Integrity (eg PhotoDNA project) + Federated AI across Enterprise Silos + What problems will you solve?
- 42. • Captum: https://captum.ai/ • Captum Blog: https://bit.ly/2vHBxJI • Captum Algorithms Matrix: https://captum.ai/docs/algorithms_comparison_matrix • Interpreting MultiModal models: https://captum.ai/tutorials/Multimodal_VQA_Interpret • Interpretable ML Book: https://christophm.github.io/interpretable-ml-book/ • Crypten: https://crypten.ai/ • CrypTen Tutorials: https://github.com/facebookresearch/CrypTen#how-crypten-works • OpenMined: https://www.openmined.org/ • OpenMined for Covid-19 Apps: https://blog.openmined.org/providing-opensource-privacy-for-covid19/ • Udacity Course: https://www.udacity.com/course/secure-and-private-ai--ud185 • Active Federated Learning Paper: https://arxiv.org/pdf/1909.12641.pdf • Microsoft PhotoDNA Project: https://www.microsoft.com/en-us/photodna REFERENCES