SlideShare a Scribd company logo

Multitask Adversarial Learning of Deep Neural Networks for Medical Imaging and Image Analysis

Download as PPTX, PDF
Debdoot Sheet, profile picture
Debdoot Sheet

The document presents a talk on multitask adversarial learning of deep neural networks (DNN) for medical imaging, focusing on methods like generative adversarial networks (GAN) for enhancing image analysis in areas such as retinal imaging and ultrasound. It discusses various research contributions, including semantic segmentation and simulating patho-realistic ultrasound images, highlighting the challenges and methodologies involved in using DNNs for improving medical imaging quality. The speaker, Dr. Debdoot Sheet, outlines collaborations and sponsors supporting this research.

Science
Related topics:
Deep Learning
1 of 48
Downloaded 12 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 0 One out this pair of images is an original image of intravascular ultrasound (IVUS) and the other is generated by a special type of artificial neural network (ANN) known as generative adversarial network (GAN). Can you identify the original?
Multitask Adversarial Learning of Deep Neural Networks for Medical Imaging and Image Analysis Dr. Debdoot Sheet Assistant Professor, Department of Electrical Engineering Principal Investigator, Kharagpur Learning, Imaging and Visualization Group Indian Institute of Technology Kharagpur www.facweb.iitkgp.ac.in/~debdoot/
Disclosure 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 2 • Conflicts and Interests – SkinCurate Research Pvt. Ltd. – Founder, stock options – Intel Technology India Pvt. Ltd. – Research Sponsor, Startup Incubation Mentor – Dept. of Biotechnology, Govt. of India – Research Sponsor – Sigtuple Technologies Pvt. Ltd. – Research Sponsor – Tata Steel Ltd. – Research Sponsor – Amazon Web Services (AWS) Inc. – Research Sponsor – Nesa Medtech Pvt. Ltd. – Research Sponsor – Samsung Inc. – Research Sponsor – Nvidia Inc. – Lab. Resources Sponsor – Microsoft – Collaborator and Lab. Resources Sponsor – Texas Instruments India Pvt. Ltd. – Lab. Resources Sponsor – Analog Devices India Pvt. Ltd. – Lab. Resources Sponsor – Indian Council of Medical Research, Govt. of India – Travel Grants – Dept. of Science and Technology, Govt. of India – Travel Grants, Startup Incubation Mentor – Biotechnology Industry Research Assistance Council (BIRAC) – Startup Incubation Grant – Society for Innovation and Entrepreneurship (SINE) IIT Bombay – Startup Incubation Grant
Learning? A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E -Tom Mitchell Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 31 Sept. 2019
Contributions to be Discussed 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 4 Learning with Multitask Adversaries using Weakly Labelled Data for Semantic Segmentation in Retinal Images (MIDL 2019) Oindrila Saha, Rachana Sathish, Debdoot Sheet Simulating Patho-realistic Ultrasound Images using Deep Generative Networks with Adversarial Learning (ISBI 2018) Francis Tom, Debdoot Sheet Learning a Deep Convolution Network with Turing Test Adversaries for Microscopy Image Super Resolution (ISBI 2019) Francis Tom, Himanshu Sharma, Dheeraj Mundhra, Tathagata Rai Dastidar, Debdoot Sheet UltraCompression: Framework for High Density Compression of Ultrasound Volumes using Physics Modeling Deep Neural Networks (ISBI 2019) Debarghya China, Francis Tom, Sumanth Nandamuri, Aupendu Kar, Mukundhan Srinivasan, Pabitra Mitra, Debdoot Sheet
RECALL 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 5
Statistically Informed Decision 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 6 𝑃 Ω = 𝜔|𝐱 = 𝒙 = 𝑃 Ω = 𝜔, 𝐱 = 𝒙 𝑃 𝐱 = 𝒙 𝑃 Ω = 𝜔|𝐱 = 𝒙 = 𝑝 𝐱 = 𝒙|Ω = 𝜔 𝑃 Ω = 𝜔 𝑃 𝐱 = 𝒙 Posterior probability Likelihood Prior probability EvidenceBayes’ Rule
Decision 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 7 𝑃 Ω = 𝜔|𝐱 = 𝒙 = 𝑝 𝐱 = 𝒙|Ω = 𝜔 𝑃 Ω = 𝜔 𝑃 𝐱 = 𝒙 Posterior probability Likelihood Prior probability EvidenceBayes’ Rule Shade of color Length 𝑥1 𝑥2 𝜔 = arg max 𝑃 Ω = 𝜔|𝐱 = 𝒙 Maximum aposteriori (MAP) Decision boundary
Challenges with a Decision Boundary 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 8 𝑥1 𝑥2 𝑥1 𝑥2 𝑥1 𝑥2 Abundant samples Not-so abundant samples Scarce samples
Understanding these Challenges 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 9 𝑥1 𝑥2 𝑥1 𝑥2 𝑃 Ω = 𝜔|𝐱 = 𝒙 = 𝑝 𝐱 = 𝒙|Ω = 𝜔 𝑃 Ω = 𝜔 𝑃 𝐱 = 𝒙 Likelihood 𝑝 𝒙 ~𝜙 𝒙3 Cubic 𝑝 𝒙 ~𝜙 𝒙2 Quadratic 𝑝 𝒙 ~𝜙 𝑥Linear 𝑥2 = 𝑚𝑥1 + 𝑐
AI from Heuristics to DL 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 10 Input Hand Designed Rules Output Input Hand Designed Features Output Learned Decision Input Learned Features Output Learned Decision Input Learned Features Output Learned Decision Learned Abstract Features Rule based AI Classical ML Representation Learn. Deep Learning
Objectives of Machine Learning 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 11 𝑝 𝒙 ~𝜙 𝒙3 Cubic 𝑝 𝒙 ~𝜙 𝒙2 Quadratic 𝑝 𝒙 ~𝜙 𝑥 Linear Increasing order of complexity 𝑥1 𝑥2 𝑥1 𝑥2 Data space plane MLE Mean Squared Error (MSE) Perception Loss (PL)
DL addressing these ML Objectives 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 12 𝑥1 𝑥2 𝑥1 𝑥2 Data space plane Van Dyk, David A., and Xiao-Li Meng. “The art of data augmentation.” Journal of Computational and Graphical Statistics 10.1 (2001): 1-50. rotate flipud fliplr flipud
DL addressing these ML Objectives 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 13 𝑝 𝒙 ~𝜙 𝒙3 Cubic 𝑝 𝒙 ~𝜙 𝒙2 Quadratic 𝑝 𝒙 ~𝜙 𝑥 Linear Increasing order of complexity 𝑥1 𝑥2 𝑥1 𝑥2 Learned Features
DL addressing these ML Objectives 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 14 MLE Mean Squared Error (MSE) Perception Loss (PL) low MSE, high PL high MSE, low PL Sketch2Photo CNN Discriminator Neural Network Real vs. Fake Photograph Generated
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 15 Learning with Multitask Adversaries using Weakly Labelled Data for Semantic Segmentation in Retinal Images Oindrila Saha, Rachana Sathish, Debdoot Sheet International Conference on Medical Imaging with Deep Learning (MIDL), London, July 2019 Oindrila Saha Rachana Sathish
Preamble 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 16 SemanticSegmentation Network Retinal vessels, Optic disc, Optic cup, Edema, Soft exudates, Hard exudates
Challenge 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 17 Image size #768✕584 Dataset 1 Retinal vessels Train # 20, Test # 20 Image size # 3,504 ✕ 2,336 Retinal vessels, optic disc Images # 18 Image size # 1,500 ✕ 1,152 Hard exudates, soft exudates, Microaneurysms, Hemorrhages Images # 89 Dataset 2 Dataset 3
Solution 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 18 Semantic Segmentation Network 3✕M✕N I(:,:,:) C✕M✕N Discriminator 2 C✕M✕N Manual vs. Synthetic ChannelShuffler(.)ChannelShuffler(.) Discriminator 1 Presence of Class 𝐲𝐱
Solution 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 19 Semantic Segmentation Network 3✕M✕N I(:,:,:) Discriminator 2 C✕M✕N Manual vs. Synthetic ChannelShuffler(.)ChannelShuffler(.) Discriminator 1 Segmented Channel Idx C✕M✕N I(:,:,:) I(:,:,:) 𝐱 𝐲 𝐲 𝐲𝐱 𝐿 𝑠𝑒𝑔 ∙ = ∀𝐲 𝑐 ≠∅ 𝐵𝐶𝐸 𝐲 𝑐 , 𝐲 𝑐 𝛻𝐿 𝑠𝑒𝑔 ∙
Discriminator 1 Semantic Segmentation Network 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 20 Discriminator 2 C✕M✕N Manual vs. Synthetic ChannelShuffler(.) Segmented Channel Idx 𝐲𝐱 3✕M✕N I(:,:,:) Solution ChannelShuffler(.) C✕M✕N 𝐱 1 𝐲𝐱 2 𝐱 3 𝐱 1 𝐱 2 𝐱 3 𝐲𝐲 𝑘 = 𝐲 𝑠ℎ𝑢𝑓𝑓𝑙𝑒 1,2, ⋯ , 𝐶
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 21 Semantic Segmentation Network 3✕M✕N I(:,:,:) C✕M✕N Discriminator 2 C✕M✕N Manual vs. Synthetic ChannelShuffler(.)ChannelShuffler(.) 𝐲𝐱 Solution Discriminator 1 Presence of Class 𝐱 1 𝐱 2 𝐱 3 𝐲 𝐲 𝐧1 = 1,0,0,0,1,1 𝐧1 = 1,1,0,1,1,1 𝐿 𝐷1 ∙ = 𝐵𝐶𝐸 𝐧1, 𝐧1 𝛻𝐿 𝐷1 ∙ 𝐧1 Ganin, Yaroslav, et al. "Domain-adversarial training of neural networks." The Journal of Machine Learning Research 17.1 (2016): 2096-2030.
Discriminator 1 Segmented Channel Idx 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 22 Semantic Segmentation Network 𝐲𝐱 Solution 𝐿 𝐷2 ∙ = 𝐵𝐶𝐸 𝐧2, 𝐧2 Discriminator 2 𝛻𝐿 𝐷2 ∙ 3✕M✕N I(:,:,:) C✕M✕N ChannelShuffler(.) C✕M✕NChannelShuffler(.) 𝐧2 = 0, 1 𝐧2 = 1, 0 𝐧2 Ganin, Yaroslav, et al. "Domain-adversarial training of neural networks." The Journal of Machine Learning Research 17.1 (2016): 2096-2030. {Manual, Synthetic} vs. {Synthetic, Manual}
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 23 3✕M✕N C✕M✕N C✕M✕N Manual vs. Synthetic ChannelShuffler(.)ChannelShuffler(.) Segmented Channel Idx 𝐲𝐱 Semantic Segmentation Network I(:,:,:) Discriminator 2 Discriminator 1 Solution 𝛻𝐿 𝑠𝑒𝑔 ∙ 𝛻𝐿 𝐷2 ∙ 𝛻𝐿 𝐷1 ∙ 𝛻𝐿 𝑎𝑑𝑣 ∙ = 𝛼1 𝛻𝐿 𝐷1 ∙ −𝛼2 𝛻𝐿 𝐷2 ∙ 𝐿 𝑎𝑑𝑣 ∙ = 𝛼1 𝐿 𝐷1 ∙ +𝛼2 1 − 𝐿 𝐷2 ∙
Results 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 24
Computed Super-Resolution Microscopy Learning a Deep Convolution Network with Turing Test Adversaries for Microscopy Image Super Resolution Francis Tom, Himanshu Sharma, Dheeraj Mundhra, Tathagato Rai Dastidar, Debdoot Sheet IEEE International Symposium on Biomedical Imaging (ISBI), 2019. Francis Tom Himanshu Sharma Dheeraj Mundhra Tathagato Rai Dastidar Computed Super-Resolution Microscopy
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 26 Turing Test 1 Turing Test 2 Region Proposals {Real, SR} vs. {SR, Real} {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙ Super Resolution CNN 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ Real LR SR
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 27 Super Resolution CNN Turing Test 2 Region Proposals 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ {Real, SR} vs. {SR, Real} LR 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙ Turing Test 1 SR {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ Real
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 28 Super Resolution CNN Turing Test 1 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ LR {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙ Turing Test 2 Region Proposals {Real, SR} vs. {SR, Real} Real SR 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 29 Turing Test 1 Turing Test 2 Region Proposals 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ {Real, SR} vs. {SR, Real} Real {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask Super Resolution CNN LR SR 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 30 Super Resolution CNN Turing Test 1 Turing Test 2 Region Proposals 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ {Real, SR} vs. {SR, Real} Real LR SR {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙
Some Results 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 31
… and our AI powers Digital Pathology in India 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 32
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 33 One out this pair of images is an original image of intravascular ultrasound (IVUS) and the other is generated by a special type of artificial neural network (ANN) known as generative adversarial network (GAN). Can you identify the original?
Can Generative Adversarial Networks Model Imaging Physics? Some experiences with simulating patho-realistic ultrasound images Francis Tom DeepSIP DeepKLIV “Simulating Patho-realistic Ultrasound Images using Deep Generative Networks with Adversarial Learning” Francis Tom, Debdoot Sheet IEEE International Symposium on Biomedical Imaging (ISBI), 2018.
Simulating Ultrasound 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 35 Jørgen Arendt Jensen and Peter Munk, ”Computer phantoms for simulating ultrasound B-mode and cfm images”, Acoustical Imaging”, vol. 23, pp. 75-80, Eds.: S. Lees and L. A. Ferrari, Plenum Press, 1997. J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med. Biol., vol. 25, no. 3, pp. 463– 479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]
Pseudo B-mode US Image Simulation 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 36 𝑇 𝑥, 𝑦 = 𝐸 𝑥, 𝑦 𝐺 𝑛 𝐺 𝑛 ~𝒩 0,1 𝑘0 = 2𝜋𝑓0 𝑐 ℎ 𝑥 𝑥, 𝑦 = sin 𝑘0 𝑥 𝑒 − 𝑥2 2𝜎 𝑥 2 ℎ 𝑦 𝑥, 𝑦 = 𝑒 − 𝑦2 2𝜎 𝑦 2 𝑉 𝑥, 𝑦 = 𝑇 𝑥, 𝑦 ∗ ℎ 𝑥 𝑥, 𝑦 ∗ ℎ 𝑦 𝑥, 𝑦 𝑏 𝑥, 𝑦 = 𝐻𝑖𝑙𝑏𝑒𝑟𝑡 𝑉 𝑥, 𝑦 J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med. Biol., vol. 25, no. 3, pp. 463–479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]
Where do we stand with Simulations? 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 37 Digital Phantom Pseudo B-mode US Real B-mode US How to bridge this disparity in appearance?
The Challenge 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 38 𝑇 𝑥, 𝑦 = 𝐸 𝑥, 𝑦 𝐺 𝑛 𝐺 𝑛 ~𝒩 0,1 𝑉 𝑥, 𝑦 = 𝑇 𝑥, 𝑦 ∗ ℎ 𝑥 𝑥, 𝑦 ∗ ℎ 𝑦 𝑥, 𝑦 𝑏 𝑥, 𝑦 = 𝐻𝑖𝑙𝑏𝑒𝑟𝑡 𝑉 𝑥, 𝑦 J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med. Biol., vol. 25, no. 3, pp. 463–479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006] Are these convolution kernels not descriptive enough to model the signal mixing process resulting in image formation? Can we learn the convolution kernels? 𝑘0 = 2𝜋𝑓0 𝑐 ℎ 𝑥 𝑥, 𝑦 = sin 𝑘0 𝑥 𝑒 − 𝑥2 2𝜎 𝑥 2 ℎ 𝑦 𝑥, 𝑦 = 𝑒 − 𝑦2 2𝜎 𝑦 2
Learning Convolution with a Neural Network 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 39 * = * =f 𝐖𝑖 𝒙 𝒚 𝒛 𝐖𝑖+1 𝒑 𝐖𝑖 𝑘+1 = 𝐖𝑖 𝑘 − 𝜂 𝜕𝐽 𝐖 𝜕𝐖𝑖 𝐽 𝐖 = 𝒑 − 𝒑 𝜕𝐽 𝐖 𝜕𝐖𝑖 = 𝜕𝐽 𝐖 𝜕𝒑 𝜕𝒑 𝜕𝐖𝑖+1 𝜕𝒛 𝜕𝒚 𝜕𝒚 𝜕𝐖𝑖 # Training Samples?
Adversarial Transformation for US Simulation 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 40 Generator Net 𝐺 𝜃 Discriminator Net 𝐷 𝜙 𝐿 𝐺 𝜃 Simulated vs. Real 𝐿 𝐷 𝜙 𝛻𝐿 𝐷 𝜙 𝛻𝐿 𝐺 𝜃 Digital Phantom Simulated Real
Adversarial Transformation for US Simulation 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 41 Hu, Y., Gibson, E., Lee, L. L., Xie, W., Barratt, D. C., Vercauteren, T., & Noble, J. A. (2017). Freehand Ultrasound Image Simulation with Spatially-Conditioned Generative Adversarial Networks. In Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment (pp. 105-115). Springer, Cham.
The Challenges 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 42 Transducer • Non uniform sampling in Cartesian coordinate domain • Signal interpolated for image formation, leading to smearing of speckles
Our Solution 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 43 Digital Phantom Digital Phantom cart2pol( ) Pseudo B-mode Stage 0 Stage I 𝐺𝐼 𝜃 Stage II 𝐺𝐼𝐼 𝜃 𝐿 𝐺 𝐼 𝜃 𝐿 𝐺 𝐼𝐼 𝜃 cart2pol() Stage I 𝐷𝐼 𝜙 Stage II 𝐷𝐼𝐼 𝜙 𝐿 𝐷 𝐼 𝜙 𝐿 𝐷 𝐼𝐼 𝜙 Stage I Sim Stage II Sim Real Real Simulated vs. Real 64 x 64 256 x 256256 x 256256 x 256256 x 256 𝛻𝐿 𝐺 𝐼 𝛻𝐿 𝐺 𝐼𝐼 𝛻𝐿 𝐷 𝐼 𝛻𝐿 𝐷 𝐼𝐼
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 44 Lets Quiz: Real vs. Simulated
More Resources • Neural Information Processing Systems (NeurIPS) • International Conference on Learning Representations (ICLR) • International Conference on Machine Learning (ICML) • Association for Advancement of Artificial Intelligence (AAAI) • Computer Vision and Pattern Recognition (CVPR) • International Conference on Medical Imaging with Deep Learning (MIDL) • IEEE Int. Symp. Biomed. Imaging (ISBI) • Journal of Machine Learning Research (JMLR) • Machine Learning • IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) • IEEE Trans. Neural Networks and Learning Systems (TNNLS) • IEEE Trans. Medical Imaging (TMI) • Medical Image Analysis (MedIA) • Medical Image Computing and Computer Assisted Intervention (MICCAI) 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 45
Take home message 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 46
1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 47 Thank you from #iitkliv http://iitkliv.github.io

More Related Content

PDF
Can Generative Adversarial Networks Model Imaging Physics?
Debdoot Sheet
 
PPTX
Deep Learning - What's the buzz all about
Debdoot Sheet
 
PDF
파이콘 한국 2019 튜토리얼 - 설명가능인공지능이란? (Part 1)
XAIC
 
PDF
Satellite and Land Cover Image Classification using Deep Learning
ijtsrd
 
PDF
IRJET- A Real Time Yolo Human Detection in Flood Affected Areas based on Vide...
IRJET Journal
 
PDF
76201950
IJRAT
 
PDF
Xin Yao: "What can evolutionary computation do for you?"
ieee_cis_cyprus
 
PDF
Skin Lesion Detection from Dermoscopic Images using Convolutional Neural Netw...
Universitat Politècnica de Catalunya
 
Can Generative Adversarial Networks Model Imaging Physics?
Debdoot Sheet
 
Deep Learning - What's the buzz all about
Debdoot Sheet
 
파이콘 한국 2019 튜토리얼 - 설명가능인공지능이란? (Part 1)
XAIC
 
Satellite and Land Cover Image Classification using Deep Learning
ijtsrd
 
IRJET- A Real Time Yolo Human Detection in Flood Affected Areas based on Vide...
IRJET Journal
 
76201950
IJRAT
 
Xin Yao: "What can evolutionary computation do for you?"
ieee_cis_cyprus
 
Skin Lesion Detection from Dermoscopic Images using Convolutional Neural Netw...
Universitat Politècnica de Catalunya
 

What's hot (14)

PDF
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
Universitat Politècnica de Catalunya
 
PDF
IRJET- DNA Fragmentation Pattern and its Application in DNA Sample Type Class...
IRJET Journal
 
PDF
THE EFFECT OF PHYSICAL BASED FEATURES FOR RECOGNITION OF RECAPTURED IMAGES
ijcsit
 
PDF
11.0003www.iiste.org call for paper.survey on wireless intelligent video surv...
Alexander Decker
 
PDF
3.survey on wireless intelligent video surveillance system using moving objec...
Alexander Decker
 
PDF
IRJET- A Survey on Medical Image Interpretation for Predicting Pneumonia
IRJET Journal
 
PDF
Acoustic Scene Classification by using Combination of MODWPT and Spectral Fea...
ijtsrd
 
PDF
Deep learning for person re-identification
哲东 郑
 
PDF
Madalina Fiterau - Hybrid Machine Learning Methods for the Interpretation and...
MLconf
 
PDF
Cross-domain complementary learning with synthetic data for multi-person part...
哲东 郑
 
PPT
2017 07 03_meetup_d
Dana Brophy
 
DOCX
The power of deep learning models applications
Sameera Sk
 
DOCX
MasterBioJS
jaime seguel
 
PDF
SSII2021 [SS2] Deepfake Generation and Detection – An Overview （ディープフェイクの生成と検出）
SSII
 
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
Universitat Politècnica de Catalunya
 
IRJET- DNA Fragmentation Pattern and its Application in DNA Sample Type Class...
IRJET Journal
 
THE EFFECT OF PHYSICAL BASED FEATURES FOR RECOGNITION OF RECAPTURED IMAGES
ijcsit
 
11.0003www.iiste.org call for paper.survey on wireless intelligent video surv...
Alexander Decker
 
3.survey on wireless intelligent video surveillance system using moving objec...
Alexander Decker
 
IRJET- A Survey on Medical Image Interpretation for Predicting Pneumonia
IRJET Journal
 
Acoustic Scene Classification by using Combination of MODWPT and Spectral Fea...
ijtsrd
 
Deep learning for person re-identification
哲东 郑
 
Madalina Fiterau - Hybrid Machine Learning Methods for the Interpretation and...
MLconf
 
Cross-domain complementary learning with synthetic data for multi-person part...
哲东 郑
 
2017 07 03_meetup_d
Dana Brophy
 
The power of deep learning models applications
Sameera Sk
 
MasterBioJS
jaime seguel
 
SSII2021 [SS2] Deepfake Generation and Detection – An Overview （ディープフェイクの生成と検出）
SSII
 
Ad

Similar to Multitask Adversarial Learning of Deep Neural Networks for Medical Imaging and Image Analysis (20)

PPT
CI image processing
Meenakshi Sood
 
PDF
Master Thesis Presentation (Subselection of Topics)
Alina Leidinger
 
PDF
Deep learning for biomedicine
Deakin University
 
PPTX
Sign Language Recognition based on Hands symbols Classification
Triloki Gupta
 
PDF
A ROBUST JOINT-TRAINING GRAPHNEURALNETWORKS MODEL FOR EVENT DETECTIONWITHSYMM...
kevig
 
PDF
A Robust Joint-Training Graph Neural Networks Model for Event Detection with ...
kevig
 
PDF
A ROBUST JOINT-TRAINING GRAPHNEURALNETWORKS MODEL FOR EVENT DETECTIONWITHSYMM...
kevig
 
PDF
A ROBUST JOINT-TRAINING GRAPHNEURALNETWORKS MODEL FOR EVENT DETECTIONWITHSYMM...
kevig
 
PDF
Semi-Targeted Model Poisoning Attack on Federated Learning via Backward Error...
YuweiSun5
 
PPTX
If a picture is worth a thousand words, Interactive data visualizations are w...
Olga Scrivner
 
PDF
Deep Learning for Computer Vision: Medical Imaging (UPC 2016)
Universitat Politècnica de Catalunya
 
PDF
Deep Learning based Segmentation Pipeline for Label-Free Phase-Contrast Micro...
UMBC
 
PDF
June 2020: Top Read Articles in Advanced Computational Intelligence
aciijournal
 
PDF
MitoGame: Gamification Method for Detecting Mitosis from Histopathological I...
IRJET Journal
 
PPTX
Enhancing Remote Adversarial Patch Attacks on Face Detectors with Tiling and ...
okano7
 
PDF
ICSE 2025 Keynote: Software Sustainability and its Engineering: How far have ...
patricialago3459
 
PPTX
Philosophy of Deep Learning
Melanie Swan
 
PDF
Brownbag_The_Role_of_Generalists_and_Specialists.pdf
Junyi19
 
PDF
IRJET- Prediction of Autism Spectrum Disorder using Deep Learning: A Survey
IRJET Journal
 
PDF
Question & Answer 2.0: A System to facilitate interaction during and after a ...
Kai Michael Höver
 
CI image processing
Meenakshi Sood
 
Master Thesis Presentation (Subselection of Topics)
Alina Leidinger
 
Deep learning for biomedicine
Deakin University
 
Sign Language Recognition based on Hands symbols Classification
Triloki Gupta
 
A ROBUST JOINT-TRAINING GRAPHNEURALNETWORKS MODEL FOR EVENT DETECTIONWITHSYMM...
kevig
 
A Robust Joint-Training Graph Neural Networks Model for Event Detection with ...
kevig
 
A ROBUST JOINT-TRAINING GRAPHNEURALNETWORKS MODEL FOR EVENT DETECTIONWITHSYMM...
kevig
 
A ROBUST JOINT-TRAINING GRAPHNEURALNETWORKS MODEL FOR EVENT DETECTIONWITHSYMM...
kevig
 
Semi-Targeted Model Poisoning Attack on Federated Learning via Backward Error...
YuweiSun5
 
If a picture is worth a thousand words, Interactive data visualizations are w...
Olga Scrivner
 
Deep Learning for Computer Vision: Medical Imaging (UPC 2016)
Universitat Politècnica de Catalunya
 
Deep Learning based Segmentation Pipeline for Label-Free Phase-Contrast Micro...
UMBC
 
June 2020: Top Read Articles in Advanced Computational Intelligence
aciijournal
 
MitoGame: Gamification Method for Detecting Mitosis from Histopathological I...
IRJET Journal
 
Enhancing Remote Adversarial Patch Attacks on Face Detectors with Tiling and ...
okano7
 
ICSE 2025 Keynote: Software Sustainability and its Engineering: How far have ...
patricialago3459
 
Philosophy of Deep Learning
Melanie Swan
 
Brownbag_The_Role_of_Generalists_and_Specialists.pdf
Junyi19
 
IRJET- Prediction of Autism Spectrum Disorder using Deep Learning: A Survey
IRJET Journal
 
Question & Answer 2.0: A System to facilitate interaction during and after a ...
Kai Michael Höver
 
Ad

More from Debdoot Sheet (9)

PPTX
Whispers of Speckles ( Part I: Building Computational Imaging Frameworks for ...
Debdoot Sheet
 
PPTX
Machine Learning for Medical Image Analysis: What, where and how?
Debdoot Sheet
 
PPTX
Decision trees and random forests
Debdoot Sheet
 
PPTX
Deep Learning of Tissue Specific Speckle Representations in Optical Coherence...
Debdoot Sheet
 
PPTX
Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomograph...
Debdoot Sheet
 
PPTX
Art and craft of effective scientific writing
Debdoot Sheet
 
PPT
Detection of Retinal Vessels in Fundus Images through Transfer Learning of Ti...
Debdoot Sheet
 
PPTX
Ultrasonic Histology
Debdoot Sheet
 
PPT
Brightness Preserving Contrast Enhancement Of Medical Images
Debdoot Sheet
 
Whispers of Speckles ( Part I: Building Computational Imaging Frameworks for ...
Debdoot Sheet
 
Machine Learning for Medical Image Analysis: What, where and how?
Debdoot Sheet
 
Decision trees and random forests
Debdoot Sheet
 
Deep Learning of Tissue Specific Speckle Representations in Optical Coherence...
Debdoot Sheet
 
Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomograph...
Debdoot Sheet
 
Art and craft of effective scientific writing
Debdoot Sheet
 
Detection of Retinal Vessels in Fundus Images through Transfer Learning of Ti...
Debdoot Sheet
 
Ultrasonic Histology
Debdoot Sheet
 
Brightness Preserving Contrast Enhancement Of Medical Images
Debdoot Sheet
 

Recently uploaded (20)

PPTX
ognitive-behavioral therapy, mindfulness-based approaches, coping skills trai...
OliveJolly1
 
PPTX
microscope-Lecturecjchchchchcuvuvhc.pptx
jaylaurencenobleza
 
PPTX
Derivatives of integument scales, beaks, horns,.pptx
Dr Showkat Ahmad Wani
 
PPTX
Classification Systems_TAXONOMY_SCIENCE8.pptx
glofernignacio
 
PDF
VARICELLA VACCINATION: A POTENTIAL STRATEGY FOR PREVENTING MULTIPLE SCLEROSIS
ijab2
 
PPTX
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
PrabakaranMP1
 
PPT
Chemical bonding and molecular structure
karthikeyan
 
PPT
protein biochemistry.ppt for university classes
bayewodajo27
 
PPTX
DRUG THERAPY FOR SHOCK gjjjgfhhhhh.pptx.
vipvivek704
 
PPTX
SCIENCE10 Q1 5 WK8 Evidence Supporting Plate Movement.pptx
ROLANARIBATO3
 
PPTX
INTRODUCTION TO EVS | Concept of sustainability
manumaddy15198
 
PPTX
2. Earth - The Living Planet earth and life
markjustinebarolobau
 
PPTX
GEN. BIO 1 - CELL TYPES & CELL MODIFICATIONS
breeysal7
 
PPTX
2. Earth - The Living Planet Module 2ELS
markjustinebarolobau
 
PPTX
TOTAL hIP ARTHROPLASTY Presentation.pptx
hararghemssc
 
PPTX
ANEMIA WITH LEUKOPENIA MDS 07_25.pptx htggtftgt fredrctvg
vedvyas5668
 
PDF
. Radiology Case Scenariosssssssssssssss
MiraMusleh
 
PPT
The World of Physical Science, • Labs: Safety Simulation, Measurement Practice
joybejerano
 
PPTX
famous lake in india and its disturibution and importance
judesebina
 
PDF
Phytochemical Investigation of Miliusa longipes.pdf
IrfanShahirSharafi
 
ognitive-behavioral therapy, mindfulness-based approaches, coping skills trai...
OliveJolly1
 
microscope-Lecturecjchchchchcuvuvhc.pptx
jaylaurencenobleza
 
Derivatives of integument scales, beaks, horns,.pptx
Dr Showkat Ahmad Wani
 
Classification Systems_TAXONOMY_SCIENCE8.pptx
glofernignacio
 
VARICELLA VACCINATION: A POTENTIAL STRATEGY FOR PREVENTING MULTIPLE SCLEROSIS
ijab2
 
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
PrabakaranMP1
 
Chemical bonding and molecular structure
karthikeyan
 
protein biochemistry.ppt for university classes
bayewodajo27
 
DRUG THERAPY FOR SHOCK gjjjgfhhhhh.pptx.
vipvivek704
 
SCIENCE10 Q1 5 WK8 Evidence Supporting Plate Movement.pptx
ROLANARIBATO3
 
INTRODUCTION TO EVS | Concept of sustainability
manumaddy15198
 
2. Earth - The Living Planet earth and life
markjustinebarolobau
 
GEN. BIO 1 - CELL TYPES & CELL MODIFICATIONS
breeysal7
 
2. Earth - The Living Planet Module 2ELS
markjustinebarolobau
 
TOTAL hIP ARTHROPLASTY Presentation.pptx
hararghemssc
 
ANEMIA WITH LEUKOPENIA MDS 07_25.pptx htggtftgt fredrctvg
vedvyas5668
 
. Radiology Case Scenariosssssssssssssss
MiraMusleh
 
The World of Physical Science, • Labs: Safety Simulation, Measurement Practice
joybejerano
 
famous lake in india and its disturibution and importance
judesebina
 
Phytochemical Investigation of Miliusa longipes.pdf
IrfanShahirSharafi
 

Multitask Adversarial Learning of Deep Neural Networks for Medical Imaging and Image Analysis

  • 1. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 0 One out this pair of images is an original image of intravascular ultrasound (IVUS) and the other is generated by a special type of artificial neural network (ANN) known as generative adversarial network (GAN). Can you identify the original?
  • 2. Multitask Adversarial Learning of Deep Neural Networks for Medical Imaging and Image Analysis Dr. Debdoot Sheet Assistant Professor, Department of Electrical Engineering Principal Investigator, Kharagpur Learning, Imaging and Visualization Group Indian Institute of Technology Kharagpur www.facweb.iitkgp.ac.in/~debdoot/
  • 3. Disclosure 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 2 • Conflicts and Interests – SkinCurate Research Pvt. Ltd. – Founder, stock options – Intel Technology India Pvt. Ltd. – Research Sponsor, Startup Incubation Mentor – Dept. of Biotechnology, Govt. of India – Research Sponsor – Sigtuple Technologies Pvt. Ltd. – Research Sponsor – Tata Steel Ltd. – Research Sponsor – Amazon Web Services (AWS) Inc. – Research Sponsor – Nesa Medtech Pvt. Ltd. – Research Sponsor – Samsung Inc. – Research Sponsor – Nvidia Inc. – Lab. Resources Sponsor – Microsoft – Collaborator and Lab. Resources Sponsor – Texas Instruments India Pvt. Ltd. – Lab. Resources Sponsor – Analog Devices India Pvt. Ltd. – Lab. Resources Sponsor – Indian Council of Medical Research, Govt. of India – Travel Grants – Dept. of Science and Technology, Govt. of India – Travel Grants, Startup Incubation Mentor – Biotechnology Industry Research Assistance Council (BIRAC) – Startup Incubation Grant – Society for Innovation and Entrepreneurship (SINE) IIT Bombay – Startup Incubation Grant
  • 4. Learning? A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E -Tom Mitchell Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 31 Sept. 2019
  • 5. Contributions to be Discussed 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 4 Learning with Multitask Adversaries using Weakly Labelled Data for Semantic Segmentation in Retinal Images (MIDL 2019) Oindrila Saha, Rachana Sathish, Debdoot Sheet Simulating Patho-realistic Ultrasound Images using Deep Generative Networks with Adversarial Learning (ISBI 2018) Francis Tom, Debdoot Sheet Learning a Deep Convolution Network with Turing Test Adversaries for Microscopy Image Super Resolution (ISBI 2019) Francis Tom, Himanshu Sharma, Dheeraj Mundhra, Tathagata Rai Dastidar, Debdoot Sheet UltraCompression: Framework for High Density Compression of Ultrasound Volumes using Physics Modeling Deep Neural Networks (ISBI 2019) Debarghya China, Francis Tom, Sumanth Nandamuri, Aupendu Kar, Mukundhan Srinivasan, Pabitra Mitra, Debdoot Sheet
  • 6. RECALL 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 5
  • 7. Statistically Informed Decision 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 6 𝑃 Ω = 𝜔|𝐱 = 𝒙 = 𝑃 Ω = 𝜔, 𝐱 = 𝒙 𝑃 𝐱 = 𝒙 𝑃 Ω = 𝜔|𝐱 = 𝒙 = 𝑝 𝐱 = 𝒙|Ω = 𝜔 𝑃 Ω = 𝜔 𝑃 𝐱 = 𝒙 Posterior probability Likelihood Prior probability EvidenceBayes’ Rule
  • 8. Decision 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 7 𝑃 Ω = 𝜔|𝐱 = 𝒙 = 𝑝 𝐱 = 𝒙|Ω = 𝜔 𝑃 Ω = 𝜔 𝑃 𝐱 = 𝒙 Posterior probability Likelihood Prior probability EvidenceBayes’ Rule Shade of color Length 𝑥1 𝑥2 𝜔 = arg max 𝑃 Ω = 𝜔|𝐱 = 𝒙 Maximum aposteriori (MAP) Decision boundary
  • 9. Challenges with a Decision Boundary 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 8 𝑥1 𝑥2 𝑥1 𝑥2 𝑥1 𝑥2 Abundant samples Not-so abundant samples Scarce samples
  • 10. Understanding these Challenges 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 9 𝑥1 𝑥2 𝑥1 𝑥2 𝑃 Ω = 𝜔|𝐱 = 𝒙 = 𝑝 𝐱 = 𝒙|Ω = 𝜔 𝑃 Ω = 𝜔 𝑃 𝐱 = 𝒙 Likelihood 𝑝 𝒙 ~𝜙 𝒙3 Cubic 𝑝 𝒙 ~𝜙 𝒙2 Quadratic 𝑝 𝒙 ~𝜙 𝑥Linear 𝑥2 = 𝑚𝑥1 + 𝑐
  • 11. AI from Heuristics to DL 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 10 Input Hand Designed Rules Output Input Hand Designed Features Output Learned Decision Input Learned Features Output Learned Decision Input Learned Features Output Learned Decision Learned Abstract Features Rule based AI Classical ML Representation Learn. Deep Learning
  • 12. Objectives of Machine Learning 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 11 𝑝 𝒙 ~𝜙 𝒙3 Cubic 𝑝 𝒙 ~𝜙 𝒙2 Quadratic 𝑝 𝒙 ~𝜙 𝑥 Linear Increasing order of complexity 𝑥1 𝑥2 𝑥1 𝑥2 Data space plane MLE Mean Squared Error (MSE) Perception Loss (PL)
  • 13. DL addressing these ML Objectives 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 12 𝑥1 𝑥2 𝑥1 𝑥2 Data space plane Van Dyk, David A., and Xiao-Li Meng. “The art of data augmentation.” Journal of Computational and Graphical Statistics 10.1 (2001): 1-50. rotate flipud fliplr flipud
  • 14. DL addressing these ML Objectives 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 13 𝑝 𝒙 ~𝜙 𝒙3 Cubic 𝑝 𝒙 ~𝜙 𝒙2 Quadratic 𝑝 𝒙 ~𝜙 𝑥 Linear Increasing order of complexity 𝑥1 𝑥2 𝑥1 𝑥2 Learned Features
  • 15. DL addressing these ML Objectives 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 14 MLE Mean Squared Error (MSE) Perception Loss (PL) low MSE, high PL high MSE, low PL Sketch2Photo CNN Discriminator Neural Network Real vs. Fake Photograph Generated
  • 16. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 15 Learning with Multitask Adversaries using Weakly Labelled Data for Semantic Segmentation in Retinal Images Oindrila Saha, Rachana Sathish, Debdoot Sheet International Conference on Medical Imaging with Deep Learning (MIDL), London, July 2019 Oindrila Saha Rachana Sathish
  • 17. Preamble 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 16 SemanticSegmentation Network Retinal vessels, Optic disc, Optic cup, Edema, Soft exudates, Hard exudates
  • 18. Challenge 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 17 Image size #768✕584 Dataset 1 Retinal vessels Train # 20, Test # 20 Image size # 3,504 ✕ 2,336 Retinal vessels, optic disc Images # 18 Image size # 1,500 ✕ 1,152 Hard exudates, soft exudates, Microaneurysms, Hemorrhages Images # 89 Dataset 2 Dataset 3
  • 19. Solution 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 18 Semantic Segmentation Network 3✕M✕N I(:,:,:) C✕M✕N Discriminator 2 C✕M✕N Manual vs. Synthetic ChannelShuffler(.)ChannelShuffler(.) Discriminator 1 Presence of Class 𝐲𝐱
  • 20. Solution 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 19 Semantic Segmentation Network 3✕M✕N I(:,:,:) Discriminator 2 C✕M✕N Manual vs. Synthetic ChannelShuffler(.)ChannelShuffler(.) Discriminator 1 Segmented Channel Idx C✕M✕N I(:,:,:) I(:,:,:) 𝐱 𝐲 𝐲 𝐲𝐱 𝐿 𝑠𝑒𝑔 ∙ = ∀𝐲 𝑐 ≠∅ 𝐵𝐶𝐸 𝐲 𝑐 , 𝐲 𝑐 𝛻𝐿 𝑠𝑒𝑔 ∙
  • 21. Discriminator 1 Semantic Segmentation Network 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 20 Discriminator 2 C✕M✕N Manual vs. Synthetic ChannelShuffler(.) Segmented Channel Idx 𝐲𝐱 3✕M✕N I(:,:,:) Solution ChannelShuffler(.) C✕M✕N 𝐱 1 𝐲𝐱 2 𝐱 3 𝐱 1 𝐱 2 𝐱 3 𝐲𝐲 𝑘 = 𝐲 𝑠ℎ𝑢𝑓𝑓𝑙𝑒 1,2, ⋯ , 𝐶
  • 22. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 21 Semantic Segmentation Network 3✕M✕N I(:,:,:) C✕M✕N Discriminator 2 C✕M✕N Manual vs. Synthetic ChannelShuffler(.)ChannelShuffler(.) 𝐲𝐱 Solution Discriminator 1 Presence of Class 𝐱 1 𝐱 2 𝐱 3 𝐲 𝐲 𝐧1 = 1,0,0,0,1,1 𝐧1 = 1,1,0,1,1,1 𝐿 𝐷1 ∙ = 𝐵𝐶𝐸 𝐧1, 𝐧1 𝛻𝐿 𝐷1 ∙ 𝐧1 Ganin, Yaroslav, et al. "Domain-adversarial training of neural networks." The Journal of Machine Learning Research 17.1 (2016): 2096-2030.
  • 23. Discriminator 1 Segmented Channel Idx 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 22 Semantic Segmentation Network 𝐲𝐱 Solution 𝐿 𝐷2 ∙ = 𝐵𝐶𝐸 𝐧2, 𝐧2 Discriminator 2 𝛻𝐿 𝐷2 ∙ 3✕M✕N I(:,:,:) C✕M✕N ChannelShuffler(.) C✕M✕NChannelShuffler(.) 𝐧2 = 0, 1 𝐧2 = 1, 0 𝐧2 Ganin, Yaroslav, et al. "Domain-adversarial training of neural networks." The Journal of Machine Learning Research 17.1 (2016): 2096-2030. {Manual, Synthetic} vs. {Synthetic, Manual}
  • 24. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 23 3✕M✕N C✕M✕N C✕M✕N Manual vs. Synthetic ChannelShuffler(.)ChannelShuffler(.) Segmented Channel Idx 𝐲𝐱 Semantic Segmentation Network I(:,:,:) Discriminator 2 Discriminator 1 Solution 𝛻𝐿 𝑠𝑒𝑔 ∙ 𝛻𝐿 𝐷2 ∙ 𝛻𝐿 𝐷1 ∙ 𝛻𝐿 𝑎𝑑𝑣 ∙ = 𝛼1 𝛻𝐿 𝐷1 ∙ −𝛼2 𝛻𝐿 𝐷2 ∙ 𝐿 𝑎𝑑𝑣 ∙ = 𝛼1 𝐿 𝐷1 ∙ +𝛼2 1 − 𝐿 𝐷2 ∙
  • 25. Results 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 24
  • 26. Computed Super-Resolution Microscopy Learning a Deep Convolution Network with Turing Test Adversaries for Microscopy Image Super Resolution Francis Tom, Himanshu Sharma, Dheeraj Mundhra, Tathagato Rai Dastidar, Debdoot Sheet IEEE International Symposium on Biomedical Imaging (ISBI), 2019. Francis Tom Himanshu Sharma Dheeraj Mundhra Tathagato Rai Dastidar Computed Super-Resolution Microscopy
  • 27. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 26 Turing Test 1 Turing Test 2 Region Proposals {Real, SR} vs. {SR, Real} {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙ Super Resolution CNN 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ Real LR SR
  • 28. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 27 Super Resolution CNN Turing Test 2 Region Proposals 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ {Real, SR} vs. {SR, Real} LR 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙ Turing Test 1 SR {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ Real
  • 29. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 28 Super Resolution CNN Turing Test 1 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ LR {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙ Turing Test 2 Region Proposals {Real, SR} vs. {SR, Real} Real SR 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask
  • 30. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 29 Turing Test 1 Turing Test 2 Region Proposals 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ {Real, SR} vs. {SR, Real} Real {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask Super Resolution CNN LR SR 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙
  • 31. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 30 Super Resolution CNN Turing Test 1 Turing Test 2 Region Proposals 𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ 𝛻𝐽 𝑅𝑒𝑐𝑜𝑛 ∙ {Real, SR} vs. {SR, Real} Real LR SR {Real, SR} vs. {SR, Real}𝐽 𝑇1 ∙ 𝛻𝐽 𝑇1 ∙ 𝐽 𝑇2 ∙ 𝛻𝐽 𝑇2 ∙ Mask 𝛻𝐽 𝐴𝑑𝑣 ∙ = −𝛼𝛻𝐽 𝑇1 ∙ −𝛽𝛻𝐽 𝑇2 ∙
  • 32. Some Results 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 31
  • 33. … and our AI powers Digital Pathology in India 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 32
  • 34. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 33 One out this pair of images is an original image of intravascular ultrasound (IVUS) and the other is generated by a special type of artificial neural network (ANN) known as generative adversarial network (GAN). Can you identify the original?
  • 35. Can Generative Adversarial Networks Model Imaging Physics? Some experiences with simulating patho-realistic ultrasound images Francis Tom DeepSIP DeepKLIV “Simulating Patho-realistic Ultrasound Images using Deep Generative Networks with Adversarial Learning” Francis Tom, Debdoot Sheet IEEE International Symposium on Biomedical Imaging (ISBI), 2018.
  • 36. Simulating Ultrasound 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 35 Jørgen Arendt Jensen and Peter Munk, ”Computer phantoms for simulating ultrasound B-mode and cfm images”, Acoustical Imaging”, vol. 23, pp. 75-80, Eds.: S. Lees and L. A. Ferrari, Plenum Press, 1997. J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med. Biol., vol. 25, no. 3, pp. 463– 479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]
  • 37. Pseudo B-mode US Image Simulation 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 36 𝑇 𝑥, 𝑦 = 𝐸 𝑥, 𝑦 𝐺 𝑛 𝐺 𝑛 ~𝒩 0,1 𝑘0 = 2𝜋𝑓0 𝑐 ℎ 𝑥 𝑥, 𝑦 = sin 𝑘0 𝑥 𝑒 − 𝑥2 2𝜎 𝑥 2 ℎ 𝑦 𝑥, 𝑦 = 𝑒 − 𝑦2 2𝜎 𝑦 2 𝑉 𝑥, 𝑦 = 𝑇 𝑥, 𝑦 ∗ ℎ 𝑥 𝑥, 𝑦 ∗ ℎ 𝑦 𝑥, 𝑦 𝑏 𝑥, 𝑦 = 𝐻𝑖𝑙𝑏𝑒𝑟𝑡 𝑉 𝑥, 𝑦 J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med. Biol., vol. 25, no. 3, pp. 463–479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]
  • 38. Where do we stand with Simulations? 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 37 Digital Phantom Pseudo B-mode US Real B-mode US How to bridge this disparity in appearance?
  • 39. The Challenge 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 38 𝑇 𝑥, 𝑦 = 𝐸 𝑥, 𝑦 𝐺 𝑛 𝐺 𝑛 ~𝒩 0,1 𝑉 𝑥, 𝑦 = 𝑇 𝑥, 𝑦 ∗ ℎ 𝑥 𝑥, 𝑦 ∗ ℎ 𝑦 𝑥, 𝑦 𝑏 𝑥, 𝑦 = 𝐻𝑖𝑙𝑏𝑒𝑟𝑡 𝑉 𝑥, 𝑦 J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med. Biol., vol. 25, no. 3, pp. 463–479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006] Are these convolution kernels not descriptive enough to model the signal mixing process resulting in image formation? Can we learn the convolution kernels? 𝑘0 = 2𝜋𝑓0 𝑐 ℎ 𝑥 𝑥, 𝑦 = sin 𝑘0 𝑥 𝑒 − 𝑥2 2𝜎 𝑥 2 ℎ 𝑦 𝑥, 𝑦 = 𝑒 − 𝑦2 2𝜎 𝑦 2
  • 40. Learning Convolution with a Neural Network 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 39 * = * =f 𝐖𝑖 𝒙 𝒚 𝒛 𝐖𝑖+1 𝒑 𝐖𝑖 𝑘+1 = 𝐖𝑖 𝑘 − 𝜂 𝜕𝐽 𝐖 𝜕𝐖𝑖 𝐽 𝐖 = 𝒑 − 𝒑 𝜕𝐽 𝐖 𝜕𝐖𝑖 = 𝜕𝐽 𝐖 𝜕𝒑 𝜕𝒑 𝜕𝐖𝑖+1 𝜕𝒛 𝜕𝒚 𝜕𝒚 𝜕𝐖𝑖 # Training Samples?
  • 41. Adversarial Transformation for US Simulation 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 40 Generator Net 𝐺 𝜃 Discriminator Net 𝐷 𝜙 𝐿 𝐺 𝜃 Simulated vs. Real 𝐿 𝐷 𝜙 𝛻𝐿 𝐷 𝜙 𝛻𝐿 𝐺 𝜃 Digital Phantom Simulated Real
  • 42. Adversarial Transformation for US Simulation 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 41 Hu, Y., Gibson, E., Lee, L. L., Xie, W., Barratt, D. C., Vercauteren, T., & Noble, J. A. (2017). Freehand Ultrasound Image Simulation with Spatially-Conditioned Generative Adversarial Networks. In Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment (pp. 105-115). Springer, Cham.
  • 43. The Challenges 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 42 Transducer • Non uniform sampling in Cartesian coordinate domain • Signal interpolated for image formation, leading to smearing of speckles
  • 44. Our Solution 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 43 Digital Phantom Digital Phantom cart2pol( ) Pseudo B-mode Stage 0 Stage I 𝐺𝐼 𝜃 Stage II 𝐺𝐼𝐼 𝜃 𝐿 𝐺 𝐼 𝜃 𝐿 𝐺 𝐼𝐼 𝜃 cart2pol() Stage I 𝐷𝐼 𝜙 Stage II 𝐷𝐼𝐼 𝜙 𝐿 𝐷 𝐼 𝜙 𝐿 𝐷 𝐼𝐼 𝜙 Stage I Sim Stage II Sim Real Real Simulated vs. Real 64 x 64 256 x 256256 x 256256 x 256256 x 256 𝛻𝐿 𝐺 𝐼 𝛻𝐿 𝐺 𝐼𝐼 𝛻𝐿 𝐷 𝐼 𝛻𝐿 𝐷 𝐼𝐼
  • 45. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 44 Lets Quiz: Real vs. Simulated
  • 46. More Resources • Neural Information Processing Systems (NeurIPS) • International Conference on Learning Representations (ICLR) • International Conference on Machine Learning (ICML) • Association for Advancement of Artificial Intelligence (AAAI) • Computer Vision and Pattern Recognition (CVPR) • International Conference on Medical Imaging with Deep Learning (MIDL) • IEEE Int. Symp. Biomed. Imaging (ISBI) • Journal of Machine Learning Research (JMLR) • Machine Learning • IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) • IEEE Trans. Neural Networks and Learning Systems (TNNLS) • IEEE Trans. Medical Imaging (TMI) • Medical Image Analysis (MedIA) • Medical Image Computing and Computer Assisted Intervention (MICCAI) 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 45
  • 47. Take home message 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 46
  • 48. 1 Sept. 2019 Multitask Adversarial Learning of DNN for MedIA [Debdoot Sheet] [Talk at Midnapore College] 47 Thank you from #iitkliv http://iitkliv.github.io

Editor's Notes

  • #27: Credits to Francis etc…
  • #36: Credits to Francis etc…