Can Generative Adversarial Networks Model Imaging Physics?

Can Generative Adversarial Networks
Model Imaging Physics?
Dr. Debdoot Sheet
Assistant Professor, Department of Electrical Engineering
Principal Investigator, Kharagpur Learning, Imaging and Visualization Group
Indian Institute of Technology Kharagpur
www.facweb.iitkgp.ernet.in/~debdoot/
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 2

Disclosure
• Conflicts and Interests
– SkinCurate Research Pvt. Ltd. – Founder, stock options
– Intel Technology India Pvt. Ltd. – ResearchSponsor
– Dept. of Biotechnology, Govt. of India – Research Sponsor
– Nesa Medtech Pvt. Ltd. – Research Sponsor
– Sigtuple Technologies Pvt. Ltd. – ResearchSponsor
– Samsung Inc. – Research Sponsor
– Nvidia Inc. – Lab. Resources Sponsor
– Texas Instruments India Pvt. Ltd. – Lab. Resources Sponsor
– Analog Devices India Pvt. Ltd. – Lab. Resources Sponsor
– Indian Council of Medical Research, Govt. of India – Travel Grants
– Dept. of Science and Technology, Govt. of India – Travel Grants
– GE John F. Welch Technology Center (GE-JFWTC) – Startup Incubation Grant
– Biotechnology Industry Research Assistance Council (BIRAC) – Startup Incubation Grant

Can Generative Adversarial Networks
Model Imaging Physics?
Some experiences with simulating patho-realistic ultrasound images
In association with Francis Tom Debarghya China DeepSIP DeepKLIV
“Simulating Patho-realistic Ultrasound Images using Deep Generative
Networks with Adversarial Learning”
Francis Tom, Debdoot Sheet
IEEE International Symposium on Biomedical Imaging (ISBI), 2018.

Overview of the Talk
• Ultrasound Imaging
– Physics and Instrumentation
– Signal Processing and Visualization
– Simulating Ultrasound Images
• Artificial Neural Networks (ANN)
– Convolutional Neural Network (CNN)
– Generative Modeling
– Adversarial Learning
• Simulation via Adversarial Transformation
• Take home message

Ultrasound Imaging

Physics of Ultrasound Imaging
Transducer
• M. Ali, D. Magee and U. Dasgupta, "Signal Processing Overview of Ultrasound Systems for Medical Imaging", Texas Instruments
White Paper, no. SPRAB12, Nov., 2008.
• M. A. Haidekker, "Ultrasound imaging", Medical Imaging Technology, pp. 97-110, 2013.

Sensing and Analog Signal Processing
Transducer
T/R Switch
HV Transmitter
Analog Front End
Tx Beamformer,
Front-end digital
controller,
Rx Beamformer
z ! " = $%&,( ), *
%+,, ), * = %-,(. ) /01 ), * /01 ), * = 234 5 ,
sin 9:
sin 9;
=
<:
<;
%&,, ), * = %+,, ), *
<; cos9: − <: cos 9;
<; cos9: + <: cos 9;
;
%&,( ), * = %&,, ), * /01 ), * A ) = B ) + CD )
B ) = B( + B: ) E
D ) = F( ) + DG )

Digital Signal Processing
Transducer
9
*
H *, 9 = ! " I *, 9 = J log H *, 9 + <

Visualization

Who is this Person?
Ian Donald was a Professor of Midwifery at Glasgow University, when he
first explored the clinical use of ultrasound in the 1950s after seeing it
being used in the Glasgow shipyards to look for flaws in metallurgy.

Simulating Ultrasound
Jørgen Arendt Jensen and Peter Munk, ”Computer phantoms
for simulating ultrasound B-mode and cfm images”, Acoustical
Imaging”, vol. 23, pp. 75-80, Eds.: S. Lees and L. A. Ferrari,
Plenum Press, 1997.
J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A
computer simulation", Phys. Med. Biol., vol. 25, no. 3, pp. 463–
479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]

Pseudo B-mode US Image Simulation
0 M, N = O M, N P Q
P Q ~S 0,1 F( =
2W)(
<
ℎY M, N = sin F(M 2
3
YZ
;[
Z
ℎE M, N = 2
3
EZ
;[]
Z
^ M, N = 0 M, N ∗ ℎY M, N ∗ ℎE M, N
I M, N = `abI2H" ^ M,N
J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med.
Biol., vol. 25, no. 3, pp. 463–479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]

Where do we stand with Simulations?
Digital Phantom Pseudo B-mode US Real B-mode US
How to bridge this disparity in appearance?

The Challenge
0 M, N = O M, N P Q
P Q ~S 0,1
^ M, N = 0 M, N ∗ ℎY M, N ∗ ℎE M, N
I M, N = `abI2H" ^ M,N
J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med.
Biol., vol. 25, no. 3, pp. 463–479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]
Are&these&convolution)kernels)not&descriptive&
enough&to&model&the&signal&mixing&process&
resulting&in&image&formation?
Can&we&learn the&convolution)kernels?
F( =
2W)(
<
ℎY M, N = sin F(M 2
3
YZ
;[
Z
ℎE M, N = 2
3
EZ
;[]
Z

Convolution
Convolutional Neural Networks [Debdoot Sheet] 15
cd =
/ − e + 2fd
gd
+ 1
ch =
i − ℎ + 2fh
gh
+ 1
cd
ch
fd
fh
gd
gh

Convolution
Convolutional Neural Networks [Debdoot Sheet] 16
3×/×i
3×e×ℎ
3×e×ℎ
/
i
2× / − e + 1 × i − ℎ + 1

Who is this Person?
Yann LeCun is well known for his work
on optical character recognition and
computer vision using convolutional neural
networks (CNN), also famously referred to as
the LeNet, and is a founding father of
convolutional nets.
He is also one of the main creators of
the DjVu image compression technology
(together with Léon Bottou and Patrick
Haffner).

Learning Convolution with a Neural Network
* =
* =f
lm
n
o
p
lmq:
rs
lm
tq:
= lm
t
− u
vw l
vlm
w l = r − rs
vw l
vlm
=
vw l
vr
vr
vlmq:
vp
vo
vo
vlm
#&Training&
Samples?

Generative Models
Adversarial Autencoder [Debdoot Sheet] 19
Encoder Decoderz

Adversarial Learning
Adversarial Autencoder [Debdoot Sheet] 20
Encoder Decoderz
Discriminator Fake vs. Real

Adversarial Transformation of Photographs
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, & Russell Webb, “Learning from Simulated and
Unsupervised Images through Adversarial Training”, Proc. IEEE/CVF Conf. Comp. Vis. Patt. Recog. (CVPR), pp. 2107-2116, 2018.
(Best Paper @ CVPR2018)

Adversarial Transformation for US Simulation
Generator&Net
P 9
Discriminator&Net
x y
z{ 9
Simulated&vs.&Real
z| y
}z| y
}z{ 9
Digital&
Phantom
Simulated
Real

Adversarial Transformation for US Simulation
Hu, Y., Gibson, E., Lee, L. L., Xie, W., Barratt, D. C., Vercauteren, T., & Noble, J. A. (2017). Freehand Ultrasound Image Simulation
with Spatially-Conditioned Generative Adversarial Networks. In Molecular Imaging, Reconstruction and Analysis of Moving Body
Organs, and Stroke Imaging and Treatment (pp. 105-115). Springer, Cham.

The Challenges
Transducer
• Non&uniform&sampling&in&Cartesian&
coordinate&domain
• Signal&interpolated&for&image&
formation,&leading&to&smearing&of&
speckles

Our Solution
Digital&Phantom Digital&Phantom
cart2pol(&)
Pseudo&BLmode
Stage 0
Stage I
P+ 9
Stage II
P++ 9
z{~
9 z{~~
9
cart2pol(&)
Stage I
x+ y
Stage II
x++ y
z|~
y z|~~
y
Stage&I&Sim Stage&II&Sim
Real Real Simulated&vs.&Real
64&x&64 256&x&256256&x&256256&x&256256&x&256
}z{~
}z{~~
}z|~
}z|~~

Architecture of the Networks
Stage&I&G()
Stage&I&D()
Stage&II&G()
Stage&II&D()

Lets Quiz: Real vs. Simulated

Results
Intra-tissue similarity of speckle (JS div.) Inter-tissue dissimilarity of speckle (JS div.)

Take home message
• Full paper on US simulation:
https://arxiv.org/abs/1712.07881
• Adversarial autoencoders:
https://arxiv.org/abs/1511.05644
• Generative adversarial networks:
https://papers.nips.cc/paper/542
3-generative-adversarial-nets.pdf

Can Generative Adversarial Networks Model Imaging Physics?

More Related Content

What's hot (20)

Similar to Can Generative Adversarial Networks Model Imaging Physics? (20)

Recently uploaded (20)

Can Generative Adversarial Networks Model Imaging Physics?