Vladislav Kolbasin “Introduction to Generative Adversarial Networks (GANs)”
1 like215 views
This document provides an introduction to generative adversarial networks (GANs). It begins with an agenda that covers what GANs are, applications of GANs such as image generation and inpainting, pros and cons of GANs, how to train a GAN, and example applications including face generation and lesion segmentation. GANs use two neural networks, a generator and discriminator, that compete against each other in a game theoretic framework. The generator learns to generate realistic samples to fool the discriminator, while the discriminator learns to distinguish generated from real samples.
Vladislav Kolbasin “Introduction to Generative Adversarial Networks (GANs)”
- 1. 1 Introduction to Generative Adversarial Networks Vladyslav Kolbasin Lead Software Developer, Consultant 14.03.2018
- 2. 2 Agenda 1. Introduction to generative adversarial network 2. What can be done with GANs? 3. Pros and cons 4. How to train it? How to construct your own GAN 5. Application examples 1. Faces in-painting 2. Lesion segmentation 6. Cool results 7. Conclusion
- 3. 3 Generative vs discriminative models • Pros: • Know about data distribution • Cons: • Very expensive to get • Need lots of data • Pros: • Easy to model • Cons: • Can classify, but not generate data Generative Discriminative P(c|x)
- 4. 4 Generative adversarial network. Architecture • GAN – two neural networks competing against each other in a zero-sum game framework. (Ian Goodfellow et al. in 2014) • G tries to “trick” D by generating samples that are hard for D to distinguish from data • Some kind of unsupervised learning • Networks try to: • D(G(z)) => max • D(x)(1 — D(G(z))) => max => Nash equilibrium: Credit: https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generati ve-models-and-adversarial-training-upc-2016
- 5. 5 What can be done with GANs? • Simulated environments and training data • Missing data • Multiple correct answers • Realistic generation tasks • Super-resolution • Learn useful embeddings
- 6. 6 Simulated environments and training data • Good generator, but too few real data • Can’t use real data • Medical data (EHR, EMR, ...) • Drug discovery • Autonomous driving data • Many others..
- 7. 7 Missing data. Image inpainting
- 9. 9 Multiple correct answers (Lotter et al 2016)
- 12. 12 Linear Latent Space G(z0 ) Linear Interpolation in z space: G(z0 +t*(z1 -z0 )) G(z1 )
- 13. 13 • “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Models”. https://arxiv.org/pdf/1511.06434.pdf Vector Space Arithmetics
- 14. 14 GAN Pros and Cons • Pros • Provide great results • Vector Space Arithmetics • Designing loss function • Cons • Generality penalty: for any given problem, application-tailored solutions might work better • It is very hard to train it (properly) ■ Sometimes training longer makes it worse ■ Discriminator almost always wins ■ Mode-collapse • Hard to evaluate
- 15. 15 Design loss function • No need in hand-engineering our loss functions for Img2Img task
- 16. 16 Design loss function • No need in hand-engineering our loss functions for Img2Img task
- 17. 17 Design loss function • No need in hand-engineering our loss functions for Img2Img task
- 19. 19 • Optimization criteria: • We want to make distributions equal • It is equivalent to maximizing log-likelihood or • KL-divergence: • We can use other divergence too • Jensen-Shannon divergence GAN. How to train? True data Noise for generating data
- 20. 20 KL-divergence
- 21. 21 KL-divergence
- 22. 22 • Introduced other metric: Wasserstein metric (earth mover's distance) • Summary about wGAN: • For mathematicians: it uses Wasserstein distance instead of Jensen-Shannon divergence to compare distributions • For engineers: it gets rid of a few unnecessary logarithms, and clips weights • For others: it employs an art critic instead of forgery expert • More math details: https://www.cph-ai-lab.com/wasserstein-gan-wgan Fixing mode collapse. wGAN(s) Wasserstein GAN. https://arxiv.org/pdf/1701.07875.pdf
- 23. 23 Fixing mode collapse. Improved wGAN. • Penalize the norm of the gradient of the critic with respect to its input instead of clipping weights. • This ‘gradient penalty’ is simply added to the Wasserstein distance for the total loss. Improved training of Wasserstein GAN. https://arxiv.org/pdf/1704.00028.pdf
- 24. 24 Improved WGAN: +gradient penalty
- 25. 2525 Demo. Mode-collapse. Fixing oscillations in GANs
- 26. 2626 How to train it?
- 27. 27 GAN. Training Discriminator Credit: https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
- 28. 28 GAN. Training Generator Credit: https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
- 31. 31 Generator
- 32. 32 Discriminator
- 33. 33 Init GAN
- 34. 34 Init GAN
- 35. 35 Train
- 36. 36 Train
- 37. 37 Train
- 38. 38 GAN hacks • https://github.com/soumith/ganhacks • About 17 hacks: • Normalize the inputs • Use Soft and Noisy Labels • Avoid Sparse Gradients: ReLU, MaxPool. Use LeakyReLU • Use SGD for discriminator and ADAM for generator • … • Sample from a gaussian distribution • Batch Normalization
- 39. 39 GANs Taxonomy
- 41. 41 1. Faces in-painting • Can we restore image with distortion? • Faces dataset: CelebA • Solution: Conditional GAN
- 42. 42 1. Faces mode-collapse 3800 4250 11000 12000 12300 14300
- 43. 43 • Competition Skin Lesion Analysis Towards Melanoma Detection • Segment image - find lesion in the picture • https://challenge.kitware.com/#challenge/560d7856cad3a57cfde481ba • Solution: Analogue of pix2pix architecture 2. Lesion segmentation
- 44. 44
- 45. 45
- 47. 47 Updates at the GAN Zoo https://github.com/hindupuravinash/the-gan-zoo
- 48. 48 StackGAN • Text-to-Image Synthesis https://github.com/hanzhanggit/StackGAN
- 49. 49
- 50. 50
- 51. 51 medGAN • “Generating Multi-label Discrete Patient Records using Generative Adversarial Network” https://arxiv.org/pdf/1703.06490.pdf
- 52. 52 Pix2Pix • Image-to-image translation with conditional adversarial nets • https://github.com/phillipi/pix2pix
- 53. 53 CycleGAN • Does image-to-image translation without input-output pairs • Unsupervised learning • https://github.com/junyanz/CycleGAN
- 54. 54 CycleGAN
- 55. 55 UNIT • Unsupervised Image-to-Image Translation Networks • https://arxiv.org/pdf/1703.00848.pdf • https://github.com/mingyuliutw/unit
- 56. 56 UNIT
- 57. 57 Conclusions • GANs can do very cool things • GANs training is hard and slow • GANs is an architecture that is rapidly developing • Three view of GANs: • Generative models: G(z) - unsupervised learning • Trainable regression loss . Learn L - what should I do. • Domain matching loss ■ Domain adaptation - train on source, adapt to target ■ Adversarial Discriminative Domain Adaptation https://arxiv.org/pdf/1702.05464.pdf
- 58. 58 References • Good tutorials from Ian Goodfellow: • https://arxiv.org/pdf/1701.00160.pdf • https://arxiv.org/pdf/1406.2661.pdf • https://youtu.be/RvgYvHyT15E • Training hacks: • https://github.com/soumith/ganhacks • Simplest GAN: • https://deeplearning4j.org/generative-adversarial-network • GAN collections: • https://github.com/nightrome/really-awesome-gan • https://github.com/wiseodd/generative-models • https://github.com/hindupuravinash/the-gan-zoo
- 59. 59 Vladyslav Kolbasin Lead Software Developer, Consultant vladyslav.kolbasin@globallogic.com ва Thank you!