Deep Advances in Generative Modeling
Download as PPTX, PDF9 likes4,573 views
Deep generative models are making progress in modeling complex, high-dimensional data in an unsupervised manner. Two promising approaches are variational autoencoders (VAEs) and generative adversarial networks (GANs). VAEs impose a prior on the code space to regularize and allow for sampling, while GANs use a generator and discriminator in an adversarial training procedure. Recent work has focused on extensions of these models, including conditional generation and combining aspects of VAEs and GANs. However, generative modeling of natural images remains challenging, with models still underfitting the complexity of unconstrained data.
![Useful Generative Model - Skipthought [1506.06726]](https://image.slidesharecdn.com/generativemodeling-alec-160321095536/85/Deep-Advances-in-Generative-Modeling-5-320.jpg)
![Two promising approaches
Variational
Autoencoders (VAE)
Kingma and Welling
[1312.6114]
Generative Adversarial
Networks (GAN)
Goodfellow et al. [1406.2661]
encoder Z decoder x̂X
z generator x̂
discriminator
X](https://image.slidesharecdn.com/generativemodeling-alec-160321095536/85/Deep-Advances-in-Generative-Modeling-6-320.jpg)
![Variational Autoencoder
from Kingma and Welling [1312.6114]
● Theoretically elegant autoencoder
● Straightforward to implement
● Impose a prior on code space
○ regularization
○ allows for sampling
● Optimizes variational lower bound on
likelihood
encoder Z decoder x̂X](https://image.slidesharecdn.com/generativemodeling-alec-160321095536/85/Deep-Advances-in-Generative-Modeling-7-320.jpg)
![VAE Extensions
from Kingma et al. [1406.5298]
Semi-Supervised Learning
from Gregor et al. [1502.04623]
DRAW](https://image.slidesharecdn.com/generativemodeling-alec-160321095536/85/Deep-Advances-in-Generative-Modeling-10-320.jpg)
![Deep convolutional GANs (DCGAN) [1511.06434]
Luke Metz Soumith ChintalaAlec Radford
tl;dr add more layers
indico indico FAIR](https://image.slidesharecdn.com/generativemodeling-alec-160321095536/85/Deep-Advances-in-Generative-Modeling-12-320.jpg)
![Deep convolutional GANs (DCGAN) [1511.06434]](https://image.slidesharecdn.com/generativemodeling-alec-160321095536/85/Deep-Advances-in-Generative-Modeling-13-320.jpg)
![DCGAN Architecture tricks
No fully connected layers
Batch Normalization Ioffe and Szegedy [1502.03167]
Leaky Rectifier in the discriminator
Use Adam Kingma and Ba [1412.6980]
Tweak Adam hyperparameters a bit (lr=0.0002, b1=0.5)](https://image.slidesharecdn.com/generativemodeling-alec-160321095536/85/Deep-Advances-in-Generative-Modeling-14-320.jpg)
![Hybridizing VAEs and GANs (best of both worlds?)
from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]](https://image.slidesharecdn.com/generativemodeling-alec-160321095536/85/Deep-Advances-in-Generative-Modeling-24-320.jpg)
![Hybridizing VAEs and GANs (best of both worlds?)
from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]](https://image.slidesharecdn.com/generativemodeling-alec-160321095536/85/Deep-Advances-in-Generative-Modeling-25-320.jpg)
Deep Advances in Generative Modeling
- 1. Deep Advances in Generative Modeling Alec Radford @AlecRad March 5th 2016
- 2. Generative modeling Modeling complex high dimensional data is an open problem. Deep generative models are currently making progress here. Various areas of study/application: unsupervised/representation/manifold learning generative counterparts of discriminative models density/likelihood estimation conditional generation
- 3. Examples of Generative Modeling
- 5. Useful Generative Model - Skipthought [1506.06726]
- 6. Two promising approaches Variational Autoencoders (VAE) Kingma and Welling [1312.6114] Generative Adversarial Networks (GAN) Goodfellow et al. [1406.2661] encoder Z decoder x̂X z generator x̂ discriminator X
- 7. Variational Autoencoder from Kingma and Welling [1312.6114] ● Theoretically elegant autoencoder ● Straightforward to implement ● Impose a prior on code space ○ regularization ○ allows for sampling ● Optimizes variational lower bound on likelihood encoder Z decoder x̂X
- 8. Generative Adversarial Networks z x̂ discriminator X generator
- 9. Generative Adversarial Networks z x̂ discriminator X generator
- 10. VAE Extensions from Kingma et al. [1406.5298] Semi-Supervised Learning from Gregor et al. [1502.04623] DRAW
- 11. GAN Extensions - LAPGAN
- 12. Deep convolutional GANs (DCGAN) [1511.06434] Luke Metz Soumith ChintalaAlec Radford tl;dr add more layers indico indico FAIR
- 13. Deep convolutional GANs (DCGAN) [1511.06434]
- 14. DCGAN Architecture tricks No fully connected layers Batch Normalization Ioffe and Szegedy [1502.03167] Leaky Rectifier in the discriminator Use Adam Kingma and Ba [1412.6980] Tweak Adam hyperparameters a bit (lr=0.0002, b1=0.5)
- 15. Really really really ridiculously good looking samples on constrained image distributions :(
- 17. Vector arithmetic properties of generator
- 18. Generator disentangles objects from scene?
- 19. Discriminator learns generalizing object detectors These are responses on validation examples!
- 20. Results on standard supervised tasks
- 22. Conditional DCGAN (unpublished) Sunrise over the ocean Beautiful falls and stream sahara desert sand dunes Tropical rainforest brazil Stars of the milkyway at night
- 23. Issues Still not completely stable especially for deep and higher res Unconstrained natural images Even the biggest models underfit Hard to evaluate no reliable/straightforward metrics No inference model limits kinds of analysis Little work on conv VAE equivalents makes comparison difficult Some funky stuff going on separate data/sample batchnorm statistics train with heuristic cost not GAN theory
- 24. Hybridizing VAEs and GANs (best of both worlds?) from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]
- 25. Hybridizing VAEs and GANs (best of both worlds?) from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]
- 27. indico.io