Generative Models for General Audiences
1 like747 views
Generative models learn the distribution of data without labels, allowing for new data creation and modification across various applications such as image, video, and speech generation. Major breakthroughs in deep generative models include the Boltzmann machine, Variational Autoencoder (VAE), and Generative Adversarial Networks (GANs), each with distinct advantages and challenges. These models serve applications ranging from image generation to data augmentation and anomaly detection.
Generative Models for General Audiences
- 2. What is Generative Model? • Generative model learns the distribution of data without label • Create new data & Modify existing data • Image/video/language/speech generation • Data augmentation & semi-supervised learning • Data privacy (e.g., public release of medical dataset)
- 3. What is Generative Model? • Generative model learns the distribution of data without label • Unsupervised representation learning • Learning “good” representation with unlabeled data • Design an auxiliary task (hence often called self-supervised learning) • Generative model is a popular approach for unsupervised learning
- 4. Major Breakthroughs in Deep Generative Models 1980 1985 1990 2000 2005 2010 2015 1985 2006 Boltzmann machine (1985) • By G. Hinton et al. • Undirected graphical model • Computationally expensive Helmholtz machine (1986) • Directed graphical model Contrastivedivergence(1989) • G. Hinton et.al • Easy method for training RBM 1986 Deep Boltzmann machine (2009) • Undirected deep generative model consists of stacks of RBM • Layerwise training followed by joint learning Restricted Boltzmann machine (1986) • Bipartite version of BM 1995 Variational Autoencoder (2013) • By Durk Kingma et al. • Easy NN like back-propagation learning in deep generative model Greedilylayer-wisepre-training(2006) • Deep Belief Networks • Major breakthrough in learning deep generative model Generative Adversarial Network (2014) • Large scale image generative model G. Hinton, S. Ruslan D. Kingma, M. Welling I. GoodfellowG. Hinton, T. Sejnowski P. Smolensky G. Hinton, R. Neal • Hierarchical feature learning• Restricted Boltzmann Machine • Contrastive Divergence • Variatianal Autoencoder 2002 2009 2013 2014 2015 Ladder Network (2015) • Performance breakthrough in Semi-supervised learning
- 5. Approaches for Generative Models 1. Flow-based (autoregressive) model • Pros: exactly compute the probability of the data (many applications) • Cons: slow inference (autoregressive) or low quality (non-autoregressive) Autoregressive (e.g., PixelCNN) Non-autoregressive (e.g., Normalizing Flow)
- 6. Approaches for Generative Models 2. Variational autoencoder (VAE) • Pros: stable training & theoretical properties (lower bound of likelihood) • Cons: known to produce blurry outputs1 1. Recent methods combine VAE and other methods, e.g., IAF-VAE (+ flow) or WAE (+ GAN) to improve the performance Blurry!
- 7. Approaches for Generative Models 3. Generative adversarial network (GAN) • Pros: good performance (most SOTA models are based on GAN) • Cons: hard to train (alternating two networks leads instability)
- 8. Application: Image Generation • BigGAN
- 9. Application: Image Generation • StyleGAN (https://www.youtube.com/watch?v=kSLJriaOumA)
- 10. Application: Image-to-Image Translation • pix2pix (paired)
- 11. Application: Image-to-Image Translation • CycleGAN (unpaired)
- 12. Application: Image-to-Image Translation • StarGAN (multi-domain)
- 13. Application: Image-to-Image Translation • MUNIT (diverse output)
- 14. Application: Image-to-Image Translation • InstaGAN (shape modification) (from our lab)
- 15. Application: Emoji Generation • DTN (create personal avatar)
- 16. Application: Semantic Manipulation • pix2pixHD (https://www.youtube.com/watch?v=3AIpPlzM_qs)
- 17. Application: Pose Guided Generation • PG2 (change pose of person)
- 18. Application: Cloth Extraction • PixelDTGAN (extract cloth from image)
- 19. Application: Text-to-Image Synthesis • Reed et al.
- 20. Application: Text-to-Image Synthesis • Hong et al. (control location with bounding box)
- 21. Application: Video-to-Video Translation • vid2vid (paired) (https://www.youtube.com/watch?v=HCqXJth9t_k)
- 22. Application: Video-to-Video Translation • everybody dance now (with pose) (https://www.youtube.com/watch?v=PCBTZh41Ris)
- 23. Application: Video-to-Video Translation • Recycle-GAN (unpaired) (https://www.youtube.com/watch?v=F51RCdDIuUw)
- 24. Application: Data Augmentation • DAGAN (augment data to improve neural network performance)
- 25. Application: Anomaly Detection • AnoGAN (find anomaly from given data)