Multi-modal embeddings: from discriminative to generative models and creative ai
4 likes2,345 views
This document discusses multi-modal embeddings and generative models. It begins by covering common generative architectures like VAEs, DBNs, RNNs and CNNs. It then discusses specific applications including text generation with RNNs, image generation using techniques like DeepDream and style transfer, and audio generation using LSTMs and mixture density networks. The document advocates for creative AI as a "brush" for rapid experimentation in human-machine collaboration.
Multi-modal embeddings: from discriminative to generative models and creative ai
- 2. Modalities 2
- 3. • AE/VAE • DBN • Latent Vectors/ Manifold Walking • RNN / LSTM /GRU • Modded RNNs (ie Biaxial RNN) • CNN + LSTM/GRU • X + Mixture density network (MDN) Common Generative Architectures • DRAW • GRAN • DCGAN • DeepDream & other CNN visualisations • Splitting/Remixing NNs: • Image Analogy • Style Transfer • Semantic Style Transfer • Texture Synthesis • Compositional Pattern-Producing Networks (CPPN) • NEAT • CPPN w/ GAN+VAE.
- 4. • AE/VAE • DBN • Latent Vectors/ Manifold Walking • RNN / LSTM /GRU • Modded RNNs (ie Biaxial RNN) • CNN + LSTM/GRU • X + Mixture density network (MDN) Common Generative Architectures • DRAW • GRAN • DCGAN • DeepDream & other CNN visualisations • Splitting/Remixing NNs: • Image Analogy • Style Transfer • Semantic Style Transfer • Texture Synthesis • Compositional Pattern-Producing Networks (CPPN) • NEAT • CPPN w/ GAN+VAE. what we’ll cover
- 6. Alex Graves (2014) Generating Sequences With Recurrent Neural Networks Wanna Play ? Text Prediction (1)
- 7. Alex Graves (2014) Generating Sequences With Recurrent Neural Networks Wanna Play ? Text Prediction (1)
- 8. Alex Graves (2014) Generating Sequences With Recurrent Neural Networks Wanna Play ? Text Prediction (1)
- 9. Alex Graves (2014) Generating Sequences With Recurrent Neural Networks Wanna Play ? Text Prediction (2)
- 10. Alex Graves (2014) Generating Sequences With Recurrent Neural Networks Wanna Play ? Text Prediction (2)
- 11. Alex Graves (2014) Generating Sequences With Recurrent Neural Networks
- 12. Alex Graves (2014) Generating Sequences With Recurrent Neural Networks Wanna Play ? Handwriting Prediction
- 13. Alex Graves (2014) Generating Sequences With Recurrent Neural Networks Wanna Play ? Handwriting Prediction (we’re skipping the density mixture network details for now)
- 15. Wanna Play ? Text generation 15 Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
- 16. Wanna Play ? Text generation 16 Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
- 19. Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
- 20. Andrej Karpathy, Justin Johnson, Li Fei-Fei (2015) Visualizing and Understanding Recurrent Networks
- 21. Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
- 25. Turn Convnet Around: “Deep Dream” Image -> NN -> What do you (think) you see -> Whats the (text) label Image -> NN -> What do you (think) you see -> feed back activations -> optimize image to “fit” to the ConvNets “hallucination” (iteratively)
- 26. Google, Inceptionism: Going Deeper into Neural Networks Turn Convnet Around: “Deep Dream” see also: www.csc.kth.se/~roelof/deepdream/
- 27. Turn Convnet Around: “Deep Dream” see also: www.csc.kth.se/~roelof/deepdream/ Google, Inceptionism: Going Deeper into Neural Networks
- 30. Multifaceted Feature Visualization Anh Nguyen, Jason Yosinski, Jeff Clune (2016) Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
- 33. Preferred stimuli generation Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune (2016) AI Neuroscience: Understanding Deep Neural Networks by Synthetically Generating the Preferred Stimuli for Each of Their Neurons
- 35. Inter-modal: Style Transfer (“Style Net” 2015) Leon A. Gatys, Alexander S. Ecker, Matthias Bethge , 2015. A Neural Algorithm of Artistic Style (GitXiv)
- 37. Inter-modal: Image Analogies (2001) A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Salesin. (2001) Image Analogies, SIGGRAPH 2001 Conference Proceedings. A. Hertzmann (2001) Algorithms for Rendering in Artistic Styles Ph.D thesis. New York University. May, 2001.
- 38. Inter-modal: Image Analogies (2001)
- 39. Inter-modal: Style Transfer (“Style Net” 2015) Leon A. Gatys, Alexander S. Ecker, Matthias Bethge , 2015. A Neural Algorithm of Artistic Style (GitXiv)
- 46. style layers: 5_1 + 5_2 + 5_3 + 5_4
- 47. Gene Kogan, 2015. Why is a Raven Like a Writing Desk? (vimeo)
- 48. Inter-modal: Style Transfer+MRF (2016) Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis, 2016, Chuan Li, Michael Wand
- 49. Inter-modal: Style Transfer+MRF (2016) Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis, 2016, Chuan Li, Michael Wand
- 50. Inter-modal: Pretrained Style Transfer (2016) Texture Networks: Feed-forward Synthesis of Textures and Stylized Images, 2016, Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, Victor Lempitsky 500x speedup! (avg min loss from 10s to 20ms)
- 51. Inter-modal: Pretrained Style Transfer #2 (2016) Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks, Chuan Li, Michael Wand, 2016 similarly 500x speedup
- 52. Inter-modal: Perceptual Loss ST (2016) Perceptual Losses for Real-Time Style Transfer and Super-Resolution, 2016, Justin Johnson, Alexandre Alahi, Li Fei-Fei
- 53. Perceptual Losses for Real-Time Style Transfer and Super-Resolution, 2016, Justin Johnson, Alexandre Alahi, Li Fei-Fei
- 54. 54 Inter-modal: Style Transfer (“Style Net” 2015) @DeepForger
- 55. 55 Inter-modal: Style Transfer (“Style Net” 2015) @DeepForger
- 56. + + = Inter-modal: Semantic Style Transfer
- 57. https://github.com/alexjc/neural-doodle Semantic Style Transfer (“Neural Doodle”)
- 59. Synthesise textures (random weights) - which activation function should i use? - pooling? - min nr of units? Experiment: https://nucl.ai/blog/extreme-style-machines/ Random Weights (kinda like “Extreme Learning Machines”)
- 60. Synthesise textures (random weights) totally random initialised weights: https://nucl.ai/blog/extreme-style-machines/
- 61. Synthesise textures (randim weights) https://nucl.ai/blog/extreme-style-machines/ Activation Functions
- 62. Synthesise textures (randim weights) https://nucl.ai/blog/extreme-style-machines/ Activation Functions
- 63. Synthesise textures (randim weights) https://nucl.ai/blog/extreme-style-machines/ Activation Functions
- 64. Synthesise textures (randim weights) https://nucl.ai/blog/extreme-style-machines/ Down-sampling
- 65. Synthesise textures (randim weights) https://nucl.ai/blog/extreme-style-machines/ Down-sampling
- 66. Synthesise textures (randim weights) https://nucl.ai/blog/extreme-style-machines/ Down-sampling
- 67. Synthesise textures (randim weights) https://nucl.ai/blog/extreme-style-machines/ Nr of units
- 68. Synthesise textures (randim weights) https://nucl.ai/blog/extreme-style-machines/ Nr of units
- 69. Synthesise textures (randim weights) https://nucl.ai/blog/extreme-style-machines/ Nr of units
- 70. • Image Analogies, 2001, A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Sales • A Neural Algorithm of Artistic Style, 2015. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge • Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis, 2016, Chuan Li, Michael Wand • Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks, 2016, Alex J. Champandard • Texture Networks: Feed-forward Synthesis of Textures and Stylized Images, 2016, Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, Victor Lempitsky • Perceptual Losses for Real-Time Style Transfer and Super-Resolution, 2016, Justin Johnson, Alexandre Alahi, Li Fei-Fei • Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks, 2016, Chuan Li, Michael Wand • @DeepForger 70 “Style Transfer” papers
- 71. “A stop sign is flying in blue skies.” “A herd of elephants flying in the blue skies.” Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015. Generating Images from Captions with Attention (arxiv) (examples) Caption -> Image generation
- 73. More… more at: http://gitxiv.com/category/computer-vision http://www.creativeai.net/ (no category for images just yet)
- 75. Early LSTM music composition (2002) Douglas Eck and Jurgen Schmidhuber (2002) Learning The Long-Term Structure of the Blues?
- 78. 78 Audio Generation: Midi https://soundcloud.com/graphific/pyotr-lstm-tchaikovsky A Recurrent Latent Variable Model for Sequential Data, 2016, J. Chung, K. Kastner, L. Dinh, K. Goel, A. Courville, Y. Bengio + “modded VRNN:
- 79. 79 Audio Generation: Midi https://soundcloud.com/graphific/neural-remix-net A Recurrent Latent Variable Model for Sequential Data, 2016, J. Chung, K. Kastner, L. Dinh, K. Goel, A. Courville, Y. Bengio + “modded VRNN:
- 80. 80 Audio Generation: Raw Gated Recurrent Unit (GRU)stanford cs224d project Aran Nayebi, Matt Vitelli (2015) GRUV: Algorithmic Music Generation using Recurrent Neural Networks
- 81. • LSTM improvements • Recurrent Batch Normalization http:// gitxiv.com/posts/MwSDm6A4wPG7TcuPZ/ recurrent-batch-normalization • also: hidden-to-hidden transition (earlier only input-to-hidden transformation of RNNs) • faster convergence and improved generalization. LSTM improvements
- 82. • LSTM improvements • weight normalisation http://gitxiv.com/posts/ p9B6i9Kzbkc5rP3cp/weight-normalization-a- simple-reparameterization-to LSTM improvements
- 83. • LSTM improvements • Associative Long Short-Term Memory http:// gitxiv.com/posts/jpfdiFPsu5c6LLsF4/associative- long-short-term-memory LSTM improvements
- 84. • LSTM improvements • Bayesian RNN dropout http://gitxiv.com/posts/ CsCDjy7WpfcBvZ88R/bayesianrnn LSTM improvements
- 85. Chor-RNN Continuous Generation Luka Crnkovic-Friis & Louise Crnkovic-Friis (2016) Generative Choreography using Deep Learning
- 86. • Mixture Density LSTM Generative 1 2 3 4 5 6
- 87. • Mixture Density LSTM Generative 1 2 3 4 5 6
- 88. • Mixture Density LSTM Generative 1 2 3 4 5 6
- 89. • Mixture Density LSTM Generative 1 2 3 4 5 6
- 90. • Mixture Density LSTM Generative 1 2 3 4 5 6
- 91. Wanna be Doing Deep Learning?
- 92. python has a wide range of deep learning-related libraries available Deep Learning with Python Low level High level deeplearning.net/software/theano caffe.berkeleyvision.org tensorflow.org/ lasagne.readthedocs.org/en/latest and of course: keras.io
- 95. Questions? love letters? existential dilemma’s? academic questions? gifts? find me at: www.csc.kth.se/~roelof/ roelof@kth.se @graphific Oh, and soon we’re looking for Creative AI enthusiasts ! - job - internship - thesis work in AI (Deep Learning) & Creativity
- 97. Creative AI > a “brush” > rapid experimentation human-machine collaboration
- 98. Creative AI > a “brush” > rapid experimentation (YouTube, Paper)
- 99. Creative AI > a “brush” > rapid experimentation (YouTube, Paper)
- 100. Creative AI > a “brush” > rapid experimentation (Vimeo, Paper)
- 101. 101 Generative Adverserial Nets Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus, 2015. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (GitXiv)
- 102. 102 Generative Adverserial Nets Alec Radford, Luke Metz, Soumith Chintala , 2015. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
- 103. 103 Generative Adverserial Nets Alec Radford, Luke Metz, Soumith Chintala , 2015. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
- 104. 104 Generative Adverserial Nets Alec Radford, Luke Metz, Soumith Chintala , 2015. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv) ”turn” vector created from four averaged samples of faces looking left vs looking right.
- 105. walking through the manifold Generative Adverserial Nets
- 106. top: unmodified samples bottom: same samples dropping out ”window” filters Generative Adverserial Nets