SlideShare a Scribd company logo

Multi modal retrieval and generation with deep distributed models

Roelof Pieters, profile picture
Roelof Pieters

The document discusses multi-modal retrieval and generation using deep learning models, emphasizing the challenge of handling the deluge of digital media across text, video, and audio. It explores the application of neural networks to create embeddings that can represent these different modalities and facilitate efficient searching and collaboration between humans and machines. Additionally, it highlights various advanced techniques and models used in natural language processing and generative AI for understanding and generating content across different media types.

Education
Related topics:
Information Retrieval • Neural Networks • Deep Learning
1 of 110
Downloaded 57 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
@graphific Roelof Pieters Mul--modal Retrieval and Genera-on with Deep Distributed Models 26 April 2016 
 KTH www.csc.kth.se/~roelof/ roelof@kth.se
Creative AI > a “brush” > rapid experimentation human-machine collaboration
Multi-modal retrieval 3
Modalities 4
[Karlgren 2014, NLP Sthlm Meetup]5 Digital Media Deluge: text
[ http://lexicon.gavagai.se/lookup/en/lol ]6 Digital Media Deluge: text lol ? …
[Youtube Blog, 2010]7 Digital Media Deluge: video
[Reelseo, 2015]8 Digital Media Deluge: video
[Reelseo, 2015]9 Digital Media Deluge: audio
[Reelseo, 2015]10 Digital Media Deluge: audio
Challenges 11 • Volume • Velocity • Variety
Can we make it searchable? 12 Language
Language: Compositionality Principle of compositionality: the “meaning (vector) of a complex expression (sentence) is determined by: — Gottlob Frege 
 (1848 - 1925) - the meanings of its constituent expressions (words) and - the rules (grammar) used to combine them” 13
• NLP treats words mainly (rule-based/statistical approaches at least) as atomic symbols:
 • or in vector space:
 • also known as “one hot” representation. • Its problem ? Word Representation Love Candy Store [0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 …] Candy [0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 …] AND Store [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 …] = 0 ! 14
Word Representation 15
Distributional semantics Distributional meaning as co-occurrence vector: 16
Deep Distributional representations • Taking it further: • Continuous word embeddings • Combine vector space semantics with the prediction of probabilistic models • Words are represented as a dense vector: Candy = 17
• Can theoretically (given enough units) approximate “any” function • and fit to “any” kind of data • Efficient for NLP: hidden layers can be used as word lookup tables • Dense distributed word vectors + efficient NN training algorithms: • Can scale to billions of words ! Neural Networks for NLP 18
Multi modal retrieval and generation with deep distributed models
Word Embeddings: SocherVector Space Model adapted rom Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: 20
Word Embeddings: SocherVector Space Model adapted rom Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: the country of my birth the place where I was born 21
Word Embeddings: SocherVector Space Model Figure (edited) from Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: the country of my birth the place where I was born ? … 22
Word Embeddings: Turian (2010) Turian, J., Ratinov, L., Bengio, Y. (2010). Word representations: A simple and general method for semi-supervised learning code & info: http://metaoptimize.com/projects/wordreprs/23
Word Embeddings: Turian (2010) Turian, J., Ratinov, L., Bengio, Y. (2010). Word representations: A simple and general method for semi-supervised learning code & info: http://metaoptimize.com/projects/wordreprs/ 24
Word Embeddings: Collobert & Weston (2011) Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P. (2011) . Natural Language Processing (almost) from Scratch 25
Multi-embeddings: Stanford (2012) Eric H. Huang, Richard Socher, Christopher D. Manning, Andrew Y. Ng (2012)
 Improving Word Representations via Global Context and Multiple Word Prototypes 26
Linguistic Regularities: Mikolov (2013) code & info: https://code.google.com/p/word2vec/ Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations 27
Word Embeddings for MT: Mikolov (2013) Mikolov, T., Le, V. L., Sutskever, I. (2013) . 
 Exploiting Similarities among Languages for Machine Translation 28
Word Embeddings for MT: Kiros (2014) 29
Recursive Embeddings for Sentiment: Socher (2013) Socher, R., Perelygin, A., Wu, J., Chuang, J.,Manning, C., Ng, A., Potts, C. (2013) 
 Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. code & demo: http://nlp.stanford.edu/sentiment/index.html 30
Paragraph Vectors: Dai et al. (2014) 31
Paragraph Vectors: Dai et al. (2014) 32
Can we make it searchable? 33 Other modalities
• Image -> vector -> embedding ? ? • Video -> vector -> embedding ? ? • Audio -> vector -> embedding ? ? 34 Other modalities: Embeddings?
•A host of statistical machine learning techniques •Enables the automatic learning of feature hierarchies •Generally based on artificial neural networks Deep Learning?
• Manually designed features are often over-specified, incomplete and take a long time to design and validate • Learned Features are easy to adapt, fast to learn
 • Deep learning provides a very flexible, (almost?) universal, learnable framework for representing world, visual and linguistic information. • Deep learning can learn unsupervised (from raw text/ audio/images/whatever content) and supervised (with specific labels like positive/negative) (as summarised by Richard Socher 2014) Deep Learning?
37 2006+ : The Deep Learning Conspirators
Multi modal retrieval and generation with deep distributed models
• Image -> vector -> embedding • Video -> vector -> embedding ? ? • Audio -> vector -> embedding ? ? 39 Image Embeddings
40 Convolutional Neural Nets for Images classification demo
41 Convolutional Neural Nets for Images http://ml4a.github.io/dev/demos/demo_convolution.html
42 Convolutional Neural Nets for Images Zeiler and Fergus 2013, 
 Visualizing and Understanding Convolutional Networks
43 Convolutional Neural Nets for Images
44 Convolutional Neural Nets for Images
45 Deep Nets
46 Deep Nets
47 Convolutional Neural Nets: Embeddings? [-0.34, 0.28, …] 4096-dimensional fc7 AlexNet CNN
48 (Karpathy)
49 Convolutional Neural Nets: Embeddings? http://ml4a.github.io/dev/demos/tsne-viewer.html
• Image -> vector -> embedding ?? • Video -> vector -> embedding • Audio -> vector -> embedding ? ? 50 Video Embeddings
51 Convolutional Neural Nets for Video 3D Convolutional Neural Networks for Human Action Recognition, Ji et al., 2010
52 Convolutional Neural Nets for Video Sequential Deep Learning for Human Action Recognition, Baccouche et al., 2011
53 Convolutional Neural Nets for Video Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014
54 Convolutional Neural Nets for Video Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014
55 Convolutional Neural Nets for Video [Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014 [Le et al. '11] vs classic 2d convnet:
56 Convolutional Neural Nets for Video [Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014
57 Convolutional Neural Nets for Video Sequential Deep Learning for Human Action Recognition, Baccouche et al., 2011
58 Convolutional Neural Nets for Video Long-term Recurrent Convolutional Networks for Visual Recognition and Description, Donahue et al., 2015
59 Convolutional Neural Nets for Video Beyond Short Snippets: Deep Networks for Video Classification, Ng et al., 2015]
60 Convolutional Neural Nets for Video Delving Deeper into Convolutional Networks for Learning Video Representations, Ballas et al., 2016
• Image -> vector -> embedding ?? • Video -> vector -> embedding ?? • Audio -> vector -> embedding 61 Audio Embeddings
62 Zero-shot Learning [Sander Dieleman, 2014]
63 Audio Embeddings [Sander Dieleman, 2014]
demo
• Can we take this further? 65 Multi Modal Embeddings?
• unsupervised pre-training (on many images) • in parallel train a neural network (Language) Model • train linear mapping between (image) representations and (word) embeddings, representing the different “classes” 66 Zero-shot Learning
DeViSE model (Frome et al. 2013) • skip-gram text model on wikipedia corpus of 5.7 million documents (5.4 billion words) - approach from (Mikolov et al. ICLR 2013) 67 Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Mikolov, T., Ranzato, M.A. (2013) 
 Devise: A deep visual-semantic embedding model
Encoder: A deep convolutional network (CNN) and long short- term memory recurrent network (LSTM) for learning a joint image-sentence embedding. Decoder: A new neural language model that combines structure and content vectors for generating words one at a time in sequence. Encoder-Decoder pipeline (Kiros et al 2014) 68 Kiros, R., Salakhutdinov, R., Zemerl, R. S. (2014) 
 Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., Zemerl, R. S. (2014) 
 Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models • matches state-of-the-art performance on Flickr8K and Flickr30K without using object detections • new best results when using the 19-layer Oxford convolutional network. • linear encoders: learned embedding space captures multimodal regularities (e.g. *image of a blue car* - "blue" + "red" is near images of red cars) Encoder-Decoder pipeline (Kiros et al 2014) 69
Image-Text Embeddings 70 Socher et al (2013) Zero Shot Learning Through Cross-Modal Transfer (info)
Image-Captioning • Andrej Karpathy Li Fei-Fei , 2015. 
 Deep Visual-Semantic Alignments for Generating Image Descriptions (pdf) (info) (code) • Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan , 2015. Show and Tell: A Neural Image Caption Generator (arxiv) • Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (arxiv) (info) (code)
“A person riding a motorcycle on a dirt road.”??? Image-Captioning
“Two hockey players are fighting over the puck.”??? Image-Captioning
• Let’s turn it around! • Generative Models • (we wont cover, but common architectures): • Auto encoders (AE), variational variants: VAE • Generative Adversarial Nets (GAN) • Variational Recurrent Neural Net (VRNN) 74 Generative Models
Wanna Play ? Text generation (RNN) 75 Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
Wanna Play ? Text generation 76 Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
Multi modal retrieval and generation with deep distributed models
Multi modal retrieval and generation with deep distributed models
Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
“A stop sign is flying in blue skies.” “A herd of elephants flying in the blue skies.” Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015. Generating Images from Captions with Attention (arxiv) (examples) Caption -> Image generation
Turn Convnet Around: “Deep Dream” Image -> NN -> What do you (think) you see 
 -> Whats the (text) label Image -> NN -> What do you (think) you see -> 
 feed back activations -> 
 optimize image to “fit” to the ConvNets “hallucination” (iteratively)
see also: www.csc.kth.se/~roelof/deepdream/ 
 Turn Convnet Around: “Deep Dream”
Turn Convnet Around: “Deep Dream” see also: www.csc.kth.se/~roelof/deepdream/
see also: www.csc.kth.se/~roelof/deepdream/ codeyoutubeRoelof Pieters 2015 Turn Convnet Around: “Deep Dream”
https://www.flickr.com/photos/graphific/albums/72157657250972188 Single Units
Inter-modal: “Style Net” Leon A. Gatys, Alexander S. Ecker, Matthias Bethge , 2015. 
 A Neural Algorithm of Artistic Style (GitXiv)
Multi modal retrieval and generation with deep distributed models
88
89
90 + + =
https://github.com/alexjc/neural-doodle Neural Doodle
Gene Kogan, 2015. Why is a Raven Like a Writing Desk? (vimeo)
• Image Analogies, 2001, A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Sales • A Neural Algorithm of Artistic Style, 2015. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge • Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis, 2016, Chuan Li, Michael Wand • Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks, 2016, Alex J. Champandard • Texture Networks: Feed-forward Synthesis of Textures and Stylized Images, 2016, Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, Victor Lempitsky • Perceptual Losses for Real-Time Style Transfer and Super-Resolution, 2016, Justin Johnson, Alexandre Alahi, Li Fei-Fei • Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks, 2016, Chuan Li, Michael Wand • @DeepForger 93 “Style Transfer” papers
• https://soundcloud.com/graphific/neural-music-walk • https://soundcloud.com/graphific/pyotr-lstm- tchaikovsky • https://soundcloud.com/graphific/neural-remix-net 94 Audio Generation A Recurrent Latent Variable Model for Sequential Data, 2016, 
 J. Chung, K. Kastner, L. Dinh, K. Goel, A. Courville, Y. Bengio
Wanna be Doing Deep Learning?
python has a wide range of deep learning-related libraries available Deep Learning with Python Low level High level deeplearning.net/software/theano caffe.berkeleyvision.org tensorflow.org/ lasagne.readthedocs.org/en/latest and of course: keras.io
Questions? love letters? existential dilemma’s? academic questions? gifts? 
 find me at:
 www.csc.kth.se/~roelof/ roelof@kth.se Code & Papers? Collaborative Open Computer Science .com @graphific
Multi modal retrieval and generation with deep distributed models
Questions? love letters? existential dilemma’s? academic questions? gifts? 
 find me at:
 www.csc.kth.se/~roelof/ roelof@kth.se Generative “creative” AI “stuff”? .net @graphific
Multi modal retrieval and generation with deep distributed models
Creative AI > a “brush” > rapid experimentation human-machine collaboration
Creative AI > a “brush” > rapid experimentation (YouTube, Paper)
Creative AI > a “brush” > rapid experimentation (YouTube, Paper)
Creative AI > a “brush” > rapid experimentation (Vimeo, Paper)
105 Generative Adverserial Nets Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus, 2015. 
 Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (GitXiv)
106 Generative Adverserial Nets Alec Radford, Luke Metz, Soumith Chintala , 2015. 
 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
107 Generative Adverserial Nets Alec Radford, Luke Metz, Soumith Chintala , 2015. 
 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
108 Generative Adverserial Nets Alec Radford, Luke Metz, Soumith Chintala , 2015. 
 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv) ”turn” vector created from four averaged samples of faces looking left vs looking right.
walking through the manifold Generative Adverserial Nets
top: unmodified samples bottom: same samples dropping out ”window” filters Generative Adverserial Nets

More Related Content

PDF
Deep learning for natural language embeddings
Roelof Pieters
 
PDF
Visual-Semantic Embeddings: some thoughts on Language
Roelof Pieters
 
PDF
Deep Learning for Information Retrieval
Roelof Pieters
 
PDF
Deep Learning for Natural Language Processing: Word Embeddings
Roelof Pieters
 
PDF
Deep Learning for NLP: An Introduction to Neural Word Embeddings
Roelof Pieters
 
PDF
Deep learning for nlp
Viet-Trung TRAN
 
PDF
Zero shot learning through cross-modal transfer
Roelof Pieters
 
PDF
Deep Learning and Text Mining
Will Stanton
 
Deep learning for natural language embeddings
Roelof Pieters
 
Visual-Semantic Embeddings: some thoughts on Language
Roelof Pieters
 
Deep Learning for Information Retrieval
Roelof Pieters
 
Deep Learning for Natural Language Processing: Word Embeddings
Roelof Pieters
 
Deep Learning for NLP: An Introduction to Neural Word Embeddings
Roelof Pieters
 
Deep learning for nlp
Viet-Trung TRAN
 
Zero shot learning through cross-modal transfer
Roelof Pieters
 
Deep Learning and Text Mining
Will Stanton
 

What's hot (20)

PDF
Engineering Intelligent NLP Applications Using Deep Learning – Part 2
Saurabh Kaushik
 
PDF
Engineering Intelligent NLP Applications Using Deep Learning – Part 1
Saurabh Kaushik
 
PDF
Information Retrieval with Deep Learning
Adam Gibson
 
PDF
Learning to understand phrases by embedding the dictionary
Roelof Pieters
 
PDF
Deep Learning, an interactive introduction for NLP-ers
Roelof Pieters
 
PDF
Deep Learning for NLP Applications
Samiur Rahman
 
PDF
Deep Learning & NLP: Graphs to the Rescue!
Roelof Pieters
 
PPTX
Neural Text Embeddings for Information Retrieval (WSDM 2017)
Bhaskar Mitra
 
PDF
Anthiil Inside workshop on NLP
Satyam Saxena
 
PDF
Representation Learning of Vectors of Words and Phrases
Felipe Moraes
 
PDF
Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)
Márton Miháltz
 
PPTX
NLP Bootcamp
Anuj Gupta
 
PDF
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...
Daniele Di Mitri
 
PDF
(Deep) Neural Networks在 NLP 和 Text Mining 总结
君 廖
 
PPTX
Talk from NVidia Developer Connect
Anuj Gupta
 
PPTX
Tomáš Mikolov - Distributed Representations for NLP
Machine Learning Prague
 
PPTX
Deep Learning for Natural Language Processing
Jonathan Mugan
 
PDF
BIng NLP Expert - Dl summer-school-2017.-jianfeng-gao.v2
Karthik Murugesan
 
PPTX
Using Text Embeddings for Information Retrieval
Bhaskar Mitra
 
PDF
NLP from scratch
Bryan Gummibearehausen
 
Engineering Intelligent NLP Applications Using Deep Learning – Part 2
Saurabh Kaushik
 
Engineering Intelligent NLP Applications Using Deep Learning – Part 1
Saurabh Kaushik
 
Information Retrieval with Deep Learning
Adam Gibson
 
Learning to understand phrases by embedding the dictionary
Roelof Pieters
 
Deep Learning, an interactive introduction for NLP-ers
Roelof Pieters
 
Deep Learning for NLP Applications
Samiur Rahman
 
Deep Learning & NLP: Graphs to the Rescue!
Roelof Pieters
 
Neural Text Embeddings for Information Retrieval (WSDM 2017)
Bhaskar Mitra
 
Anthiil Inside workshop on NLP
Satyam Saxena
 
Representation Learning of Vectors of Words and Phrases
Felipe Moraes
 
Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)
Márton Miháltz
 
NLP Bootcamp
Anuj Gupta
 
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...
Daniele Di Mitri
 
(Deep) Neural Networks在 NLP 和 Text Mining 总结
君 廖
 
Talk from NVidia Developer Connect
Anuj Gupta
 
Tomáš Mikolov - Distributed Representations for NLP
Machine Learning Prague
 
Deep Learning for Natural Language Processing
Jonathan Mugan
 
BIng NLP Expert - Dl summer-school-2017.-jianfeng-gao.v2
Karthik Murugesan
 
Using Text Embeddings for Information Retrieval
Bhaskar Mitra
 
NLP from scratch
Bryan Gummibearehausen
 
Ad

Viewers also liked (18)

PDF
Multi-modal embeddings: from discriminative to generative models and creative ai
Roelof Pieters
 
PDF
Creative AI & multimodality: looking ahead
Roelof Pieters
 
PDF
Graph, Data-science, and Deep Learning
Roelof Pieters
 
PDF
Deep Learning as a Cat/Dog Detector
Roelof Pieters
 
PDF
Deep Neural Networks 
that talk (Back)… with style
Roelof Pieters
 
PDF
Explore Data: Data Science + Visualization
Roelof Pieters
 
PDF
Building a Deep Learning (Dream) Machine
Roelof Pieters
 
PDF
Deep Learning: a birds eye view
Roelof Pieters
 
PDF
Python for Image Understanding: Deep Learning with Convolutional Neural Nets
Roelof Pieters
 
PDF
Deep Learning for industrial Prognostics & Health Management (PHM)
Michael Giering
 
PPTX
Multimodal Learning Analytics
Xavier Ochoa
 
PDF
Multimodal Residual Learning for Visual Question-Answering
NAVER D2
 
PDF
introduce to Multimodal Deep Learning for Robust RGB-D Object Recognition
WEBFARMER. ltd.
 
PDF
Deep Neural Networks for Multimodal Learning
Marc Bolaños Solà
 
PDF
Tutorial on Deep Learning
inside-BigData.com
 
PDF
Deep Learning Cases: Text and Image Processing
Grigory Sapunov
 
PPTX
Universidad nacional de chimborazo 7
JAVIER CHACATER
 
PPT
Аліна Марусик "Конфлікти в команді і методи їх вирішення"
Lviv Startup Club
 
Multi-modal embeddings: from discriminative to generative models and creative ai
Roelof Pieters
 
Creative AI & multimodality: looking ahead
Roelof Pieters
 
Graph, Data-science, and Deep Learning
Roelof Pieters
 
Deep Learning as a Cat/Dog Detector
Roelof Pieters
 
Deep Neural Networks 
that talk (Back)… with style
Roelof Pieters
 
Explore Data: Data Science + Visualization
Roelof Pieters
 
Building a Deep Learning (Dream) Machine
Roelof Pieters
 
Deep Learning: a birds eye view
Roelof Pieters
 
Python for Image Understanding: Deep Learning with Convolutional Neural Nets
Roelof Pieters
 
Deep Learning for industrial Prognostics & Health Management (PHM)
Michael Giering
 
Multimodal Learning Analytics
Xavier Ochoa
 
Multimodal Residual Learning for Visual Question-Answering
NAVER D2
 
introduce to Multimodal Deep Learning for Robust RGB-D Object Recognition
WEBFARMER. ltd.
 
Deep Neural Networks for Multimodal Learning
Marc Bolaños Solà
 
Tutorial on Deep Learning
inside-BigData.com
 
Deep Learning Cases: Text and Image Processing
Grigory Sapunov
 
Universidad nacional de chimborazo 7
JAVIER CHACATER
 
Аліна Марусик "Конфлікти в команді і методи їх вирішення"
Lviv Startup Club
 
Ad

Similar to Multi modal retrieval and generation with deep distributed models (20)

PDF
Interactive Video Search: Where is the User in the Age of Deep Learning?
klschoef
 
PPT
Compact and Distinctive Visual Vocabularies for Efficient Multimedia Data Ind...
Symeon Papadopoulos
 
PPTX
Deep Learning @ ZHAW Datalab (with Mark Cieliebak & Yves Pauchard)
Thilo Stadelmann
 
PDF
Language Modelling in Natural Language Processing-Part II.pdf
Deptii Chaudhari
 
PDF
Deep Convnets for Video Processing (Master in Computer Vision Barcelona, 2016)
Universitat Politècnica de Catalunya
 
PDF
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
Matthew Lease
 
PDF
Deep Learning for Computer Vision (3/4): Video Analytics @ laSalle 2016
Universitat Politècnica de Catalunya
 
PDF
UDL 2.0 7-18-10
Christopher Bugaj
 
PPTX
Bridging the gap between AI and UI - DSI Vienna - full version
Liad Magen
 
PDF
SCAI invited talk @EMNLP2020
Verena Rieser
 
PDF
Superheroes SXSW 2013
Yvonne So
 
PDF
Multi-modal Neural Machine Translation - Iacer Calixto
Sebastian Ruder
 
PDF
Deep Language and Vision - Xavier Giro-i-Nieto - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
PDF
How can text-mining leverage developments in Deep Learning? Presentation at ...
jcscholtes
 
PDF
Understanding user interactivity for immersive communications and its impact ...
lauratoni4
 
PDF
Understanding user interactivity for immersive communications and its impact ...
Alpen-Adria-Universität
 
PPT
Malden Slideshow 08 26 09
Cynthia Curry
 
PDF
Building an Academic Virtual Reality System
Bill E
 
PDF
Semantic Interoperability - grafi della conoscenza
Giorgia Lodi
 
PPTX
Gesture detection
JanitChawla1
 
Interactive Video Search: Where is the User in the Age of Deep Learning?
klschoef
 
Compact and Distinctive Visual Vocabularies for Efficient Multimedia Data Ind...
Symeon Papadopoulos
 
Deep Learning @ ZHAW Datalab (with Mark Cieliebak & Yves Pauchard)
Thilo Stadelmann
 
Language Modelling in Natural Language Processing-Part II.pdf
Deptii Chaudhari
 
Deep Convnets for Video Processing (Master in Computer Vision Barcelona, 2016)
Universitat Politècnica de Catalunya
 
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
Matthew Lease
 
Deep Learning for Computer Vision (3/4): Video Analytics @ laSalle 2016
Universitat Politècnica de Catalunya
 
UDL 2.0 7-18-10
Christopher Bugaj
 
Bridging the gap between AI and UI - DSI Vienna - full version
Liad Magen
 
SCAI invited talk @EMNLP2020
Verena Rieser
 
Superheroes SXSW 2013
Yvonne So
 
Multi-modal Neural Machine Translation - Iacer Calixto
Sebastian Ruder
 
Deep Language and Vision - Xavier Giro-i-Nieto - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
How can text-mining leverage developments in Deep Learning? Presentation at ...
jcscholtes
 
Understanding user interactivity for immersive communications and its impact ...
lauratoni4
 
Understanding user interactivity for immersive communications and its impact ...
Alpen-Adria-Universität
 
Malden Slideshow 08 26 09
Cynthia Curry
 
Building an Academic Virtual Reality System
Bill E
 
Semantic Interoperability - grafi della conoscenza
Giorgia Lodi
 
Gesture detection
JanitChawla1
 

Recently uploaded (20)

PDF
102 student loan defaulters named and shamed – Is someone you know on the list?
Kweku Zurek
 
PDF
RMMM.pdf make it easy to upload and study
sajedul2
 
PPTX
Final Presentation General Medicine 03-08-2024.pptx
ZaheerAhmad228692
 
PPTX
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
chetansingh379583
 
PDF
grade 11-chemistry_fetena_net_5883.pdf teacher guide for all student
banteamlakmebratu738
 
PPTX
PPH.pptx obstetrics and gynecology in nursing
SrideviDevaraj5
 
PPTX
GDM (1) (1).pptx small presentation for students
shrawanbpkihs
 
PPTX
IMMUNITY IMMUNITY refers to protection against infection, and the immune syst...
shilpa939953
 
PDF
O7-L3 Supply Chain Operations - ICLT Program
labyankof
 
PDF
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf
sreejithcareers
 
PDF
Sports Quiz easy sports quiz sports quiz
nikunj2757
 
PDF
VCE English Exam - Section C Student Revision Booklet
jpinnuck
 
PDF
Complications of Minimal Access Surgery at WLH
mohitsuren827
 
PDF
The Lost Whites of Pakistan by Jahanzaib Mughal.pdf
jahanzaibmughal6
 
PDF
TR - Agricultural Crops Production NC III.pdf
avgilmegino3
 
PDF
Black Hat USA 2025 - Micro ICS Summit - ICS/OT Threat Landscape
Chris Sistrunk
 
PDF
01-Introduction-to-Information-Management.pdf
PrinceIbarra
 
PDF
O5-L3 Freight Transport Ops (International) V1.pdf
labyankof
 
PDF
FourierSeries-QuestionsWithAnswers(Part-A).pdf
Raji Murugan
 
PDF
STATICS OF THE RIGID BODIES Hibbelers.pdf
pascualasturiaskaye4
 
102 student loan defaulters named and shamed – Is someone you know on the list?
Kweku Zurek
 
RMMM.pdf make it easy to upload and study
sajedul2
 
Final Presentation General Medicine 03-08-2024.pptx
ZaheerAhmad228692
 
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
chetansingh379583
 
grade 11-chemistry_fetena_net_5883.pdf teacher guide for all student
banteamlakmebratu738
 
PPH.pptx obstetrics and gynecology in nursing
SrideviDevaraj5
 
GDM (1) (1).pptx small presentation for students
shrawanbpkihs
 
IMMUNITY IMMUNITY refers to protection against infection, and the immune syst...
shilpa939953
 
O7-L3 Supply Chain Operations - ICLT Program
labyankof
 
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf
sreejithcareers
 
Sports Quiz easy sports quiz sports quiz
nikunj2757
 
VCE English Exam - Section C Student Revision Booklet
jpinnuck
 
Complications of Minimal Access Surgery at WLH
mohitsuren827
 
The Lost Whites of Pakistan by Jahanzaib Mughal.pdf
jahanzaibmughal6
 
TR - Agricultural Crops Production NC III.pdf
avgilmegino3
 
Black Hat USA 2025 - Micro ICS Summit - ICS/OT Threat Landscape
Chris Sistrunk
 
01-Introduction-to-Information-Management.pdf
PrinceIbarra
 
O5-L3 Freight Transport Ops (International) V1.pdf
labyankof
 
FourierSeries-QuestionsWithAnswers(Part-A).pdf
Raji Murugan
 
STATICS OF THE RIGID BODIES Hibbelers.pdf
pascualasturiaskaye4
 

Multi modal retrieval and generation with deep distributed models

  • 2. Creative AI > a “brush” > rapid experimentation human-machine collaboration
  • 5. [Karlgren 2014, NLP Sthlm Meetup]5 Digital Media Deluge: text
  • 7. [Youtube Blog, 2010]7 Digital Media Deluge: video
  • 12. Can we make it searchable? 12 Language
  • 13. Language: Compositionality Principle of compositionality: the “meaning (vector) of a complex expression (sentence) is determined by: — Gottlob Frege 
 (1848 - 1925) - the meanings of its constituent expressions (words) and - the rules (grammar) used to combine them” 13
  • 14. • NLP treats words mainly (rule-based/statistical approaches at least) as atomic symbols:
 • or in vector space:
 • also known as “one hot” representation. • Its problem ? Word Representation Love Candy Store [0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 …] Candy [0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 …] AND Store [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 …] = 0 ! 14
  • 16. Distributional semantics Distributional meaning as co-occurrence vector: 16
  • 17. Deep Distributional representations • Taking it further: • Continuous word embeddings • Combine vector space semantics with the prediction of probabilistic models • Words are represented as a dense vector: Candy = 17
  • 18. • Can theoretically (given enough units) approximate “any” function • and fit to “any” kind of data • Efficient for NLP: hidden layers can be used as word lookup tables • Dense distributed word vectors + efficient NN training algorithms: • Can scale to billions of words ! Neural Networks for NLP 18
  • 20. Word Embeddings: SocherVector Space Model adapted rom Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: 20
  • 21. Word Embeddings: SocherVector Space Model adapted rom Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: the country of my birth the place where I was born 21
  • 22. Word Embeddings: SocherVector Space Model Figure (edited) from Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: the country of my birth the place where I was born ? … 22
  • 23. Word Embeddings: Turian (2010) Turian, J., Ratinov, L., Bengio, Y. (2010). Word representations: A simple and general method for semi-supervised learning code & info: http://metaoptimize.com/projects/wordreprs/23
  • 24. Word Embeddings: Turian (2010) Turian, J., Ratinov, L., Bengio, Y. (2010). Word representations: A simple and general method for semi-supervised learning code & info: http://metaoptimize.com/projects/wordreprs/ 24
  • 25. Word Embeddings: Collobert & Weston (2011) Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P. (2011) . Natural Language Processing (almost) from Scratch 25
  • 26. Multi-embeddings: Stanford (2012) Eric H. Huang, Richard Socher, Christopher D. Manning, Andrew Y. Ng (2012)
 Improving Word Representations via Global Context and Multiple Word Prototypes 26
  • 27. Linguistic Regularities: Mikolov (2013) code & info: https://code.google.com/p/word2vec/ Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations 27
  • 28. Word Embeddings for MT: Mikolov (2013) Mikolov, T., Le, V. L., Sutskever, I. (2013) . 
 Exploiting Similarities among Languages for Machine Translation 28
  • 29. Word Embeddings for MT: Kiros (2014) 29
  • 30. Recursive Embeddings for Sentiment: Socher (2013) Socher, R., Perelygin, A., Wu, J., Chuang, J.,Manning, C., Ng, A., Potts, C. (2013) 
 Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. code & demo: http://nlp.stanford.edu/sentiment/index.html 30
  • 31. Paragraph Vectors: Dai et al. (2014) 31
  • 32. Paragraph Vectors: Dai et al. (2014) 32
  • 33. Can we make it searchable? 33 Other modalities
  • 34. • Image -> vector -> embedding ? ? • Video -> vector -> embedding ? ? • Audio -> vector -> embedding ? ? 34 Other modalities: Embeddings?
  • 35. •A host of statistical machine learning techniques •Enables the automatic learning of feature hierarchies •Generally based on artificial neural networks Deep Learning?
  • 36. • Manually designed features are often over-specified, incomplete and take a long time to design and validate • Learned Features are easy to adapt, fast to learn
 • Deep learning provides a very flexible, (almost?) universal, learnable framework for representing world, visual and linguistic information. • Deep learning can learn unsupervised (from raw text/ audio/images/whatever content) and supervised (with specific labels like positive/negative) (as summarised by Richard Socher 2014) Deep Learning?
  • 37. 37 2006+ : The Deep Learning Conspirators
  • 39. • Image -> vector -> embedding • Video -> vector -> embedding ? ? • Audio -> vector -> embedding ? ? 39 Image Embeddings
  • 40. 40 Convolutional Neural Nets for Images classification demo
  • 41. 41 Convolutional Neural Nets for Images http://ml4a.github.io/dev/demos/demo_convolution.html
  • 42. 42 Convolutional Neural Nets for Images Zeiler and Fergus 2013, 
 Visualizing and Understanding Convolutional Networks
  • 47. 47 Convolutional Neural Nets: Embeddings? [-0.34, 0.28, …] 4096-dimensional fc7 AlexNet CNN
  • 49. 49 Convolutional Neural Nets: Embeddings? http://ml4a.github.io/dev/demos/tsne-viewer.html
  • 50. • Image -> vector -> embedding ?? • Video -> vector -> embedding • Audio -> vector -> embedding ? ? 50 Video Embeddings
  • 51. 51 Convolutional Neural Nets for Video 3D Convolutional Neural Networks for Human Action Recognition, Ji et al., 2010
  • 52. 52 Convolutional Neural Nets for Video Sequential Deep Learning for Human Action Recognition, Baccouche et al., 2011
  • 53. 53 Convolutional Neural Nets for Video Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014
  • 54. 54 Convolutional Neural Nets for Video Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014
  • 55. 55 Convolutional Neural Nets for Video [Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014 [Le et al. '11] vs classic 2d convnet:
  • 56. 56 Convolutional Neural Nets for Video [Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014
  • 57. 57 Convolutional Neural Nets for Video Sequential Deep Learning for Human Action Recognition, Baccouche et al., 2011
  • 58. 58 Convolutional Neural Nets for Video Long-term Recurrent Convolutional Networks for Visual Recognition and Description, Donahue et al., 2015
  • 59. 59 Convolutional Neural Nets for Video Beyond Short Snippets: Deep Networks for Video Classification, Ng et al., 2015]
  • 60. 60 Convolutional Neural Nets for Video Delving Deeper into Convolutional Networks for Learning Video Representations, Ballas et al., 2016
  • 61. • Image -> vector -> embedding ?? • Video -> vector -> embedding ?? • Audio -> vector -> embedding 61 Audio Embeddings
  • 64. demo
  • 65. • Can we take this further? 65 Multi Modal Embeddings?
  • 66. • unsupervised pre-training (on many images) • in parallel train a neural network (Language) Model • train linear mapping between (image) representations and (word) embeddings, representing the different “classes” 66 Zero-shot Learning
  • 67. DeViSE model (Frome et al. 2013) • skip-gram text model on wikipedia corpus of 5.7 million documents (5.4 billion words) - approach from (Mikolov et al. ICLR 2013) 67 Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Mikolov, T., Ranzato, M.A. (2013) 
 Devise: A deep visual-semantic embedding model
  • 68. Encoder: A deep convolutional network (CNN) and long short- term memory recurrent network (LSTM) for learning a joint image-sentence embedding. Decoder: A new neural language model that combines structure and content vectors for generating words one at a time in sequence. Encoder-Decoder pipeline (Kiros et al 2014) 68 Kiros, R., Salakhutdinov, R., Zemerl, R. S. (2014) 
 Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
  • 69. Kiros, R., Salakhutdinov, R., Zemerl, R. S. (2014) 
 Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models • matches state-of-the-art performance on Flickr8K and Flickr30K without using object detections • new best results when using the 19-layer Oxford convolutional network. • linear encoders: learned embedding space captures multimodal regularities (e.g. *image of a blue car* - "blue" + "red" is near images of red cars) Encoder-Decoder pipeline (Kiros et al 2014) 69
  • 70. Image-Text Embeddings 70 Socher et al (2013) Zero Shot Learning Through Cross-Modal Transfer (info)
  • 71. Image-Captioning • Andrej Karpathy Li Fei-Fei , 2015. 
 Deep Visual-Semantic Alignments for Generating Image Descriptions (pdf) (info) (code) • Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan , 2015. Show and Tell: A Neural Image Caption Generator (arxiv) • Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (arxiv) (info) (code)
  • 72. “A person riding a motorcycle on a dirt road.”??? Image-Captioning
  • 73. “Two hockey players are fighting over the puck.”??? Image-Captioning
  • 74. • Let’s turn it around! • Generative Models • (we wont cover, but common architectures): • Auto encoders (AE), variational variants: VAE • Generative Adversarial Nets (GAN) • Variational Recurrent Neural Net (VRNN) 74 Generative Models
  • 75. Wanna Play ? Text generation (RNN) 75 Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
  • 76. Wanna Play ? Text generation 76 Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
  • 79. Karpathy (2015), The Unreasonable Effectiveness of Recurrent Neural Networks (blog)
  • 80. “A stop sign is flying in blue skies.” “A herd of elephants flying in the blue skies.” Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015. Generating Images from Captions with Attention (arxiv) (examples) Caption -> Image generation
  • 81. Turn Convnet Around: “Deep Dream” Image -> NN -> What do you (think) you see 
 -> Whats the (text) label Image -> NN -> What do you (think) you see -> 
 feed back activations -> 
 optimize image to “fit” to the ConvNets “hallucination” (iteratively)
  • 82. see also: www.csc.kth.se/~roelof/deepdream/ 
 Turn Convnet Around: “Deep Dream”
  • 83. Turn Convnet Around: “Deep Dream” see also: www.csc.kth.se/~roelof/deepdream/
  • 84. see also: www.csc.kth.se/~roelof/deepdream/ codeyoutubeRoelof Pieters 2015 Turn Convnet Around: “Deep Dream”
  • 86. Inter-modal: “Style Net” Leon A. Gatys, Alexander S. Ecker, Matthias Bethge , 2015. 
 A Neural Algorithm of Artistic Style (GitXiv)
  • 88. 88
  • 89. 89
  • 92. Gene Kogan, 2015. Why is a Raven Like a Writing Desk? (vimeo)
  • 93. • Image Analogies, 2001, A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Sales • A Neural Algorithm of Artistic Style, 2015. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge • Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis, 2016, Chuan Li, Michael Wand • Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks, 2016, Alex J. Champandard • Texture Networks: Feed-forward Synthesis of Textures and Stylized Images, 2016, Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, Victor Lempitsky • Perceptual Losses for Real-Time Style Transfer and Super-Resolution, 2016, Justin Johnson, Alexandre Alahi, Li Fei-Fei • Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks, 2016, Chuan Li, Michael Wand • @DeepForger 93 “Style Transfer” papers
  • 94. • https://soundcloud.com/graphific/neural-music-walk • https://soundcloud.com/graphific/pyotr-lstm- tchaikovsky • https://soundcloud.com/graphific/neural-remix-net 94 Audio Generation A Recurrent Latent Variable Model for Sequential Data, 2016, 
 J. Chung, K. Kastner, L. Dinh, K. Goel, A. Courville, Y. Bengio
  • 95. Wanna be Doing Deep Learning?
  • 96. python has a wide range of deep learning-related libraries available Deep Learning with Python Low level High level deeplearning.net/software/theano caffe.berkeleyvision.org tensorflow.org/ lasagne.readthedocs.org/en/latest and of course: keras.io
  • 97. Questions? love letters? existential dilemma’s? academic questions? gifts? 
 find me at:
 www.csc.kth.se/~roelof/ roelof@kth.se Code & Papers? Collaborative Open Computer Science .com @graphific
  • 99. Questions? love letters? existential dilemma’s? academic questions? gifts? 
 find me at:
 www.csc.kth.se/~roelof/ roelof@kth.se Generative “creative” AI “stuff”? .net @graphific
  • 101. Creative AI > a “brush” > rapid experimentation human-machine collaboration
  • 102. Creative AI > a “brush” > rapid experimentation (YouTube, Paper)
  • 103. Creative AI > a “brush” > rapid experimentation (YouTube, Paper)
  • 104. Creative AI > a “brush” > rapid experimentation (Vimeo, Paper)
  • 105. 105 Generative Adverserial Nets Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus, 2015. 
 Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (GitXiv)
  • 106. 106 Generative Adverserial Nets Alec Radford, Luke Metz, Soumith Chintala , 2015. 
 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
  • 107. 107 Generative Adverserial Nets Alec Radford, Luke Metz, Soumith Chintala , 2015. 
 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
  • 108. 108 Generative Adverserial Nets Alec Radford, Luke Metz, Soumith Chintala , 2015. 
 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv) ”turn” vector created from four averaged samples of faces looking left vs looking right.
  • 109. walking through the manifold Generative Adverserial Nets
  • 110. top: unmodified samples bottom: same samples dropping out ”window” filters Generative Adverserial Nets