SlideShare a Scribd company logo
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
What we will learn in this section:
• The idea behind Recurrent Neural Networks
• The Vanishing Gradient Problem
• Long Short-Term Memory (LSTM)
• Practical Intuition
• EXTRA: LSTM Variations
© SuperDataScienceDeep Learning A-Z
Used for Regression & ClassificationArtificial Neural Networks
Used for Computer VisionConvolutional Neural Networks
Used for Time Series AnalysisRecurrent Neural Networks
Used for Feature DetectionSelf-Organizing Maps
Used for Recommendation SystemsDeep Boltzmann Machines
Used for Recommendation SystemsAutoEncoders
SupervisedUnsupervised
© SuperDataScienceDeep Learning A-Z
Image Source: Wikipedia
Cerebellum
Brainstem
Cerebrum
Frontal Lobe
Temporal Lobe
Parietal Lobe
Occipital Lobe
© SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value 3
Output values
Input
Layer
Hidden
Layer
Output
Layer
4
5
6
7
5
6
2
1
1
© SuperDataScienceDeep Learning A-Z
456
Input value 1
Input value 2
Input value 3
Output values
Input
Layer
Hidden
Layer
Output
Layer
7 56211
© SuperDataScienceDeep Learning A-Z
456Input Vector
Output Vector
Input
Layer
Hidden
Layer
Output
Layer
7 56211
© SuperDataScienceDeep Learning A-Z
Input Vector
Output Vector
Input
Layer
Hidden
Layer
Output
Layer
7 51
© SuperDataScienceDeep Learning A-Z
751
© SuperDataScienceDeep Learning A-Z
517
© SuperDataScienceDeep Learning A-Z
517
Time
© SuperDataScienceDeep Learning A-Z
517
Time
517
517
517
© SuperDataScienceDeep Learning A-Z
517
Time
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
© SuperDataScienceDeep Learning A-Z
517
Time
517
517
517
© SuperDataScienceDeep Learning A-Z
Time
Many to Many
Time
One to Many
Time
Many to One
Time
Many to Many
Reference: karpathy.github.io
karpathy.github.io
dev.havenondemand.com
© SuperDataScienceDeep Learning A-Z
Sunspring (movie, 2016)
• Directed By Oscar Sharp
• Written by Benjamin
Link:
https://arstechnica.com/the-multiverse/2016/06/an-ai-wrote-this-movie-and-
its-strangely-moving/
Additional Reading:
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
Image Source: people.idsia.ch
© SuperDataScienceDeep Learning A-Z
Image Source: Université Montréal
© SuperDataScienceDeep Learning A-Z
Image Source: recode.net
© SuperDataScienceDeep Learning A-Z
C
ŷ
C = ½(ŷ- y)2
© SuperDataScienceDeep Learning A-Z
y
X2
X1
W11,1
W11,2
W11,3
W12,1
W12,2
W12,3
W21,1
W22,1
W23,1
© SuperDataScienceDeep Learning A-Z
517
Time
517
517
517
517
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
yt
εt
xtxt-1xt-2xt-3
Formula Source: Razvan Pascanu et al. (2013)
εt+1εt-1εt-2εt-3
Wrec ~ small
Wrec ~ large
Vanishing
Exploding
Wrec Wrec Wrec
© SuperDataScienceDeep Learning A-Z
Solutions:
1. Exploding Gradient
• Weight Initialization
• Echo State Networks
• Long Short-Term Memory Networks (LSTMs)
• Truncated Backpropagation
• Penalties
• Gradient Clipping
2. Vanishing Gradient
© SuperDataScienceDeep Learning A-Z
Untersuchungen zu dynamischen
neuronalen Netzen
By Sepp (Josef) Hochreiter (1991)
Link:
http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidh
uber.pdf
Additional Reading:
© SuperDataScienceDeep Learning A-Z
Learning Long-Term
Dependencies with Gradient
Descent is Difficult
By Yoshua Bengio et al. (1994)
Link:
http://www-dsi.ing.unifi.it/~paolo/ps/tnn-94-gradient.pdf
Additional Reading:
© SuperDataScienceDeep Learning A-Z
On the difficulty of training recurrent
neural networks
By Razvan Pascanu et al. (2013)
Link:
http://www.jmlr.org/proceedings/papers/v28/pascanu13.pdf
Additional Reading:
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
Today:
• A bit of history
• LSTM Architecture
• Example walkthrough
© SuperDataScienceDeep Learning A-Z
517
Time
517
517
517
517
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
yt
εt
xtxt-1xt-2xt-3
Formula Source: Razvan Pascanu et al. (2013)
εt+1εt-1εt-2εt-3
Wrec ~ small
Wrec ~ large
Vanishing
Exploding
Wrec Wrec Wrec
Wrec = 1
< 1
> 1
© SuperDataScienceDeep Learning A-Z
Image Sources: people.idsia.ch, ics.usi.ch
© SuperDataScienceDeep Learning A-Z
517
Time
517
517
517
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
© SuperDataScienceDeep Learning A-Z
Image Source: arxiv.org/pdf/1503.04069.pdf
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
© SuperDataScienceDeep Learning A-Z
Ct-1
Image Source: colah.github.io
Ct
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
Ct-1
Image Source: colah.github.io
Ct
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
Ct-1
Image Source: colah.github.io
Ct
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
CtCt-1
© SuperDataScienceDeep Learning A-Z
Long Short-Term Memory
By Sepp Hochreiter &
Jurgen Schmidhuber (1997)
Link:
http://www.bioinf.jku.at/publications/older/2604.pdf
Additional Reading:
© SuperDataScienceDeep Learning A-Z
Understanding LSTM Networks
By Christopher Olah (2015)
Link:
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Additional Reading:
© SuperDataScienceDeep Learning A-Z
Understanding LSTM and its
diagrams
By Shi Yan (2016)
Link:
https://medium.com/@shiyan/understanding-lstm-and-its-diagrams-
37e2f46f1714
Additional Reading:
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
© SuperDataScienceDeep Learning A-Z
Image Source: karpathy.github.io
© SuperDataScienceDeep Learning A-Z
Image Source: karpathy.github.io
© SuperDataScienceDeep Learning A-Z
Image Source: karpathy.github.io
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
© SuperDataScienceDeep Learning A-Z
Image Source: karpathy.github.io
© SuperDataScienceDeep Learning A-Z
The Unreasonable Effectiveness of
Recurrent Neural Networks
By Andrej Karpathy (2015)
Link:
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Additional Reading:
© SuperDataScienceDeep Learning A-Z
Visualizing and Understanding
Recurrent Networks
By Andrej Karpathy et al. (2015)
Link:
https://arxiv.org/pdf/1506.02078.pdf
Additional Reading:
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
Ct-1
Image Source: colah.github.io
Ct
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct
© SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct
© SuperDataScienceDeep Learning A-Z
© SuperDataScienceDeep Learning A-Z
LSTM: A Search Space Odyssey
By Klaus Greff et al. (2015)
Link:
https://arxiv.org/pdf/1503.04069.pdf
Additional Reading:

More Related Content

PPTX
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Module 2
PPTX
Deep Learning A-Z™: Artificial Neural Networks (ANN) - Module 1
PPTX
Deep Learning A-Z™: AutoEncoders - Module 6
PPTX
Deep Learning A-Z™: Artificial Neural Networks (ANN) - The Neuron
PPTX
Deep Learning A-Z™: Artificial Neural Networks (ANN) - Plan of Attack
PPTX
Deep Learning A-Z™: Boltzmann Machines - Module 5
PPTX
Deep Learning A-Z™: Boltzmann Machine - Module 5 - Boltzmann Machine
PPTX
Deep Learning A-Z™: Artificial Neural Networks (ANN) - The Activation Function
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Module 2
Deep Learning A-Z™: Artificial Neural Networks (ANN) - Module 1
Deep Learning A-Z™: AutoEncoders - Module 6
Deep Learning A-Z™: Artificial Neural Networks (ANN) - The Neuron
Deep Learning A-Z™: Artificial Neural Networks (ANN) - Plan of Attack
Deep Learning A-Z™: Boltzmann Machines - Module 5
Deep Learning A-Z™: Boltzmann Machine - Module 5 - Boltzmann Machine
Deep Learning A-Z™: Artificial Neural Networks (ANN) - The Activation Function

What's hot (20)

PPTX
Deep Learning A-Z™: Self Organizing Maps (SOM) - Module 4
PPTX
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Step 1(b): ReLU Layer
PPTX
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - What Are Convolutio...
PPTX
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Step 1: Convolution...
PPTX
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - LSTMs
PPTX
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Step 2: Pooling
PPTX
Deep Learning A-Z™: Self Organizing Maps (SOM) - How Do SOMs Work?
PPTX
Deep Learning A-Z™: Boltzmann Machines - Boltzmann Machine
PPTX
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Summary
PPTX
Deep Learning A-Z™: Boltzmann Machines - Restricted Boltzmann Machine
PPTX
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Step 3: Flattening
PPTX
Deep Learning A-Z™: Autoencoders - Contractive Autoencoders
PPTX
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - The Vanishing Gradient ...
PPTX
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - Practical Intuition
PPTX
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - The Idea Behind Recurre...
PPTX
Deep Learning A-Z™: AutoEncoders - Training an AutoEncoder
PPTX
Deep Learning A-Z™: Artificial Neural Networks (ANN) - Backpropagation
PPTX
Deep Learning A-Z™: Autoencoders - Sparse Autoencoders
PDF
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...
PPTX
Electricity price forecasting with Recurrent Neural Networks
Deep Learning A-Z™: Self Organizing Maps (SOM) - Module 4
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Step 1(b): ReLU Layer
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - What Are Convolutio...
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Step 1: Convolution...
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - LSTMs
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Step 2: Pooling
Deep Learning A-Z™: Self Organizing Maps (SOM) - How Do SOMs Work?
Deep Learning A-Z™: Boltzmann Machines - Boltzmann Machine
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Summary
Deep Learning A-Z™: Boltzmann Machines - Restricted Boltzmann Machine
Deep Learning A-Z™: Convolutional Neural Networks (CNN) - Step 3: Flattening
Deep Learning A-Z™: Autoencoders - Contractive Autoencoders
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - The Vanishing Gradient ...
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - Practical Intuition
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - The Idea Behind Recurre...
Deep Learning A-Z™: AutoEncoders - Training an AutoEncoder
Deep Learning A-Z™: Artificial Neural Networks (ANN) - Backpropagation
Deep Learning A-Z™: Autoencoders - Sparse Autoencoders
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...
Electricity price forecasting with Recurrent Neural Networks
Ad

Similar to Deep Learning A-Z™: Recurrent Neural Networks (RNN) - Module 3 (20)

PDF
rnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
PDF
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
PDF
PDF
Recurrent Neural Networks (D2L8 Insight@DCU Machine Learning Workshop 2017)
PDF
Recurrent Neural Networks. Part 1: Theory
PDF
Video Analysis with Recurrent Neural Networks (Master Computer Vision Barcelo...
PDF
Recurrent Neural Networks I (D2L2 Deep Learning for Speech and Language UPC 2...
PDF
Recurrent Neural Networks
PDF
Rnn presentation 2
PDF
Deep Learning for Computer Vision: Recurrent Neural Networks (UPC 2016)
PPT
14889574 dl ml RNN Deeplearning MMMm.ppt
PPTX
RNN and LSTM model description and working advantages and disadvantages
PDF
Recurrent Neural Networks (D2L2 2017 UPC Deep Learning for Computer Vision)
PPTX
Introduction to deep learning
PDF
Recurrent Neural Networks RNN - Xavier Giro - UPC TelecomBCN Barcelona 2020
PDF
Recurrent and Recursive Nets (part 2)
PPTX
recurrent_neural_networks_april_2020.pptx
PPTX
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
PPTX
PPTX
Recurrent Neural Network
rnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
Recurrent Neural Networks (D2L8 Insight@DCU Machine Learning Workshop 2017)
Recurrent Neural Networks. Part 1: Theory
Video Analysis with Recurrent Neural Networks (Master Computer Vision Barcelo...
Recurrent Neural Networks I (D2L2 Deep Learning for Speech and Language UPC 2...
Recurrent Neural Networks
Rnn presentation 2
Deep Learning for Computer Vision: Recurrent Neural Networks (UPC 2016)
14889574 dl ml RNN Deeplearning MMMm.ppt
RNN and LSTM model description and working advantages and disadvantages
Recurrent Neural Networks (D2L2 2017 UPC Deep Learning for Computer Vision)
Introduction to deep learning
Recurrent Neural Networks RNN - Xavier Giro - UPC TelecomBCN Barcelona 2020
Recurrent and Recursive Nets (part 2)
recurrent_neural_networks_april_2020.pptx
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Recurrent Neural Network
Ad

More from Kirill Eremenko (18)

PPTX
Deep Learning A-Z™: Regression & Classification - Module 7
PPTX
Deep Learning A-Z™: Regression & Classification - Simple Linear Regression - ...
PPTX
Deep Learning A-Z™: Regression & Classification - Simple Linear Regression - ...
PPTX
Deep Learning A-Z™: Regression & Classification - Logistic Regression
PPTX
Deep Learning A-Z™: Autoencoders - Stacked Autoencoders
PPTX
Deep Learning A-Z™: Autoencoders - Denoising Autoencoders
PPTX
Deep Learning A-Z™: Regression - Multiple Linear Regression Intuition
PPTX
Deep Learning A-Z™: Boltzmann Machines - Deep Belief Networks
PPTX
Deep Learning A-Z™: Boltzmann Machines - Contrastive Divergence
PPTX
Deep Learning A-Z™: Boltzmann Machines - Energy Based Models (ebm)
PPTX
Deep Learning A-Z™: Boltzmann Machines - Deep Boltzmann Machines
PPTX
Deep Learning A-Z™: AutoEncoders - AutoEncoders
PPTX
Deep Learning A-Z™: Boltzmann Machines - Plan of Attack
PPTX
Deep Learning A-Z™: Self Organizing Maps (SOM) - How do SOMs learn (part 2)
PPTX
Deep Learning A-Z™: Self Organizing Maps (SOM) - Reading an Advanced SOM
PPTX
Deep Learning A-Z™: Self Organizing Maps (SOM) - K-Means Clustering (part 2)
PPTX
Deep Learning A-Z™: Self Organizing Maps (SOM) - K-Means Clustering
PPTX
Deep Learning A-Z™: Self Organizing Maps (SOM) - How do SOMs learn (part 1)
Deep Learning A-Z™: Regression & Classification - Module 7
Deep Learning A-Z™: Regression & Classification - Simple Linear Regression - ...
Deep Learning A-Z™: Regression & Classification - Simple Linear Regression - ...
Deep Learning A-Z™: Regression & Classification - Logistic Regression
Deep Learning A-Z™: Autoencoders - Stacked Autoencoders
Deep Learning A-Z™: Autoencoders - Denoising Autoencoders
Deep Learning A-Z™: Regression - Multiple Linear Regression Intuition
Deep Learning A-Z™: Boltzmann Machines - Deep Belief Networks
Deep Learning A-Z™: Boltzmann Machines - Contrastive Divergence
Deep Learning A-Z™: Boltzmann Machines - Energy Based Models (ebm)
Deep Learning A-Z™: Boltzmann Machines - Deep Boltzmann Machines
Deep Learning A-Z™: AutoEncoders - AutoEncoders
Deep Learning A-Z™: Boltzmann Machines - Plan of Attack
Deep Learning A-Z™: Self Organizing Maps (SOM) - How do SOMs learn (part 2)
Deep Learning A-Z™: Self Organizing Maps (SOM) - Reading an Advanced SOM
Deep Learning A-Z™: Self Organizing Maps (SOM) - K-Means Clustering (part 2)
Deep Learning A-Z™: Self Organizing Maps (SOM) - K-Means Clustering
Deep Learning A-Z™: Self Organizing Maps (SOM) - How do SOMs learn (part 1)

Recently uploaded (20)

PDF
Complications of Minimal Access Surgery at WLH
PDF
O7-L3 Supply Chain Operations - ICLT Program
PDF
VCE English Exam - Section C Student Revision Booklet
PDF
Abdominal Access Techniques with Prof. Dr. R K Mishra
PDF
Module 4: Burden of Disease Tutorial Slides S2 2025
PPTX
Lesson notes of climatology university.
PDF
102 student loan defaulters named and shamed – Is someone you know on the list?
PDF
Microbial disease of the cardiovascular and lymphatic systems
PPTX
Cell Types and Its function , kingdom of life
PPTX
Institutional Correction lecture only . . .
PDF
01-Introduction-to-Information-Management.pdf
PDF
2.FourierTransform-ShortQuestionswithAnswers.pdf
PDF
O5-L3 Freight Transport Ops (International) V1.pdf
PPTX
Final Presentation General Medicine 03-08-2024.pptx
PDF
Basic Mud Logging Guide for educational purpose
PDF
Classroom Observation Tools for Teachers
PPTX
Cell Structure & Organelles in detailed.
PDF
RMMM.pdf make it easy to upload and study
PDF
The Lost Whites of Pakistan by Jahanzaib Mughal.pdf
PDF
TR - Agricultural Crops Production NC III.pdf
Complications of Minimal Access Surgery at WLH
O7-L3 Supply Chain Operations - ICLT Program
VCE English Exam - Section C Student Revision Booklet
Abdominal Access Techniques with Prof. Dr. R K Mishra
Module 4: Burden of Disease Tutorial Slides S2 2025
Lesson notes of climatology university.
102 student loan defaulters named and shamed – Is someone you know on the list?
Microbial disease of the cardiovascular and lymphatic systems
Cell Types and Its function , kingdom of life
Institutional Correction lecture only . . .
01-Introduction-to-Information-Management.pdf
2.FourierTransform-ShortQuestionswithAnswers.pdf
O5-L3 Freight Transport Ops (International) V1.pdf
Final Presentation General Medicine 03-08-2024.pptx
Basic Mud Logging Guide for educational purpose
Classroom Observation Tools for Teachers
Cell Structure & Organelles in detailed.
RMMM.pdf make it easy to upload and study
The Lost Whites of Pakistan by Jahanzaib Mughal.pdf
TR - Agricultural Crops Production NC III.pdf

Deep Learning A-Z™: Recurrent Neural Networks (RNN) - Module 3