SlideShare a Scribd company logo

RNN, LSTM and Seq-2-Seq Models

Emory NLP, profile picture
Emory NLP

This document provides an overview of recurrent neural network (RNN) models including long short-term memory (LSTM) networks and sequence-to-sequence (seq-2-seq) models. RNNs maintain information about previous computations through feedback connections, making them well-suited for sequence processing tasks. LSTMs address the gradient vanishing problem of standard RNNs through gated cell states. Seq-2-seq models consist of an encoder RNN that encodes the input sequence into a vector, and a decoder RNN that generates the output sequence from the vector. The document includes a TensorFlow code example of an RNN trained to predict the next character in a sequence.

Technology
1 of 11
Downloaded 176 times
1
2
3
4
5
6
7
8
9
10
11
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 1/11 Introductory Presentation on RNN, LSTM and Seq-2-Seq Models by Jayeol Chun and Sang-Hyun Eun 1. Brief Overview of Theory behind RNN Q: What is RNN?
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 2/11 Feed-Forward vs. Feed-Back : Static vs. Dynamic As opposed to Convolution Neural Network (CNN) where there are no cycles, Recurrent Neural Network (RNN) maintains the Persistence of Information by linking the outputs of previous computations to the later computations, and is thus well suited for processing sequence of characters, naturally making it an ideal tool in NLP. Basic RNN Computation in Theory class RNN: # ... def step(self, x): # update the hidden state self.h = np.tanh(np.dot(self.W_hh, self.h) + np.dot(self.W_xh, x)) # compute the output vector y = np.dot(self.W_hy, self.h) return y # main instruction rnn = RNN() y = rnn.step(x) # x is an input vector, y is the RNN's output vector ~ Point to Take Away : Quite Simple !! Challenge Unstable Gradient Problem -> "The gradient in deep neural networks is unstable, tending to either explode or vanish in earlier layers." In at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers. Question : The more the hidden layers, the better ??
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 3/11 "Backpropagation computes gradients by chain rule -> This has the effect of multiplying n of these small numbers to compute gradients of the front layers in an n-layer network, meaning that the gradient (error signal) decreases exponentially with n and the front layers train very slowly." 2. Long Short Term Memory Network (LSTM)
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 4/11 The most commonly used type of RNN that addresses the above challenge Can learn to recognize context-sensitive languages Key is the cell state. It runs down the straight down the entire chain, with only minor linear interactions It updates its information with a structure called gates There are the 3 main types of gates Forget Gate Layer - Sigmoid layer and chooses what information to forget. Input Gate Layer - Choose what values to update and whats values to add Output Gate Layer - Based on our cell state, filter it to decide which values we want to output 3. Sequence to Sequence Model
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 5/11 Seq-2-Seq Model consists of two RNNs : an encoder that processes the input and maps it to a vector, and a decoder that generates the output sequence of symbols from the vector representation. Specifically, the encoder maps a variable-length source sequence to a fixed-length vector, and the decoder maps the vector representation back to a variable-length target sequence of symbols. The two networks are trained jointly to maximize the conditional probability of the target sequence given a source sequence. Each box in the picture above represents a cell of the RNN.
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 6/11 Example Sample RNN / Seq-2-Seq Code
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 7/11 In [1]: import tensorflow as tf from tensorflow.models.rnn import rnn, rnn_cell import numpy as np char_rdic = ['h','e','l','o'] # id -> char char_dic = {w: i for i, w in enumerate(char_rdic)} # char -> id sample = [char_dic[c] for c in "hello"] # to index x_data = np.array([ [1,0,0,0], # h [0,1,0,0], # e [0,0,1,0], # l [0,0,1,0]], # l dtype='f') # Configuration char_vocab_size = len(char_dic) rnn_size = 4 #char_vocab_size # 1 hot coding (one of 4) time_step_size = 4 # 'hell' -> predict 'ello' batch_size = 1 # one sample # RNN model rnn_cell = rnn_cell.BasicRNNCell(rnn_size) state = tf.zeros([batch_size, rnn_cell.state_size]) X_split = tf.split(0, time_step_size, x_data) outputs, state = tf.nn.seq2seq.rnn_decoder ( X_split, state, rnn_cell) print (state) print (outputs) # logits: list of 2D Tensors of shape [batch_size x num_decoder_symbol s]. # targets: list of 1D batch-sized int32 Tensors of the same length as lo gits. # weights: list of 1D batch-sized float-Tensors of the same length as lo gits. logits = tf.reshape(tf.concat(1, outputs), [-1, rnn_size]) targets = tf.reshape(sample[1:], [-1]) weights = tf.ones([time_step_size * batch_size]) loss = tf.nn.seq2seq.sequence_loss_by_example([logits], [targets], [weig hts]) cost = tf.reduce_sum(loss) / batch_size train_op = tf.train.RMSPropOptimizer(0.01, 0.9).minimize(cost) # Launch the graph in a session with tf.Session() as sess: # you need to initialize all variables tf.initialize_all_variables().run() for i in range(100): sess.run(train_op) result = sess.run(tf.arg_max(logits, 1)) print (result, [char_rdic[t] for t in result])
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 8/11
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 9/11 Tensor("rnn_decoder/BasicRNNCell_3/Tanh:0", shape=(1, 4), dtype=float3 2) [<tf.Tensor 'rnn_decoder/BasicRNNCell/Tanh:0' shape=(1, 4) dtype=float3 2>, <tf.Tensor 'rnn_decoder/BasicRNNCell_1/Tanh:0' shape=(1, 4) dtype=f loat32>, <tf.Tensor 'rnn_decoder/BasicRNNCell_2/Tanh:0' shape=(1, 4) dt ype=float32>, <tf.Tensor 'rnn_decoder/BasicRNNCell_3/Tanh:0' shape=(1, 4) dtype=float32>] (array([2, 0, 0, 0]), ['l', 'h', 'h', 'h']) (array([2, 0, 0, 0]), ['l', 'h', 'h', 'h']) (array([2, 0, 3, 0]), ['l', 'h', 'o', 'h']) (array([2, 0, 3, 0]), ['l', 'h', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 2, 2]), ['l', 'l', 'l', 'l']) (array([2, 2, 2, 2]), ['l', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l'])
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 10/11 (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o'])
6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 11/11 4. RNN Applications - Language Modeling - Conversation Modeling / Question Answering - Machine Translation - Speech Recognition - Image / Video Captioning - Image / Music Generation 5. References - http://colah.github.io/posts/2015-08-Understanding-LSTMs/ - https://en.wikipedia.org/wiki/Convolutional_neural_network - https://en.wikipedia.org/wiki/Recurrent_neural_network - http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-in troduction-to-rnns/ - https://github.com/hunkim/ml - https://www.tensorflow.org/versions/r0.9/tutorials/index.html - http://karpathy.github.io/2015/05/21/rnn-effectiveness/ - http://arxiv.org/pdf/1409.3215v3.pdf - http://arxiv.org/pdf/1406.1078v3.pdf Code References: - https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ker nel_tests/rnn_test.py - https://github.com/hans/ipython-notebooks/blob/master/tf/TF%20tutorial.ipy nb - https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rn n/ptb/ptb_word_lm.py - https://gist.github.com/karpathy/d4dee566867f8291f086#file-min-char-rnn-py

More Related Content

PDF
Seq2Seq (encoder decoder) model
佳蓉 倪
 
PDF
Deep Learning: Recurrent Neural Network (Chapter 10)
Larry Guo
 
PDF
Attention mechanisms with tensorflow
Keon Kim
 
PPTX
Introduction For seq2seq(sequence to sequence) and RNN
Hye-min Ahn
 
PDF
Learning Financial Market Data with Recurrent Autoencoders and TensorFlow
Altoros
 
PDF
Attention-based Models (DLAI D8L 2017 UPC Deep Learning for Artificial Intell...
Universitat Politècnica de Catalunya
 
PPTX
Neural network basic and introduction of Deep learning
Tapas Majumdar
 
PPTX
RNN & LSTM: Neural Network for Sequential Data
Yao-Chieh Hu
 
Seq2Seq (encoder decoder) model
佳蓉 倪
 
Deep Learning: Recurrent Neural Network (Chapter 10)
Larry Guo
 
Attention mechanisms with tensorflow
Keon Kim
 
Introduction For seq2seq(sequence to sequence) and RNN
Hye-min Ahn
 
Learning Financial Market Data with Recurrent Autoencoders and TensorFlow
Altoros
 
Attention-based Models (DLAI D8L 2017 UPC Deep Learning for Artificial Intell...
Universitat Politècnica de Catalunya
 
Neural network basic and introduction of Deep learning
Tapas Majumdar
 
RNN & LSTM: Neural Network for Sequential Data
Yao-Chieh Hu
 

What's hot (19)

PPTX
TypeScript and Deep Learning
Oswald Campesato
 
PDF
Attention is all you need (UPC Reading Group 2018, by Santi Pascual)
Universitat Politècnica de Catalunya
 
PPTX
Recurrent Neural Networks for Text Analysis
odsc
 
PPTX
[Paper Reading] Attention is All You Need
Daiki Tanaka
 
PPTX
Java and Deep Learning (Introduction)
Oswald Campesato
 
PDF
Introduction to Tree-LSTMs
Daniel Perez
 
PDF
Recurrent Neural Networks, LSTM and GRU
ananth
 
PDF
Recurrent Neural Networks (DLAI D7L1 2017 UPC Deep Learning for Artificial In...
Universitat Politècnica de Catalunya
 
PDF
Recurrent Neural Networks
CloudxLab
 
PPTX
Tutorial on convolutional neural networks
Hojin Yang
 
PDF
The Munich LSTM-RNN Approach to the MediaEval 2014 “Emotion in Music” Task
multimediaeval
 
PPTX
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Eun Ji Lee
 
PPTX
Deep Learning, Keras, and TensorFlow
Oswald Campesato
 
PDF
Recurrent Neural Networks I (D2L2 Deep Learning for Speech and Language UPC 2...
Universitat Politècnica de Catalunya
 
PDF
Multilayer Perceptron - Elisa Sayrol - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
PDF
The Perceptron (D1L2 Deep Learning for Speech and Language)
Universitat Politècnica de Catalunya
 
PDF
Neural Network as a function
Taisuke Oe
 
PPTX
Angular and Deep Learning
Oswald Campesato
 
PDF
Convolutional Neural Networks - Veronica Vilaplana - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
TypeScript and Deep Learning
Oswald Campesato
 
Attention is all you need (UPC Reading Group 2018, by Santi Pascual)
Universitat Politècnica de Catalunya
 
Recurrent Neural Networks for Text Analysis
odsc
 
[Paper Reading] Attention is All You Need
Daiki Tanaka
 
Java and Deep Learning (Introduction)
Oswald Campesato
 
Introduction to Tree-LSTMs
Daniel Perez
 
Recurrent Neural Networks, LSTM and GRU
ananth
 
Recurrent Neural Networks (DLAI D7L1 2017 UPC Deep Learning for Artificial In...
Universitat Politècnica de Catalunya
 
Recurrent Neural Networks
CloudxLab
 
Tutorial on convolutional neural networks
Hojin Yang
 
The Munich LSTM-RNN Approach to the MediaEval 2014 “Emotion in Music” Task
multimediaeval
 
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Eun Ji Lee
 
Deep Learning, Keras, and TensorFlow
Oswald Campesato
 
Recurrent Neural Networks I (D2L2 Deep Learning for Speech and Language UPC 2...
Universitat Politècnica de Catalunya
 
Multilayer Perceptron - Elisa Sayrol - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
The Perceptron (D1L2 Deep Learning for Speech and Language)
Universitat Politècnica de Catalunya
 
Neural Network as a function
Taisuke Oe
 
Angular and Deep Learning
Oswald Campesato
 
Convolutional Neural Networks - Veronica Vilaplana - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
Ad

Similar to RNN, LSTM and Seq-2-Seq Models (20)

PDF
Detecting Misleading Headlines in Online News: Hands-on Experiences on Attent...
Kunwoo Park
 
PDF
Sequencing and Attention Models - 2nd Version
ssuserbd372d
 
PDF
Cheatsheet recurrent-neural-networks
Steve Nouri
 
PDF
Generating Sequences with Deep LSTMs & RNNS in julia
Andre Pemmelaar
 
PPTX
10.0 SequenceModeling-merged-compressed_edited.pptx
ykchia03
 
PDF
Rnn presentation 2
Shubhangi Tandon
 
PDF
Recurrent Neural Networks
Sharath TS
 
PDF
Recurrent Neural Networks RNN - Xavier Giro - UPC TelecomBCN Barcelona 2020
Universitat Politècnica de Catalunya
 
PDF
Recent Progress in RNN and NLP
hytae
 
PPTX
Introduction to deep learning
Junaid Bhat
 
PDF
RNN sharing at Trend Micro
Chun Hao Wang
 
PDF
Recurrent and Recursive Networks (Part 1)
sohaib_alam
 
PDF
RNNs for Timeseries Analysis
Bruno Gonçalves
 
PPTX
Tensorflow, deep learning and recurrent neural networks without a ph d
DanielGinot
 
PDF
Cs231n 2017 lecture10 Recurrent Neural Networks
Yanbin Kong
 
PDF
Artificial Intelligence Practical Manual.pdf
priyanshi25121980
 
PDF
Recurrent Neural Networks (RNN): Unlocking Sequential Data Processing
studyandinnovation
 
PDF
Lecture 9 - Deep Sequence Models, Learn Recurrent Neural Networks (RNN), GRU ...
Maninda Edirisooriya
 
PDF
Towards typesafe deep learning in scala
Tongfei Chen
 
PDF
Recurrent Neural Networks
Jun Young Park
 
Detecting Misleading Headlines in Online News: Hands-on Experiences on Attent...
Kunwoo Park
 
Sequencing and Attention Models - 2nd Version
ssuserbd372d
 
Cheatsheet recurrent-neural-networks
Steve Nouri
 
Generating Sequences with Deep LSTMs & RNNS in julia
Andre Pemmelaar
 
10.0 SequenceModeling-merged-compressed_edited.pptx
ykchia03
 
Rnn presentation 2
Shubhangi Tandon
 
Recurrent Neural Networks
Sharath TS
 
Recurrent Neural Networks RNN - Xavier Giro - UPC TelecomBCN Barcelona 2020
Universitat Politècnica de Catalunya
 
Recent Progress in RNN and NLP
hytae
 
Introduction to deep learning
Junaid Bhat
 
RNN sharing at Trend Micro
Chun Hao Wang
 
Recurrent and Recursive Networks (Part 1)
sohaib_alam
 
RNNs for Timeseries Analysis
Bruno Gonçalves
 
Tensorflow, deep learning and recurrent neural networks without a ph d
DanielGinot
 
Cs231n 2017 lecture10 Recurrent Neural Networks
Yanbin Kong
 
Artificial Intelligence Practical Manual.pdf
priyanshi25121980
 
Recurrent Neural Networks (RNN): Unlocking Sequential Data Processing
studyandinnovation
 
Lecture 9 - Deep Sequence Models, Learn Recurrent Neural Networks (RNN), GRU ...
Maninda Edirisooriya
 
Towards typesafe deep learning in scala
Tongfei Chen
 
Recurrent Neural Networks
Jun Young Park
 
Ad

Recently uploaded (20)

PDF
Approach and Philosophy of On baking technology
30anthuong17
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PDF
Modernizing your data center with Dell and AMD
Principled Technologies
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PDF
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Approach and Philosophy of On baking technology
30anthuong17
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
Modernizing your data center with Dell and AMD
Principled Technologies
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Teaching material agriculture food technology
LiaRayya
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 

RNN, LSTM and Seq-2-Seq Models

  • 1. 6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 1/11 Introductory Presentation on RNN, LSTM and Seq-2-Seq Models by Jayeol Chun and Sang-Hyun Eun 1. Brief Overview of Theory behind RNN Q: What is RNN?
  • 2. 6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 2/11 Feed-Forward vs. Feed-Back : Static vs. Dynamic As opposed to Convolution Neural Network (CNN) where there are no cycles, Recurrent Neural Network (RNN) maintains the Persistence of Information by linking the outputs of previous computations to the later computations, and is thus well suited for processing sequence of characters, naturally making it an ideal tool in NLP. Basic RNN Computation in Theory class RNN: # ... def step(self, x): # update the hidden state self.h = np.tanh(np.dot(self.W_hh, self.h) + np.dot(self.W_xh, x)) # compute the output vector y = np.dot(self.W_hy, self.h) return y # main instruction rnn = RNN() y = rnn.step(x) # x is an input vector, y is the RNN's output vector ~ Point to Take Away : Quite Simple !! Challenge Unstable Gradient Problem -> "The gradient in deep neural networks is unstable, tending to either explode or vanish in earlier layers." In at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers. Question : The more the hidden layers, the better ??
  • 3. 6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 3/11 "Backpropagation computes gradients by chain rule -> This has the effect of multiplying n of these small numbers to compute gradients of the front layers in an n-layer network, meaning that the gradient (error signal) decreases exponentially with n and the front layers train very slowly." 2. Long Short Term Memory Network (LSTM)
  • 4. 6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 4/11 The most commonly used type of RNN that addresses the above challenge Can learn to recognize context-sensitive languages Key is the cell state. It runs down the straight down the entire chain, with only minor linear interactions It updates its information with a structure called gates There are the 3 main types of gates Forget Gate Layer - Sigmoid layer and chooses what information to forget. Input Gate Layer - Choose what values to update and whats values to add Output Gate Layer - Based on our cell state, filter it to decide which values we want to output 3. Sequence to Sequence Model
  • 5. 6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 5/11 Seq-2-Seq Model consists of two RNNs : an encoder that processes the input and maps it to a vector, and a decoder that generates the output sequence of symbols from the vector representation. Specifically, the encoder maps a variable-length source sequence to a fixed-length vector, and the decoder maps the vector representation back to a variable-length target sequence of symbols. The two networks are trained jointly to maximize the conditional probability of the target sequence given a source sequence. Each box in the picture above represents a cell of the RNN.
  • 7. 6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 7/11 In [1]: import tensorflow as tf from tensorflow.models.rnn import rnn, rnn_cell import numpy as np char_rdic = ['h','e','l','o'] # id -> char char_dic = {w: i for i, w in enumerate(char_rdic)} # char -> id sample = [char_dic[c] for c in "hello"] # to index x_data = np.array([ [1,0,0,0], # h [0,1,0,0], # e [0,0,1,0], # l [0,0,1,0]], # l dtype='f') # Configuration char_vocab_size = len(char_dic) rnn_size = 4 #char_vocab_size # 1 hot coding (one of 4) time_step_size = 4 # 'hell' -> predict 'ello' batch_size = 1 # one sample # RNN model rnn_cell = rnn_cell.BasicRNNCell(rnn_size) state = tf.zeros([batch_size, rnn_cell.state_size]) X_split = tf.split(0, time_step_size, x_data) outputs, state = tf.nn.seq2seq.rnn_decoder ( X_split, state, rnn_cell) print (state) print (outputs) # logits: list of 2D Tensors of shape [batch_size x num_decoder_symbol s]. # targets: list of 1D batch-sized int32 Tensors of the same length as lo gits. # weights: list of 1D batch-sized float-Tensors of the same length as lo gits. logits = tf.reshape(tf.concat(1, outputs), [-1, rnn_size]) targets = tf.reshape(sample[1:], [-1]) weights = tf.ones([time_step_size * batch_size]) loss = tf.nn.seq2seq.sequence_loss_by_example([logits], [targets], [weig hts]) cost = tf.reduce_sum(loss) / batch_size train_op = tf.train.RMSPropOptimizer(0.01, 0.9).minimize(cost) # Launch the graph in a session with tf.Session() as sess: # you need to initialize all variables tf.initialize_all_variables().run() for i in range(100): sess.run(train_op) result = sess.run(tf.arg_max(logits, 1)) print (result, [char_rdic[t] for t in result])
  • 9. 6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 9/11 Tensor("rnn_decoder/BasicRNNCell_3/Tanh:0", shape=(1, 4), dtype=float3 2) [<tf.Tensor 'rnn_decoder/BasicRNNCell/Tanh:0' shape=(1, 4) dtype=float3 2>, <tf.Tensor 'rnn_decoder/BasicRNNCell_1/Tanh:0' shape=(1, 4) dtype=f loat32>, <tf.Tensor 'rnn_decoder/BasicRNNCell_2/Tanh:0' shape=(1, 4) dt ype=float32>, <tf.Tensor 'rnn_decoder/BasicRNNCell_3/Tanh:0' shape=(1, 4) dtype=float32>] (array([2, 0, 0, 0]), ['l', 'h', 'h', 'h']) (array([2, 0, 0, 0]), ['l', 'h', 'h', 'h']) (array([2, 0, 3, 0]), ['l', 'h', 'o', 'h']) (array([2, 0, 3, 0]), ['l', 'h', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 0]), ['l', 'l', 'o', 'h']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 3, 2]), ['l', 'l', 'o', 'l']) (array([2, 2, 2, 2]), ['l', 'l', 'l', 'l']) (array([2, 2, 2, 2]), ['l', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l'])
  • 10. 6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 10/11 (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 2]), ['e', 'l', 'l', 'l']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o']) (array([1, 2, 2, 3]), ['e', 'l', 'l', 'o'])
  • 11. 6/23/2016 Presentation Final file:///Users/jayeolchun/Downloads/Presentation+Final%20(1).html 11/11 4. RNN Applications - Language Modeling - Conversation Modeling / Question Answering - Machine Translation - Speech Recognition - Image / Video Captioning - Image / Music Generation 5. References - http://colah.github.io/posts/2015-08-Understanding-LSTMs/ - https://en.wikipedia.org/wiki/Convolutional_neural_network - https://en.wikipedia.org/wiki/Recurrent_neural_network - http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-in troduction-to-rnns/ - https://github.com/hunkim/ml - https://www.tensorflow.org/versions/r0.9/tutorials/index.html - http://karpathy.github.io/2015/05/21/rnn-effectiveness/ - http://arxiv.org/pdf/1409.3215v3.pdf - http://arxiv.org/pdf/1406.1078v3.pdf Code References: - https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ker nel_tests/rnn_test.py - https://github.com/hans/ipython-notebooks/blob/master/tf/TF%20tutorial.ipy nb - https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rn n/ptb/ptb_word_lm.py - https://gist.github.com/karpathy/d4dee566867f8291f086#file-min-char-rnn-py