Deep learning

Introduction2
Deep Architecture of ANN
Convolutional Neural Networks
4
5
Fundamentals3
Deep Learning Requirement1
Recurrent Neural Networks6
Deep Autoencoders7
Conclusions and Further Readings8
Table of Contents

Deep Learning Requirement
1. Over the period of time, more drive for Automation,
Artificial Intelligence (E.g.: Autonomous Car, Alphago
from Google Deepmind)
2. Some problems cannot be mathematically
programmed exclusively, instead make machines
learn by itself E.g.: Face recognition
3. Over the period, percentage of unstructured data has
grown to about 90% of total data E.g.: Pictures,
Twitter chats, YouTube videos, WhatsApp Logs etc.
Deep Learning is well suited for Picture, Audio and
Language processing etc.
4. Highly non linear models can be fitted on Big Data
without much issue of over fitting
5. High capacity computational power for cheap makes
the tedious calculations very possible to implement
Fitting Highly Non linear model on Small Data Vs
Big Data

Diversified Utility of Deep Learning

Introduction
• Deep learning is a form of machine learning that uses a
model of computing that's very much inspired by the
structure of the brain. Hence we call this model a neural
network. The basic foundational unit of a neural network is
the neuron)
• Each neuron has a set of inputs, each of which is given a
specific weight. The neuron computes some function on
these weighted inputs. A linear neuron takes a linear
combination of the weighted inputs and apply activation
function (sigmoid, tanh etc.)
• Network feeds the weighted sum of the inputs into
the logistic function (in case of sigmoid activation
function). The logistic function returns a value between 0
and 1. When the weighted sum is very negative, the return
value is very close to 0. When the weighted sum is very
large and positive, the return value is very close to 1
Biological Neuron Artificial Neuron
Number of Neurons in Species

Introduction
• Softwares used in Deep Learning
• Theano: Python based Deep Learning Library
• TensorFlow: Google’s Deep Learning library runs on top of Python/C++
• Keras / Lasagne: Light weight wrapper which sits on top of Theano/TensorFlow, enables faster model
prototyping
• Torch: Lua based Deep Learning library with wide support for machine learning algorithms
• Caffe: Deep Learning library primarily used for processing pictures
• Useful online Courses
• CS231n: Convolutional Neural Networks for Visual Recognition from Stanford university by Andrej Karpathy,
Justin Johnson (http://cs231n.stanford.edu/syllabus.html)
• Machine Learning from Oxford university by Nando de Freitas
(https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/)
• Neural Networks for Machine Learning from University of Toronto by Geff Hinton
(https://www.coursera.org/course/neuralnets)
• CS224d: Deep Learning for Natural Language Processing from Stanford university by Richard Socher
(http://cs224d.stanford.edu/)

Fundamentals
• Activation Functions: Every activation function takes a single
number and performs a certain fixed mathematical operation on it.
Below are popularly used activation functions in Deep Learning
• Sigmoid
• Tanh
• Relu
• Linear
• Sigmoid: Sigmoid has mathematical form σ(x) = 1 / (1+e−x). It takes
real valued number and squashes it into range between 0 and 1.
Sigmoid is popular choice, which makes ease of calculating
derivatives and easy to interpret
• Tanh: Tanh squashes the real valued number to the range [-1,1].
Output is zero centered. In practice tanh non-linearity is always
preferred to the sigmoid nonlinearity. Also, it can be proved that
Tanh is scaled sigmoid neuron tanh(x) = 2σ(2x) − 1
Sigmoid Activation Function
.
Tanh Activation Function

Fundamentals
• ReLU (Rectified Linear Unit): ReLU has become very popular in last
few years. It computes the function f(x)=max(0,x). Activation is
simply thresholds at zero
• Linear: Linear activation function is used in Linear regression
problems where it provides derivative always as 1 due to the
function used is f(x) = x
Relu is now popularly being used in place of Sigmoid or Tanh
due to its better property of convergence
ReLU Activation Function
.
Linear Activation Function

Weight9
Activation = g(BiasWeight1 + Weight1 * Input1 + Weight2 * Input2)
Weight1
Weight2
BiasWeight1
Fundamentals
• Forward propagation & Backpropagation: During the forward
propagation stage, features are input to the network and feed
through the subsequent layers to produce the output activations.
• However, we can calculate error of the network only at the
output units but not in the middle/hidden layers. In order to
update the weights to optimal, we must propagate the network’s
errors backwards through its layers
Forward Propagation of Layer 1 Neurons
Hidden
1
Hidden
2
Hidden
3
Input 1
Input 2
Hidden
4
Hidden
5
Hidden
6
Output
1
Output
2
Hidden Layer 1 Hidden Layer 2
Output LayerInput Layer
Activation = g(BiasWeight4 + Weight7 * Hidden1 + Weight8 * Hidden2+ Weight9 *
Hidden3)
Weight7
Weight8
BiasWeight4
Forward Propagation of Layer 2 Neurons
Hidden
1
Hidden
2
Hidden
3
Input 1
Input 2
Hidden
4
Hidden
5
Hidden
6
Output
1
Output
2
Output LayerInput Layer

Fundamentals
Weight18
Activation = g(BiasWeight7 + Weight16 * Hidden4 + Weight17 * Hidden5+ Weight18 *
Hidden6)
Weight16
Weight17
BiasWeight7
Forward Propagation of Output Layer Neurons
Hidden
1
Hidden
2
Hidden
3
Input 1
Input 2
Hidden
4
Hidden
5
Hidden
6
Output
1
Output
2
Output Layer
Input Layer
Error (Output 1)= g’(Output 1) * (True1 – Output1)
Backpropagation of Output Layer Neurons
Hidden
1
Hidden
2
Hidden
3
Input 1
Input 2
Hidden
4
Hidden
5
Hidden
6
Output
1
Output
2
Output Layer
Input Layer

Weight13
Weight10
Weight7
Weight19
Fundamentals
Error (Hidden4)= g’(Hidden4) + (Weight16 * Error(Output1) + Weight19 * Error(Output2))
Weight16
Backpropagation of Hidden Layer2 Neurons
Hidden
1
Hidden
2
Hidden
3
Input 1
Input 2
Hidden
4
Hidden
5
Hidden
6
Output
1
Output
2
Output Layer
Input Layer
Error (Hidden1)= g’(Hidden1) * (Weight7 * Error(Hidden4) + Weight10 *
Error(Hidden5) + Weight13*Error(Hidden6))
Backpropagation of Hidden Layer1 Neurons
Hidden
1
Hidden
2
Hidden
3
Input 1
Input 2
Hidden
4
Hidden
5
Hidden
6
Output
1
Output
2
Output Layer
Input Layer

Weight8
BiasWeight4
Weight1
Input Layer
BiasWeight1
Weight2
Fundamentals
BiasWeight1= BiasWeight1 + a * Error(Hidden1) * 1
Weight1 = Weight1 + a * Error(Hidden1) * Input1
Weight2 = Weight2 + a * Error(Hidden1) * Input2
Updating Weights of Hidden Layer 1 – Input Layer
during Backpropagation
Hidden
1
Hidden
2
Hidden
3
Input 1
Input 2
Hidden
4
Hidden
5
Hidden
6
Output
1
Output
2
Output Layer
Weight7
Input Layer
Weight9
BiasWeight4= BiasWeight4 + a * Error(Hidden4) * 1
Weight7 = Weight7 + a * Error(Hidden4) * Hidden1
Updating Weights of Hidden Layer 1 – Hidden
Layer 2 during Forward Propagation
Hidden
1
Hidden
2
Hidden
3
Input 1
Input 2
Hidden
4
Hidden
5
Hidden
6
Output
1
Output
2
Hidden Layer 1
Output Layer
Hidden Layer 2

Fundamentals
• Dropout: Dropout is a regularization in Neural networks to avoid
over fitting the data. Typically Dropout is 0.8 (80 % neurons
present randomly all the time) in initial layers and 0.5 in middle
layers
• Optimization: Various techniques used to optimize the weights
including
• SGD (Stochastic Gradient Descent)
• Momentum
• Nag (Nesterov Accelerated Gradient)
• Adagrad (Adaptive gradient)
• Adadelta
• Rmsprop
• Adam (Adaptive moment estimation)
In practice Adam is good default choice, if you cant afford full
batch updates, then try out L-BFGS
Application of Dropout in Neural network
Optimization of Error Surface

Fundamentals
• Stochastic Gradient Descent (SGD): Gradient descent is a way to
minimize an objective function J(θ) parameterized by a model’s
parameter θ∈Rd by updating the parameters in the opposite
direction of the gradient of the objective function w.r.to the
parameters. Learning rate determines the size of steps taken to
reach minimum.
• Batch Gradient Descent (all training observations per each iteration)
• SGD (1 observation per iteration)
• Mini Batch Gradient Descent (size of about 50 training observations for each
iteration)
• Momentum: SGD has trouble navigating surface curves much more
steeply in one dimension than in other, in these scenarios SGD
oscillates across the slopes of the ravine while only making hesitant
progress along the bottom towards the local optimum
(When using momentum we push a ball down a hill. Ball accumulates momentum as it rolls
downhill, becoming faster and faster on the way until it stops (due to air resistance etc.)
similarly momentum
term increases for dimensions whose gradients point in the same direction and reduces
updates for dimensions whose gradients change directions. As a result, we gain faster
convergence and reduced oscillations)
Gradient Descent
Comparison of SGD without & with Momentum

Fundamentals
• Nesterov Accelerated Gradient (NAG): If a ball rolls down a hill and
blindly follows a slope, is highly unsatisfactory and it should have a
notion of where it is going so that it knows to slow down before the
hill slopes up again. NAG is a way to give momentum term this kind
of prescience
(While momentum first computes the current gradient (small blue vector) and then takes a big
jump in the direction of the updated accumulated gradient (big blue vector), NAG first makes a
big jump in the direction of the previous accumulated gradient (brown vector), measures the
gradient and then makes a correction (green vector). This anticipatory update prevents the ball
from going too fast and results in increased responsiveness and performance)
• Adagrad: Adagrad is an algorithm for gradient-based optimization
that adapts the differential learning rate to parameters, performing
larger updates for infrequent and smaller updates for frequent
parameters
(Adagrad greatly improves the robustness of SGD and used it to training large-scale neural
nets. One of the Adagrad’s main benefits is that it eliminates the need to manually tune the
learning rate. Most implementations use a default value of 0.01 and leave it at that.
Adagrad's main weakness is its accumulation of the squared gradients in the denominator:
Since every added term is positive, the accumulated sum keeps growing during training. This
in turn causes the learning rate to shrink and eventually become infinitesimally small, at which
point the algorithm is no longer able to acquire additional knowledge. The following
algorithms aim to resolve this flaw.)
Nesterov Momentum Update

Fundamentals
• Adadelta: Adadelta is an extension of Adagrad that seeks to reduce
its aggressive, monotonically decreasing learning rate. Instead of
accumulating all past squared gradients, Adadelta restricts the
window of accumulated past gradients to some fixed size w
(Instead of inefficiently storing W previous squared gradients, the sum of gradients is
recursively defined as a decaying average of all past squared gradients)
• RMSprop: RMSprop and Adadelta have both developed independently
around the same time to resolve Adagrad’s radically diminishing
learning rates
(RMSprop as well divides the learning rate by an exponentially decaying average of squared
gradients)
• Adam (Adaptive Moment Estimation): Adam is another method that
computes adaptive learning rates for each parameter. In addition to
storing an exponentially decaying average of past squared gradients
like Adadelta and RMSprop, Adam also keeps an exponentially
decaying average of past gradients similar to momentum
In practice Adam gives best results. For complete details on all methods refer:
http://sebastianruder.com/optimizing-gradient-descent/index.html#batchgradientdescent

Deep Architecture of ANN (Artificial Neural Network)
• In Multi Layer /Deep Architecture, each layer is fully
connected with the subsequent layer. Output of each
artificial neuron in a layer is an input to every artificial
neuron in the next layer towards the output
• Solving Methodology: Back propagation used to solve deep
layers by calculating the error of the network at output
units and propagate back through layers
• Thumb rules:
• All hidden layers should have same number of
neurons per layer
• Typically 2 hidden layers are good enough to solve
majority of problems
• Using scaling/batch normalization (mean 0, variance
1) for all input variables after each layer improves
convergence effectiveness
• Reduction in step size after each iteration improves
convergence, in addition to usage of momentum &
Dropout
Deep Architecture
Decision Boundary of Deep Architecture

Deep Architecture of ANN (Artificial Neural Network)
• Case Study: To predict the survival (0 or 1) on Titanic
based on few characteristics like Class, Age, Gender,
Fare etc.
Probability of Survival in Titanic Disaster
Method Test score Settings
ANN 0.7799 512-512-512-1, nb_epoch = 100, batchsize = 32
Adaboost 0.77033 ntree = 100, lrate = 0.04, algo = SAMME.R
Randomforest 0.77033
ntree = 100, maxdepth = 4, criteria =
gini,max_features = auto
Gradientboost 0.76555
ntree = 100, lrate = 0.04, maxdep =5, maxfeatures =
auto
XGBoost 0.76077 ntree = 100, lrate = 0.04, maxdep = 5
Logistic
Regression 0.7512 NA
Hidden
Layer 1
Hidden
Layer 2
Hidden
Layer 3
Output
Layer

• Convolutional Neural Networks used in picture analysis,
including image captioning, digit recognizer and various
visual system processing E.g.: Vision detection in Self
driving cars, Hand written Digit recognizer, Google
Deepmind’ s Alphago
Object recognition and classification using
Convolutional Networks
CNN application in Self Driving Cars CNN application in Handwritten Digit Recognizer

• Hubel & Wiesel inserted microscopic electrodes into the
visual cortex of anesthetized cat to read activity of the
single cells in visual cortex while presenting various
stimuli to it’s eyes during experiment on 1959. For which
received noble prize under Medicine category on 1981
• Hubel & Wiesel discovered that vision is hierarchical,
consists of simple cells, complex cells & hyper-complex
cells
Hubel & Weisel experiments on Cat’s Vision
Vision is Hierarchical phenomenon
Formation of features over layers using Neural Networks
Object detection using Edges

• Input layer/picture consists of 32 x 32 pixels with 3 colors
(Red, Green & Blue) (32 x 32 x 3)
• Convolution layer is formed by running a filter (5 x 5 x 3)
over Input layer which will result in (28 x 28 x 1)
Input Layer & Filter
Running filter over Input Layer to form Convolution layer Complete Convolution Layer from filter

• 2nd Convolution layer has been created in similar way with
another filter
• After striding/convolving with 6 filters, new layer has been
created with 28 x 28 x 6 dimension
Complete Convolution layer from Filter 2
Convolution layers created with 6 Filters Formation of complete 2nd layer

• Pooling Layer: Pooling layer makes the representation
smaller and more manageable. Operates over each
activation map independently. Pooling applies on width
and breadth of the layer and depth will remains the same
during pooling stage
• Padding: Size of the image (width & breadth) is getting
shrunk consecutively, this issue is undesirable during deep
networks, padding keeps the size of picture constant or
controllable in size throughout the network
Max pooling working methodology
Max pool layer after performing pooling
Zero padding on 6 x 6 picture

• Alex net Architecture: Alex Net won the IMAGENET challenge
competition during 2012
• Layer 0: Input image (227 * 227 * 3 ~= 150k)
• Layer 1: Convolution with 96 filters, size 11×11, stride 4, padding 0
• Layer 2: Max-Pooling with 3×3 filter, stride 2
• Layer 9: Fully Connected with 4096 neuron
• Layer 10: Fully Connected with 4096 neuron
• Layer 11: Fully Connected with 1000 neurons (classes to predict)
Total memory required 24M * 4 bytes ~= 93 MB/image (only forward !~
*2 for bwd)
Alex Net for IMAGENET Challenge 2012

• Case Study: kaggle Digit recognizer to recognize
handwritten digits
• Following implementation made the score 0.99314
(ideal score 1) 46th rank (3.5 %) out of 1314 teams
in Public Leaderboard
Digit Recognizer to classify Hand Written digits
Layer 1 consists of 2
Convolutional layers
followed by Max pooling
layer
}
}
Convolutional layers
followed by Max pooling
layer
} Layer 3 consists of
Dense network with
Dropout 0.5
Layer 4 is Softmax layer
for multiclass (10)
outputs

Recurrent Neural Networks
• Recurrent neural networks are very much useful in sequence
remembering, time series forecasting, Image captioning,
machine translation etc.
• RNNs are useful in building A.I. Chabot in which sequence of
words with all syntaxes & semantics would be remembered and
subsequently provide answers to given questions
Image Captioning using Convolutional and
Recurrent Neural Network
Application of RNN in A.I. Chatbot

• Recurrent neural network is used for processing sequence
of vectors x by applying a recurrence formula at every time
step
Recurrent Neural Network
Vanilla Network Image Captioning
(image -> Seq. of words)
Sentiment Classification
(Seq. of words -> Sentiment)
Machine Translation
(Seq. of words -> Seq. of words)
Video Classification on
frame level
yt
x
t
RNN
y0
x0
RNN
y1
x1
RNN
y2
x2
RNN
yt
x
t
RNN

• Vanishing gradient problem with RNN: Gradients do
vanishes quickly with more number of layers and this issue
is severe with RNN. Vanishing gradients leads to slow
training rates. LSTM & GRU are used to avoid this issue
• LSTM (Long Short Term Memory): LSTM is an artificial
neural network contains LSTM blocks in addition to regular
network units. LSTM block contains gates that determine
when the input is significant enough to remember, when it
should continue to remember or when it should forget the
value and when it should output the value
LSTM Working Principle (Backpropagation
through a memory cell)
LSTM Cell
RNN & LSTM formula

• Case Study: NIFTY prediction NIFTY 1 Year EOD data
}
}
}
}
Recurrent LSTM neurons
with return sequence
False
Output Layer consists of
1 neuron with linear
activation function

Deep Autoencoders
• Deep Autoencoder: Autoencoder neural network is an
unsupervised learning algorithm that applies
backpropagation. Stacking layers of Autoencoders
produces a deeper architecture known as Stacked or Deep
Autoencoders
• Application of Encoders in Face recognition, Speech
recognition, Signal Denoising etc.
PCA vs Deep Autoencoder for MNIST Data
Face Recognition using Deep Autoencoders

Deep Autoencoders
• Deep Autoencoder: Autoencoder neural network is an
unsupervised learning algorithm that applies
backpropagation, setting the target values to be equal to
the inputs. i.e. it uses y ( i ) = x ( i )
• Typically deep Autoencoder is composed of two segments,
encoding network and decoding network.
Deep Autoencoder Examples Training Deep Autoencoder
Autoencoder with Classifier
Reconstruction of features with weight transpose

Deep learning

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Deep learning (20)

Recently uploaded (20)

Deep learning