Neural networks

Submitted By :-
Harshit Gupta (8816103024)
Muaaz Ehteshamuddin (8816103045)
Saurabh Tiwari (8816103051)
Mentor :-
Dr. Nishant Srivastava
Department of Computer Science and Engineering,
Jaypee University Anoopshahr.

Introduction
• Handwritten digits recognition being a challenging problem
is area of research in recent years.
• There are many concern areas regarding this problem, for
example - automatic processing of bank cheques, number
plates of vehicles, postal addresses, in mobile phones etc.

Related Work
• Calin Enachescu et. al proposed a neural computing method for
recognizing handwritten digits. With original NIST dataset, it
provided 96.74% accuracy, and with the images that are without
background it provided 96.56% accuracy.
•Mexican hat wavelet transformation technique was used for
preprocessing data and it gave the 99.17% accuracy on data set

•A wide range of researches has been performed on the MNIST
database to explore the potential and drawbacks of the best
recommended approach. The best methodology till date offers a
training accuracy of 99.81%.

Data Set
•Each training example is a 20 pixel by 20 pixel grayscale
image of the digit.
•Each pixel is represented by a floating point number indicating
the grayscale intensity at that location.
•The 20 by 20 grid of pixels is unrolled into a 400-dimensional
vector. Each of these training examples becomes a single row in
our data matrix X. This gives us a 5000 by 400 matrix X.

Logistic Regression
Form of regression that allows
the prediction of discrete
variables by a mix of
continuous and discrete
predictors. Logistic regression
is often used because the
relationship between the DV
(a discrete variable) and a
predictor is non-linear
Fig 2:-A Logistic Regression model

The activation function that is used is known as
the sigmoid function. The plot of the sigmoid function looks like
Fig 3 :-Sigmoid Function
Hypothesis

We can see that the value of the sigmoid function always lies
between 0 and 1. The value is exactly 0.5 at X=0. We can use
0.5 as the probability threshold to determine the classes. If
the probability is greater than 0.5, we classify it as Class-1
(Y=1) or else as Class-0 (Y=0). sigmoid function is
represented by,
The hypothesis for logistic regression then becomes,
(1)
(2)

If the weighted sum of inputs is greater than zero, the
predicted class is 1 and vice-versa. So the decision boundary
separating both the classes can be found by setting the
weighted sum of inputs to 0.

Cost Function
The cost function for a single training example can be given
by:

If the actual class is 1 and the model predicts 0, we should
highly penalize it and vice-versa. As you can see from the
below picture, for the plot -log(h(x))as h(x) approaches 1,
the cost is 0 and as h(x) nears 0, the cost is infinity(that is
we penalize the model heavily). Similarly for the plot -
log(1-h(x)) when the actual value is 0 and the model
predicts 0, the cost is 0 and the cost becomes infinity as
h(x) approaches 1.

Fig 4 :-Cost Function
We can combine both of the equations using:
(3)

The cost for all the training examples denoted by J(θ) can be
computed by taking the average over the cost of all the
training samples
We will use gradient descent to minimize the cost function.
The gradient w.r.t any parameter can be given by
(1)
(3)
(4)

Neural Network
A neural network is a series of
algorithm that attempts to
recognize underlying
relationships in a set of data
through a process that mimics
the way the human brain
operates.
Fig 5:-A simple Neural Network model

Proposed Methodology
Our neural network architecture consist of 3 layers. One input
layer with 400 units, hidden layer with 25 units and output layer
with 10 units each for one digits from 0 to 9. In this neural
network sigmoid function (eq1) is used. This sigmoid function
gives the value in the range [0,1].
(5)

Input Layer
• The 400 pixels extracted from each image is arranged as a
single row in the input vector X.
• The input layer in the neural network architecture consists of
400 neurons, with each neuron representing each pixel value of
vector X for the entire sample.

Hidden Layer
• Many structure is decided through assumption and then best
one is picked up by the test of cross validation test graph and
error graph.
• So after going through these process it decided to choose 1
hidden layer with 25 units.

Output Layer
• The targets for the entire 5000 sample dataset were arranged
in a vector Y of size (5000×1).
• One unit will give the value 1 and the rest will be 0.
• Therefore 10 units for each digits from 9 to 0.

Fig 6: The neural network architecture

• We can initialize all our weights as 0 but it will help in
logistic regression not here.
• When we back propagate all our nodes will update to all
same value repeatedly.
• Instead of this we can randomly initialize our weights .
Initializing Weights

Forward Pass
The 400 pixels was provided to the input layer which is further
multiplied with the weights connecting input and hidden layer
which is represented by
(6)
This net input is provided to the sigmoid function eq(1) and the
output is the value of the hidden layer
(7)

The values of hidden layer is further multiplied with the weights
connecting hidden and output layer which is represented by
(8)
This net input is provided to the sigmoid function eq(1) and the
output is the value of the output layer
(9)

Start
Initialize the weights
to some random
variables
For each
training
pair x , t
Receive Input signal x1 &
transmit to hidden unit
In hidden unit, calculate
o/p,
" 𝑍𝑖𝑛𝑗 = V0j + 𝑖=1
𝑛
=𝑋𝑖 𝑉𝑖𝑗
𝑍𝑖=f(𝑍𝑖𝑛𝑗),
j = 1 to p
i= 1 to n
Send 𝑍𝑖 to the output layer units
Calculate output signal
from
output layer,
𝑌𝑖𝑛𝑘 = W0k+ 𝑦=1
𝑝
𝑍𝑗𝑊𝑗𝑘
𝑌𝑘 = f(𝑌𝑖𝑛𝑘),
k = 1 to m
Target pair 𝑓𝑘
enters
C
B
A
NO
Yes

Cost Function
(10)
• This is the cost function we have to minimize.
• The additional term is used for Regularization.
K – No. of layers
m – No of training data

Reverse Pass
The gradient value for both output and hidden layers were
calculated for updating the weights
where δk and δj are the gradients of the output layer and hidden
layer respectively.
(11)
(12)

(13)
(14)
(15)
(16)
where, Wjk denotes the weight updates of the weights
connecting the hidden and output layer and Wij represents the
weight updates of the weights connecting the input and hidden
layer
• Since Backpropagation has some bugs therefore we can use
gradient checking to monitor it.

A
Compute error correction factor
𝑓𝑘= (𝑡 𝑘 − 𝑌𝑘 ) f’(𝑌𝑖𝑛𝑘)
(between output and hidden)
Find weight & bias correction
term
∆𝑊𝑗𝑘 = 𝛼𝛿 𝑘 𝑍𝑗, ∆𝑊0𝑘 = 𝛼𝛿 𝑘
Calculate error term 𝛿𝑖
(between hidden and input)
𝛿𝑖𝑛𝑗 = 𝑘=1
𝑚
𝛿 𝑘 𝑊𝑗𝑘
𝛿𝑗 = 𝛿𝑖𝑛𝑗 f’(𝑍𝑖𝑛𝑗)
Compute change in weights & bias based
On 𝛿𝑗, ∆𝑉𝑖𝑗 = 𝛼𝛿𝑗 𝑋𝑗, ∆𝑉0𝑗 = 𝛼𝛿𝑗
Update weight and bias on
hidden unit
𝑉𝑖𝑗 (new) =𝑉𝑖𝑗(old) +∆𝑉𝑖𝑗
𝑉0𝑗(new) =𝑉0𝑗 (old) + ∆𝑉0𝑗
Update weight and bias on
output unit
𝑊𝑗𝑘 (new) = 𝑊𝑗𝑘 (old) + ∆𝑊𝑗𝑘
𝑊0𝑘 (new)= 𝑊0𝑘 (old)+ ∆𝑊0𝑘
If specified
number of
epochs
reached or
𝑡 𝑘 = 𝑦 𝑘
C
B
Stop
NO
YES

Implementation
Language Used :
• Python 3.7
Libraries Used :
• Numpy
• Matplotlib
• Pandas
• Sckitlearn

Result
• The performance of the above neural network architecture was
tested on 1000 data set of 20*20 gray scale images.
• It was tested in Python 3.7 under windows 8 environment on
Intel i3 processor with 4GB RAM. The accuracy is the criteria
for assessment of the performance. The accuracy rate is given
by:-

• Training and testing was continued at the interval of 50 iteration
until we got the consistent accuracy for Neural Networks.
92
93
94
95
96
97
98
99
100
50 100 150 200 250
Fig 7:-No of iteration vs. accuracy
• The Accuracy is above 96.5% and highest is 99.32% on the test
data in 200 iterations for Neural Networks.

• The Accuracy is 91.5% for Logistic Regression.

Future Work
•In future working will be done on natural images likes
number plates, debit cards etc with different backgrounds.
•These images will need some image preprocessing so that
images are converted into 20 pixel * 20 pixel format.
•Different classification will be use to compare with this
classification

Conclusion
•The Neural Network architecture with hidden neurons 25 and
maximum number of iterations 200 were found to provide more
accuracy than Logistic Regression.

References
[1] Dr.Kusum Gupta” Neural network Handwritten Digit Recognition using
various Neural Network Approaches” Sakshica1, IJARCCE Vol. 4, Issue 2,
February 2015
[2] Saeed AL-Mansoori “Digit Recognition using Neural Networks” Int. Journal
of Engineering Research and Applications ISSN : 2248-9622, Vol. 5, Issue 5, (
Part -3), pp.46-51 2015
[3] M. S. Akhtar and H. A. Qureshi, "Handwritten digit recognition through
wavelet decomposition and wavelet packet decomposition," Digital Information
Management (ICDIM), Eighth International Conference on, Islamabad, 2013,
pp. 143-148 2013
[4] https://www.researchgate.net/publication/221522748
[5] Saleh Ali K. Al-Omari, Putra Sumari, Sadik A. Al-Taweel and Anas J.A.
Husain
Journal of Computer Science 5 (6): 427-434, 2009 ISSN 1549-3636 © 2009
Science Publications

[6] Y.Le Cuu, B.Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard and
L.D. Jackel,” Handwritten Digit Recognition with a Back-Propagation Network,”
AT&T Bell Laboratories, Holmdel, N.J.07733 1990.
[7] Daniel Cruces Alvarez, Fernando Martin Rodriguez, Xulio Fernandez
Hermida,”Printed and Handwritten Digits Recognition Using Neural Networks”,
E.T.S.I.T. Ciudad Universitaria S/N.36200 Vigo.SPAIN 1998.
[8] Dewi Nasien, Siti S. Yuhaniz, Habibollah Haron,”Recognition of Isolated
Handwritten Latin Characters Using One Continuous Route of Freeman Chain
Code Representation and Feed forward Neural Network Classifier”, World
Academy of Science, Engineering and Technology 67 (2 ),2010
[9] Calin Enachescu, Cristian-Dumitru Miron,”Handwritten Digits Recognition
Using Neural Computing”, Scientific Bulletin of the Petru Maior University of
Tirgu Mures, Vol.6(XXIII), 2009, ISSN 1841-9267.
[10] Xiao-Xiao Niu and Ching Y. Suen, A novel hybrid CNN–SVM classifier for
recognizing handwritten digits, ELSEVIER, The Journal of the Pattern Recognition
Society, Vol. 45, 2012, 1318– 1325.
[11] Diego J. Romero, Leticia M. Seijas, Ana M. Ruedin, Directional Continuous
Wavelet Transform Applied to Handwritten Numerals Recognition Using Neural
Networks, JCS&T, Vol. 7 No. 1, 2007.

Neural networks

More Related Content

What's hot (20)

Similar to Neural networks (20)

Recently uploaded (20)

Neural networks