Automatic Attendance System using Deep Learning Framework

Automatic Attendance System using
Deep Learning Framework
Pinaki Ranjan Sarkara
, Deepak Mishraa
and Gorthi R.K.S.S Manyamb
a,b: Indian Institute of Space Science and Technology, Trivandrum
Indian Institute of Technology, Tirupati
International Conference On
Machine Intelligence and Signal Processing, IIT-Indore
December 24, 2017

Outline
1 Introduction
Motivation
Diﬃculties
Objective
2 Background Theories
Deep Learning
Approach
Face Detection
Face Recognition
Spatial Transformer Network
Classroom Dataset
3 Results
4 Contribution
P. R. Sarkar Automatic Attendance System using Deep Learning Framework December 24, 2017 2/34

Introduction Motivation
Motivation
Motivation
Taking attendance in large classes is cumbersome, repetitive,
consumes valuable class time.
What if we make an eﬃcient face detection and recognition
system for this task?
So the problem is to develop and implement an eﬃcient face
detection and recognition system.

Introduction Difficulties
Difficulties
Difficulties
Large pose variation
Hidden faces and tiny faces
Different illumination conditions and occlusions.
(a) (b)
Figure 1: Examples taken from classroom

Introduction Objective
Objective
Objectives
Robust face detection in wild condition.
Low False Positive Rate and High True Positive Rate in face
veriﬁcation for fool-proof attendance system.

Background Theories Deep Learning
Deep Learning
“Deep Learning is an algorithm which has no theoretical limitations of
what it can learn; the more data you give and the more computational
time you provide, the better it is.”
- Geoﬀrey Hinton
Deep learning maybe loosely deﬁned as an attempt to train a
hierarchy of feature detectors with each layer learning a higher
representation of the preceding layer.
Deep learning discovers intricate structure in large data sets by
using the backpropagation algorithm to indicate how a machine
should change its internal parameters that are used to compute the
representation in each layer from the representation in the previous
layer1
.
1
LeCun, Y., Bengio, Y. and Hinton, G., 2015. Deep learning. Nature, 521(7553), pp.436-444..

Successful Architectures in DL
Many variants of deep learning architectures are being proposed
and some of them are proved to be successful such as:
Convolutional Neural Network (CNN)2
Deep Boltzmann Machine (DBM)3
Deep Belief Networks (DBN)4
Stacked Denoising Auto-encoders (SDAE)5
Recently, Hinton et al. has published another breakthrough paper
in the ﬁeld of deep learning which is called “Dynamic routing
between capsules”, NIPS 2017.
2
A. Krizhevsky, “Imagenet classiﬁcation with deep convolutional neural networks”.
3
R. Salakhutdinov and G. E. Hinton, “Deep boltzmann machines, ” in AISTATS, vol. 1, p. 3, 2009.
4
G. E. Hinton, “Deep belief networks, ” Scholarpedia, vol. 4, no. 5, p. 5947, 2009.
5
P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol,
“Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion”.

Why DL works
It is very high level modelling of brain. (Please Note: NNs are
NOT inspired from Human brain only, it is inspired from any
animal brain)
It discovers intrinsic structure in large dataset. Understands
the hierarchy of representation involved in the data. That is
how our brain learns, isn’t it?
For more info please go through this paper: ”Visualizing and
understanding convolutional networks”, ECCV - 2014

Convolutional Neural Network
Figure 2: The CNN architecture is composed hierarchical units and each unit extracts
different level of features. Combining more units will produce deeper network along
with more semantic features.6
6
P.R.Sarkar, Deepak Mishra. and Gorthi R.K.S.S. Manyam,
“Classification of Breast Masses Using Convolutional Neural Network as Feature Extractor and Classifier”,
Second International Conference on Computer Vision and Image Processing (CVIP-17) - IIT-Roorkee.

Design Details
The system has two phase i.e. Face Detection and Face
Recognition.
Figure 3: Automatic Attendance System

Background Theories Approach
Face Detection
Recently two researchers Peiyun Hu and Deva Ramanan from
Carnegie Mellon University came up with a novel deep
learning framework7 to ﬁnd tiny faces which was found to be
very eﬀective in our class room data.
Their detector combines a novel combination of scale,
context and resolution to detect faces.
Their face detector produces an average precision of 80%
while prior state-of-the-art ranges from 29-64% on WIDER
FACE database (a massively benchmarked database for face
detection).
7
P. Hu and D. Ramanan, “Finding Tiny Faces, ” CVPR 2017.

Background Theories Face Detection
An example output

Face Detection
This is a binary multi-channel heatmap prediction problem.
Figure 4: Overall architecture for face detection8
Now the question is how to select the templates?
8
P. Hu and D. Ramanan, “Finding Tiny Faces, ” CVPR 2017.

Multi-task modelling of scales
(a) Diﬀerent detectors for
diﬀerent object scales.
(b) Coarse pyramid to
capture extreme scales
(c) Uses of additional
context for tiny faces
(d) Templates over deep
features from ResNet-50
Figure 5: Approach for scale invariance

Effect of context
Figure 6: Effect of context. The green box represents the actual face size, while
dotted boxes represent receptive fields associated with features from different layers
(cyan = res2, light = blue = res3, dark-blue = res4, black = res5)

Eﬀect of resolution
Figure 7: Eﬀect of resolution to build template sizes

Template selection
Let t(x, y, σ) to represent a template which is tuned to detect
faces of size (x/σ, y/σ) at resolution σ. In Figure 7 both
t(25, 20, 1) and t(50, 40, 0.5) are used to find 25x20 face.
To maximize the performance of ti (σi hi , σi wi , σi ) templates
at σi resolution they trained separate multi-task models for
each value of σ ∈ (some fixed set) and take the maximum
for each object size of (hi , wi )
The performance plot of each resolution-specific multi-task
model is shown in the next slide

Template selection
Figure 8: Template resolution analysis. X-axis represents target object sizes, derived
by clustering. Left Y-axis shows AP around each target size (ignoring objects with
more than 0.5 Jaccard distance)

Results on our classroom data
(a) Classroom data-1 (b) Detection result
(c) Classroom data-2 (d) Detection result
Figure 9: Produced results

Missed detection on our classroom data
(a) +45o rotated image (b) 11/14 true and 1 false detection
Figure 10: False detection results

Background Theories Face Recognition
Face Recognition
Deep learning proved that automatically learnt deep features
from personal identity are more eﬀective in robust recognition
than the traditional handcrafted features
We experimented with 2D and 3D transformations to see the
eﬀectiveness of alignment learning (AAM and 3D Dense Face
Alignment) in the performance of face recognition task but
they do not produce very high recognition results.
We have achieved 98.67% accuracy in LFW database using
our proposed network

Background Theories Face Recognition
Proposed Architecture
Figure 11: Overall architecture of the automatic attendance system

Background Theories Spatial Transformer Network
What is Spatial Transformer Network(STN)?
Formulating Spatial Transforms
Three main diﬀerentiable blocks:
Localisation network
Grid generator
Sampler
Why do we need?
To make CNN invariant to scale, rotation and translation.

Spatial Transformer Network
Figure 12: Spatial Transformer Network9
9
M. Jaderberg, K. Simonyan, A. Zisserman et al. , “Spatial transformer networks”,
in Advances in Neural Information Processing Systems, 2015, pp. 20172025..

Intuition behind STN
(a) The sampling grid is the regular grid
G = TI (G), where I is the identity
transformation parameters
(b) The sampling grid is the result of
warping the regular grid with an aﬃne
transformation Tθ(G)
Figure 13: Parametrised sampling grid to an image U producing the output V

Recognition Result on LFW Database
LFW database consists of 5423 unique classes with large pose
and illumination variations. Out of 5423 classes, we randomly
took 1000 classes because of our limited computation power
Augmentation (random rotation between [−45o, +45o]) to
increase our training data into 15399 training images, 3501
testing images and 2100 validation images.
(a) Training accuracy plot (b) Loss plot
Figure 14: Performance on LFW database

Output of STN on LFW Database
(a) Original (b)
(c) (d)
Figure 15: Output of the STN in LFW database

Background Theories Classroom Dataset
Classroom Data
Setup
A video is taken (13.84 seconds ,50 fps) using NIKON D5200
camera from one of the classrooms.
The students were told to give various face pose during the
video
We randomly took 25 frames from the 692 frames and
detected the faces using the state-of-the-art algorithm
Out of the 25 frames, we took all the faces from ﬁrst 20
frames to train our recognition network and all the face
images from the remaining 5 frames to test the network
The classroom which we have selected includes most of the
challenging factors such as extreme lighting condition,
occlusion, presence of tiny faces etc.

Background Theories Classroom Dataset
Classroom Data
(a) Classroom data. Frame: 204 (b) Detection result
Figure 16: Detection result produced using the state-of-the-art paper

Results
Performance on Classroom Data
(a) Training accuracy plot (b) Loss plot
Figure 17: Performance on classroom database

Results
Recognition Result on Classroom Data
(a) (b)
(c) (d)
Figure 18: Recognition output. Left side: Trained face, Right side: Test face

Contribution
Contribution
Methods Train set Database Recognition
(in million) accuracy
DeepFace10 4M LFW 97.35%
FaceNet11 200M LFW 99.63%
DeepID2+12 0.2M LFW 99.47%
Alignment- 0.46M LFW 99.08%
Learning13
Ours 0.02M LFW 98.67%
Ours Classroom 100%
Table 1: Comparison of face verification performances
10
Y. Taigman, M. Yang, M. Ranzato, and L. Wolf,
Deepface: Closing the gap to human-level performance in face verification,
in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 17011708, 2014.
11
F. Schroff, D. Kalenichenko, and J. Philbin,
Facenet: A unified embedding for face recognition and clustering,
12
Y. Sun, X. Wang, and X. Tang, Deeply learned face representations are sparse, selective, and robust,
13
Y. Zhong, J. Chen, and B. Huang, Towards End-to-End Face Recognition through Alignment Learning,
arXiv:1701.07174 [cs], Jan. 2017. arXiv: 1701.07174..

Questions?
sarkar0499pinaki@gmail.com

Automatic Attendance System using Deep Learning Framework

More Related Content

What's hot (20)

Similar to Automatic Attendance System using Deep Learning Framework (20)

Recently uploaded (20)

Automatic Attendance System using Deep Learning Framework