Deep learning for understanding faces

66 IEEE Signal Processing Magazine | January 2018 |
Deep Learning for Visual Understanding:
Part 2
1053-5888/18©2018IEEE
Rajeev Ranjan, Swami Sankaranarayanan, Ankan Bansal,
Navaneeth Bodla, Jun-Cheng Chen, Vishal M. Patel,
Carlos D. Castillo, and Rama Chellappa
Deep Learning for Understanding Faces
Machines may be just as good, or better, than humans
R
ecent developments in deep convolutional neural networks
(DCNNs) have shown impressive performance improve-
ments on various object detection/recognition problems.
Thishasbeenmadepossibleduetotheavailabilityoflargeanno-
tated data and a better understanding of the nonlinear mapping
between images and class labels, as well as the affordability of
powerful graphics processing units (GPUs). These developments
in deep learning have also improved the capabilities of machines
in understanding faces and automatically executing the tasks
of face detection, pose estimation, landmark localization, and
face recognition from unconstrained images and videos. In this
article, we provide an overview of deep-learning methods used
for face recognition. We discuss different modules involved in
designing an automatic face recognition system and the role of
deep learning for each of them. Some open issues regarding
DCNNs for face recognition problems are then discussed. This
article should prove valuable to scientists, engineers, and end
users working in the fields of face recognition, security, visual
surveillance, and biometrics.
What we can learn from faces
Facial analytics is a challenging problem in computer vision and
has been actively researched for more than two decades [1]. The
goal is to extract as much information as possible, such as loca-
tion, pose, gender, ID, age, emotion, and so forth from a face.
Applications of this technology include detecting and identifying
a person of interest from surveillance videos, active authentica-
tion of users’ cell phones, payment transactions using face bio-
metrics, smart cars, etc. In addition, there has been a growing
interest in face recognition and verification from unconstrained
images and videos that also involve subtasks such as face detec-
tion, facial landmark localization, etc.
In this article, we present an overview of recent automatic
face identification and verification systems based on deep
learning. Three modules are typically needed for such an auto-
matic system. First, a face detector is applied to localize faces in
images or videos. For a robust system, the face detector should
be capable of detecting faces with varying pose, illumination,
Digital Object Identifier 10.1109/MSP.2017.2764116
Date of publication: 9 January 2018
©Istockphoto.com/zapp2photo

67IEEE Signal Processing Magazine | January 2018 |
and scale. Also, the locations and sizes of face bounding boxes
should be precisely determined so that they contain a minimal
amount of background. Second, a fiducial point detector is used
to localize the important facial landmarks such as eye centers,
nose tip, mouth corners, etc. These points are used to align the
faces to normalized canonical coordinates to mitigate the effects
of in-plane rotation and scaling. Third, a feature descriptor that
encodes the identity information is extracted from the aligned
face. Given the face representations, similarity scores are then
obtained between them using a metric. If this metric is lower
than a threshold, it signifies that the two faces are from the same
subject. Since the early 1990s, many face identification/verifi-
cation algorithms have been shown to work well on images and
videos that are collected in controlled settings. However, the
performance of these algorithms often degrades significantly
on face images or videos that have large variations in pose, illu-
mination, resolution, expression, age, background clutter, and
occlusion. Moreover, for the application of video surveillance
where a subject needs to be identified from hundreds of low-
resolution videos, the algorithm should be fast and robust.
To overcome these challenges, researchers have employed
deep learning for computing the features required for facial
analysis. DCNNs have been shown to be effective for image
analysis tasks from the ImageNet Large Scale Visual Recogni-
tion Challenge (ILSVRC) 2012 [3]. Over the last five years,
DCNNs have been used for solving numerous computer vision
problems such as object recognition [3]–[5] and object detec-
tion [6]–[8]. A typical DCNN is a hierarchy of convolutional
layers with a rectified linear unit activation function, consist-
ing of millions of trainable parameters. Owing to the learned
rich and discriminative representations, DCNNs have recently
been successfully applied for face detection [2], [9], [10], fidu-
cial point localization [2], [10], [11], and face recognition and
verification [12]. A key contributing factor to this exceptional
performance is the availability of a large number of annotated
unconstrained face data sets like CASIA-WebFace [13], Mega-
Face [14], [15], MS-Celeb-1M [16], and VGGFace [17] for the
face recognition task and WIDER FACE [18] for the face detec-
tion task. The large amount of training data represents significant
variations in pose, illumination, expression, and occlusion, which
enables the DCNNs to be robust to these variations and extract
meaningful features required for the task.
Face detection in unconstrained images
Face detection plays a crucial role in a face recognition pipeline
and forms the first step in an automatic face recognition system.
Given an input image, a face detector needs to detect all of the
faces in the image and return bounding-box coordinates for each
of them. The key challenge in unconstrained face detection is that
traditional features like Haar wavelets and histogram of oriented
gradient (HOG) do not capture salient facial information at dif-
ferent resolution, viewpoint, illumination, expression, skin color,
occlusions, and cosmetics conditions. The limitation is more due
to the features used than the classifiers. However, with recent
advances in deep-learning techniques and the availability of
GPUs, it is becoming possible to use DCNNs for feature extrac-
tion. It has been shown in [3] that a DCNN pretrained with a large
generic data set can be used as a meaningful feature extractor.
The deep features thus obtained have been used extensively for
object and face detection. The DCNN-based methods for face
detection can be divided into two subcategories: the region-based
approach and the sliding-window approach.
Region based
The region-based approach generates a pool of generic object-
proposals (approximately 2,000 per image), and a DCNN is used
to classify whether or not a given proposal contains a face. Most
DCNN-based approaches are proposal based [2], [10], [19]. A
general pipeline for region-based face detectors consists of off-
the-shelf object proposal generators like Selective Search [20],
followed by a DCNN that classifies these proposals as face/non-
face. HyperFace [10] and All-in-One Face [2] follow this pipeline
for face detection.
Faster R-CNN
A more recent face detector using faster R-CNN [19] uses a com-
mon DCNN to generate the proposals and classify faces. It can
also simultaneously regress the bounding-box coordinates for
each face proposal. Li et al. [21] proposed a multitask face detec-
tor using the framework of faster R-CNN, which integrates a
DCNN and a three-dimensional (3-D) mean face model. The 3-D
mean face model is used to improve the performance of fiducial
detection of the region proposal network (RPN), which greatly
helps face proposal pruning and refinement after face normaliza-
tion. Similarly, Chen et al. [22] proposed a face detector by train-
ing a multitask RPN for face and fiducial detections to generate
high-quality face proposals while reducing the redundant ones
to strike a balance between high recall and precision. The face
proposals are then normalized by the detected fiducial points and
refined using a DCNN face classifier for improved performance.
Recently, Najibi et al. proposed the Single-Stage Headless face
detector [23], which uses the RPN on the VGGNet [24] network
without the fully connected layers.
A major drawback of region-based face detectors is that diffi-
cult faces are hard to capture in any object proposal, which results
in a low recall. Also, the additional proposal generation task
increases the overall computation time.
Sliding-window based
The sliding-window-based method computes a face detection
scoreandbounding-boxcoordinatesateverylocationinafeature
map at a given scale. This procedure is fast compared to region-
based methods and can be implemented using just a convolution
operation that works in a sliding-window fashion. Detections at
different scales are typically carried out by creating an image
pyramid at multiple scales. DCNN-based face detectors built
using the sliding-window technique include DP2MFD [9] and
DDFD [25]. Faceness [26] produces face-part responses along
with full face response and combines them based on their spa-
tial configuration to determine the final face score. Li et al. [27]
proposed a cascade architecture based on DCNNs that oper-
ates at multiple resolutions, quickly rejects background objects

at low-resolution stages, and only evaluates a small number of
challenging candidates at high-resolution stages.
Single-shot detector
Liu et al. [8] proposed the single-shot detector (SSD) for the
task of object detection. The SSD is a sliding-window-based
detector, but instead of creating image pyramids at differ-
ent scales, it utilizes the inbuilt pyramid structure present in
DCNNs. Features are pooled from the intermediate layers of
DCNN at different scales that are used for object classifica-
tion and bounding-box regression. Because the detections are
obtained in a single forward pass of the network, the overall
computation time of SSD is lower than faster R-CNN [19].
Few SSD-based face detectors have been proposed recently.
Yang et al. proposed ScaleFace [28] to detect faces of different
scales from multiple layers of the network and fuse them in
the end. Zhang et al. proposed S3FD [29], which uses a scale-
equitable framework and scale compensation anchor-matching
strategy for improved detection of small faces. Figure 1 shows
the architectures for the aforementioned approaches.
The availability of large training data sets for unconstrained
face detection has also contributed significantly to the success
of DCNN-based face detectors. The FDDB [30] data set is the
most widely used benchmark for unconstrained face detection.
It consists of 2,845 images containing a total of 5,171 faces col-
lected from news articles on yahoo.com. The MALF [31] data
set consists of 5,250 high-resolution images containing a total
of 11,931 faces. The images were collected from Flickr and an
image search service provided by Baidu, Inc. These data sets
contain faces in unconstrained settings with large variations in
occlusion, pose, and illumination.
The WIDER Face [18] data set contains a total of 32,203
images, with 50% samples for training and 10% for valida-
tion. This data set contains faces with large variations in pose,
illumination, occlusion, and scale. Face detectors trained on
this data set have achieved improved performance [19], [23],
[28], [29], [32], [33]. The evaluation results for this data set
reveal that finding small faces in the crowd is still challeng-
ing. A recent method proposed by Hu et al. [33] shows that
contextual information is necessary for detecting tiny faces.
It captures semantics from lower-level features and context
from higher-level features of a DCNN to detect tiny faces (see
Figure 2). Due to space limitations in this article, we are unable
to discuss more traditional face detection methods. We refer the
interested readers to [34] for more detailed discussions of tradi-
tional cascade-based methods and deformable part-based model
(DPM) methods. In addition, for videos with multiple faces, one
also needs to perform face associations to extract face tracks of
each subject. We also refer the readers to [12] for a discussion
on the role of face association in video-based face recognition.
Figure 3(a) provides a comparative performance evaluation of
different face detectors on the FDDB [30] data set.
Finding crucial facial keypoints and head orientation
Facial keypoint detection is also an important preprocessing
component for face recognition and verification tasks. Facial
keypoints such as eye centers, nose tip, mouth corners, etc. can
be used to align the face into canonical coordinates, and such
face normalization helps face recognition [35] and attribute
detection. Head-pose estimation is required for pose-based
face analysis. Both of these problems have been researched
extensively over the last few years. Most of the existing facial
Input Image
Input Image
Extract Region
Proposals (~2 k/Image)
Image Pyramid
Deep Pyramid DPM
Detection
Scores at
Each Location
CNN
CNN
Compute CNN Features Classify Regions
(Linear SVM)
Face? : Yes
Face
Nonface
(a)
(b)
Figure 1. A typical DCNN-based face detector using the (a) region-proposal approach and (b) the sliding-window approach.

keypoint localization algorithms use either a model-based
approach or a cascaded regression-based approach. Wang et al.
[36] provide a comprehensive survey of traditional model-
based methods, including the active appearance model (AAM),
active shape model (ASM), constrained local model (CLM),
and some regression methods such as the supervised descent
method (SDM). Chrysos et al. [37] also summarize fiducial
point tracking across videos using traditional fiducial detec-
tion methods. We refer the interested readers to these works for
more details. In this article, we summarize recently developed
fiducial detection methods based on DCNNs.
Model based
Model-based approaches, such as AAM, ASM, and CLM,
learn a shape model during training and use it to fit new faces
aduring testing. Recently, Antonakos et al. [43] proposed a
method by modeling the appearance of the object using mul-
tiple graph-based pairwise normal distributions (Gaussian
Markov random field) between patches extracted from the
regions. However, the learned models lack the power to cap-
ture complex face image variations in pose, expression, and
illumination. Also, they are sensitive to the initialization in
gradient descent optimization. Besides two-dimensional (2-D)
shape, face alignment methods based on 3-D models have
been recently developed. PIFA [44] by Jourabloo et al. sug-
gested a 3-D approach that employed cascaded regression to
predict the coefficients of a 3-D to 2-D projection matrix and
the base shape coefficients. Another recent work from Jour-
abloo et al. [45] formulated the face alignment problem as a
dense 3-D model-fitting problem, where the camera projection
matrix and 3-D shape parameters were estimated by a cascade
of DCNN-based regressors. 3DDFA [46] by Zhu et al. fitted a
dense 3-D face model to the image via DCNN, where the depth
data is modeled in a Z-buffer.
1
0.95
0.9
0.85
0.8
0.75
0.7
1
0.9
0.6
0.7
0.8
0.5
0.3
0.4
0.1
0.2
0
0 100 200 300 400 500 600 700 800
False Positive
0 1 2 3 4 5 6 7 108 9
Normalized Mean Error (%)
(a) (b)
TruePositiveRate
Fractionof
TestFaces(468inTotal)
NDPFace
CascadeCNN
Faceness
Headhunter
DP2MFD
Faster R-CNN
HyperFace
Zhang et al.
JointCascade
All-In-One Face
HR
RCPR (78.87%)
ERT (81.70%)
SDM (76.75%)
LBF (83.54%)
CCL (91.22%)
HyperFace (84.62%)
All-In-One Face (95.51%)
Figure 3. (a) The receiver operating characteristic (ROC) curves for different face detection methods on the FDDB data set. (b) Landmarks localization evaluation on
the AFW [55] data set using the protocol described in [38] for recently published methods such as CCL [38], HyperFace [10], All-in-One Face [2], LBF [39], SDM [40],
ERT [41], and RCPR [42]. The numbers in the legend are the fraction of faces having a normalized mean error of less than 5%. AFW: Annotated Faces in the Wild; CCL:
cascade compositional learning; LBF: local binary features; ERT: Ensemble of Regression Trees; RCPR: robust cascaded pose regression.
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Figure 2. The hybrid resolution [33] face detector output. The algorithm is able to detect very small faces by making use of novel characterizations of
scale, resolution, and context. Detector confidence is given by the color bar on the right.

Cascaded regression based
Since face alignment is naturally a regression problem, a
multitude of regression-based approaches has been proposed
in recent years. In general, these methods learn a model that
directly maps the image appearance to the target output. Nev-
ertheless, the performance of these methods depends on the
robustness of local descriptors. Sun et al. [47] proposed a cas-
cade of carefully designed DCNNs in which, at each level,
outputs of multiple networks are fused for landmark estima-
tion and achieve good performance. Zhang et al. [48] pro-
posed a coarse-to-fine autoencoder networks approach, which
cascades several successive stacked autoencoder networks
(SANs). The first few SANs predict the rough location of each
facial landmark, and subsequent SANs progressively refine
the landmarks by taking as input the local features extracted
around the detected landmarks at the current stage with higher
resolution. Kumar et al. [11] used a single DCNN, carefully
designed to provide a unique keypoint descriptor and achieved
better performance (see Figure 4). Xiong et al. [49] suggested
domain-dependent descent maps. Zhu et al. [38] observed that
optimizing the base shape coefficients and projection is indi-
rect and suboptimal since smaller parameter errors are not nec-
essarily equivalent to smaller alignment errors. Therefore, they
proposed CCL [38] to develop head-pose-based and domain-
selective regressors by partitioning the optimization domain
into multiple directions based on head pose and combining the
results of multiple domain regressors through the composition
estimator function. Trigeorgis et al. [50] proposed a combined
and jointly trained convolutional recurrent neural network
architecture that enables end-to-end training of the regressors
jointly used in the cascaded regression framework. It avoids
training each regressor independently and canceling of the
descent direction of the regressors. Bulat et al. [51] proposed a
DCNN architecture that first performs facial part detection by
using a score map generated from the DCNN features of the
first few layers to roughly localize each facial landmark and
is then followed by a regression branch to refine the detection
results. Therefore, the algorithm is not sensitive to the qual-
ity of detected face bounding boxes, and the system can be
trained end to end. Kumar et al. [52] also proposed an effective
iterative method for keypoint estimation and pose prediction
of unconstrained faces by learning efficient heat-map-based
DCNN regressors for the face alignment problem, where the
heat maps provide a probability value, indicating the existence
of a certain joint at a specific location.
On the other hand, since different data sets provide differ-
ent facial landmark annotations, 300 Faces in the Wild data-
base (300 W) [53] has been developed as a benchmark for
enabling a fair comparison among various fiducial detection
methods, and it contains more than 12,000 images annotated
with 68 landmarks including Labeled Face Parts in the Wild
[36], Helen [36], AFW [36], Ibug [36], and 600 test images.
(i.e., 300 indoor and 300 outdoor.)
Besides using a 2-D transformation for face alignment, Has-
sner et al. [54] proposed an effective method to frontalize faces
with the help of a generic 3-D face model. However, the effec-
tiveness of the method also highly relies on the quality of the
detected facial landmarks (i.e., the method usually introduces
undesirable artifacts when the quality of facial landmarks is
poor). In addition, quite a few methods have been developed
for multitask learning (MTL) approaches for face detection,
which involve simultaneous training of the face detection task
along with a correlated task such as facial keypoint estimation.
MTL helps the network build a synergy and learn robust fea-
tures since the network is aided by additional supervision. For
instance, information about eye centers, nose tip, etc. obtained
Train
Test
Deep
Descriptor Network
Linear Projection
Global Shape
Indexed Features
Learning Linear
Projection Wt
Deep Descriptor Network
Estimated Shape St–1
Ground-Truth Shape S
~
Estimated Shape St
Ground-Truth Shape S
~
[...]256∗L
[...]256∗L
Wt
Figure 4. A model diagram of regression-based keypoint estimation method using DCNNs. (Figure used with permission from [11].)

from the keypoints can help the network determine the struc-
ture of the face. Zhang et al. [32], Chen et al. [22], Li et al. [21],
and HyperFace [10] train the face detection network along with
the keypoint estimation task, and All-in-One Face [2] extends
the MTL approach to additional tasks such as face verification,
gender, smile, and age estimation. Figure 3(b) provides a com-
parative performance evaluation of different algorithms for the
task of facial keypoint estimation on the AFW [55] data set.
Face identification and verification
In this section, we review the relevant works on face identi-
fication and verification. In Figure 5, we illustrate a general
pipeline for training and testing a face identification/verifica-
tion system using DCNNs. There are two major components
for face identification/verification: 1) robust face representa-
tion and 2) a discriminative classification model (face iden-
tification) or similarity measure (face verification). Since we
focus on deep-learning-based approaches, we refer the inter-
ested readers to [56], which summarizes traditional approaches
including local binary pattern (LBP) and the Fisher vector, as
well as metric learning such as the one-shot similarity (OSS)
kernel, Mahalanobis metric learning, cosine metric learning,
large-margin nearest neighbor, attribute-based classifier, and
joint Bayesian (JB).
Robust feature learning for faces using deep learning
Learning invariant and discriminative feature representations
is a critical step in a face recognition system. Deep-learning
methods have shown that compact and discriminative repre-
sentations can be learned using a DCNN trained with very
large data sets. We first review some important face recogni-
tion works using deep learning followed by a brief summary of
recent methods for feature representation learning.
Instead of relying on handcrafted features such as the popu-
lar face descriptor LBP, Huang et al. [57] proposed to learn the
face representation using convolutional deep belief networks
based on local convolutional restricted Boltzmann machines.
Their method first learns useful representation in an unsuper-
vised manner, from an unlabeled natural image data set con-
taining images of natural scenes, and then transfers the learned
representation to a face identification/verification task through
classification models [e.g., support vector machine (SVM)]
and metric-learning approaches (e.g., OSS). The approach
achieved satisfactory results on the LFW data set without using
large-scale annotated face data sets.
One of the earlier large-scale applications of 3-D-model-
based alignment along with DCNNs for face recognition was
proposed by Taigman et al. in their DeepFace [58] approach.
They derive a face representation using a nine-layer deep neu-
ral network, which involves more than 120 million parameters
and uses several locally connected layers without weight sharing,
rather than the standard convolutional layers. A proprietary face
data set that consists of 4 million facial images belonging to more
than 4,000 identities was used to train their system.
Since collecting a large-scale annotated data set is costly,
Sun et al. proposed DeepID frameworks [59]–[61] with JB for
face verification, which utilized an ensemble of shallower and
smaller deep convolutional networks (i.e., each DCNN con-
sists of four convolutional layers and uses 39 31 1# # patches
as the input) than DeepFace and were trained using 202,599
images of 10,177 subjects. The large number of distinct identi-
ties of the data set and an ensemble of DCNNs for different
local and global face patches make DeepID learn discrimina-
tive and informative face representation. The approach was
the first to achieve results that surpass human performance for
face verification on the LFW data set.
DCNN Model
W, b
Metric Learning
Pos
Neg
Compute
Similarity
Training
DCNN Model
Testing
Figure 5. A general pipeline for training and testing a face verification system using DCNNs.

Schroff et al. proposed a CNN-based approach for face
recognition called FaceNet [62] that directly optimizes the
embedding itself, rather than an intermediate bottleneck layer
as in other deep-learning approaches. To train, they use triplets
of roughly aligned matching/nonmatching face patches gen-
erated using an online triplet mining method. Their approach
was trained on a large proprietary face data set consisting of
100–200 million thumbnails of faces consisting of about 8 mil-
lion different identities.
Yang et al. [13] collected a public large-scale annotated
face data set, CASIA-WebFace, which consists of 494,414 face
images for 10,575 subjects collected from the IMDB website.
They trained a DCNN with 5 million parameters. The model
followed with JB achieved satisfactory results on the LFW data
set. CASIA-WebFace is widely used to train various DCNN
models in the face recognition community.
Parkhi et al. [17] also collected a public large-scale anno-
tated face data set, VGGFace, which consists of 2.6 million
face images for 2,600 subjects, and trained a DCNN based
on the well-known VGGNet [24] for object recognition, fol-
lowed by triplet embedding for face verification. The trained
DCNN model using VGGFace achieves comparable results on
both still-face (i.e., LFW) and video-face [i.e., YouTube Faces
(YTF)] data sets with other state-of-the-art approaches using
only a single model and a publicly available data set. This dem-
onstrates the usefulness of the VGGFace data set to the face
recognition community.
In a recent work, AbdAlmageed et al. [63] handle pose
variations by training separate DCNN models for frontal, half-
profile, and full-profile poses to improve face recognition per-
formance in unconstrained settings. Furthermore, Masi et al.
[64] utilized 3-D morphable models to augment the CASIA-
WebFace data set with large amounts of synthetic faces to
improve the recognition performance instead of collecting
more data through crowdsourcing the annotation tasks. Ding
et. al. [65] proposed to fuse the deep features around facial
landmarks from different layers followed by a new triplet loss
function that achieves state-of-the-art performance for video-
based face recognition. Wen et. al. [66] proposed a new loss
function that takes the centroid for each class into consideration
and uses it as a regularization constraint to the softmax loss in
addition to a residual neural network for learning more dis-
criminative face representation. Liu et al. [67] proposed a novel
angular loss based on the modified softmax loss. It results in a
discriminative angular feature representation for faces that is
optimized to the commonly used similarity metric and cosine
distance, and the model achieves recognition performance
comparable to state-of-the-art methods that use a much smaller
training data set. Ranjan et al. [68] also trained with softmax
loss regularized with a scaled L -2 norm constraint on a sub-
set of the recently released MS-Celeb-1M face data set. The
authors show that the regularized loss optimizes the angular
margin between classes. The method achieves state-of-the-
art results on the IARPA Benchmark A (IJB-A) data set [69].
Besides the commonly used average aggregation for per-frame
video face representation, Yang et al. proposed a neural aggre-
gated network (NAN) [70] to perform dynamically weighted
aggregation on the features from multiple face images or face
frames of a face video to yield a succinct and robust represen-
tation for video face recognition. The approach demonstrated
state-of-the-art results on multiple image-set and video face
data sets. Bodla et al. [71] proposed a fusion network to com-
bine face representations from two different DCNN models to
improve the recognition performance.
Discriminative metric learning for faces
Learning a classifier or a similarity measure from data is
another key component for improving the performance of a
face recognition system. Many approaches have been proposed
in the literature that essentially exploit the label information
from face images or face pairs. Hu et al. [72] learned a dis-
criminative metric within the deep neural network framework.
Schroff et al. [62] and Parkhi et al. [17] optimized the DCNN
parameters based on the triplet loss, which directly embeds the
DCNN features into a discriminative subspace and presented
promising results for face verification. In [73], a discriminant
low-rank embedding is learned using a probabilistic model for
face verification and clustering. Song et al. [74] proposed a
method to fully utilize the training data in a batch by consider-
ing the full pairwise distances between samples.
Besides supervised learning of DCNNs for face recognition,
Yang et al. [75] proposed learning deep representations and
image clusters jointly in a recurrent framework. Each image is
treated as separate clusters at the beginning, and a deep network
is trained using this initial grouping. Deep representation and
cluster members are then iteratively refined until the number of
clusters reach the predefined value. The unsupervised learned
representation has demonstrated promising results on various
tasks, including face recognition, digit classification, etc. Zhang
et al. [76] proposed to cluster face images in videos by alternat-
ing between deep representation adaption and clustering. Tri-
georgis et al. [77] proposed a deep semisupervised nonnegative
matrix factorization to learn hidden representations that allow
themselves to an interpretation of clustering according to dif-
ferent, unknown attributes of a given face data set, such as pose,
emotion, and identity. Their method also demonstrates promis-
ing results on challenging face data sets. On the other hand,
Lin et al. [78] proposed an unsupervised clustering algorithm
that exploits the neighborhood structure between samples and
implicitly performs domain adaptation to achieve improved
clustering performance. They also demonstrated an application
of the clustering algorithm to curate a large-scale noisy face
data set, such as MS-Celeb-1M [79].
Implementation
Face recognition can be subdivided into two tasks: 1) face veri-
fication and 2) face identification. For the task of face verifica-
tion, given a pair of face images, the system needs to predict
whether or not they belong to the same person. For the task of
face identification, given a face image of unknown identity, the
system should determine the subject’s identity by matching the
features from the database.

For both of these tasks, obtaining discriminative and robust
identity features is important. For the task of face verification,
the faces are first localized by the face detector and normalized
to the canonical coordinate through similarity transform using
the detected fiducial points. Then, each of the face images is
passed through a DCNN to obtain a feature representation.
Once the features are obtained, a score measure is computed
based on a similarity metric. Most commonly used similarity
metrics are 1) L2 distance between the face features and 2)
cosine similarity, which provides a measure of how close the
features are in the angular space. One can also use multiple
DCNNs for improved performance by fusing the network fea-
tures or the similarity scores such as the DeepID framework
[59]–[61] or the fusion network [71] approach. For the task of
face identification, the face images in the gallery are passed
through DCNNs, and the features for each identity are stored
in the database. When a new face image is provided to the
system, it computes its feature representation, and a similarity
score is obtained for each of the identities in the gallery.
Training data sets for face recognition
In Table 1, we summarize the public data sets used for testing
the algorithm performance and for training the DCNN mod-
els on face recognition tasks. MS-Celeb-1M [79] is the cur-
rent largest public face recognition data set, containing more
than 10 million labeled face images of the top 100,000 distinct
identities from the 1 million celebrity list with significant pose,
illumination, occlusion, and other variations. Since this data
set also contains a large amount of label noise, interested read-
ers are referred to [78] for an example of data-set curation by
applying subject clustering. For other data sets, the CelebA data
set [80] is annotated with 40 face attributes and five keypoints by a
professional labeling company for 202,599 face images of 10,000
subjects. The data set can be used for training face recognition,
face attribute classification, and fiducial prediction models. The
CASIA-WebFace [13] is also one of popular public large-scale
annotated data sets for face recognition. It consists of 494,414
face images for 10,575 subjects collected from the IMDB web-
site. TheVGGFace [17] is another public data set for training face
recognition algorithms and consists of 2.6 million face images for
2,600 subjects. The MegaFace [14], [15] is the data set used to
test the robustness of face recognition algorithms in the open-set
setting with 1 million distractors. There are two parts for the data
set—the first one allows the use of any external training data
sets, and the other provides 4.7 million face images of 672,000
subjects. The LFW [81] data set contains 13,233 face images
of 5,749 subjects taken from the Internet, where 1,680 subjects
have two or more images. The data set is mainly used to evalu-
ate the performance of the still face recognition algorithms, and
most of the faces are frontal. The IJB-A [69] data set contains
500 subjects with 5,397 images and 2,042 videos splitting into
20,412 frames. The data set is designed to test the robustness of
the algorithms under large pose, illumination, and image/video
frame quality variations in the mixed-media setting (i.e., each
gallery and probe set consists of one or more images and video
frames). The YTF [82] data set contains 3,425 videos of 1,595
different subjects and is the standard data set used to evalu-
ate video-face recognition algorithms. The PaSC [83] data set
consists of 2,802 videos of 293 subjects and is used to test the
performances of video-face algorithms under large pose, illu-
mination, and blur variations where the videos are captured
in controlled and point-and-shoot manners. The Celebrities
in Frontal-Profile (CFP) data set [84] contains 7,000 images
of 500 subjects and is used for evaluating face verification
approaches on extreme pose variations. The UMDFaces [85]
and UMDFace Video [35] are data sets that contain 367,888
still images of 8,277 subjects and 22,075 videos of 3,107
subjects. The data sets can be used to train still- and video-
face recognition algorithms. The subjects in UMDFace Video
are also in UMDFaces, and this is useful to adapt the models
trained in still-image face data sets to the video domain.
Recently, Bansal et al. [35] studied various features of a
good large-scale data set for face recognition, including the
following questions:
■■ Can we train on still images and expect the systems to
work on videos?
■■ Are deeper data sets better than wider data sets where,
given a set of images, deeper data sets mean more images
per subject, and wider data sets mean more subjects?
■■ Does adding label noise always lead to improvement in
performance of deep networks?
■■ Is alignment needed for face recognition?
The authors used CASIA-WebFace [13], UMDFaces [85] and
its video extension [35] Youtube Face [82], and the IJB-A data
set [69]. They found that DCNN models trained on a combina-
tion of still images and video frames perform better than those
trained on only still images or on only video frames. Based on
the experiments, it appears that, for smaller models, training
using wider data sets is better than training using deeper data
sets, while for deeper models, training using wider data sets is
better. The authors of [35] showed that label noise usually hurts
Table 1. Recent data sets for face recognition.
Face Recognition
Name Number of Faces
Number of
Subjects
MS-Celeb-1M [16] 10 million 100,000
CelebA [80] 202,599 10,177
CASIA-WebFace [13] 494,414 10,575
VGGFace [17] 2.6 million 2,622
Megaface [14], [15] 4.7 million 672,000
LFW [81] 13,233 5,749
IJB-A [69] 25,809 500
YTF [82] 3,425 videos 1,595
PaSC [83] 2,802 videos 293
CFP [84] 7,000 500
UMDFaces [85] 367,888 8,277
UMDFace Video [35] 22,075 videos 3,107

the performance of face recognition. It was also found that
good alignment can greatly help face recognition performance.
Performance summary
We summarize the performance results of recent face identi-
fication and verification algorithms for the well-known LFW
[81] and IJB-A [69] data sets.
LFW data set
The LFW data set contains 13,233 face images of 5,749 subjects
where 1,680 subjects have two or more images.We show the per-
formance comparisons using the standard protocol of face verifi-
cation which defines 3,000 positive pairs and 3,000 negative pairs
in total and further splits them into ten disjoint subsets for cross
validation. Each subset contains 300 positive and 300 negative
pairs. It contains 7,701 images of 4,281 subjects. In Table 2, we
show the mean accuracy of state-of-the-art deep-learning-based
methods: DeepFace [58], DeepID2 [61], DeepID3 [86], FaceNet
[62],Yi et al. [13], Wang et al. [87], Ding et al. [88], Parkhi et al.
[17],Wen et al. [66], Liu et al. [67], Ranjan et al. [68], and human
performance on the “funneled” LFW images.
IJB-A benchmark a data set
Besides the LFW data set, we also present the performance com-
parisons for the challenging IJB-A data set. The IJB-A data set
contains 500 subjects with 5,397 images and 2,042 videos split-
ting into 20,412 frames. Sample images and video frames from
the data sets are shown in Figure 6. The ROC curve measures
the performance in the verification scenarios, and the cumula-
tive match characteristic (CMC) score measures the accuracy in
closed set identification scenarios. In addition, the IJB-A evalu-
ation protocol consists of face verification (1:1 matching) over
ten splits. Each split contains approximately 11,748 pairs of tem-
plates (1,756 positive and 9,992 negative pairs) on average. Sim-
ilarly, the face identification (1:N search) protocol also consists
of ten splits, which are used to evaluate the search performance.
In each search split, there are about 112 gallery templates and
1,763 probe templates (i.e., 1,187 genuine probe templates and
576 impostor probe templates). The training set contains 333
subjects, and the test set contains 167 subjects without any over-
lapping subjects. Ten random splits of training and testing are
provided. Unlike the LFW [81] and YTF [89] data sets, which
only use a sparse set of negative pairs to evaluate the verification
performance, the IJB-A data set divides the images/video frames
into gallery and probe sets so that all the available positive and
negative pairs are used for the evaluation. Also, each gallery and
probe set consist of multiple templates. Each template contains a
combination of images or frames sampled from multiple image
sets or videos of a subject. In contrast to the LFW and YTF data
sets, which only include faces detected by the Viola Jones face
detector [90], the images in the IJB-A contain extreme pose, illu-
mination,andexpressionvariations.Thesefactorsessentiallymake
the IJB-A a challenging face recognition data set.
The CMC scores and the ROC are used to evaluate the perfor-
mance of different algorithms for face identification/verification.
The identification/verification results for the IJB-A data set are
Table 2. The accuracy of different verification methods on the LFW data set.
Method
Number of
Networks Training Set Metric Mean Accuracy ± Std
DeepFace [58] One 4.4 million images of 4,030 subjects, private Cosine 95.92% ± 0.29%
DeepFace Seven 4.4 million images of 4,030 subjects, private Unrestricted, SVM 97.35% ± 0.25%
DeepID2 [61] One 202,595 images of 10,117 subjects, private Unrestricted, JB 95.43%
DeepID2 25 202,595 images of 10,117 subjects, private Unrestricted, JB 99.15% ± 0.15%
DeepID3 [86] 50 202,595 images of 10,117 subjects, private Unrestricted, JB 99.53% ± 0.10%
FaceNet [62] One 260 million images of 8 million subjects, private L2 99.63% ± 0.09%
Yi et al. [13] One 494,414 images of 10,575 subjects, public Cosine 96.13% ± 0.30%
Yi et al. One 494,414 images of 10,575 subjects, public Unrestricted, JB 97.73% ± 0.31%
Wang et al. [87] One 494,414 images of 10,575 subjects, public Cosine 96.95% ± 1.02%
Wang et al. Seven 494,414 images of 10,575 subjects, public Cosine 97.52% ± 0.76%
Wang et al. One 494,414 images of 10,575 subjects, public Unrestricted, JB 97.45% ± 0.99%
Wang et al. Seven 494,414 images of 10,575 subjects, public Unrestricted, JB 98.23% ± 0.68%
Ding et al. [88] Eight 471,592 images of 9,000 subjects, public Unrestricted, JB 99.02% ± 0.19%
Parkhi et al. [17] One 2.6 million images of 2,622 subjects, public Unrestricted, TDE 98.95%
Wen et al. [66] One 0.7 million images of 17,189 subjects, public Cosine 99.28%
Liu et al. [67] One 494,414 images of 10,575 subjects, public Cosine 99.42%
Ranjan et al. [68] One 3.7 million images of 58,207 subjects, public Cosine 99.78%
Human, funneled [87] N/A N/A N/A 99.20%

shown in Table 3. In addition to using the average feature repre-
sentation, we also perform media averaging which is to first aver-
age the features from the same media (image or video) and then
further average the media average features to generate the final
feature representation followed by triplet probabilistic embed-
ding proposed in [73].
Table 3 summarizes the scores (i.e., both ROC and CMC
numbers) produced by different face identification/verification
methods on the IJB-A data set. We compare the results with
■■ DCNNcasia [87]
■■ DCNNbl (bilinear CNN [92])
■■ DCNNpose (multipose DCNN models [63]), [70])
■■ DCNN d3 [64]
■■ template adaptation (TP) [93]
■■ DCNNtpe [73]
■■ DCNNall [2] (All-in-One Face)
■■ DCNNl tpe2 + [68]
■■ [91], reported recently by the National Institute of Standards
and Technology.
We also summarize the details of each method in Table 4.
Facial attributes
From a single face, we are able to identify facial attributes such as
gender, expression, age, skin tone, etc. Those attributes are very
usefulforapplicationslikeimageretrieval,emotiondetection,and
mobile security. In biometrics literature, facial attributes are also
Figure 6. The IJB-A data set contains faces in large variations of pose, illumination, image quality, occlusion, etc.
Table 3. The results on the IJB-A data set. The TAR of all the approaches at FAR = 0.1, 0.01, and 0.001 for the ROC curves (IJB-A 1:1 verification).
The Rank-1 and Rank-10 retrieval accuracies of the CMC curves (IJB-A 1:N identification). We report average and standard deviation of the ten splits
for All-in-One Face [2], DCNNall, and, after triplet probabilistic embedding (TPE), DCNNall+tpe . All of the performance results reported in [91], DCNNcasia
[87], Janus B (JanusB-092015), DCNNbl [92], DCNNfusion [12], DCNN3d [64], NAN [70], DCNNpose [63], DCNNtpe [73], TP [93], and DCNNl2+tpe [68],
where RX101 means ResNeXt-101 [94].
IJB-A-Verif DCNNcasia [87] JanusB [91] DCNNpose [63] DCNNbl [92] NAN [70] DCNN3d [64]
FAR = 1e-3 0.514 0.65 — — 0.881 0.725
FAR = 1e-2 0.732 0.826 0.787 — 0.941 0.886
FAR = 1e-1 0.895 0.932 0.911 — 0.978 —
IJB-A-Ident DCNNcasia [87] JanusB [91] DCNNpose [63] DCNNbl [92] NAN [70] DCNN3d [64]
Rank-1 0.820 0.87 0.846 0.895 0.958 0.906
Rank-10 — 0.95 0.947 — 0.986 0.977
IJB-A-Verif DCNNfusion [12] DCNNtpe [73] DCNNall [2] DCNNall+tpe [2] TP [93] DCNNl2+tpe(RX101) [68]
FAR = 1e-3 0.76 0.813 0.787 0.823 — 0.943
FAR = 1e-2 0.889 0.9 0.893 0.922 0.939 0.970
FAR = 1e-1 0.968 0.964 0.968 0.976 — 0.984
IJB-A-Ident DCNNfusion [12] DCNNtpe [73] DCNNall [2] DCNNall+tpe [2] TP [93] DCNNl2+tpe(RX101) [68]
Rank-1 0.942 0.932 0.941 0.947 0.928 0.973
Rank-10 0.988 0.977 0.988 0.988 0.986 0.988

referred to as soft biometrics [95]. Kumar et al. [56] introduced
the concept of attributes as image descriptors for face verification.
They used a collection of 65 binary attributes to describe each
face image. Berg et al. created classifiers for each pair of people
in a data set and then used these classifiers to create features for a
face verification classifier [56]. Here, rather than manually iden-
tifying attributes, each person was described by their likeness to
other people. This is a way of automatically creating a set of attri-
butes without having to exhaustively hand-label attributes on a
large data set. Recently, DCNNs have been used for attribute
classification, demonstrating impressive results. Pose Aligned
Networks for Deep Attributes (PANDA) achieved state-of-the-
art performance by combining part-based models with deep
learning to train pose-normalized DCNNs for attribute classi-
fication [96]. Focusing on age and gender, [97] applied DCNNs
to the Adience data set. Liu et al. used two DCNNs—one for
face localization and the other for attribute recognition—and
achieved impressive results on the CelebA and LFWA data
sets, outperforming PANDA on many attributes [80].
Instead of treating each attribute as independent, [99]
exploits the correlation among attributes to improve image
ranking and retrieval by using independently trained attribute
classifiers and then learning pairwise correlations based on the
outputs of these classifiers. Hand et al. [100] train a single attri-
bute network that classifies 40 attributes, sharing information
throughout the network, by learning the relationships among
all the 40 attributes, not just attribute pairs. Ranjan et al. [2]
train a single deep network with MTL that can simultaneously
perform face detection, facial landmark detection, face recog-
nition, 3-D head-pose estimation, gender classification, age
estimation, and smile detection. Recently, Gunther et al. pro-
posed the Alignment-Free Facial Attribute Classification Tech-
nique (AFFACT) [101] algorithm that performs alignment-free
facial attribute classification. It uses a data augmentation tech-
nique that allows a network to classify facial attributes without
requiring alignment beyond detected face bounding boxes.
The algorithm achieves state-of-the-art results on the CelebA
[80] data set with an ensemble of three networks.
In addition, some facial attributes can be used to boost mobile
authentication performance [17]. The recent method of attribute-
based continuous authentication [102], [103] shows that large
number of attributes by themselves can give good authentica-
tion results on mobile phones. They are semantic features, which
are easier to learn than facial identities. Also, if they are learned
on facial parts, they become less complex. By leveraging these
two qualities, Samangouei et. al. [98] designed efficient DCNN
architectures suitable for mobile devices. The pipeline in Figure 7
illustrates how facial attributes are used for mobile authentication.
MTL for facial analysis
In this section, we present the design details of various MTL
methods for different facial analysis tasks. The MTL frame-
work for machine learning was first analyzed by Caruana [104]
in detail. Since then, it has been used for solving multiple prob-
lems in computer vision. One of the earlier works of MTL-based
facial analysis was proposed by Zhu et al. [55]. The algorithm
was used for solving the tasks of face detection, landmarks local-
ization, and head-pose estimation. It used a mixture of a trees
model with a shared pool of parts, where a part represents a
landmark location. Another method called JointCascade [105]
improved the face detection algorithm by combining the training
with the landmarks localization task. Both these methods used
handcrafted features, which made it difficult to extend the MTL
approach to a wide variety of tasks.
Table 4. A comparison of face recognition algorithms on the IJB-A data set, where OSS-SVM [93] means SVM with the OSS framework,
TSE [12] means triplet similarity embedding, OSS-SVM [93] means SVM using the OSS framework, and AttentionNet means the attention network
that is used to compute the weights of each face for feature aggregation and substitute the average representation in all of the other compared
methods. The boldface in Rank-1 and Rank-10 signify the best results for the respective metrics.
Method Base Network Loss Training Set Metric Rank-1 Rank-10
DCNNcasia [87] CASIANet Softmax 494,414 images of 10,575 subjects, public [13] JB 82 —
DCNNpose [63] VGGNet Softmax 494,414 images of 10,575 subjects, public [13] cosine 84.6 94.7
DCNNbl [92] VGGNet Softmax 494,414 images of 10,575 subjects, public [13] SVM 89.5 —
NAN [70] GoogLeNet(BN) +
AttentionNet
Contrastive 3 million images of 50,000 subjects, private [70] cosine 95.8 98.6
DCNN d3 [64] VGGNet Softmax 2,472,070 synthesized images of 10,575 subjects using a
3-D morphable model (of which 494,414 images are natu-
ral face images), public [13]
cosine 90.6 97.7
DCNNfusion [12] AlexNet+CASIANet Softmax 494,414 images of 10,575 subjects, public [13] TSE 94.2 98.8
DCNNtpe [73] AlexNet Softmax 494,414 images of 10,575 subjects, public [13] TPE 93.2 97.7
DCNNall tpe+ [2] AlexNet Multitask 993,153 images from multiple data sets, (only 494,414
images out of them contain identity labels from 10,575
subjects), public [2]
TPE 94.7 98.8
TP [93] VGGNet Softmax 2.6 million images of 2,622 subjects, public [17] OSS-SVM 92.8 98.6
DCNNl tpe2 + [68] ResNet L -2 Softmax 3.7 million images of 58,207 subjects, public [79] TPE 97.3 98.8

Before the advent of deep-learning methods, MTL was
restricted to a few sets of tasks because the feature representa-
tions required for solving each of the tasks were different. For
instance, the task of face detection typically used HOG, whereas
the task of face recognition used LBPs. Similarly, for the tasks of
landmarks localization, age and gender estimation, and attributes
classification, different handcrafted features were derived. How-
ever, with the rise of deep learning, the handcrafted features were
replaced by DCNN features for solving the aforementioned facial
analysis tasks. This made it possible to train a single DCNN
model for multiple tasks such as face detection, landmarks local-
ization, facial attributes prediction, and face recognition.
In general, when a human looks at a face in an image, he/
she can detect where the face is and determine the gender, rough
pose, age, expressions, etc. When machines are designed to per-
form these tasks, one often constructs independent algorithms
for solving each. However, one can design a deep network that
can simultaneously accomplish all of these tasks by sharing the
deep features and exploiting the relationships among these tasks.
Goodfellow et al. [106] interprets MTL as a regularization meth-
od for DCNNs. With the MTL approach, the learned parameters
of a network are in consensus with all the tasks in hand, which
reduces overfitting and converges to a robust solution.
HyperFace [10] and the Tasks-Constrained Deep Convolu-
tional Network (TCDCN) [107] are some of the earliest methods
to use DCNNs in the MTL framework. HyperFace was proposed
for solving the tasks of face detection, landmarks localization,
head-pose estimation, and gender classification. It fused the
intermediate layers of a DCNN such that the tasks could lever-
age semantically rich features from the deeper layers as well as
location-specific features from the lower layers. The synergy
developed by the MTL approach improved the performance
of individual tasks. Zhang et al. [107] proposed the TCDCN
algorithm that used MTL-based DCNN for facial landmark
detection, along with the tasks of discrete head yaw estimation,
gender recognition, smile prediction, and glass detection. In their
method, the predictions for all theses tasks were pooled from the
same feature space. They showed that using auxiliary tasks such
as glass detection and smile prediction improved the landmarks
localization for specific parts of the face.
Ranjan et al. recently proposed All-in-One Face [2], which
is a single DCNN model for simultaneous face detection, land-
mark localization, face recognition, 3-D head-pose estimation,
smile detection, facial age estimation, and gender classification.
The All-in-One Face architecture [illustrated in Figure 8(a)]
starts with the pretrained face identification network from San-
karanarayanan et al. [73]. The network consists of seven convo-
lutional layers followed by three fully connected layers, and it is
used as a backbone network for training the face identification
task and sharing the parameters of its first six convolution layers
with other face-related tasks. The central tenet is that a CNN pre-
trained on a face identification task provides better initialization
for a generic face analysis task, since the filters retain discrimina-
tive face information.
To leverage multiple data sets with all of the annotations for
the face bounding box, fiducial points, pose, gender, age, smile,
and identity information, multiple subnetworks are trained with
respect to task-related data sets and share the parameters among
them, since no single data set contains annotations for all of the
facial analysis tasks at the same time. In this way, the shared
parameters adapt to the complete set of domains instead of fit-
ting to a task-specific domain. At test time, these subnetworks
Efficient Deep Part-Based Attribute Detection
CNNAA
Chubby
Beard
Mustache
Blond
Male
Eyeglasses
Chubby
Beard
Mustache
Blond
Male
Eyeglasses
0.4
0.4
0.8
1
==
Yes
No
Continue Access
Lock the Phone
Enrolled
Attributes
Figure 7. A sample pipeline of a face attribute detection algorithm for mobile authentication [98]. CNNAA: CNN for Facial Attribute-Based Active Authentication.

Figure 8. (a) The All-in-One Face network architecture and (b) an overview of the All-in-One Face system for face recognition and facial analysis.
Face Region (227 × 227 × 3)
11 × 11 Conv, 96, /4
5 × 5 Conv, 256, /2
2×2Conv,
256,/2
4×4Conv,
256,/4
1×1Conv,256
fc2,048
fc512fc512
fc21
fc512
fc2fc42
3 × 3 Conv, 384
3 × 3 Conv, 384
3 × 3 Conv, 512
3 × 3 Conv, 512
3 × 3 Conv, 512
Pool, /2
Pool,/2
Pool, /2
Norm +Pool, /2
Norm +Pool, /2
fc 1,024
fc 512
fc 10,548 (Age)
(Gender Probability)
fc 1
fc 128
fc 1,024
fc 2
fc 128
fc 1,024
(Identity Descriptor)
fc512
(Smile
Probability)
fc2
fc512
fc3
(Roll,Pitch,Yaw)(Keypoints)(Visibility)
(Detection
Probability)
Input Image Region Proposals
All-In-One
CNN
All-In-One
CNN
All-In-One
CNN
Detected Faces, Fiducial
Points, Pose Estimates
Aligned
Faces
Identify :
Age :
Gender :
Smiling :
Y. Shinawatra
39 Years
Female
No
Identify :
Age :
Gender :
Smiling :
B. Aquino III
52 Years
Male
Yes
(a)
(b)

are fused together into a single All-in-One Face. Table 5 lists
the different data sets used for training the All-in-One Face. Task-
specific loss functions are used to train the complete network
end to end. Some sample outputs for All-in-One Face are shown
in Figure 9.
MTL-based DCNNs have also been used for determining
multiple facial attributes. Dehghan et al. proposed Deep Age,
Gender, and Emotion Recognition (DAGER) [111] for recog-
nizing age, gender, and emotion using DCNNs. Similar to All-
in-One Face [2], it uses different training sources labeled for
different tasks for training the DCNN. He et al. [112] jointly
learn a deep architecture for face detection and facial attri-
bute analysis. Unlike other MTL-based methods, they use full
image as an input to the network instead of a face region. A
faster R-CNN based approach is used to obtain the detections
along with the attributes for the faces. Table 6 summarizes the
facial analysis tasks performed by some of the recent MTL-
based methods.
Open issues
We briefly discussed several design considerations for each
component of an automated face identification/verification
system, including the following:
■■ Face detection: In contrast to a generic object detection task,
face detection is more challenging due to the wide range of
variations in the appearance of faces. The variability is
caused mainly by changes in illumination, facial expression,
viewpoints, occlusions, etc. Other factors such as blur and
low resolution challenge the face detection task.
■■ Fiducial detection: Most data sets contain only a few thou-
sand images. A large-scale annotated and unconstrained data
set will make the face alignment system more robust to the
challenges, including extreme pose, low illumination, and
small, blurry face images. Researchers have hypothesized
that deeper layers of CNNs can encode more abstract infor-
mation such as identity, pose, and attributes; however, it has
not yet been thoroughly studied which layers exactly corre-
spond to local features for fiducial detection.
■■ Face identification/verification: For face identification/ver-
ification, the performance can be improved by learning a
discriminative distance measure. However, due to memory
constraints limited by graphics cards, how to choose infor-
mative pairs or triplets and train the network end to end
using online methods (e.g., stochastic gradient descent) on
large-scale data sets is still an open problem. Another chal-
lenging problem to address is to incorporate full motion
video processing in deep networks for enabling video-based
face analytics.
Conclusions
In this article, we presented an overview of recent develop-
ments in designing an automatic face recognition system. We
also discussed various multitask CNN-based approaches for
face analytics. The examples shown in the article demonstrate
that subject-independent tasks benefit from MTL, and network
initialization from the face recognition task. They also show
howthese tasks help to improve face recognition performance.
Given the sufficient amount of annotated data and GPUs,
DCNNs have been shown to yield impressive performance
improvements. Still many issues remain to be addressed to
make the DCNN-based systems robust and practical, such as
reducing reliance on large training data sets, handling data bias
and degradation in training data, incorporating domain knowl-
edge, reducing the training time when the network goes deeper
and wider, and building the theoretical foundations to under-
stand the characteristics and behaviors of DCNN models. We
refer the interested readers to [12] for more details about other
challenges for DCNN models.
Table 5. Data sets used for training All-in-One Face.
Data Set Facial Analysis Task
Number of
Training Samples
CASIA [13] Identification, gender 490,356
MORPH [108] Age, gender 55,608
IMDB+WIKI [109] Age, gender 224,840
Adience [97] Age 19,370
CelebA [80] Smile, gender 182,637
AFLW [110] Detection, pose, fiducials 20,342
Total 993,153
Table 6. A list of various MTL-based facial analysis algorithms along with the types of face tasks they can perform.
Method Face Detection Fiducials Head Pose Gender Age Expression Other Attributes Face Recognition
Zhu et al. [55] ✓ ✓ ✓
JointCascade [105] ✓ ✓
Zhang et al. [32] ✓ ✓
TCDCN [107] ✓ ✓ ✓ ✓ ✓
HyperFace [10] ✓ ✓ ✓ ✓
He et al. [112] ✓ ✓ ✓ ✓ ✓
DAGER [111] ✓ ✓ ✓
All-in-One Face [2] ✓ ✓ ✓ ✓ ✓ ✓ ✓

Figure 9. (a)–(f) Sample results of the All-in-One Face [2] for the IJB-A data set [69] with detected face bounding boxes, fiducial points, and identity
along with 3-D head pose, gender, smile, and facial age estimation. Although the algorithm predicts identity, age, gender and smile attributes for all faces,
we show them only for subjects that are present in the IJB-A data set for better image clarity.
Y. Shinawatra
Age: 39
Gender: Female
Smiling: No
Kurt Warner
Age: 35
Gender: Male
Smiling: No
Angelina Jolie
Age: 43
Gender: Female
Smiling: No
Ellen Johnson Sirleaf
Age: 71
Gender: Female
Smiling: Yes
Helen Clark
Age: 35
Gender: Female
Smiling: Yes
Sarah-Jane Dias
Age: 23
Gender: Female
Smiling: Yes
Riteish Deshmukh
Age: 29
Gender: Male
Smiling: Yes
Park Geun-hye
Age: 48
Gender: Female
Smiling: Yes
Barack Obama
Age: 55
Gender: Male
Smiling: Yes
B. Aquino III
Age: 51
Gender: Male
Smiling: Yes
(a) (b)
(c) (d)
(e) (f)

Acknowledgments
We acknowledge the help of Anirudh Nanduri, Amit Kumar,
Wei-An Lin, Pouya Samangouei, Emily M. Hand, and Ching-
Hui Chen. This research is based upon work supported by the
Office of the Director of National Intelligence (ODNI), Intel-
ligence Advanced Research Projects Activity (IARPA), via
IARPA R&D contract number 2014-14071600012. The views
and conclusions contained herein are those of the authors and
should not be interpreted as necessarily representing the offi-
cial policies or endorsements, either expressed or implied, of
the ODNI, IARPA, or the U.S. Government. The U.S. Gov-
ernment is authorized to reproduce and distribute reprints for
governmental purposes notwithstanding any copyright annota-
tion thereon.
Authors
Rajeev Ranjan (rranjan1@umd.edu) received his B.Tech
degree in electronics and electrical communication engineer-
ing from the Indian Institute of Technology, Kharagpur, in
2012. He is a research assistant at the University of Maryland
(UMD), College Park. His research interests include face
detection, face recognition, and machine learning. He received
the 2015 UMD Outstanding Invention of theYear Award.
Swami Sankaranarayanan (swamiviv@umiacs.umd.edu)
received his B.Tech degree in electronics and communication
engineering from the National Institute of Technology, Trichy,
India, in 2010 and his M.S. degree specializing in pattern rec-
ognition and machine learning from the Delft University of
Technology, The Netherlands, in 2012. He is a Ph.D. candi-
date at the University of Maryland, College Park. His current
research interests include face analysis and adversarial
machine learning.
Ankan Bansal (ankan@umiacs.umd.edu) received his
B.Tech. and M.Tech. degrees in electrical engineering from
the Indian Institute of Technology, Kanpur, in 2015. Currently,
he is a Ph.D. degree student at the University of Maryland,
College Park. His research interests include multimodal learn-
ing, action understanding, and face analysis. He was awarded
the Clark School of Engineering Distinguished Graduate
Fellowship in 2015–2016.
Navaneeth Bodla (nbodla@umiacs.umd.edu) received his
B.E (honors) degree in electrical and electronics engineering
from Birla Institute of Technology and Science, Pilani, India,
in 2012 and his M.S. degree in computer engineering from the
University of Florida, Gainesville, in 2014. His research inter-
ests include face recognition, object detection, and adversarial
image synthesis. Currently, he is a research assistant at the
University of Maryland, College Park.
Jun-Cheng Chen (pullpull@umiacs.umd.edu) received
his bachelor’s and master’s degrees in 2004 and 2006, respec-
tively, both from Department of Computer Science and
Information Engineering, National Taiwan University, Taipei.
He received his Ph.D. degree from the University of Maryland,
College Park, in 2016. He is a postdoctoral research fellow at
the University of Maryland Institute for Advanced Computer
Studies. His current research interests include computer vision
and machine learning with applications to face recognition
and facial analysis. He was a recipient of the 2006 Association
for Computing Machinery Multimedia Best Technical Full
Paper Award.
Vishal M. Patel (vishal.m.patel@rutgers.edu) received his
B.S. degrees in electrical engineering and applied mathemat-
ics (with honors) and his M.S. degree in applied mathematics
from North Carolina State University, Raleigh, in 2004 and
2005, respectively. He is an A. Walter Tyson Assistant Professor
in the Department of Electrical and Computer Engineering at
Rutgers University, Piscataway, New Jersey. Prior to joining
Rutgers, he was a member of the research faculty at the
University of Maryland Institute for Advanced Computer
Studies. His research interests are in signal processing, comput-
er vision, and machine learning with applications to radar imag-
ing and biometrics. He received the 2016 Office of Naval
Research Young Investigator Award and the 2010 Oak Ridge
Associated Universities postdoctoral fellowship.
Carlos D. Castillo (carlos@umiacs.umd.edu) received his
B.S. degree in computer science from the Universidad Simón
Bolívar in Caracas, Venezuela, in 2003 and his M.S. and Ph.D.
degrees in 2007 and 2012, respectively, from the University of
Maryland, College Park. He is an assistant research scientist at
the University of Maryland Institute for Advanced Computer
Studies. His current research interests include stereo matching,
multiview geometry, face detection, alignment, and recognition.
Rama Chellappa (rama@umiacs.umd.edu) received his
B.E. (honors) degree in electronics and communication engi-
neering from the University of Madras, India, in 1975 and his
M.E. (with distinction) degree from the Indian Institute of
Science, Bangalore, in 1977. He received his M.S.E.E. and
Ph.D. degrees in electrical engineering from Purdue University,
West Lafayette, Indiana, in 1978 and 1981, respectively. He is a
Distinguished University Professor, a Minta Martin Professor of
Engineering, and the chair of the Electrical and Computer
Engineering Department at the University of Maryland, College
Park. His current research interests are face and gait analysis,
three-dimensional modeling from video, image- and video-based
recognition and exploitation, compressive sensing, and hyper-
spectral processing. He has received the Society, Technical
Achievement, and Meritorious Service Awards from the IEEE
Signal Processing Society.
References
[1] W. Y. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld, “Face recognition: a
literature survey,” ACM Comput. Surveys, vol. 35, no. 4, pp. 399–458, 2003.
[2] R. Ranjan, S. Sankaranarayanan, C. D. Castillo, and R. Chellappa, “An all-in-
one convolutional neural network for face analysis,” in Proc. IEEE Int. Conf.
Automatic Face Gesture Recognition, 2017, pp. 17–24.
[3] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with
deep convolutional neural networks,” in Proc. Advances Neural Information
Processing Systems Conf., 2012, pp. 1097–1105.
[4] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V.
Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” arXiv Preprint,
arXiv:1409.4842, 2014.
[5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recogni-
tion,” in Proc. IEEE Conf. Computer Vision Pattern Recognition, 2016, pp. 770–778.
[6] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for
accurate object detection and semantic segmentation,” in Proc. IEEE Conf.
Computer Vision Pattern Recognition, 2014, pp. 580–587.

[7] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time
object detection with region proposal networks,” in Proc. Advances Neural
Information Processing Systems Conf., 2015, pp. 91–99.
[8] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg,
“SSD: Single shot multibox detector,” in Proc. European Conf. Computer Vision,
2016, pp. 21–37.
[9] R. Ranjan, V. M. Patel, and R. Chellappa, “A deep pyramid deformable part
model for face detection,” in Proc. IEEE 7th Int. Conf. Biometrics Theory,
Applications and Systems, 2015, pp. 1–8.
[10] R. Ranjan, V. Patel, and R. Chellappa, “Hyperface: a deep multi-task learning
framework for face detection, landmark localization, pose estimation, and gender
recognition,” arXiv Preprint, arXiv:1603.01249, 2016.
[11] A. Kumar, R. Ranjan, V. Patel, and R. Chellappa, “Face alignment by local
deep descriptor regression,” arXiv Preprint, arXiv:1601.07950, 2016.
[12] J. Chen, R. Ranjan, S. Sankaranarayanan, A. Kumar, C. Chen, V. M. Patel,
C. D. Castillo, and R. Chellappa, “Unconstrained still/video-based face verifica-
tion with deep convolutional neural networks,” Int. J. Comput. Vis., pp. 1–20.
2017.
[13] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from
scratch,” arXiv Preprint, arXiv:1411.7923, 2014.
[14] I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller, and E. Brossard, “The
megaface benchmark: 1 million faces for recognition at scale,” in Proc. IEEE Conf.
[15] A. Nech and I. Kemelmacher-Shlizerman, “Level playing field for million
scale face recognition,” in Proc. IEEE Int. Conf. Computer Vision Pattern
Recognition, 2017, pp. 873–4882.
[16] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, “Ms-celeb-1m: A data set and
benchmark for large-scale face recognition,” in Proc. European Conf. Computer
Vision, 2016, pp. 87–102.
[17] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in Proc.
British Machine Vision Conf., vol. 1, no. 3, 2015, p. 6.
[18] S. Yang, P. Luo, C.-C. Loy, and X. Tang, “Wider face: A face detection bench-
mark,” in Proc. IEEE Conf. Computer Vision Pattern Recognition, 2016, pp.
5525–5533.
[19] H. Jiang and E. Learned-Miller, “Face detection with the faster R-CNN,” arXiv
Preprint, arXiv:1606.03473, 2016.
[20] J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W. Smeulders, “Selective
search for object recognition,” Int. J. Comput. Vis., vol. 104, no. 2, pp. 154–171,
2013.
[21] Y. Li, B. Sun, T. Wu, and Y. Wang, “Face detection with end-to-end integration
of a convnet and a 3D model,” in Proc. European Conf. Computer Vision, 2016,
pp. 420–436.
[22] D. Chen, G. Hua, F. Wen, and J. Sun, “Supervised transformer network for effi-
cient face detection,” in Proc. European Conf. Computer Vision, 2016, pp. 122–
138.
[23] M. Najibi, P. Samangouei, R. Chellappa, and L. Davis, “SSH: Single stage
headless face detector,” arXiv Preprint, arXiv:1708.03979, 2017.
[24] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-
scale image recognition,” arXiv Preprint, arXiv:1409.1556, 2014.
[25] S. S. Farfade, M. J. Saberian, and L.-J. Li, “Multi-view face detection using
deep convolutional neural networks,” in Proc. ACM Int. Conf. Multimedia
Retrievals, 2015, pp. 643–650.
[26] S. Yang, P. Luo, C.-C. Loy, and X. Tang, “From facial parts responses to face
detection: A deep learning approach,” in Proc. IEEE Int. Conf. Computer Vision,
2015, pp. 3676–3684.
[27] H. Li, Z. Lin, X. Shen, J. Brandt, and G. Hua, “A convolutional neural network
cascade for face detection,” in Proc. IEEE Conf. Computer Vision Pattern
Recognition, 2015, pp. 5325–5334.
[28] S. Yang, Y. Xiong, C. C. Loy, and X. Tang, “Face detection through scale-
friendly deep convolutional networks,” arXiv Preprint, arXiv:1706.02863,
2017.
[29] S. Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, and S. Z. Li, “S3 FD: Single shot
scale-invariant face detector,” arXiv Preprint, arXiv:1708.05237, 2017.
[30] V. Jain and E. Learned-Miller, “FDDB: A benchmark for face detection in
unconstrained settings,” Tech. Rep. vol. 88, Univ. Massachusetts, Amherst,
UM-CS-2010-009, 2010.
[31] B. Yang, J. Yan, Z. Lei, and, S. Z. Li, “Fine-grained evaluation on face detec-
tion in the wild,” in, Proc. 11th IEEE Int. Conf. WorkshopsAutomatic Face and
Gesture Recognition, vol. 1, 2015, pp. 1–7.
[32] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment
using multitask cascaded convolutional networks,” IEEE Signal Processing Lett.,
vol. 23, no. 10, pp. 1499–1503, 2016.
[33] P. Hu and D. Ramanan, “Finding tiny faces,” arXiv Preprint,
arXiv:1612.04402, 2016.
[34] S. Zafeiriou, C. Zhang, and Z. Zhang, “A survey on face detection in the wild:
Past, present and future,” Comput. Vis. Image Understand., vol. 138, pp. 1–24,
Sept. 2015.
[35] A. Bansal, C. D. Castillo, R. Ranjan, and R. Chellappa, “The do’s and don’ts
for CNN-based face verification,” arXiv Preprint, arXiv:1705.07426, 2017.
[36] N. Wang, X. Gao, D. Tao, H. Yang, and X. Li, “Facial feature point detection:
A comprehensive survey,” Neurocomputing, June 2017.
[37] G. G. Chrysos, E. Antonakos, P. Snape, A. Asthana, and S. Zafeiriou, “A com-
prehensive performance evaluation of deformable face tracking in-the-wild,” Int. J.
Comput. Vis., pp. 1–35, 2016.
[38] S. Zhu, C. Li, C.-C. Loy, and X. Tang, “Unconstrained face alignment via cas-
caded compositional learning,” in Proc. IEEE Conf. Computer Vision Pattern
Recognition, 2016, pp. 3409–3417.
[39] S. Ren, X. Cao, Y. Wei, and J. Sun, “Face alignment at 3000 fps via regressing
local binary features,” in Proc. IEEE Conf. Computer Vision Pattern Recognition,
June 2014, pp. 1685–1692.
[40] X. Xiong and F. D. la Torre, “Supervised descent method and its applications to
face alignment,” in Proc. IEEE Conf. Computer Vision Pattern Recognition, 2013,
pp. 532–539.
[41] V. Kazemi and J. Sullivan, “One millisecond face alignment with an ensemble
of regression trees,” in Proc. IEEE Conf. Computer Vision Pattern Recognition,
2014, pp. 1867–1874.
[42] X. Burgos-Artizzu, P. Perona, and P. Dollár, “Robust face landmark estimation
under occlusion,” in Proc. IEEE Int. Conf. Computer Vision, 2013, pp. 1513–1520.
[43] E. Antonakos, J. Alabort-i Medina, and S. Zafeiriou, “Active pictorial struc-
tures,” in Proc. IEEE Conf. Computer Vision Pattern Recognition, 2015, pp. 5435–
5444.
[44] A. Jourabloo and X. Liu, “Pose-invariant 3D face alignment,” in Proc. IEEE
Int. Conf. Computer Vision, 2015, pp. 3694–3702.
[45] A.Jourabloo and X. Liu, “Large-pose face alignment via CNN-based dense 3D
model fitting,” in Proc. IEEE Conf. Computer Vision Pattern Recognition, 2016,
pp. 4188–4196.
[46] X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li, “Face alignment across large
poses: A 3D solution,” in Proc. IEEE Conf. Computer Vision Pattern Recognition,
2016, pp. 146–155.
[47] Y. Sun, X. Wang, and X. Tang, “Deep convolutional network cascade for
facial point detection,” in Proc. IEEE Conf. Computer Vision Pattern Recognition,
2013, pp. 3476–3483.
[48] J. Zhang, S. Shan, M. Kan, and X. Chen, “Coarse-to-fine auto-encoder net-
works for real-time face alignment,” in Proc. European Conf. Computer Vision,
2014, pp. 1–16.
[49] X. Xiong and F. D. la Torre, “Global supervised descent method,” in Proc.
IEEE Conf. Computer Vision Pattern Recognition, 2015, pp. 2664–2673.
[50] G. Trigeorgis, P. Snape, M. A. Nicolaou, E. Antonakos, and S. Zafeiriou,
“Mnemonic descent method: A recurrent process applied for end-to-end face align-
ment,” in Proc. IEEE Conf. Computer Vision Pattern Recognition, 2016, pp. 4177–
4187.
[51] A. Bulat and G. Tzimiropoulos, “Convolutional aggregation of local evidence
for large pose face alignment,” in Proc. British Machine Vision Conference
(BMVC), Sept. 2016, pp. 86.1–86.12.
[52] A. Kumar, A. Alavi, and R. Chellappa, “Kepler: Keypoint and pose estimation
of unconstrained faces by learning efficient H-CNN regressors,” in Proc. IEEE Int.
Conf. Automatic Face Gesture Recognition, 2017. doi: 10.1109/FG.2017.149
[53] C. Sagonas, E. Antonakos, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic,
“300 Faces in-the-wild challenge: database and results,” Image Vis. Comput., vol.
47, pp. 3–18, Mar. 2016.
[54] T. Hassner, S. Harel, E. Paz, and R. Enbar, “Effective face frontalization in
unconstrained images,” in Proc. IEEE Int. Conf. Computer Vision Pattern
Recognition, 2015, pp. 4295–4304.
[55] X. Zhu and D. Ramanan, “Face detection, pose estimation, and landmark local-
ization in the wild,” in Proc. IEEE Conf. Computer Vision Pattern Recognition,
June 2012, pp. 2879–2886.
[56] E. Learned-Miller, G. B. Huang, A. RoyChowdhury, H. Li, and G. Hua,
“Labeled faces in the wild: A survey,” in Proc. Advances Face Detection Facial
Image Analysis Conf., 2016, pp. 189–248.
[57] G. B. Huang, H. Lee, and E. Learned-Miller, “Learning hierarchical represen-
tations for face verification with convolutional deep belief networks,” in Proc. IEEE
Int. Conf. Computer Vision Pattern Recognition, 2012, pp. 2518–2525.
[58] Y. Taigman, M. Yang, M. A. Ranzato, and L. Wolf, “Deepface: Closing the
gap to human-level performance in face verification,” in Proc. IEEE Conf.
[59] Y. Sun, X. Wang, and X. Tang, “Deep learning face representation from pre-
dicting 10000 classes,” in Proc. IEEE Conf. Computer Vision Pattern Recognition,
2014, pp. 1891–1898.

[60] Y. Sun, Y. Chen, X. Wang, and X. Tang, “Deep learning face representation by
joint identification-verification,” in Proc. Advances Neural Information Processing
Systems Conf., 2014, pp. 1988–1996.
[61] Y. Sun, X. Wang, and X. Tang, “Deeply learned face representations are sparse,
selective, and robust,” arXiv Preprint, arXiv:1412.1265, 2014.
[62] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for
face recognition and clustering,” arXiv Preprint, arXiv:1503.03832, 2015.
[63] W. AbdAlmageed, Y. Wu, S. Rawls, S. Harel, T. Hassne, I. Masi, J. Choi, J.
Lekust, J. Kim, P. Natarajana, R. Nevatia, and G. Medioni, “Face recognition using
deep multi-pose representations,” in Proc. IEEE Winter Conf. Applications
Computer Vision, 2016, pp. 1–9.
[64] I. Masi, A. T. Tran, J. T. Leksut, T. Hassner, and G. Medioni, “Do we really
need to collect millions of faces for effective face recognition?” arXiv Preprint,
arXiv:1603.07057, 2016.
[65] C. Ding and D. Tao, “Trunk-branch ensemble convolutional neural networks
for video-based face recognition,” arXiv Preprint, arXiv:1607.05427, 2016.
[66] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning
approach for deep face recognition,” in Proc. European Conf. Computer Vision,
2016, pp. 499–515.
[67] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hyper-
sphere embedding for face recognition,” in Proc. IEEE Int. Conf. Computer Vision
Pattern Recognition, 2017, pp. 212–220.
[68] R. Ranjan, C. D. Castillo, and R. Chellappa, “L2-constrained softmax loss for
discriminative face verification,” arXiv Preprint, arXiv:1703.09507, 2017.
[69] B. F. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen, P.
Grother, A. Mah, M. Burge, and A. K. Jain, “Pushing the frontiers of unconstrained
face detection and recognition: IARPA Janus Benchmark A,” in Proc. IEEE Conf.
[70] J. Yang, P. Ren, D. Chen, F. Wen, H. Li, and G. Hua, “Neural aggregation net-
work for video face recognition,” arXiv Preprint, arXiv:1603.05474, 2016.
[71] N. Bodla, J. Zheng, H. Xu, J.-C. Chen, C. Castillo, and R. Chellappa, “Deep
heterogeneous feature fusion for template-based face recognition,” in Proc. IEEE
Winter Conf. Applications Computer Vision, 2017, pp. 586–595.
[72] J. Hu, J. Lu, and Y.-P. Tan, “Discriminative deep metric learning for face verifi-
cation in the wild,” in Proc. IEEE Conf. Computer Vision Pattern Recognition,
2014, pp. 1875–1882.
[73] S. Sankaranarayanan, A. Alavi, C. Castillo, and R. Chellappa, “Triplet proba-
bilistic embedding for face verification and clustering,” arXiv Preprint,
arXiv:1604.05417, 2016.
[74] H. O. Song, Y. Xiang, S. Jegelka, and S. Savarese, “Deep metric learning via
lifted structured feature embedding,” in Proc. IEEE Conf. Computer Vision Pattern
Recognition, 2016, pp. 4004–4012.
[75] J. Yang, D. Parikh, and D. Batra, “Joint unsupervised learning of deep repre-
sentations and image clusters,” in Proc. IEEE Conf. Computer Vision Pattern
Recognition, 2016, pp. 5147–5156.
[76] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, “Joint face representation adaptation and
clustering in videos,” in Proc. European Conf. Computer Vision, 2016, pp. 236–251.
[77] G. Trigeorgis, K. Bousmalis, S. Zafeiriou, and B. W. Schuller, “A deep matrix
factorization method for learning attribute representations,” IEEE Trans. Pattern
Anal. Mach. Intell., vol. 39, no. 3, pp. 417–429, 2017.
[78] W.-A. Lin, J.-C. Chen, and R. Chellappa, “A proximity-aware hierarchical
clustering of faces,” in Proc. IEEE Conf. Automatic Face Gesture Recognition,
2017. doi: 10.1109/FG.2017.134
[79] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, “MS-celeb-1m: A data set and
benchmark for large scale face recognition,” in Proc. European Conf. Computer
Vision, 2016, pp. 87–102.
[80] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the
wild,” in Proc. IEEE Int. Conf. Computer Vision, 2015, pp. 3730–3738.
[81] G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller, “Labeled faces in the
wild: A database for studying face recognition in unconstrained environments,” vol.
1, no. 2, p. 3, Tech. Rep. 07-49, Univ. Massachusetts, Amherst, 2007.
[82] L. Wolf, T. Hassner, and, I. Maoz, “Face recognition in unconstrained videos
with matched background similarity,” in Proc. IEEE Conf. Computer Vision
[83] J. R. Beveridge, P. J. Phillips, D. S. Bolme, B. A. Draper, G. H. Givens, Y. M.
Lui, M. N. Teli, H. Zhang, W. T. Scruggs, K. W. Bowyer, and P. J. Flynn, “The
challenge of face recognition from digital point-and-shoot cameras,” in Proc. IEEE
Int. Conf. Biometrics: Theory, Applications and Systems, 2013, pp. 1–8.
[84] S. Sengupta, J.-C. Chen, C. Castillo, V. M. Patel, R. Chellappa, and D. W.
Jacobs, “Frontal to profile face verification in the wild,” in Proc. IEEE Winter
Conf. Applications of Computer Vision, 2016, pp. 1–9.
[85] A. Bansal, A. Nanduri, C. Castillo, R. Ranjan, and R. Chellappa, “Umdfaces:
An annotated face data set for training deep networks,” arXiv Preprint,
arXiv:1611.01484, 2016.
[86] Y. Sun, D. Liang, X. Wang, and X. Tang, “Deepid3: Face recognition with very
deep neural networks,” arXiv Preprint, arXiv:1502.00873, 2015.
[87] D. Wang, C. Otto, and A. K. Jain, “Face search at scale: 80 million gallery,”
arXiv Preprint, arXiv:1507.07242, 2015.
[88] C. Ding and D. Tao, “Robust face recognition via multimodal deep face repre-
sentation,” arXiv Preprint, arXiv:1509.00244, 2015.
[89] L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos
with matched background similarity,” in Proc. IEEE Conf. Computer Vision
[90] P. Viola and M. J. Jones, “Robust real-time face detection,” Int. J. Comput.
Vis., vol. 57, no. 2, pp. 137–154, 2004.
[91] “IARPA Janus benchmark: A performance report,” National Institute of
Standards and Technology (NIST), 2016.
[92] A. RoyChowdhury, T.-Y. Lin, S. Maji, and E. Learned-Miller, “One-to-many
face recognition with bilinear CNNs,” in Proc. IEEE Winter Conf. Applications of
Computer Vision, 2016, pp. 1–9.
[93] N. Crosswhite, J. Byrne, O. M. Parkhi, C. Stauffer, Q. Cao, and A. Zisserman,
“Template adaptation for face verification and identification,” Proc. IEEE Int. Conf.
Automatic Face Gesture Recognition, 2017, pp. 1–8.
[94] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transfor-
mations for deep neural networks,” arXiv Preprint, arXiv:1611.05431, 2016.
[95] A. K. Jain, S. C. Dass, and K. Nandakumar, “Can soft biometric traits assist
user recognition?” in Defense and Security. Orlando, FL: Int. Society Optics and
Photonics, 2004, pp. 561–572.
[96] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev, “Panda: Pose
aligned networks for deep attribute modeling,” in Proc. IEEE Conf. Computer
Vision Pattern Recognition, 2014, pp. 1637–1644.
[97] G. Levi and T. Hassner, “Age and gender classification using convolutional
neural networks,” in Proc. IEEE Conf. Computer Vision Pattern Recognition
Workshops, 2015, pp. 34–42.
[98] P. Samangouei and R. Chellappa, “Convolutional neural networks for attribute-
based active authentication on mobile devices,” in Proc. IEEE Int. Conf. Biometrics
Theory Applications Systems, 2016, pp. 1–8.
[99] B. Siddiquie, R. S. Feris, and L. S. Davis, “Image ranking and retrieval based
on multi-attribute queries,” in Proc. IEEE Conf. Computer Vision Pattern
Recognition, 2011, pp. 801–808.
[100] E. M. Hand and R. Chellappa, “Attributes for improved attributes: A multi-
task network utilizing implicit and explicit relationships for facial attribute classifica-
tion,” in Proc. AAAI Conf. Artificial Intelligence, 2017, pp. 4068–4074.
[101] M. Günther, A. Rozsa, and T. E. Boult, “Affact-alignment free facial attribute
classification technique,” arXiv Preprint, arXiv:1611.06158, 2016.
[102] P. Samangouei, V. M. Patel, and R. Chellappa, “Attribute-based continuous
user authentication on mobile devices,” in Proc. IEEE Int. Conf. Biometrics Theory
Applications Systems, 2015, pp. 1–8.
[103] P. Samangouei, V. Patel, and R. Chellappa, “Facial attributes for active
authentication on mobile devices,” Image Vis. Computing, vol. 58, pp. 181–192,
Feb. 2017.
[104] R. Caruana, “Multitask learning,” in Learning to Learn. New York: Springer,
1998, pp. 95–133.
[105] Y. W. X. C. D. Chen, S. Ren, and J. Sun, “Joint cascade face detection and
alignment,” in Proc. European Conf. Computer Vision, 2014, vol. 8694, pp. 109–
122.
[106] I. Goodfellow, Y. Bengio, and A. Courville. (2016). Deep Learning.
Cambridge, MA: MIT Press. [Online]. Available: http://www.deeplearningbook
.org
[107] Z. Zhang, P. Luo, C. Loy, and X. Tang, “Facial landmark detection by deep
multi-task learning,” in Proc. European Conf. Computer Vision, 2014, pp. 94–108.
[108] K. Ricanek and T. Tesafaye, “Morph: A longitudinal image database of nor-
mal adult age-progression,” in Proc. Int. Conf. Automatic Face Gesture
Recognition, Apr. 2006, pp. 341–345.
[109] R. Rothe, R. Timofte, and L. V. Gool, “DEX: Deep expectation of apparent
age from a single image,” in Proc. IEEE Int. Conf. Computer Vision Workshop
ChaLearn Looking at People, 2015, pp. 10–15.
[110] M. Koestinger, P. Wohlhart, P. M. Roth, and H. Bischof, “Annotated facial
landmarks in the wild: A large-scale, real-world database for facial landmark local-
ization,” in Proc. IEEE 1st Int. Workshop Benchmarking Facial Image Analysis
Technologies, 2011, pp. 2144–2151.
[111] A. Dehghan, E. G. Ortiz, G. Shu, and S. Z. Masood, “DAGER: Deep age,
gender and emotion recognition using convolutional neural network,” arXiv
Preprint, arXiv:1702.04280, 2017.
[112] K. He, Y. Fu, and X. Xue, “A jointly learned deep architecture for facial attri-
bute analysis and face detection in the wild,” arXiv Preprint, arXiv:1707.08705,
2017.
SP

Deep learning for understanding faces

More Related Content

What's hot (19)

Similar to Deep learning for understanding faces (20)

Recently uploaded (20)

Deep learning for understanding faces