Learning_Deep_Models_for_Face_Anti-Spoofing_Binary.pdf

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/329740354
Learning Deep Models for Face Anti-Spooﬁng: Binary or Auxiliary Supervision
Conference Paper · June 2018
DOI: 10.1109/CVPR.2018.00048
CITATIONS
350
READS
914
3 authors:
Some of the authors of this publication are also working on these related projects:
Gait Recognition View project
Image Inpainting View project
Yaojie Liu
Google Inc.
19 PUBLICATIONS   1,087 CITATIONS
SEE PROFILE
Amin Jourabloo
Michigan State University
SEE PROFILE
Xiaoming Liu
Michigan State University
SEE PROFILE
All content following this page was uploaded by Yaojie Liu on 09 April 2019.
The user has requested enhancement of the downloaded file.

Learning Deep Models for Face Anti-Spoofing: Binary or Auxiliary Supervision
Yaojie Liu∗
Amin Jourabloo∗
Xiaoming Liu
Department of Computer Science and Engineering
Michigan State University, East Lansing MI 48824
{liuyaoj1,jourablo,liuxm}@msu.edu
Abstract
Face anti-spoofing is crucial to prevent face recognition
systems from a security breach. Previous deep learning ap-
proaches formulate face anti-spoofing as a binary classifi-
cation problem. Many of them struggle to grasp adequate
spoofing cues and generalize poorly. In this paper, we ar-
gue the importance of auxiliary supervision to guide the
learning toward discriminative and generalizable cues. A
CNN-RNN model is learned to estimate the face depth with
pixel-wise supervision, and to estimate rPPG signals with
sequence-wise supervision. The estimated depth and rPPG
are fused to distinguish live vs. spoof faces. Further, we
introduce a new face anti-spoofing database that covers a
large range of illumination, subject, and pose variations.
Experiments show that our model achieves the state-of-the-
art results on both intra- and cross-database testing.
1. Introduction
With applications in phone unlock, access control, and
security, biometric systems are widely used in our daily
lives, and face is one of the most popular biometric modal-
ities. While face recognition systems [42,45] gain popular-
ity, attackers present face spoofs (i.e., presentation attacks,
PA) to the system and attempt to be authenticated as the
genuine user. The face PA include printing a face on pa-
per (print attack), replaying a face video on a digital device
(replay attack), wearing a mask (mask attack), etc. To coun-
teract PA, face anti-spoofing techniques [16, 22, 23, 29] are
developed to detect PA prior to a face image being recog-
nized. Therefore, face anti-spoofing is vital to ensure that
face recognition systems are robust to PA and safe to use.
RGB image and video are the standard input to face anti-
spoofing systems, similar to face recognition systems. Re-
searchers start the texture-based anti-spoofing approaches
by feeding handcrafted features to binary classifiers [13,18,
19,27,33,34,38,51]. Later in the deep learning era, several
Convolutional Neural Networks (CNN) approaches utilize
∗denotes equal contribution by the authors.
Live Face
Presentation
Attack
✗ Binary
Supervision
0
1
✓ Auxiliary
Supervision
Depth map rPPG signal
Figure 1. Conventional CNN-based face anti-spoof approaches uti-
lize the binary supervision, which may lead to overfitting given the
enormous solution space of CNN. This work designs a novel net-
work architecture to leverage two auxiliary information as super-
vision: the depth map and rPPG signal, with the goals of improved
generalization and explainable decisions during inference.
softmax loss as the supervision [21, 30, 37, 50]. It appears
almost all prior work regard the face anti-spoofing problem
as merely a binary (live vs. spoof) classification problem.
There are two main issues in learning deep anti-spoofing
models with binary supervision. First, there are different
levels of image degradation, namely spoof patterns, com-
paring a spoof face to a live one, which consist of skin de-
tail loss, color distortion, moiré pattern, shape deformation
and spoof artifacts (e.g., reflection) [29, 38]. A CNN with
softmax loss might discover arbitrary cues that are able to
separate the two classes, such as screen bezel, but not the
faithful spoof patterns. When those cues disappear during
testing, these models would fail to distinguish spoof vs. live
faces and result in poor generalization. Second, during the
testing, models learnt with binary supervision will only gen-
erate a binary decision without explanation or rationale for
the decision. In the pursuit of Explainable Artificial Intelli-
gence [1], it is desirable for the learnt model to generate the
spoof patterns that support the final binary decision.
To address these issues, as shown in Fig. 1, we propose
a deep model that uses the supervision from both the spa-
tial and temporal auxiliary information rather than binary
supervision, for the purpose of robustly detecting face PA
1

from a face video. These auxiliary information are acquired
based on our domain knowledge about the key differences
between live and spoof faces, which include two perspec-
tives: spatial and temporal. From the spatial perspective,
it is known that live faces have face-like depth, e.g., the
nose is closer to the camera than the cheek in frontal-view
faces, while faces in print or replay attacks have flat or pla-
nar depth, e.g., all pixels on the image of a paper have the
same depth to the camera. Hence, depth can be utilized
as auxiliary information to supervise both live and spoof
faces. From the temporal perspective, it was shown that the
normal rPPG signals (i.e., heart pulse signal) are detectable
from live, but not spoof, face videos [31, 35]. Therefore,
we provide different rPPG signals as auxiliary supervision,
which guides the network to learn from live or spoof face
videos respectively. To enable both supervisions, we design
a network architecture with a short-cut connection to cap-
ture different scales and a novel non-rigid registration layer
to handle the motion and pose change for rPPG estimation.
Furthermore, similar to many vision problems, data
plays a significant role in training the anti-spoofing mod-
els. As we know, camera/screen quality is a critical factor
to the quality of spoof faces. Existing face anti-spoofing
databases, such as NUAA [44], CASIA [52], Replay-
Attack [17], and MSU-MFSD [47], were collected 3 − 5
years ago. Given the fast advance of consumer electron-
ics, the types of equipment (e.g., cameras and spoofing
mediums) used in those data collection are outdated com-
pared to the ones nowadays, regarding the resolution and
imaging quality. More recent MSU-USSA [38] and OULU
databases [14] have subjects with fewer variations in poses,
illuminations, expressions (PIE). The lack of necessary
variations would make it hard to learn an effective model.
Given the clear need for more advanced databases, we col-
lect a face anti-spoofing database, named Spoof in the Wild
Database (SiW). SiW database consists of 165 subjects, 6
spoofing mediums, and 4 sessions covering variations such
as PIE, distance-to-camera, etc. SiW covers much larger
variations than previous databases, as detailed in Tab. 1 and
Sec. 4. The main contributions of this work include:
We propose to leverage novel auxiliary information
(i.e., depth map and rPPG) to supervise the CNN learning
for improved generalization.
We propose a novel CNN-RNN architecture for end-
to-end learning the depth map and rPPG signal.
We release a new database that contains variations of
PIE, and other practical factors. We achieve the state-of-
the-art performance for face anti-spoofing.
2. Prior Work
We review the prior face anti-spoofing works in three
groups: texture-based methods, temporal-based methods,
and remote photoplethysmography methods.
Texture-based Methods Since most face recognition sys-
tems adopt only RGB cameras, using texture information
has been a natural approach to tackling face anti-spoofing.
Many prior works utilize hand-crafted features, such as
LBP [18, 19, 33], HoG [27, 51], SIFT [38] and SURF [13],
and adopt traditional classifiers such as SVM and LDA.
To overcome the influence of illumination variation, they
seek solutions in a different input domain, such as HSV and
YCbCr color space [11,12], and Fourier spectrum [29].
As deep learning has proven to be effective in many com-
puter vision problems, there are many recent attempts of us-
ing CNN-based features or CNNs in face anti-spoofing [21,
30, 37, 50]. Most of the work treats face anti-spoofing as a
simple binary classification problem by applying the soft-
max loss. For example, [30,37] use CNN as feature extrac-
tor and fine-tune from ImageNet-pretrained CaffeNet and
VGG-face. The work of [21, 30] feed different designs of
the face images into CNN, such as multi-scale faces and
hand-crafted features, and directly classify live vs. spoof.
One prior work that shares the similarity with ours is [5],
where Atoum et al. propose a two-steam CNN-based anti-
spoofing method using texture and depth. We advance [5] in
a number of aspects, including fusion with temporal super-
vision (i.e., rPPG), finer architecture design, novel non-rigid
registration layer, and comprehensive experimental support.
Temporal-based Methods One of the earliest solutions for
face anti-spoofing is based on temporal cues such as eye-
blinking [36,37]. Methods such as [26,43] track the motion
of mouth and lip to detect the face liveness. While these
methods are effective to typical paper attacks, they become
vulnerable when attackers present a replay attack or a paper
attack with eye/mouth portion being cut.
There are also methods relying on more general tempo-
ral features, instead of the specific facial motion. The most
common approach is frame concatenation. Many hand-
crafted feature-based methods may improve intra-database
testing performance by simply concatenating the features of
consecutive frames to train the classifiers [11,18,28]. Addi-
tionally, there are some works proposing temporal-specific
features, e.g., Haralick features [4], motion mag [7], and op-
tical flow [6]. In the deep learning era, Feng et al. feed the
optical flow map and Shearlet image feature to CNN [21].
In [49], Xu et al. propose an LSTM-CNN architecture to
utilize temporal information for binary classification. Over-
all, all prior methods still regard face anti-spoofing as a
binary classification problem, and thus they have a hard
time to generalize well in the cross-database testing. In
this work, we extract discriminative temporal information
by learning the rPPG signal of the face video.
Remote Photoplethysmography (rPPG) Remote photo-
plethysmography (rPPG) is the technique to track vital sig-
nals, such as heart rate, without any contact with human
skin [9,20,41,46,48]. Research starts with face videos with

CNN
RNN
…
…
…
…
Live/Spoof
CNN
RNN
CNN
RNN
Figure 2. The overview of the proposed method.
no motion or illumination change to videos with multiple
variations. In [20], Haan et al. estimate rPPG signals from
RGB face videos with lighting and motion changes. It uti-
lizes color difference to eliminate the specular reflection and
estimate two orthogonal chrominance signals. After apply-
ing the Band Pass Filter (BPM), the ratio of the chromi-
nance signals are used to compute the rPPG signal.
rPPG has previously been utilized to tackle face anti-
spoofing [31, 35]. In [31], rPPG signals are used for de-
tecting the 3D mask attack, where the live faces exhibit a
pulse of heart rate unlike the 3D masks. They use rPPG sig-
nals extracted by [20] and compute the correlation features
for classification. Similarly, Magdalena et al. [35] extract
rPPG signals (also via [20]) from three face regions and two
non-face regions, for detecting print and replay attacks. Al-
though in replay attacks, the rPPG extractor might still cap-
ture the normal pulse, the combination of multiple regions
can differentiate live vs. spoof faces. While the analytic so-
lution to rPPG extraction [20] is easy to implement, we ob-
serve that it is sensitive to PIE variations. Hence, we employ
a novel CNN-RNN architecture to learn a mapping from a
face video to the rPPG signal, which is not only robust to
PIE variations, but also discriminative to live vs. spoof.
3. Face Anti-Spoofing with Deep Network
The main idea of the proposed approach is to guide the
deep network to focus on the known spoof patterns across
spatial and temporal domains, rather than to extract any
cues that could separate two classes but are not generaliz-
able. As shown in Fig. 2, the proposed network combines
CNN and RNN architectures in a coherent way. The CNN
part utilizes the depth map supervision to discover subtle
texture property that leads to distinct depths for live and
spoof faces. Then, it feeds the estimated depth and the fea-
ture maps to a novel non-rigid registration layer to create
aligned feature maps. The RNN part is trained with the
aligned maps and the rPPG supervision, which examines
temporal variability across video frames.
3.1. Depth Map Supervision
Depth maps are a representation of the 3D shape of the
face in a 2D image, which shows the face location and the
depth information of different facial areas. This representa-
tion is more informative than binary labels since it indicates
one of the fundamental differences between live faces, and
print and replay PA. We utilize the depth maps in the depth
loss function to supervise the CNN part. The pixel-based
depth loss guides the CNN to learn a mapping from the face
area within a receptive field to a labeled depth value – a
scale within [0, 1] for live faces and 0 for spoof faces.
To estimate the depth map for a 2D face image, given a
face image, we utilize the state-of-the-art dense face align-
ment (DeFA) methods [25, 32] to estimate the 3D shape of
the face. The frontal dense 3D shape SF ∈ R3×Q
, with Q
vertices, is represented as a linear combination of identity
bases {Si
id}Nid
i=1 and expression bases {Si
exp}
Nexp
i=1 ,
SF = S0 +
Nid
X
i=1
αi
idSi
id +
Nexp
X
i=1
αi
expSi
exp, (1)
where αid ∈ R199
and αext ∈ R29
are the identity and
expression parameters, and α = [αid, αexp] are the shape
parameters. We utilize the Basel 3D face model [39] and
the facewearhouse [15] as the identity and expression bases.
With the estimated pose parameters P = (s, R, t),
where R is a 3D rotation matrix, t is a 3D translation, and
s is a scale, we align the 3D shape S to the 2D face image:
S = sRSF + t. (2)
Given the challenge of estimating the absolute depth
from a 2D face, we normalize the z values of 3D vertices in
S to be within [0, 1]. That is, the vertex closest to the cam-
era (e.g., nose) has a depth of one, and the vertex furthest
away has the depth of zero. Then, we apply the Z-Buffer
algorithm [53] to S for projecting the normalized z values
to a 2D plane, which results in an estimated “ground truth”
2D depth map D ∈ R32×32
for a face image.
3.2. rPPG Supervision
rPPG signals have recently been utilized for face anti-
spoofing [31,35]. The rPPG signal provides temporal infor-
mation about face liveness, as it is related to the intensity
changes of facial skin over time. These intensity changes
are highly correlated with the blood flow. The traditional
method [20] for extracting rPPG signals has three draw-
backs. First, it is sensitive to pose and expression variation,
as it becomes harder to track a specific face area for measur-
ing intensity changes. Second, it is also sensitive to illumi-
nation changes, since the extra lighting affects the amount
of reflected light from the skin. Third, for the purpose of
anti-spoof, rPPG signals extracted from spoof videos might
not be sufficiently distinguishable to signals of live videos.
One novelty aspect of our approach is that, instead of
computing the rPPG signal via [20], our RNN part learns to
estimate the rPPG signal. This eases the signal estimation

Conv
Conv
Pool
Conv
Conv
Depth
map
Feature
map
256×256×6×5
(RGB+HSV)
LSTM
Conv
Conv
Pool
Conv
Conv
Pool
RNN
CNN
Conv
Conv
Conv
Conv
Conv
Non-rigid
Registration
S
100
Hidden
Neurons
100
rPPG
Loss
Depth Map
Loss
32×32
32×32
1
3
128
1
64
128
128 196
128 196
128 196
64
Conv
128
Conv
128
Conv
128
FC
FFT
Block 1 Block 2 Block 3
Figure 3. The proposed CNN-RNN architecture. The number of filters are shown on top of each layer, the size of all filters is 3 × 3 with
stride 1 for convolutional and 2 for pooling layers. Color code used: orange=convolution, green=pooling, purple=response map.
from face videos with PIE variations, and also leads to more
discriminative rPPG signals, as different rPPG supervisions
are provided to live vs. spoof videos. We assume that the
videos of the same subject under different PIE conditions
have the same ground truth rPPG signal. This assumption
is valid since the heart beat is similar for the videos of the
same subject that are captured in a short span of time ( 5
minutes). The rPPG signal extracted from the constrained
videos (i.e., no PIE variation) are used as the “ground truth”
supervision in the rPPG loss function for all live videos
of the same subject. This consistent supervision helps the
CNN and RNN parts to be robust to the PIE changes.
In order to extract the rPPG signal from a face video
without PIE, we apply the DeFA [32] to each frame and es-
timate the dense 3D face shape. We utilize the estimated 3D
shape to track a face region. For a tracked region, we com-
pute two orthogonal chrominance signals xf = 3rf − 2gf ,
yf = 1.5rf +gf −1.5bf where rf , gf , bf are the bandpass
filtered versions of the r, g, b channels with the skin-tone
normalization. We utilize the ratio of the standard deviation
of the chrominance signals γ =
σ(xf )
σ(yf ) to compute blood
flow signals [20]. We calculate the signal p as:
p = 3(1 −
γ
2
)rf − 2(1 +
γ
2
)gf +
3γ
2
bf . (3)
By applying FFT to p, we obtain the rPPG signal f ∈ R50
,
which shows the magnitude of each frequency.
3.3. Network Architecture
Our proposed network consists of two deep networks.
First, a CNN part evaluates each frame separately and esti-
mates the depth map and feature map of each frame. Sec-
ond, a recurrent neural network (RNN) part evaluates the
temporal variability across the feature maps of a sequence.
3.3.1 CNN Network
We design a Fully Convolutional Network (FCN) as our
CNN part, as shown in Fig. 3. The CNN part contains mul-
tiple blocks of three convolutional layers, pooling and re-
sizing layers where each convolutional layer is followed by
one exponential linear layer and batch normalization layer.
Then, the resizing layers resize the response maps after each
block to a pre-defined size of 64 × 64 and concatenate the
response maps. The bypass connections help the network to
utilize extracted features from layers with different depths
similar to the ResNet structure [24]. After that, our CNN
has two branches, one for estimating the depth map and the
other for estimating the feature map.
The first output of the CNN is the estimated depth map
of the input frame I ∈ R256×256
, which is supervised by the
estimated “ground truth” depth D,
ΘD = arg min
ΘD
Nd
X
i=1
||CNND(Ii; ΘD) − Di||2
1, (4)
where ΘD is the CNN parameters and Nd is the number of
training images. The second output of the CNN is the fea-
ture map, which is fed into the non-rigid registration layer.
3.3.2 RNN Network
The RNN part aims to estimate the rPPG signal f of an input
sequence with Nf frames {Ij}
Nf
j=1. As shown in Fig. 3,
we utilize one LSTM layer with 100 hidden neurons, one
fully connected layer, and an FFT layer that converts the
response of fully connected layer into the Fourier domain.
Given the input sequence {Ij}
Nf
j=1 and the “ground truth”
rPPG signal f, we train the RNN to minimize the `1 distance
of the estimated rPPG signal to “ground truth” f,
ΘR = arg min
ΘR
Ns
X
i=1
||RNNR([{Fj}
Nf
j=1]i; ΘR) − fi||2
1, (5)
where ΘR is the RNN parameters, Fj ∈ R32×32
is the
frontalized feature map (details in Sec. 3.4), and Ns is the
number of sequences.
3.3.3 Implementation Details
Ground Truth Data Given a set of live and spoof face
videos, we provide the ground truth supervision for the
depth map D and rPPG signal f, as in Fig. 4. We follow
the procedure in Sec. 3.1 to compute “ground truth” data for
live videos. For spoof videos, we set the ground truth depth
maps to a plain surface, i.e., zero depth. Similarly, we fol-
low the procedure in Sec. 3.2 to compute the “ground truth”

Frame 3D Fitting Depth Map
Prior
Frame
Extracted rPPG
Prior
Live Video Spoof Video
Depth Map
rPPG
Figure 4. Example ground truth depth maps and rPPG signals.
rPPG signal from a patch on the forehead, for one live video
of each subject without PIE variation. Also, we normalize
the norm of estimated rPPG signal such that kfk2 = 1. For
spoof videos, we consider the rPPG signals are zero.
Note that, while the term “depth” is used here, our es-
timated depth is different to the conventional depth map in
computer vision. It can be viewed as a “pseudo-depth” and
serves the purpose of providing discriminative auxiliary su-
pervision to the learning process. The same perspective ap-
plies to the supervision based on pseudo-rPPG signal.
Training Strategy Our proposed network combines the
CNN and RNN parts for end-to-end training. The desired
training data for the CNN part should be from diverse sub-
jects, so as to make the training procedure more stable and
increase the generalizability of the learnt model. Mean-
while, the training data for the RNN part should be long se-
quences to leverage the temporal information across frames.
These two preferences can be contradictory to each other,
especially given the limited GPU memory. Hence, to sat-
isfy both preferences, we design a two-stream training strat-
egy. The first stream satisfies the preference of the CNN
part, where the input includes face images I and the ground
truth depth maps D. The second stream satisfies the RNN
part, where the input includes face sequences {Ij}
Nf
j=1, the
ground truth depth maps {Dj}
Nf
j=1, the estimated 3D shapes
{Sj}
Nf
j=1, and the corresponding ground truth rPPG signals
f. During training, our method alternates between these
two streams to converge to a model that minimizes both
the depth map and rPPG losses. Note that even though the
first stream only updates the weights of the CNN part, the
back propagation of the second stream updates the weights
of both CNN and RNN parts in an end-to-end manner.
Testing To provide a classification score, we feed the test-
ing sequence to our network and compute the depth map D̂
of the last frame and the rPPG signal f̂. Instead of designing
a classifier using D̂ and f̂, we compute the final score as:
score = ||f̂||2
2 + λ||D̂||2
2, (6)
where λ is a constant weight for combining the two outputs
of the network.
D̂
T
S 0
S
V
F
3D Face
Alignment
U
ij
m ij
m
Figure 5. The non-rigid registration layer.
3.4. Non-rigid Registration Layer
We design a new non-rigid registration layer to prepare
data for the RNN part. This layer utilizes the estimated
dense 3D shape to align the activations or feature maps from
the CNN part. This layer is important to ensure that the
RNN tracks and learns the changes of the activations for the
same facial area across time, as well as across all subjects.
As shown in Fig. 5, this layer has three inputs: the fea-
ture map T ∈ R32×32
, the depth map D̂ and the 3D shape
S. Within this layer, we first threshold the depth map and
generate a binary mask V ∈ R32×32
:
V = D̂ ≥ threshold. (7)
Then, we compute the inner product of the binary mask and
the feature map U = T

V, which essentially utilizes
the depth map as a visibility indicator for each pixel in the
feature map. If the depth value for one pixel is less than the
threshold, we consider that pixel to be invisible. Finally, we
frontalize U by utilizing the estimated 3D shape S,
F(i, j) = U(S(mij, 1), S(mij, 2)), (8)
where m ∈ RK
is the pre-defined list of K indexes of the
face area in S0, and mij is the corresponding index of pixel
i, j. We utilize m to project the masked activation map U
to the frontalized image F. This proposed non-rigid regis-
tration layer has three contributions to our network:
By applying the non-rigid registration, the input data
are aligned and the RNN can compare the feature maps
without concerning about the facial pose or expression. In
other words, it can learn the temporal changes in the activa-
tions of the feature maps for the same facial area.
The non-rigid registration removes the background
area in the feature map. Hence the background area would
not participate in RNN learning, although the background
information is already utilized in the layers of the CNN part.
For spoof faces, the depth maps are likely to be closer
to zero. Hence, the inner product with the depth maps
substantially weakens the activations in the feature maps,
which makes it easier for the RNN to output zero rPPG sig-
nals. Likewise, the back propagation from the rPPG loss

Table 1. The comparison of our collected SiW dataset with existing datasets for face anti-spoofing.
Dataset
Year # of # of # of live/attack Pose Different Extra
Display devices
Spoof
subj. sess. vid. (V), ima. (I) range expres. light. attacks
NUAA [44] 2010 15 3 5105/7509 (I) Frontal No Yes - Print
CASIA-MFSD [52] 2012 50 3 150/450 (V) Frontal No No iPad Print, Replay
Replay-Attack [17] 2012 50 1 200/1000 (V) Frontal No Yes iPhone 3GS, iPad Print, 2 Replay
MSU-MFSD [47] 2015 35 1 110/330 (V) Frontal No No iPad Air, iPhone 5S Print, 2 Replay
MSU-USSA [38] 2016 1140 1 1140/9120 (I) [−45◦
, 45◦
] Yes Yes MacBook, Nexus 5, Nvidia Shield Tablet 2 print, 6 Replay
Oulu-NPU [14] 2017 55 3 1980/3960 (V) Frontal No Yes Dell 1905FP, Macbook Retina 2 Print, 2 Replay
SiW 2018 165 4 1320/3300 (V) [−90◦
, 90◦
] Yes Yes iPad Pro, iPhone 7, Galaxy S8, Asus MB168B 2 Print, 4 Replay
Face
Size
Probability
Figure 6. The statistics of the subjects in the SiW database. Left
side: The histogram shows the distribution of the face sizes.
also encourages the CNN part to generate zero depth maps
for either all frames, or one pixel location in majority of the
frames within an input sequence.
4. Collection of Face Anti-Spoofing Database
With the advance of sensor technology, existing anti-
spoofing systems can be vulnerable to emerging high-
quality spoof mediums. One way to make the system robust
to these attacks is to collect new high-quality databases. In
response to this need, we collect a new face anti-spoofing
database named Spoof in the Wild (SiW) database, which
has multiple advantages over previous datasets as in Tab. 1.
First, it contains substantially more live subjects with di-
verse races, e.g., 3 times of the subjects of Oulu-NPU. Note
that MSU-USSA is constructed using existing images of
celebrities without capturing live faces. Second, live videos
are captured with two high-quality cameras (Canon EOS
T6, Logitech C920 webcam) with different PIE variations.
SiW provides live and spoof 30-fps videos from 165 sub-
jects. For each subject, we have 8 live and 20 spoof videos,
in total 4, 620 videos. Some statistics of the subjects are
shown in Fig. 6. The live videos are collected in four ses-
sions. In Session 1, the subject moves his head with varying
distances to the camera. In Session 2, the subject changes
the yaw angle of the head within [−90◦
, 90◦
], and makes
different face expressions. In Sessions 3, 4, the subject re-
peats the Sessions 1, 2, while the collector moves the point
light source around the face from different orientations.
The live videos captured by both cameras are of 1, 920×
1, 080 resolution. We provide two print and four replay
video attacks for each subject, with examples shown in
Fig. 7. To generate different qualities of print attacks, we
Normal Small Large Pose Expression Lighting
Print 1 Print 2 iPad iPhone PC Screen Samsung S8
Figure 7. Example live (top) and spoof (bottom) videos in SiW.
capture a high-resolution image (5, 184 × 3, 456) for each
subject and use it to make a high-quality print attack. Also,
we extract a frontal-view frame from a live video for lower-
quality print attack. We print the images with an HP color
LaserJet M652 printer. The print attack videos are captured
by holding printed papers still or warping them in front of
the cameras. To generate high-quality replay attack videos,
we select four spoof mediums: Samsung Galaxy S8, iPhone
7, iPad Pro, and PC (Asus MB168B) screens. For each sub-
ject, we randomly select two of the four high-quality live
videos to display in the spoof mediums.
5. Experimental Results
5.1. Experimental Setup
Databases We evaluate our method on multiple databases
to demonstrate its generalizability. We utilize SiW and Oulu
databases [14] as new high-resolution databases and per-
form intra and cross testing between them. Also, we use the
CASIA-MFSD [52] and Replay-Attack [17] databases for
cross testing and comparing with the state of the art.
Parameter setting The proposed method is implemented
in TensorFlow [3] with a constant learning rate of 3e−3,
and 10 epochs of the training phase. The batch size of the
CNN stream is 10 and that of the CNN-RNN stream is 2
with Nf being 5. We randomly initialize our network by
using a normal distribution with zero mean and std of 0.02.
We set λ in Eq. 6 to 0.015 and threshold in Eq. 7 to 0.1.
Evaluation metrics To compare with prior works, we re-
port our results with the following metrics: Attack Presenta-
tion Classification Error Rate APCER [2], Bona Fide Pre-
sentation Classification Error Rate BPCER [2], ACER =
AP CER+BP CER
2 [2], and Half Total Error Rate HTER.
The HTER is half of the summation of the False Rejection
Rate (FRR) and the False Acceptance Rate (FAR).

Table 2. TDR at different FDRs, cross testing on Oulu Protocol 1.
FDR 1% 2% 10% 20%
Model 1 8.5% 18.1% 71.4% 81.0%
Model 2 40.2% 46.9% 78.5% 93.5%
Model 3 39.4% 42.9% 67.5% 87.5%
Model 4 45.8% 47.9% 81% 94.2%
Table 3. ACER of our method at different Nf , on Oulu Protocol 2.
PPPPPP
P
Test
Train
5 10 20
5 4.16% 4.16% 3.05%
10 4.02% 3.61% 2.78%
20 4.10% 3.67% 2.98%
5.2. Experimental Comparison
5.2.1 Ablation Study
Advantage of proposed architecture We compare four
architectures to demonstrate the advantages of the proposed
loss layers and non-rigid registration layer. Model 1 has an
architecture similar to the CNN part in our method (Fig. 3),
except that it is extended with additional pooling layers,
fully connected layers, and softmax loss for binary classi-
fication. Model 2 is the CNN part in our method with a
depth map loss function. We simply use ||D̂||2 for classifi-
cation. Model 3 contains the CNN and RNN parts without
the non-rigid registration layer. Both of the depth map and
rPPG loss functions are utilized in this model. However, the
RNN part would process unregistered feature maps from the
CNN. Model 4 is the proposed architecture.
We train all four models with the live and spoof videos
from 20 subjects of SiW. We compute the cross-testing per-
formance of all models on Protocol 1 of Oulu database. The
TDR at different FDR are reported in Tab. 2. Model 1 has
a poor performance due to the binary supervision. In com-
parison, by only using the depth map as supervision, Model
2 achieves substantially better performance. However, after
adding the RNN part with the rPPG supervision, our pro-
posed Model 4 can further the performance improvement.
By comparing Model 4 and 3, we can see the advantage of
the non-rigid registration layer. It is clear that the RNN part
cannot use feature maps directly for tracking the changes in
the activations and estimating the rPPG signals.
Advantage of longer sequences To show the advantage of
utilizing longer sequences for estimating the rPPG, we train
and test our model when the sequence length Nf is 5, 10, or
20, using intra-testing on Oulu Protocol 2. From Tab. 3, we
can see that by increasing the sequence length, the ACER
decreases due to more reliable rPPG estimation. Despite
the benefit of longer sequences, in practice, we are limited
by the GPU memory size, and forced to decrease the image
size to 128×128 for all experiments in Tab. 3. Hence, we set
Nf to be 5 with the image size of 256 × 256 in subsequent
experiments, due to importance of higher resolution (e.g, a
lower ACER of 2.5% in Tab. 4 is achieved than 4.16%).
Table 4. The intra-testing results on four protocols of Oulu.
Prot. Method APCER (%) BPCER (%) ACER (%)
CPqD 2.9 10.8 6.9
1 GRADIANT 1.3 12.5 6.9
Proposed method 1.6 1.6 1.6
MixedFASNet 9.7 2.5 6.1
2 Proposed method 2.7 2.7 2.7
GRADIANT 3.1 1.9 2.5
MixedFASNet 5.3 ± 6.7 7.8 ± 5.5 6.5 ± 4.6
3 GRADIANT 2.6 ± 3.9 5.0 ± 5.3 3.8 ± 2.4
Proposed method 2.7 ± 1.3 3.1 ± 1.7 2.9 ± 1.5
Massy HNU 35.8 ± 35.3 8.3 ± 4.1 22.1 ± 17.6
4 GRADIANT 5.0 ± 4.5 15.0 ± 7.1 10.0 ± 5.0
Proposed method 9.3 ± 5.6 10.4 ± 6.0 9.5 ± 6.0
Table 5. The intra-testing results on three protocols of SiW.
Prot. Subset Subject # Attack APCER (%) BPCER (%) ACER (%)
1
Train 90 First 60 Frames
3.58 3.58 3.58
Test 75 All
2
Train 90 3 display
0.57 ± 0.69 0.57 ± 0.69 0.57 ± 0.69
Test 75 1 display
3
Train 90 print (display)
8.31 ± 3.81 8.31 ± 3.80 8.31 ± 3.81
Test 75 display (print)
5.2.2 Intra Testing
We perform intra testing on Oulu and SiW databases. For
Oulu, we follow the four protocols [10] and report their
APCER, BPCER and ACER. Tab. 4 shows the compari-
son of our proposed method and the best two methods for
each protocol respectively, in the face anti-spoofing compe-
tition [10]. Our method achieves the lowest ACER in 3 out
of 4 protocols. We have slightly worse ACER on Protocol
2. To set a baseline for future study on SiW, we define three
protocols for SiW. The Protocol 1 deals with variations in
face pose and expression. We train using the first 60 frames
of the training videos that are mainly frontal view faces,
and test on all testing videos. The Protocol 2 evaluates the
performance of cross spoof medium of replay attack. The
Protocol 3 evaluates the performance of cross PA, i.e., from
print attack to replay attack and vice versa. Tab. 5 shows the
protocol definition and our performance of each protocol.
5.2.3 Cross Testing
To demonstrate the generalization of our method, we per-
form multiple cross-testing experiments. Our model is
trained with live and spoof videos of 80 subjects in SiW,
and test on all protocols of Oulu. The ACER on Proto-
col 1-4 are respectively: 10.0%, 14.1%, 13.8 ± 5.7%, and
10.0 ± 8.8%. Comparing these cross-testing results to the
intra-testing results in [10], we are ranked sixth on the av-
erage ACER of four protocols, among the 15 participants of
the face anti-spoofing competition. Especially on Protocol
4, the hardest one among all protocols, we achieve the same
ACER of 10.0% as the top performer. This is a notable result
since cross testing is known to be substantially harder than
intra testing, and yet our cross-testing result is comparable
with the top intra-testing performance. This demonstrates
the generalization ability of our learnt model.

(a) (b)
Frequency
Magnitude
Frequency
Magnitude
Figure 8. (a) 8 successful anti-spoofing examples and their estimated depth maps and rPPG signals. (b) 4 failure examples: the first two
are live and the other two are spoof. Note our ability to estimate discriminative depth maps and rPPG signals.
Table 6. Cross testing on CASIA-MFSD vs. Replay-Attack.
Method
Train Test Train Test
CASIA- Replay Replay CASIA-
MFSD Attack Attack MFSD
Motion [19] 50.2% 47.9%
LBP [19] 55.9% 57.6%
LBP-TOP [19] 49.7% 60.6%
Motion-Mag [8] 50.1% 47.0%
Spectral cubes [40] 34.4% 50.0%
CNN [50] 48.5% 45.5%
LBP [11] 47.0% 39.6%
Colour Texture [12] 30.3% 37.7%
Proposed method 27.6% 28.4%
Live Spoof
Mean Standard deviation Mean Standard deviation
Figure 9. Mean/Std of frontalized feature maps for live and spoof.
Furthermore, we utilize the CASIA-MFSD and Replay-
Attack databases to perform cross testing between them,
which is widely used as a cross-testing benchmark. Tab. 6
compares the cross-testing HTER of different methods. Our
proposed method reduces the cross-testing errors on the
Replay-Attack and CASIA-MFSD databases by 8.9% and
24.6% respectively, relative to the previous SOTA.
5.2.4 Visualization and Analysis
In the proposed architecture, the frontalized feature maps
are utilized as input to the RNN part and are supervised by
the rPPG loss function. The values of these maps can show
the importance of different facial areas to rPPG estimation.
Fig. 9 shows the mean and standard deviation of frontalized
feature maps, computed from 1, 080 live and spoof videos
of Oulu. We can see that the side areas of forehead and
cheek have higher influence for rPPG estimation.
While the goal of our system is to detect PAs, our model
is trained to estimate the auxiliary information. Hence, in
addition to anti-spoof, we also like to evaluate the accuracy
of auxiliary information estimation. For this purpose, we
calculate the accuracy of estimating depth maps and rPPG
signals, for testing data in Protocol 2 of Oulu. As shown
in Fig. 10, the accuracy for both estimation in spoof data
Live Spoof
Figure 10. The MSE of estimating depth maps and rPPG signals.
is high, while that of the live data is relatively lower. Note
that the depth estimation of the mouth area has more errors,
which is consistent with the fewer activations of the same
area in Fig. 9. Examples of successful and failure cases in
estimating depth maps and rPPG signals are shown in Fig. 8.
Finally, we conduct statistical analysis on the failure
cases, since our system can determine potential causes us-
ing the auxiliary information. With Proctocol 2 of Oulu,
we identify 31 failure cases (2.7% ACER). For each case,
we calculate whether anti-spoofing using its depth map or
rPPG signal would fail if that information alone is used. In
total, 29
31 , 13
31 , and 11
31 samples fail due to depth map, rPPG
signals, or both. This indicates the future research direction.
6. Conclusions
This paper identifies the importance of auxiliary supervi-
sion to deep model-based face anti-spoofing. The proposed
network combines CNN and RNN architectures to jointly
estimate the depth of face images and rPPG signal of face
video. We introduce the SiW database that contains more
subjects and variations than prior databases. Finally, we ex-
perimentally demonstrate the superiority of our method.
Acknowledgment This research is based upon work sup-
ported by the Office of the Director of National Intelli-
gence (ODNI), Intelligence Advanced Research Projects
Activity (IARPA), via IARPA RD Contract No. 2017-
17020200004. The views and conclusions contained herein
are those of the authors and should not be interpreted as nec-
essarily representing the official policies or endorsements,
either expressed or implied, of the ODNI, IARPA, or the
U.S. Government. The U.S. Government is authorized to re-
produce and distribute reprints for Governmental purposes
notwithstanding any copyright annotation thereon.

References
[1] Explainable Artificial Intelligence (XAI).
https://www.darpa.mil/program/
explainable-artificial-intelligence.
1
[2] ISO/IEC JTC 1/SC 37 Biometrics. information technol-
ogy biometric presentation attack detection part 1: Frame-
work. international organization for standardization, 2016.
https://www.iso.org/obp/ui/iso. 6
[3] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean,
M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. Tensor-
flow: A system for large-scale machine learning. In OSDI,
volume 16, pages 265–283, 2016. 6
[4] A. Agarwal, R. Singh, and M. Vatsa. Face anti-spoofing us-
ing Haralick features. In BTAS, 2016. 2
[5] Y. Atoum, Y. Liu, A. Jourabloo, and X. Liu. Face anti-
spoofing using patch and depth-based CNNs. In IJCB, 2017.
2
[6] W. Bao, H. Li, N. Li, and W. Jiang. A liveness detection
method for face recognition based on optical flow field. In
IASP, pages 233–236, 2009. 2
[7] S. Bharadwaj, T. Dhamecha, M. Vatsa, and R. Singh.
Face anti-spoofing via motion magnification and multifea-
ture videolet aggregation. 2014. 2
[8] S. Bharadwaj, T. I. Dhamecha, M. Vatsa, and R. Singh. Com-
putationally efficient face spoofing detection with motion
magnification. In CVPRW, pages 105–110, 2013. 8
[9] S. Bobbia, Y. Benezeth, and J. Dubois. Remote photo-
plethysmography based on implicit living skin tissue seg-
mentation. In ICPR, pages 361–365, 2016. 2
[10] Z. Boulkenafet. A competition on generalized software-
based face presentation attack detection in mobile scenarios.
In IJCB, 2017. 7
[11] Z. Boulkenafet, J. Komulainen, and A. Hadid. Face anti-
spoofing based on color texture analysis. In ICIP, pages
2636–2640, 2015. 2, 8
[12] Z. Boulkenafet, J. Komulainen, and A. Hadid. Face spoof-
ing detection using colour texture analysis. IEEE Trans. Inf.
Forens. Security, 11(8):1818–1830, 2016. 2, 8
[13] Z. Boulkenafet, J. Komulainen, and A. Hadid. Face anti-
spoofing using speeded-up robust features and Fisher vec-
tor encoding. IEEE Signal Process. Letters, 24(2):141–145,
2017. 1, 2
[14] Z. Boulkenafet, J. Komulainen, L. Li, X. Feng, and A. Hadid.
OULU-NPU: A mobile face presentation attack database
with real-world variations. In FG, 2017. 2, 6
[15] C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou. Face-
warehouse: A 3D facial expression database for visual com-
puting. IEEE Trans. Vis. Comput. Graphics, 20(3):413–425,
2014. 3
[16] G. Chetty and M. Wagner. Multi-level liveness verification
for face-voice biometric authentication. In BC, 2006. 1
[17] I. Chingovska, A. Anjos, and S. Marcel. On the effectiveness
of local binary patterns in face anti-spoofing. In BIOSIG,
2012. 2, 6
[18] T. de Freitas Pereira, A. Anjos, J. M. De Martino, and S. Mar-
cel. LBP-TOP based countermeasure against face spoofing
attacks. In ACCV, pages 121–132, 2012. 1, 2
[19] T. de Freitas Pereira, A. Anjos, J. M. De Martino, and S. Mar-
cel. Can face anti-spoofing countermeasures work in a real
world scenario? In ICB, 2013. 1, 2, 8
[20] G. de Haan and V. Jeanne. Robust pulse rate from
chrominance-based rPPG. IEEE Trans. Biomedical Engi-
neering, 60(10):2878–2886, 2013. 2, 3, 4
[21] L. Feng, L.-M. Po, Y. Li, X. Xu, F. Yuan, T. C.-H. Cheung,
and K.-W. Cheung. Integration of image quality and motion
cues for face anti-spoofing: A neural network approach. J.
Visual Communication and Image Representation, 38:451–
460, 2016. 1, 2
[22] R. W. Frischholz and U. Dieckmann. BiolD: a multimodal
biometric identification system. J. Computer, 33(2):64–68,
2000. 1
[23] R. W. Frischholz and A. Werner. Avoiding replay-attacks
in a face recognition system using head-pose estimation. In
AMFGW, pages 234–235, 2003. 1
[24] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR, pages 770–778, 2016. 4
[25] A. Jourabloo and X. Liu. Pose-invariant face alignment via
CNN-based dense 3D model fitting. Int. J. Comput. Vision,
124(2):187–203, 2017. 3
[26] K. Kollreider, H. Fronthaler, M. I. Faraj, and J. Bigun. Real-
time face detection and motion analysis with application
in liveness assessment. IEEE Trans. Inf. Forens. Security,
2(3):548–558, 2007. 2
[27] J. Komulainen, A. Hadid, and M. Pietikainen. Context based
face anti-spoofing. In BTAS, 2013. 1, 2
[28] J. Komulainen, A. Hadid, M. Pietikäinen, A. Anjos, and
S. Marcel. Complementary countermeasures for detecting
scenic face spoofing attacks. In ICB, 2013. 2
[29] J. Li, Y. Wang, T. Tan, and A. K. Jain. Live face detection
based on the analysis of fourier spectra. In SPIE (BTHI),
volume 5404, pages 296–304, 2004. 1, 2
[30] L. Li, X. Feng, Z. Boulkenafet, Z. Xia, M. Li, and A. Ha-
did. An original face anti-spoofing approach using partial
convolutional neural network. In IPTA, 2016. 1, 2
[31] S. Liu, P. C. Yuen, S. Zhang, and G. Zhao. 3D mask face
anti-spoofing with remote photoplethysmography. In ECCV,
pages 85–100, 2016. 2, 3
[32] Y. Liu, A. Jourabloo, W. Ren, and X. Liu. Dense face align-
ment. In ICCVW, pages 1619–1628, 2017. 3, 4
[33] J. Määttä, A. Hadid, and M. Pietikäinen. Face spoofing de-
tection from single images using micro-texture analysis. In
IJCB, 2011. 1, 2
[34] V. Mirjalili and A. Ross. Soft biometric privacy: Retaining
biometric utility of face images while perturbing gender. In
ICB, 2017. 1
[35] E. M. Nowara, A. Sabharwal, and A. Veeraraghavan. Ppgse-
cure: Biometric presentation attack detection using photo-
pletysmograms. In FG, pages 56–62, 2017. 2, 3
[36] G. Pan, L. Sun, Z. Wu, and S. Lao. Eyeblink-based anti-
spoofing in face recognition from a generic webcamera. In
ICCV, 2007. 2

Learning_Deep_Models_for_Face_Anti-Spoofing_Binary.pdf

More Related Content

Similar to Learning_Deep_Models_for_Face_Anti-Spoofing_Binary.pdf (20)

More from MrRRajasekarCSE (13)

Recently uploaded (20)

Learning_Deep_Models_for_Face_Anti-Spoofing_Binary.pdf