![CHAPTER 1
Unconstrained Data Acquisition
Frameworks and Protocols
João C. Neves∗, Juan C. Moreno∗, Silvio Barra†, Fabio Narducci‡,
Hugo Proença∗
∗IT – Instituto de Telecomunicações, University of Beira Interior, Covilhã, Portugal
†Department of Mathematic and Computer Science, University of Cagliari, Cagliari, Italy
‡BIPLab – University of Salerno, Fisciano, Italy
Contents
1.1 Introduction 1
1.2 Unconstrained Biometric Data Acquisition Modalities 3
1.3 Typical Challenges 4
1.3.1 Optical Constraints 5
1.3.2 Non-comprehensive View of the Scene 5
1.3.3 Out-of-Focus 6
1.3.4 Calibration of Multi-camera Systems 7
1.4 Unconstrained Biometric Data Acquisition Systems 8
1.4.1 Low Resolutions Systems 9
1.4.1.1 Super-resolution Approach 9
1.4.1.2 Gait Recognition 10
1.4.2 PTZ-Based Systems 11
1.4.3 Face 15
1.4.3.1 Typical Design of Master–Slave Systems 17
1.4.3.2 Systems Based on Logical Alignment of the Cameras 19
1.4.3.3 QUIS–CAMPI System 21
1.4.3.4 Other Biometrics: Iris, Periocular, and Ear 23
1.5 Conclusions 25
References 26
1.1 INTRODUCTION
The recognition of individuals either from physical or behavioral traits,
usually denoted as biometric traits, is common practice in environments
where the subjects cooperate with the acquisition system.
In the last years, the focus has been placed on extending the robustness
of recognition methods to address less constrained scenarios with non-
cooperative subjects. Researchers have introduced different strategies to
address high variability in the age [1], pose [2], illumination [3], expres-
Human Recognition in Unconstrained Environments.
DOI: http://dx.doi.org/10.1016/B978-0-08-100705-1.00001-4
© 2017 Elsevier Ltd. All rights reserved. 1](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-22-320.jpg)
![2 Human Recognition in Unconstrained Environments
sion [4] (A-PIE), and other confounding factors such as occlusion [5] and
blur [6]. Also, these improvements have been evidenced by the performance
advances reported in unconstrained biometric datasets, such as the Labeled
Faces in the Wild (LFW) [7].
Despite these achievements, the recognition of humans in totally wild
conditions observed in visual surveillance scenarios has not been achieved,
yet. In this kind of setup, images are captured from large distances, and
the acquired data have limited discriminative capabilities, even when using
high-resolution cameras [8]. Considering that in unconstrained environ-
ments data resolution may have a greater impact on the performance than
A-PIE factors, several authors have been particularly devoted to extending
the workability of biometric data acquisition frameworks to unconstrained
scenarios in which human collaboration is not assumed.
In this chapter, we review most of the relevant frameworks and proto-
cols for acquiring biometric data in unconstrained scenarios. In Section 1.2,
we provide a comparative analysis between the different acquisition modal-
ities. The advantages of using magnifications devices, such as PTZ cameras,
are evidenced by the difference in the resolution between the eyes (in-
terpupillary distance). The resolution of biometric data acquired by PTZ
cameras at the maximum zoom is five times higher than typical surveillance
cameras. Also, the minimum resolution required to acquire high-resolution
face images (interpupillary distance greater than 60 pixels) with a stand-off
distance larger than 5 m can only be attained using PTZ devices.
Section 1.3 discusses the advantages and drawbacks of the different ac-
quisition modalities with special attention given to the use of magnification
devices in unconstrained environments. The use of a highly narrow field
of view introduces a multitude of challenges that restrict the workability
of PTZ-based systems in outdoor scenarios (inter-camera calibration) and
degrade image quality (e.g., out-of-focus, incorrect exposure).
In Section 1.4, we present a comprehensive collection of the state-
of-the-art systems for unconstrained scenarios. The systems are organized
according to the modalities of unconstrained biometric data acquisition:
(i) low-resolution systems; and (ii) PTZ-based systems. In the former, most
approaches rely on soft biometrics (e.g., gait) for recognizing individuals in
unconstrained scenarios, since the reduced discrimination of data inhibits
the use of hard biometrics. The use of these traits is only feasible when
relying on super-resolution. In the latter, the systems are grouped with re-
spect to the biometric trait that they were designed to acquire. Despite
the advantages of iris regarding recognition performance, its reduced size](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-23-320.jpg)
![Unconstrained Data Acquisition Frameworks and Protocols 3
curtails the maximum stand-off distance of these systems. Consequently,
most approaches have introduced multiple strategies to acquire facial im-
agery at large stand-off distances. The workability of these approaches in
real unconstrained environments is discussed by analyzing their feasibility
in surveillance scenarios, which are among of the most representative ex-
amples of these environments. Among these systems, particular attention
is given to the works of Park et al. [9] and Neves et al. [10], which are
two representative examples of PTZ-based systems capable of acquiring
high-resolution face images in surveillance scenarios. Finally, Section 1.5
concludes the chapter.
1.2 UNCONSTRAINED BIOMETRIC DATA ACQUISITION
MODALITIES
The acquisition of biometric data in unconstrained environments is gener-
ally performed in two distinct manners: (i) using wide-view cameras and
(ii) using magnification devices, such as PTZ cameras.
The former strategy is usually associated with CCTV cameras, which
have increased exponentially in number during the last years [11]. The
large number of such devices deployed in outdoor scenarios and their re-
duced cost are the major reasons for relying on CCTV surveillance systems.
However, in wide open scenarios, the obtained resolution is not adequate
to faithfully represent biometric data [12], restraining the recognition of
humans at-a-distance. With the rise of high-resolution cameras, they have
been considered as substitutes of old CCTV cameras and suggested as the
solution for remote human recognition. Even though high-resolution cam-
eras can be a practical solution for mid-term distances, they still cannot
outperform PTZ-based systems. Fig. 1.1 illustrates the relation between the
interpupillary distance (assumed to be 61 cm) and the stand-off distance (the
distance between the front of the lens and the subject) when using different
optical devices. In this comparison, the angle of view (AOV) of wide-view
cameras was assumed to be 70◦
, while the AOV of PTZ cameras was as-
sumed as 2.1◦
when set at the maximum zoom. This comparison shows
that, apart from the notorious differences between the resolution of the
data, the minimum number of pixels required to acquire high-resolution
face images (interpupillary distance greater than 60 pixels) with a stand-off
distance larger than 5 m can only be attained when using PTZ devices.
The use of PTZ cameras has been pointed as the most efficient and
practical way to acquire biometric data in unconstrained scenarios, since](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-24-320.jpg)
![Unconstrained Data Acquisition Frameworks and Protocols 7
using extreme zoom magnifications. Firstly, the reduced field of view of
the camera significantly reduces the amount of time the subject is imaged
(approximately 1 s), and the auto-focus mechanism is not fast enough (ap-
proximately 2 s) to seamlessly frame the subject over time. Secondly, motion
also introduces blur in the image which may mislead the contrast adjust-
ment scheme.
As an alternative, the focus lens can be manually adjusted with respect
to the distance of the subject to the camera. Given the 3D position of the
subject, it is possible to infer its distance to the camera. Then, focus is dy-
namically adjusted using a function relating the subject’s distance and the
focus lens position. In this strategy, the estimation of a 3D subject position
is regarded as the major bottleneck since it depends on the use of stereo
reconstruction techniques. However, this issue has been progressively ad-
dressed by the state-of-the-art methods since 3D information is critical for
accurate PTZ-based systems.
1.3.4 Calibration of Multi-camera Systems
Camera calibration typically refers to establishing the relationship between
the world and camera coordinate systems. Several tools have been devel-
oped to address this problem with effective results. However, when using
more than one camera, the issues increase, thus turning the calibration into
a harder problem.
In a multi-camera system, the cameras are supposed, in general, to
cooperate and share the acquisitions of the scene. Therefore, apart from
calibrating each camera separately, in such systems a mapping function be-
tween the camera streams must be defined that can turn a point in the
coordinate system of a camera into that of another.
It represents a non-trivial goal to achieve because of an important con-
straint of multi-camera systems that is related to epipolar geometry. The
epipolar geometry [13] is used to represent the geometric relations of two
points onto 2D images that come from two cameras when pointing at the
same location in the world coordinates (in a 3D space). Fig. 1.3 shows a
typical example of epipolar geometry where a shared 3D point X is ob-
served by both O1 and O2. We can see that, by changing the position of X
(see dots along the view-axis of O1), its projection X1 remains the same
but it changes in X2. Only if the relative position of the two cameras is
known, it is possible to estimate the match between the two image planes
and therefore obtain the exact measure for both cameras. Assuming that](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-28-320.jpg)
![8 Human Recognition in Unconstrained Environments
Figure 1.3 Epipolar geometry of a 3D point X over two image planes. Two cameras
with their respective centers of projection points O1 and O2 observe a point X. The
projection of X onto each of the image planes is denoted x1 and x2. Points e1 and e2 are
the epipoles.
O1 is the wide-view camera of a master–slave system and O2 is the PTZ
camera, it would not be possible to determine the pan-tilt angle necessary
to observe X by only using the information of its projection X1.
Multi-camera video surveillance systems are particularly suited to be ex-
ploited in person re-identification scenarios. Re-identification regards the
task of assigning the same identifier to all the instances of the same ob-
ject or, more specifically of the same person [14], by means of the visual
aspect obtained from an image or a video. One of the most critical chal-
lenges of person re-identification is to recognize the same person viewed by
disjoint, possibly non-overlapping cameras, at different time instants and lo-
cations [15]. Issues like tracking and indexing, camera–subject distance, and
recognition-by-parts highly degrade the performance of re-identification.
Relying on well calibrated cameras is therefore critical in order to have
an efficient video surveillance system. The challenges and approaches of
re-identification in surveillance is out of the scope of this study. Interested
readers might find a useful source in [16].
1.4 UNCONSTRAINED BIOMETRIC DATA ACQUISITION
SYSTEMS
This section presents a comprehensive collection of the state-of-the-art sys-
tems for biometric data acquisition in unconstrained video surveillance
scenarios. These frameworks can be broadly divided into two groups:
(i) CCTV systems and (ii) PTZ-based systems.
In the former, cameras are arranged using a maximum coverage strat-
egy to monitor multiple subjects in a surveillance area. These systems are
popular for their flexibility and reduced cost; however, the limited resolu-](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-29-320.jpg)
![Unconstrained Data Acquisition Frameworks and Protocols 9
tion of biometric data is regarded as their major drawback. The reduced
discriminability of the data inhibits the use of hard biometrics for recogni-
tion purposes. Consequently, two feasible approaches are commonly used
to recognize individuals in low resolution data: (i) the use of soft biomet-
ric traits (e.g., gait) and (ii) the use of super-resolution approaches to infer
a higher level of details from poor acquisitions. A short overview of such
systems is provided in Section 1.4.1.
The second group comprises systems using PTZ cameras for acquiring
high-resolution imagery of regions of interest in the scene. Challenges are
numerous but it is commonly accepted that such systems (described in
detail in Section 1.4.2) represent the most efficient and mature solution
for acquiring biometric data at-a-distance (e.g., face, iris, periocular).
1.4.1 Low Resolutions Systems
In surveillance scenarios, cameras are typically arranged in a way that max-
imizes the coverage area, thus making the biometric data acquired hard to
discriminate. Despite the vast number of factors affecting recognition per-
formance, the low-resolution of data is one of the major causes for the
hardness of human identification in surveillance environments. To over-
come this limitation, methodologies like the super-resolution or gait-based
recognition have been explored. In the following sections, some exam-
ples of recent works of these two groups are discussed. We provide a short
overview of such systems because we believe that current disadvantages of
both make them still infeasible for video surveillance scenarios.
1.4.1.1 Super-resolution Approach
Super-resolution approaches infer a high-resolution image from low-
resolution data using a pre-learnt model that relates both representa-
tions [17]. Even though the majority of the works focus on improving
data quality, some approaches have also tried to boost biometric recogni-
tion performance [18–20]. Ben-Ezra et al. [21] presented a Jitter Camera that
exploits a micro actuator for enhancing the resolution provided by a low-
resolution camera. Fig. 1.4 depicts the results attained for a low-resolution
frame acquired in a surveillance scenario. Despite these improvements, it
is commonly accepted that these approaches have to be extended to more
realistic scenarios as described in [22]. This becomes particularly evident
when trying to use such approaches to biometric recognition at-a-distance.](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-30-320.jpg)
![10 Human Recognition in Unconstrained Environments
Figure 1.4 Example of the effect of a super-resolution approach on a low resolution
image. The resolution enhancement achieved by Ben-Ezra et al. [21] starting from a low-
resolution acquisition in a surveillance scenario.
1.4.1.2 Gait Recognition
The way humans walk can also be used for identification purposes and is
usually known as gait recognition [23,24]. The advantages of gait can be
summarized in the following: (i) it can be easily measured at-a-distance,
(ii) it is difficult to disguise or occlude, and (iii) it is robust to low-
resolution images. Moreover, a recent study about the covariate factors
affecting recognition performance has found that gait is time-invariant in
the short and medium term, thus gaining special attention among reliable
biometric traits. On the other hand, gait strongly depends on the control
over clothing and footwear [25], which negatively impacts its feasibility in
surveillance scenarios.
Notwithstanding, many methods have been introduced in the literature
to optimize gait recognition systems. Ran et al. [26] used human walking
to segment and label body parts that can be helpful to perform real-time
recognition. Gait patterns were captured and stacked in a 3D data cube
containing all possible deformations. The symmetries between the patterns
were analyzed in order to measure all possible changes and correctly label
different parts of the human body.
Venkat and De Wilde [27] faced the problem of low-resolution in video
data focusing on potential information from sub-gaits that can contribute
to the recognition system. Moustakas et al. [28] exploited the height and
stride length as soft biometric traits. These features were combined in a
probabilistic framework to accurately perform gait recognition system.
Conversely, Jung et al. [29] exploited gait to estimate the head pose in
surveillance scenarios. In this approach, a 3D face model was also inferred
to improve recognition performance.](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-31-320.jpg)
![Unconstrained Data Acquisition Frameworks and Protocols 11
Choudhury and Tjahjadi [30] exploited human silhouettes extracted
from a gait system to perform recognition. By analyzing the shape of the
contours, they were able to overcome some typical side effects introduced
by the presence of noise in the gait recognition system. Considering that
this strategy is highly dependent on clothing, the authors introduced an ex-
tended version of the previous work [31] handling occlusion factors related
to variations of view, subject’s clothing and the presence of a carried item.
Nevertheless, the system requires the availability of a matching template for
any possible view of interest.
Kusakunniran [32] proposed a recognition model that directly extracts
gait features by from raw video sequences on a spatio-temporal feature
domain. They introduced the space-time interest points (STIPs) that represent
a point of interest of a dominant walking pattern, which is used to represent
the characteristics of each individual gait. The advantage of this method
is that it does not require any pre-processing of the video stream (e.g.,
background subtraction, edge detection, human silhouettes, and so on).
This makes the proposed method robust to partial occlusion caused by,
among others, carrying items or hair/clothes/footwear variations over time.
1.4.2 PTZ-Based Systems
In this section, a detailed description of PTZ-based system for uncon-
strained biometric data acquisition is provided. Existing PTZ-based systems
can be broadly divided into two groups: master–slave configuration and
single-camera configuration.
Single PTZ systems feature the advantage of trivial calibration issues.
Due to the zooming capability of these acquisition systems, once the object
of interest in the scene is detected, the pan-tilt motor can be easily managed
to keep track of it (thus ensuring that it is seamlessly centered in the video
frame). However, the engineering limitations of the pan-tilt engines should
be considered in the design. PTZ-motor introduces a significant delay that
negatively impacts the tracking performance. When using the maximum
zoom of the camera, a strong or too fine change in pan-tilt angles may
easily imply a tracking failure (details are explained in Section 1.3).
Taking into account the limitations of single PTZ systems, the major-
ity of works have focused on master–slave approaches. The typical design
of this architecture is described in Section 1.4.3.1. In spite of the multiple
advantages of the master–slave architecture, its feasibility is greatly depen-
dent on the accurate inter-camera calibration (see Section 1.3.4). The lack](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-32-320.jpg)
![12 Human Recognition in Unconstrained Environments
of depth information poses the mapping between both devices as an ill-
posed problem. To that end, several approximations have been proposed to
minimize the inter-camera mapping error.
Table 1.1 provides a comparison between PTZ-based systems for
surveillance purposes. It must be noted that these systems were not de-
signed specifically for acquiring biometric data. Notwithstanding, the per-
formance and the control of the camera(s) makes them suitable to face the
challenges of biometric detection/recognition at-a-distance.
As already mentioned, single PTZ systems have no particular con-
straints. They can be freely disposed in the working environments and do
not pose any calibration issue. The work of Kumar et al. [45] and Varcheie
and Bilodeau [46,48] are two examples of approaches using a single PTZ
device in surveillance scenarios. Pan-tilt values are adjusted to keep the
tracked subject in the central region of the camera view. In both proposals,
the zoom feature is not implemented. Therefore, they could not be used
for biometric recognition of traits like face or iris, but are indeed usable for
gait recognition.
Tracking methods based on traditional cameras or with a fixed zoom
level have the drawback of providing a variable amount of details while
an object moves far/close from/to the camera. When using PTZ cameras,
the consequence is that the details of the target become unrecognizable at
certain distances, and a larger zoom is then required. A reduced zoom is re-
quired in the opposite condition, that is, when the high zoom level implies
strong panning and tilting that might not ensure a continuous tracking. Yao
et al. [47] proposed a vision-based tracking system that exploits a PTZ cam-
era for real-time size preserving tracking. Therefore, the authors proposed
to adjust, frame by frame, the zoom level of a PTZ so that the ratio of
object’s pixels and background’s pixels is constant over time, thus preserv-
ing the resolution at which the object is tracked. Challenges are numerous:
(i) varying focal length implies a loop of parametrizations; (ii) practical
implementation of the relation between the system’s focal length and the
camera’s zoom control; (iii) feature extractions is affected by the differen-
tiation between the target’s motion and the background motion caused by
camera zooming. The authors exploited 3D affine shape methods for fast
target feature separation/grouping and a target scale estimation algorithm
based on a linear method of Structure From Motion (SFM) [49] with a
detailed perspective projection model.
Even though single PTZ systems impose few constraints for calibration
and can be freely mounted everywhere in the environment (refer to column](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-33-320.jpg)
![Table 1.1 A list of PTZ-based video surveillance systems
System Architecture Master camera Pan/tilt est. Cam. disp. Zoom Calib. marks
Lu and Payandeh [33] Master–slave Wide Exact Arbitrary Yes Yes
Xu and Song [34] Master–slave Wide Exact Arbitrary Yes No
Bodor et al. [35] Master–slave Wide Approximated Specific No Yes
Scotti et al. [36] Master–slave Catadioptric Exact Specific Yes Yes
Tarhan and Altug [37] Master–slave Catadioptric Approximated Specific No No
Chen et al. [38] Master–slave Omnidirectional Approximated Arbitrary No Yes
Krahnstoever et al. [39] Master–slave PTZ (multiple) Exact Arbitrary No No
Zhou et al. [40] Master–slave PTZ Exact Specific Yes No
Yang et al. [41] Master–slave PTZ Exact Arbitrary No Yes
Del Bimbo et al. [42] Master–slave PTZ Approximated Arbitrary Yes No
Everts et al. [43] Master–slave PTZ Approximated Arbitrary No No
Liao and Chen [44] Master–slave PTZ Approximated Specific Yes No
Kumar et al. [45] Single PTZ – – – – –
Varcheie and Bilodeau [46] Single PTZ – – – – –
Yao et al. [47] Single PTZ – – – – –
Varcheie and Bilodeau [48] Single PTZ – – – – –](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-34-320.jpg)
![14 Human Recognition in Unconstrained Environments
Camera Disposal in Table 1.1), they also have several limitations. Master–
slave systems indeed represent the most appropriate solution to address
the challenges of biometric recognition in video surveillance scenarios. As
described in Table 1.1, there are diverse configurations for master–slave sys-
tems. Most of them use two PTZ cameras where one acts like a master and
the second one like a slave. The master camera is used as a wide-view cam-
era, and therefore it is responsible for detecting and tracking objects in the
scene. The slave receives the tracking information and tracks the objects of
interest providing an alternative view of them (Yang et al. [41]).
A very complex and effective calibration procedure is proposed by Del
Bimbo et al. [42]. They exploited a pre-built map of visual 2D landmarks
of the wide area to support multi-view image matching. The landmarks
were extracted from a finite number of images taken from non-calibrated
PTZ cameras. At run-time, the features that were detected in the current
PTZ camera view are matched with those of the base set in the map. The
matches were used to localize the camera with respect to the scene and
hence estimate the position of the target body parts. Self-calibration is re-
garded as the major advantage of this approach (see column Calib. Marks in
Table 1.1). On the other hand, the dependency of stationary visual land-
marks for calibration may be problematic in dynamic surveillance scenarios
(in a crowded scene or in the presence of moving objects that significantly
change the appearance of the scene).
In [40] and [44], the authors implemented dual-PTZ systems with high
resolution images of subjects obtained by exploiting the zooming capability
of the PTZ cameras (refer to the Zoom column). No specific biometric
traits were detected, but they could be reasonably used for face detection
and tracking. Other approaches using different cameras for the wide view
of the scene have been proposed in the literature. Omnidirectional (Chen et
al. [38]) or catadioptric cameras1 (Scotti et al. [36,37]) have been exploited
in surveillance scenarios. The added value of using an omni/catadioptric
camera is that they make it feasible to seamlessly track a scene at about
360◦
.
Biometric recognition at-a-distance in surveillance (unconstrained) sce-
narios poses numerous challenges. Although the methods discussed are
good candidates, none of them has been formally proved to be effective
for human recognition. The following section explores the state-of-the-art
1 A catadioptric optical system is one where refraction and reflection are combined in an
optical system, usually via lenses (dioptrics) and curved mirrors (catoptrics).](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-35-320.jpg)
![Unconstrained Data Acquisition Frameworks and Protocols 15
and presents a collection of notable systems that achieved significantly high
level of accuracy in recognition of strong biometric traits, e.g., face and iris,
thus proving the feasibility and the potentials of such a line of research.
1.4.3 Face
By observing Table 1.2, it is evident that most systems opt to use face for
recognizing individuals in surveillance. The robustness and detectability at
long distances makes the human face the biometric trait of choice for the
surveillance scenario.
The work of Stillman et al. [51] represents one of the first attempts
where multiple cameras were combined for biometric data acquisition in
surveillance scenarios. Simple skin-color segmentation and color indexing
methods were used to locate multiple people in a calibrated space. The
proposed method demonstrated the feasibility of face detection in uncon-
trolled environment exploiting a multi-camera system. As we can see in
Table 1.2, the use of a wide-view as master camera is the most preferred op-
tion. Wide-view cameras ensure a wide coverage area thus representing the
most efficient solution in surveillance scenarios. Background subtraction is
the approach that is typically adopted for people detection and tracking.
Hampapur et al. [50] and Marchesotti et al. [53] used both background
subtraction techniques to extract the people silhouettes from the scene and
used face colors to detect and track people’s faces. Color-based techniques
are in general computationally inexpensive but are also affected by several
limitations related to illumination and occlusions. However, in surveillance
scenarios with a wide-view camera, color-based detection techniques be-
come almost the unique solution to adopt. Bernardin et al. [56] performed
human detection using fuzzy rules to simulate the natural behavior of a
human operator that allowed obtaining smoother camera handling. A KLT
tracker [63] was used to track face’s features over the time. In any case,
the detection phase of the proposed tracker relied on face colors. Mian [57]
also proposed a single PTZ-camera system to detect and track faces over the
video stream by exploiting the Camshift algorithm. As already discussed in
previous sections, using a single camera for detection and tracking avoids
the problems related to excessive calibration. However, especially when fac-
ing with biometrics, multi-camera systems become necessary to deal with
the problem of off-pose or occlusions. According to this perspective view
of the problem, Amnuaykanjanasin et al. [55] used stereo-matching and
triangulation between a pair of camera streams to estimate the 3D position](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-36-320.jpg)
![Table 1.2 A list of biometric video surveillance systems
System Architecture Master camera Pan/tilt est. Cam. disp. I.Z.S. Calib. marks
FACE
Hampapur et al. [50] Master–slave Wide (multiple) Exact Arbitrary No Yes
Stillman et al. [51] Master–slave Wide (multiple) Approximated Specific No No
Neves et al. [10] Master–slave Wide Exact Arbitrary Yes No
Wheeler et al. [52] Master–slave Wide Approximated Arbitrary No Yes
Marchesotti et al. [53] Master–slave Wide Approximated Arbitrary Yes Yes
Park et al. [9], [54] Master–slave Wide Exact Specific Yes No
Amnuaykanjanasin et al. [55] Master–slave Wide Exact Specific No No
Bernardin et al. [56] Single PTZ – – – – –
Mian [57] Single PTZ – – – – –
IRIS
Wheeler et al. [58] Master–slave Wide (multiple) Exact Specific Yes No
Yoon et al. [59] Master–slave Wide + light stripe Approximated Specific Yes Yes
Bashir et al. [60] Master–slave Wide Exact Specific No No
Venugopalan and Savvides [61] Single PTZ – – – – –
PERIOCULAR
Juefei-Xu and Savvides [62] Single PTZ – – – – –](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-37-320.jpg)
![Unconstrained Data Acquisition Frameworks and Protocols 17
of a person. The proposed method still relies on color information of the
skin to detect the faces. On the other side, the depth information from
stereo-matching ensures good estimation of the PTZ parameters to point
the camera.
Face recognition at-a-distance, although more explored than other hard
biometrics, can be still considered as an unfulfilled and promising field of
research in which improvements are expected in a recent future. In the
following sections, the design of a typical master–slave biometric system
for surveillance scenario is presented. Section 1.4.3.2 presents an inno-
vative solution to face recognition for video surveillance while the last
subsection (Section 1.4.3.3) discusses in more details a recent master–slave
system, called QUIS–CAMPI, that exploits a novel calibration technique
and automatic detection and tracking of people in-the-wild (outdoors)
video surveillance scenario.
1.4.3.1 Typical Design of Master–Slave Systems
In Fig. 1.5, an overview of a typical video surveillance system aimed at
biometric recognition is depicted. Such a system is a generalization of the
face recognition system proposed by Wheeler et al. [52], in which two
cameras cooperate in a master–slave architecture for the tracking of an in-
dividual and for the cropping of the face to achieve biometric recognition.
In master–slave architectures, the hardware usually consists of:
• A Wide Field of View camera (WFOV) that acts as a master. By pro-
viding a wide view of the scene, such cameras allow actions like the
tracking of objects/persons and detection of events of interest.
• A Narrow Field of View camera (NFOV) that acts as a slave. This
camera provides a narrowed view of the scene and allows focusing on
a single element of the scene. If such cameras provide good resolution
images, the acquisition of several biometric traits will be possible (face,
ear, periocular area, in descending size order).
The WFOV is responsible of providing the view of the whole scene
in which the system will operate. Since it is a stationary device, a back-
ground/foreground segmentation approach is applicable and thus detects
moving objects in the scene. Intrinsic and extrinsic parameters of the cam-
era have to be determined by means of a calibration procedure, so that
a mapping with the real world coordinates is provided. As well as for the
wide camera, a calibration procedure is also required for the NFOV camera.
Firstly, it needs to be calibrated with the WFOV camera:](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-38-320.jpg)
![18 Human Recognition in Unconstrained Environments
Figure 1.5 Overview of a typical video surveillance system aimed at biometric recogni-
tion. The system architecture shows a Wide-View camera and a PTZ camera operating in
a master–slave configuration to detect, track and recognize biometric data in a surveil-
lance scenario.
• The pan, tilt, and zoom values of the NFOV camera are set such that
it is in its home position;
• A homography matrix is then estimated, by creating correspondences
among the points in the wide scene and those in the narrow scene.
A further calibration is usually applied to the NFOV camera, in order to
determine how the pan, tilt, and zoom values affect the field of view of
the camera. The zoom point is calibrated in order to reduce the offset be-
tween several levels of zoom. The concept of a zoom point is introduced
in [52] and indicates the pixel location that points at the same real world
coordinates, even if the zoom factor is changing. Once the full calibration
is accomplished, it is simple to determine the pan, tilt, and zoom values of
the NFOV from the region of interest in the WFOV.
Since multiple subjects/objects may be detected and tracked in the
WFOV video, a Target Scheduler module is needed in order to keep trace of
the position of the targets (Target Records) in the video and their current
state. The scheduler passes the information regarding the position of the
target to a PTZ controller that calculates the PTZ values and zooms-in on
the detected target (the zoom value may vary depending on the resolution
of the video and on the size of the biometric trait). Once the biometric
trait is cropped, the Recognition Module handles the recognition activities](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-39-320.jpg)
![Unconstrained Data Acquisition Frameworks and Protocols 19
Figure 1.6 Schematic view of a multi-camera system using a beam splitter. The beamer
splits the light into two so that the PTZ Camera and the Static Camera can share the
same view of the scene.
(segmentation of the trait, feature extraction, matching). If a trait matches
one present in the gallery, an ID number is associated, and the Target Records
dataset is updated.
1.4.3.2 Systems Based on Logical Alignment of the Cameras
In Section 1.4.2, we described multiple surveillance systems designed to
face the issues of the calibration between pairs of cameras. Ideally, if two
identical cameras were mounted at the same point so that they could col-
lect the same view of the scene, a calibration between them would not be
necessary. Even though this configuration is not possible, the use of beam
splitter2 can mimic this process and ease the calibration between the static
and PTZ cameras. Solutions that use a beam splitter [64] are perfect ex-
amples of how to approach the problem of low-calibration constraints in
multi-camera systems. To better understand how the beam splitter works
and how a multi-camera system can be configured, see the schematic view
in Fig. 1.6.
A particularly interesting approach that relaxes the constraints of calibra-
tion was presented by Park et al. [9]. They proposed different multi-camera
systems that indirectly solve the problem of sharing the same view between
two cameras. In this approach, designated as coaxial–concentric configuration,
the cameras are mounted in a way that they are all logically aligned along
a shared view-axis. Therefore, it overcomes the problems related to the
epipolar geometry (for details, refer to Section 1.3.4). A picture of the system
proposed by Park et al. [9] is shown in Fig. 1.7.
2 A beam splitter is an optical device that splits a beam of light into two.](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-40-320.jpg)
![20 Human Recognition in Unconstrained Environments
Figure 1.7 A coaxial–concentric multi-camera system. It represents the solution pro-
posed by Park et al. [9] that uses a beam splitter in combination with a wide-view camera
and a PTZ camera to achieve a coaxial configuration.
In the system of Park et al. [9] (Fig. 1.7), the multi-camera consisted of
a hexahedral dark box with one of its sides tilted by 45 degrees and attached
to a beam splitter. PTZ camera was configured inside the dark box and the
static camera was placed outside the box. The incident beam was split at
the beam splitter and captured by PTZ and static cameras to provide almost
the same image to both of the cameras. All the camera axes were effectively
parallel in this configuration (it enables the use of a single static camera to
estimate the pan and tilt parameters of the PTZ camera). It is worth noting
that such a system ensures a high level of matching between the two camera
streams. However, the field of view does not completely overlap due to dif-
ferent camera lens and optics. As such, the authors introduced a calibration
method for estimating with minimum error the pan-tilt parameters of the
PTZ camera after a user-assisted one-time parametrization.
A similar solution was presented by Yoo et al. [65], where the wide-
view and narrow-view cameras were combined with a beam splitter to
simultaneously acquire facial and iris images. The authors combined two
sensors (an image sensor for face and infra-red sensor for irises) with a beam
splitter. The integrated dual-sensor system was therefore able to map rays
to same position in both camera sensors, thus avoiding excessive calibration
and the need of depth information.
Compared to other camera systems proposed in the literature, the ap-
proaches based on a logical alignment of the cameras feature interesting
advantages:
1. World coordinates and their matching between pairs of camera streams
are not involved in the calibration process;
2. Just a simple calibration, which mainly consists in a visual alignment
between camera streams, is required;
3. The calibrated system can be easily deployed at a different location with
no need of re-calibration.](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-41-320.jpg)
![Unconstrained Data Acquisition Frameworks and Protocols 21
Figure 1.8 Processing chain of the QUIS–CAMPI surveillance system. A master–slave ar-
chitecture is adopted for the proposed surveillance system, where the master camera is
responsible for monitoring a surveillance area and providing a set of regions of interest
(in this case the location of subjects face) to the PTZ camera.
Moreover, these approaches were already demonstrated to be feasible
for human recognition at-a-distance (rank-1 face recognition accuracy of
91.5% in case of single person tracking with a probe set of 50 subjects
against a notably larger gallery set of 10.050 subjects). However, the strict
configuration required having the camera focal points aligned which might
represent a limitation of the proposed approach in some video surveillance
scenarios since the dimensions of the system inhibit its deployment in out-
door scenarios.
1.4.3.3 QUIS–CAMPI System
Recently, Neves et al. [10,66] have introduced an alternative solution to ex-
tend PTZ-assisted facial recognition to surveillance scenarios. The authors
proposed a novel calibration algorithm [10,67] capable of accurately esti-
mating pan-tilt parameters without resorting to intermediate zoom states,
multiple optical devices or highly stringent configurations. This approach
exploits geometric cues, i.e., the vanishing points available in the scene,
to automatically estimate subjects height (h) and thus determine their 3D
position. Furthermore, the authors have built on the work of Lv et al. [68]
to ensure robustness against human shape variability during walking.
The proposed surveillance system is divided into five major mod-
ules, broadly grouped into three main phases: (i) human motion analysis,
(ii) inter-camera calibration, and (iii) camera scheduling. The workflow
chart of the surveillance system used for acquiring the QUIS–CAMPI
dataset is given in Fig. 1.8 and described in detail afterwards.
The master camera is responsible for covering the whole surveillance
area (about 650 m2) and for detecting and tracking subjects in the scene, so
that it can provide to the PTZ camera a set of facial regions. In the calibra-](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-42-320.jpg)
![22 Human Recognition in Unconstrained Environments
tion phase, the coordinates (xi(t),yi(t)) of the ith subject in the scene need
to be converted to the correspondent pan-tilt angle. However, 3D posi-
tioning is required, which involves solving the following underdetermined
equation:
λ
⎛
⎜
⎝
xi
yi
1
⎞
⎟
⎠ = Km [Rm |Tm ]
:= Pm
⎛
⎜
⎜
⎜
⎝
X
Y
Z
1
⎞
⎟
⎟
⎟
⎠
, (1.2)
where Km and [Rm |Tm ] denote the intrinsic and extrinsic matrices of
the master camera, whereas Pm represents the camera matrix.
To address this ambiguity, the existing systems either relied on highly
stringent camera disposals [52,54] or on multiple optical devices [9]. In
contrast, the authors introduced a novel calibration algorithm [10] that ex-
ploited geometric cues, i.e., the vanishing points available in the scene, to
automatically estimate subjects’ height and thus determine their 3D posi-
tion. By considering the ground as the XY plane, the Z coordinate equals
the subject height, and therefore, Eq. (1.2) can be rearranged as
λ
⎛
⎜
⎝
xi
yi
1
⎞
⎟
⎠ = [p1 p2 hp3 + p4]
⎛
⎜
⎝
X
Y
1
⎞
⎟
⎠, (1.3)
where pi is the set of column vectors of the projection matrix Pm.
The corresponding 3D position in the PTZ referential can then be
calculated using the extrinsic parameters of the camera as
⎛
⎜
⎝
Xp
Yp
Zp
⎞
⎟
⎠ = [Rp |Tp ]
⎛
⎜
⎜
⎜
⎝
X
Y
Z
1
⎞
⎟
⎟
⎟
⎠
. (1.4)
The corresponding pan and tilt angles are given by
θp = arctan
X
p
Z
p
(1.5)
and
θt = arcsin
⎛
⎜
⎝
Y
p
(X
p)2 + (Y
p)2 + (Z
p)2
⎞
⎟
⎠. (1.6)](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-43-320.jpg)
![Unconstrained Data Acquisition Frameworks and Protocols 23
When multiple targets are available in the scene, a camera scheduling
approach determines the sequence of observations that minimizes the cu-
mulative transition time, in order to start the acquisition process as soon
as possible and maximize the number of samples taken from the subjects
in the scene. Considering that this problem has no known solution that
runs in polynomial time, the authors have introduced a method capable of
inferring an approximate solution in real-time [69].
1.4.3.4 Other Biometrics: Iris, Periocular, and Ear
Commercial iris recognition systems can identify subjects with extremely
low error rates. However, they rely on highly restrictive capture volumes,
reducing their workability in less constrained scenarios. In the last years,
different works have attempted to relax the constraints of iris recognition
systems by exploiting innovative strategies to increase both the capture vol-
ume and the stand-off distance, i.e., the distance between the front of the
lens and the subject. Successful identification of humans using iris is greatly
dependent on the quality of the iris image. To be considered of acceptable
quality, the standards recommend a resolution of 200 pixels across the iris
(ISO/IEC 2004), and an in-focus image. Also, sufficient near infra-red (IR)
illumination should be ensured (more than 2 mW/cm2) without harming
human health (less than 10 mW/cm2 according to the international safety
standard IEC-60852-1). The volume of space in front of the acquisition sys-
tem where all these constraints are satisfied is denoted as the capture volume
of the system. Considering all these constraints, the design of an acquisition
framework capable of acquiring good quality iris images in unconstrained
scenarios is extremely hard, particularly at large stand-off distances. This
section reviews the most relevant works and acquisition protocols for iris
and periocular recognition at-a-distance.
In general, two strategies can be used to image iris in less constrained
scenarios: (i) the use of typical cameras and (ii) the use of magnification de-
vices. In the former, the Iris-on-the-Move system is notorious for having
significantly decreased the cooperation in image acquisition. Iris images are
acquired on-the-move while subjects walk through a portal equipped with
NIR illuminators. Another example of a widely used commercial device is
the LG IrisAccess4000. Image acquisition is performed at-a-distance; how-
ever, the user has to be directed to an optimal position so that the system
can acquire an in-focus iris image. The need for fine adjustment of the user
position arises from the limited capture volume of the system.](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-44-320.jpg)
![24 Human Recognition in Unconstrained Environments
Considering the reduced size of periocular region and iris, several ap-
proaches have exploited magnification devices, such as PTZ cameras, which
permit extending the system stand-off distance while maintaining the nec-
essary resolution for reliable iris recognition. Wheeler et al. [58] introduced
a system to acquire iris at a resolution of 200 pixels from cooperative sub-
jects at 1.5 m using a PTZ camera assisted by two wide view cameras.
Dong et al. [70] also proposed a PTZ-based system, and due to a higher
resolution of the camera they were capable of imaging iris at a distance of
3 m with more than 150 pixels. As an alternative, Yoon et al. [59] relied
on a light stripe to determine 3D position, avoiding the use of an extra-
wide camera. The eagle eye system [60] uses one wide-view camera and
three close-view cameras for capturing the facial region and the two irises.
This system uses multiple cameras with hierarchically-ordered field of view,
a highly precise pan-tilt unit, and a long focal length zoom lens. It is one
of a few example systems that can perform iris recognition at a large stand-
off distance (3–6 m). Experimental tests show good acquisition quality for
single stationary subjects of both face and irises. On the other hand, the av-
erage acquisition time is 6.1 s which does not match with the requirements
of real-time processing in non-cooperative scenarios.
Regarding periocular recognition at-a-distance, few works have been
developed. In general, the periocular region is significantly less depen-
dent on face distortions (i.e., neutral expression, smiling expression, closed
eyes, and facial occlusions) than the whole face for recognition across all
kinds of unconstrained scenarios. The work by Juefei-Xu and Savvides [62]
is considered the only notable proposal to perform periocular recogni-
tion in highly unconstrained environments. The authors utilized the 3D
generic elastic models (GEMs) [71] to correct the off-angle pose to rec-
ognize non-cooperative subjects. To deal with illumination changes, they
exploited a parallelized implementation of the anisotropic diffusion image
preprocessing algorithm running on GPUs to achieve real-time process-
ing time. In their experimental analysis, they reported a verification rate
of 60.7% (in the presence of facial expression and occlusions) but, more
notably, they attained a 16.9% performance boost over the full face ap-
proach. Notwithstanding the encouraging results achieved, the periocular
region at-a-distance still represents an unexplored field of research. The
same holds for ear recognition. Ear is another interesting small biometric
that has been proved relatively stable and has drawn researchers’ atten-
tion recently [72]. However, like other similar biometrics (e.g., iris and](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-45-320.jpg)
![CHAPTER 2
Face Recognition Using an
Outdoor Camera Network
Ching-Hui Chen, Rama Chellappa
Department of Electrical and Computer Engineering, University of Maryland, College Park, MD,
United States
Contents
2.1 Introduction 31
2.2 Taxonomy of Camera Networks 34
2.2.1 Static Camera Networks 34
2.2.2 Active Camera Networks 35
2.2.3 Characteristics of Camera Networks 35
2.3 Face Association in Camera Networks 36
2.3.1 Face-to-Face Association 36
2.3.2 Face-to-Person Association 37
2.4 Face Recognition in Outdoor Environment 38
2.4.1 Robust Descriptors for Face Recognition 38
2.4.2 Video-Based Face Recognition 39
2.4.3 Multi-view and 3D Face Recognition 39
2.4.4 Face Recognition with Context Information 41
2.4.5 Incremental Learning of Face Recognition 41
2.5 Outdoor Camera Systems 42
2.5.1 Static Camera Approach 42
2.5.2 Single PTZ Camera Approach 43
2.5.3 Master and Slave Camera Approach 44
2.5.3.1 Camera Calibration 44
2.5.3.2 Camera Control 46
2.5.3.3 Face Recognition 47
2.5.4 Distributed Active Camera Networks 48
2.6 Remaining Challenges and Emerging Techniques 49
2.7 Conclusions 50
References 50
2.1 INTRODUCTION
Outdoor camera networks have several applications in surveillance and
scene understanding. Several prior works have investigated multiple person
tracking [28,43,54], analysis of group behaviors [16,15], anomaly detec-
Human Recognition in Unconstrained Environments.
DOI: http://dx.doi.org/10.1016/B978-0-08-100705-1.00002-6
© 2017 Elsevier Ltd. All rights reserved. 31](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-52-320.jpg)
![32 Human Recognition in Unconstrained Environments
tion [49], person re-identification [17], and face recognition [29,18,19,8]
in camera networks. Face recognition in outdoor camera networks is par-
ticularly of interest in surveillance system for identifying persons of interest.
Besides, the identities of subjects in the monitored area can be useful in-
formation for high-level understanding and description of scenes [60]. As
persons in the monitored area are non-cooperative, the face of a person
is only visible to a subset of cameras. Hence, information collected from
each camera should be jointly utilized to determine the identity of the
subject. Unlike person re-identification, face recognition usually requires
high-resolution images for extracting the detailed features of the face. As
human faces possess a semi-rigid structure, this enables the face recognition
method to develop 3D face models and multi-view descriptors for robust
face representation.
Camera networks can be categorized into static camera networks and
active camera networks. In static camera networks, cameras are placed
around the monitored area with preset field of views (FOVs). The ap-
pearance of a face depends on the relative viewpoints observed from the
camera sensors and the potential occlusion in the scene, which has direct
impact on the performance of recognition algorithms. Hence, prior work
in [64] has proposed a method for optimal placement of static cameras
in the scene based on the visibility of objects. Active vision techniques
have shown improvements for the task of low-level image understanding
than conventional passive vision techniques [2] by allocating the resources
based on current observations. Active camera networks usually comprise of
a mixture of static cameras and pan-tilt-zoom (PTZ) cameras. During op-
eration, PTZ cameras are continuously reconfigured such that the coverage,
resolution (target coverage), informative view, and the risk of missing the
target are properly managed to maximize the utility of the application [19,
18].
A recent research survey on active camera network is provided in [47],
and the authors propose a high-level framework for dynamic reconfigura-
tion of camera networks. This framework consists of local cameras, fusion
unit, and a reconfiguration unit. The local cameras capture information
in the environment and submit all the information to the fusion unit.
The fusion unit abstracts the manipulation of information from local cam-
eras in a centralized or distributed processing framework and outputs the
fused information. The reconfiguration unit optimizes the reconfiguration
parameters based on the fused information, resource constraints, and ob-
jectives. In centralized processing frameworks, the information from each](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-53-320.jpg)
![Face Recognition Using an Outdoor Camera Network 33
camera is conveyed to a central node for predicting the states of the ob-
servations and reconfiguring the local cameras. On the other hand, the
distributed processing of the camera networks becomes desirable when the
bandwidth and power resources are limited. In this scenario, each camera
node receives information from its neighboring nodes and performs the
tasks of prediction and reconfiguration locally.
Face association across video frames is an important component in any
face recognition algorithm that processes videos. When there are multiple
faces appearing in a camera view, robust face-to-face association methods
should track the multiple faces across the frames and avoid the potential of
identity switching. Also, face images observed from multiple views should
be properly associated for effective face recognition. When the cameras are
calibrated, the correspondence of face images observed in multiple views
can be established by geometric localization methods, e.g., triangulation.
Nevertheless, geometric localization methods demand accurate calibration
and synchronization among the cameras, and they usually require the target
to be observed by at least two calibrated cameras. Hence, these methods are
not suitable for associating the face images captured by a single PTZ camera
operating at various zoom settings. Alternatively, the association between
face images observed in multiple views can be established by utilizing the
appearance of upper body [23,22,8]. This method is effective as the human
body is more perceivable than the face. Besides, the visibility of human
body is not restricted to certain view angle as the human face does. Based
on this fact, a face-to-person technique has been developed in [8] to asso-
ciate the face in the zoomed-in mode with the person in the zoomed-out
mode. In order to effectively utilize all the captured face images for robust
recognition, face-to-face and face-to-person associations become the fun-
damental modules to ensure that the face images captured from different
cameras and various FOVs are correctly associated.
Face images captured by cameras in outdoor environments are often not
as constrained as mug shots since persons in the scene are typically non-
cooperative. Furthermore, the face images captured by cameras deployed
in outdoor environments can be affected by illumination changes, pose
variations, dynamic backgrounds, and occlusions. Moreover, the sudden
changes in PTZ settings in active camera networks can introduce motion
blur. Although constructing a 3D face model from face images enables syn-
thesis of different views for pose-invariant recognition, it typically relies on
accurate camera calibration, synchronization, and high-resolution images.](https://image.slidesharecdn.com/23111920-250512035235-2520b48d/85/Human-Recognition-In-Unconstrained-Environments-Using-Computer-Vision-Pattern-Recognition-And-Machine-Learning-Methods-For-Biometrics-Maria-De-Marsico-54-320.jpg)
Human Recognition In Unconstrained Environments Using Computer Vision Pattern Recognition And Machine Learning Methods For Biometrics Maria De Marsico
- 1. Human Recognition In Unconstrained Environments Using Computer Vision Pattern Recognition And Machine Learning Methods For Biometrics Maria De Marsico download https://ebookbell.com/product/human-recognition-in-unconstrained- environments-using-computer-vision-pattern-recognition-and- machine-learning-methods-for-biometrics-maria-de-marsico-46223840 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Human Activity Recognition Har In Healthcare 1st Edition Luigi Bibb https://ebookbell.com/product/human-activity-recognition-har-in- healthcare-1st-edition-luigi-bibb-56459020 Human Recognition At A Distance In Video 1st Edition Bir Bhanu https://ebookbell.com/product/human-recognition-at-a-distance-in- video-1st-edition-bir-bhanu-2041078 The Mirror And The Mind A History Of Selfrecognition In The Human Sciences Katja Guenther https://ebookbell.com/product/the-mirror-and-the-mind-a-history-of- selfrecognition-in-the-human-sciences-katja-guenther-51945552 Human Activity Recognition And Behaviour Analysis For Cyberphysical Systems In Smart Environments 1st Ed Liming Chen https://ebookbell.com/product/human-activity-recognition-and- behaviour-analysis-for-cyberphysical-systems-in-smart- environments-1st-ed-liming-chen-10488098
- 3. Advances In Biometrics For Secure Human Authentication And Recognition Tyler Justin Speed https://ebookbell.com/product/advances-in-biometrics-for-secure-human- authentication-and-recognition-tyler-justin-speed-4732366 Advances In Human Activity Detection And Recognition Hadr Systems Santosh Kumar Tripathy https://ebookbell.com/product/advances-in-human-activity-detection- and-recognition-hadr-systems-santosh-kumar-tripathy-56154460 Multimodal Pattern Recognition Of Social Signals In Humancomputerinteraction First Iapr Tc3 Workshop Mprss 2012 Tsukuba Japan November 11 2012 Revised Selected Papers 1st Edition Anna Pesarin https://ebookbell.com/product/multimodal-pattern-recognition-of- social-signals-in-humancomputerinteraction-first-iapr-tc3-workshop- mprss-2012-tsukuba-japan-november-11-2012-revised-selected-papers-1st- edition-anna-pesarin-4522898 Multimodal Pattern Recognition Of Social Signals In Humancomputerinteraction Third Iapr Tc3 Workshop Mprss 2014 Stockholm Sweden August 24 2014 Revised Selected Papers 1st Edition Friedhelm Schwenker https://ebookbell.com/product/multimodal-pattern-recognition-of- social-signals-in-humancomputerinteraction-third-iapr-tc3-workshop- mprss-2014-stockholm-sweden-august-24-2014-revised-selected- papers-1st-edition-friedhelm-schwenker-4974056 Multimodal Pattern Recognition Of Social Signals In Humancomputerinteraction 4th Iapr Tc 9 Workshop Mprss 2016 Cancun Mexico December 4 2016 Revised Selected Papers 1st Edition Friedhelm Schwenker https://ebookbell.com/product/multimodal-pattern-recognition-of- social-signals-in-humancomputerinteraction-4th-iapr-tc-9-workshop- mprss-2016-cancun-mexico-december-4-2016-revised-selected-papers-1st- edition-friedhelm-schwenker-5884492
- 5. Human Recognition in Unconstrained Environments Using ComputerVision, Pattern Recognition and Machine Learning Methods for Biometrics Maria De Marsico Michele Nappi Hugo Proença Edited by
- 7. This page intentionally left blank
- 8. HUMAN RECOGNITION IN UNCONSTRAINED ENVIRONMENTS Using Computer Vision, Pattern Recognition and Machine Learning Methods for Biometrics Edited by MARIA DE MARSICO MICHELE NAPPI HUGO PROENÇA AMSTERDAM • BOSTON • HEIDELBERG • LONDON • NEW YORK OXFORD • PARIS • SAN DIEGO • SAN FRANCISCO • SINGAPORE SYDNEY • TOKYO Academic Press is an imprint of Elsevier
- 9. Academic Press is an imprint of Elsevier 125 London Wall, London EC2Y 5AS, United Kingdom 525 B Street, Suite 1800, San Diego, CA 92101-4495, United States 50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom Copyright © 2017 Elsevier Ltd. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions. This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein). Notices Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary. Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility. To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein. Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-08-100705-1 For information on all Academic Press publications visit our website at https://www.elsevier.com Publisher: Nikki Levy Acquisition Editor: Tim Pitts Editorial Project Manager: Charlotte Kent Production Project Manager: Lisa Jones Designer: Mark Rogers Typeset by VTeX
- 10. CONTENTS Contributors ix Editor Biographies xi Foreword xiii 1. Unconstrained Data Acquisition Frameworks and Protocols 1 João C. Neves, Juan C. Moreno, Silvio Barra, Fabio Narducci, Hugo Proença 1.1. Introduction 1 1.2. Unconstrained Biometric Data Acquisition Modalities 3 1.3. Typical Challenges 4 1.4. Unconstrained Biometric Data Acquisition Systems 8 1.5. Conclusions 25 References 26 2. Face Recognition Using an Outdoor Camera Network 31 Ching-Hui Chen, Rama Chellappa 2.1. Introduction 31 2.2. Taxonomy of Camera Networks 34 2.3. Face Association in Camera Networks 36 2.4. Face Recognition in Outdoor Environment 38 2.5. Outdoor Camera Systems 42 2.6. Remaining Challenges and Emerging Techniques 49 2.7. Conclusions 50 References 50 3. Real Time 3D Face-Ear Recognition on Mobile Devices: New Scenarios for 3D Biometrics “in-the-Wild” 55 Michele Nappi, Stefano Ricciardi, Massimo Tistarelli 3.1. Introduction 55 3.2. 3D Capture of Face and Ear: CURRENT Methods and Suitable Options 58 3.3. Mobile Devices for Ubiquitous Face–Ear Recognition 62 3.4. The Next Step: Mobile Devices for 3D Sensing Aiming at 3D Biometric Applications 67 3.5. Conclusions and Future Scenarios 70 References 72 v
- 11. vi Contents 4. A Multiscale Sequential Fusion Approach for Handling Pupil Dilation in Iris Recognition 77 Raghunandan Pasula, Simona Crihalmeanu, Arun Ross 4.1. Introduction 77 4.2. Previous Work 82 4.3. WVU Pupil Light Reflex (PLR) Dataset 84 4.4. Impact of Pupil Dilation 87 4.5. Proposed Method 88 4.6. Experimental Results 97 4.7. Conclusions and Future Work 100 References 101 5. Iris Recognition on Mobile Devices Using Near-Infrared Images 103 Haiqing Li, Qi Zhang, Zhenan Sun 5.1. Introduction 103 5.2. Preprocessing 105 5.3. Feature Analysis 109 5.4. Multimodal Biometrics 113 5.5. Conclusions 115 References 116 6. Fingerphoto Authentication Using Smartphone Camera Captured Under Varying Environmental Conditions 119 Aakarsh Malhotra, Anush Sankaran, Apoorva Mittal, Mayank Vatsa, Richa Singh 6.1. Introduction 119 6.2. Literature Survey 121 6.3. IIITD SmartPhone Fingerphoto Database v1 124 6.4. Proposed Fingerphoto Matching Algorithm 126 6.5. Experimental Results 132 6.6. Conclusion 142 6.7. Future Work 143 Acknowledgements 143 References 143 7. Soft Biometric Attributes in the Wild: Case Study on Gender Classification 145 Modesto Castrillón-Santana, Javier Lorenzo-Navarro 7.1. Introduction 145 7.2. Biometrics in the Wild 150 7.3. Gender Classification in the Wild 152
- 12. Contents vii 7.4. Conclusions 168 References 169 8. Gait Recognition: The Wearable Solution 177 Maria De Marsico, Alessio Mecca 8.1. Machine Vision Approach 179 8.2. Floor Sensor Approach 181 8.3. Wearable Sensor Approach 182 8.4. Datasets Available for Experiments 189 8.5. An Example of a Complete System for Gait Recognition 189 8.6. Conclusions 194 References 194 9. Biometric Authentication to Access Controlled Areas Through Eye Tracking 197 Virginio Cantoni, Nahumi Nugrahaningsih, Marco Porta, Haochen Wang 9.1. Introduction 197 9.2. ATM-Like Solutions 199 9.3. Methods Based on Fixation and Scanpath Analysis 201 9.4. Methods Based on Eye/Gaze Velocity 206 9.5. Methods Based on Pupil Size 210 9.6. Methods Based on Oculomotor Features 211 9.7. Methods Based on Head Orientation 213 9.8. Conclusions 213 References 214 10. Noncooperative Biometrics: Cross-Jurisdictional Concerns 217 Mario Savastano 10.1. Introduction 217 10.2. Biometrics for Implementing Biometric Surveillance 218 10.3. Reaction to Public Opinion 219 10.4. The Early Days 220 10.5. An Interesting Clue (2007) 224 10.6. Biometric Surveillance Today 225 10.7. Conclusions 226 References 227 Index 229
- 13. This page intentionally left blank
- 14. CONTRIBUTORS Silvio Barra Department of Mathematic and Computer Science, University of Cagliari, Cagliari, Italy Virginio Cantoni Dipartimento di Ingegneria Industriale e dell’Informazione, Università di Pavia, Pavia, Italy Modesto Castrillón-Santana SIANI, Universidad de Las Palmas de Gran Canaria, Las Palmas de Gran Canaria, Spain Rama Chellappa Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, United States Ching-Hui Chen Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, United States Simona Crihalmeanu Computer Science & Engineering, Michigan State University, East Lansing, MI, USA Maria De Marsico Department of Computer Science, Sapienza University of Rome, Rome, Italy Haiqing Li Center for Research on Intelligent Perception and Computing, Institute of Automation, Chinese Academy of Sciences, Beijing, PR China Javier Lorenzo-Navarro SIANI, Universidad de Las Palmas de Gran Canaria, Las Palmas de Gran Canaria, Spain Aakarsh Malhotra IIIT Delhi, Delhi, India Alessio Mecca Department of Computer Science, Sapienza University of Rome, Rome, Italy Apoorva Mittal IIIT Delhi, Delhi, India Juan C. Moreno IT – Instituto de Telecomunicações, University of Beira Interior, Covilhã, Portugal Michele Nappi Department of Information Technology, University of Salerno, Fisciano, Italy ix
- 15. x Contributors Fabio Narducci BIPLab – University of Salerno, Fisciano, Italy João C. Neves IT – Instituto de Telecomunicações, University of Beira Interior, Covilhã, Portugal Nahumi Nugrahaningsih Dipartimento di Ingegneria Industriale e dell’Informazione, Università di Pavia, Pavia, Italy Raghunandan Pasula Computer Science & Engineering, Michigan State University, East Lansing, MI, USA Marco Porta Dipartimento di Ingegneria Industriale e dell’Informazione, Università di Pavia, Pavia, Italy Hugo Proença IT – Instituto de Telecomunicações, University of Beira Interior, Covilhã, Portugal Stefano Ricciardi Department of Biosciences, University of Molise, Italy Arun Ross Computer Science & Engineering, Michigan State University, East Lansing, MI, USA Anush Sankaran IIIT Delhi, Delhi, India Mario Savastano Institute of Biostructures and Bioimaging (IBB)/National Research Council of Italy (CNR), Napoli, Italy Richa Singh IIIT Delhi, Delhi, India Zhenan Sun Center for Research on Intelligent Perception and Computing, Institute of Automation, Chinese Academy of Sciences, Beijing, PR China Massimo Tistarelli Department of Communication Sciences and Information Technology, University of Sassari, Sassari, Italy Mayank Vatsa IIIT Delhi, Delhi, India Haochen Wang Dipartimento di Ingegneria Industriale e dell’Informazione, Università di Pavia, Pavia, Italy Qi Zhang Center for Research on Intelligent Perception and Computing, Institute of Automation, Chinese Academy of Sciences, Beijing, PR China
- 16. EDITOR BIOGRAPHIES Maria De Marsico is an Associate Professor at Sapienza University of Rome, Department of Computer Science. She got her Master’s degree in Computer Science from University of Salerno. Her scientific interests focus on Image Processing and Human–Computer Interaction. Re- garding the first one, she works on biometric recognition, including face, iris, gate, and multimodal recognition. Re- garding the second one, she is especially interested in multimodal interac- tion, accessibility for users with special needs, and advanced techniques for personalized distance learning. She is an Associate Editor of Pattern Recog- nition Letters, and Area Editor of the IEEE Biometrics Compendium. She published about 100 scientific works in international journals, conferences, and book chapters. She has been a member of many Technical program Committees and is referee for several top journals, and Program Chair for the International Conference on Pattern Recognition Applications and Methods since 2013. Michele Nappi was born in Naples, Italy, in 1965. He received the laurea degree (cum laude) in Computer Science from the Univer- sity of Salerno, Salerno, Italy, in 1991, the MSc degree in information and communication technology from I.I.A.S.S. “E.R. Caianiello”, Vietri sul Mare, Salerno, and the PhD degree in applied mathematics and com- puter science from the University of Padova, Padova, Italy. He is currently an Associate Professor of Computer Science at the University of Salerno. His research interests include Multibiometric Systems, Pattern Recogni- tion, Image Processing, Compression and Indexing, Multimedia Databases, Human–Computer Interaction, VR/AR. He has co-authored over 120 papers in international conferences, peer review journals and book chap- ters in these fields (see http://www.informatik.uni-trier.de/~ley/pers/hd/ n/Nappi:Michele.html). He also served as a Guest Editor for several in- ternational journals and as Editor for International Books. In 2014, he was one of the founders of the spin-off BS3 (Biometric System for Security and Safety), president of the Italian Chapter of the IEEE Biometrics Council (2015–2017), member of IAPR and IEEE. He is the team leader of the Biometric and Image Processing Lab (BIPLAB). Dr. Nappi received several international awards for scientific and research activities. xi
- 17. xii Editor Biographies Hugo Proença BSc (2001), MSc (2004), PhD (2007) and Habilitation (Agregado, 2016) is an Associate Profes- sor and the current Head of the Department (2015–2017) of Computer Science, University of Beira Interior. He has been researching mainly about biometrics and visual- surveillance, particularly in developing human recognition solutions able to work on degraded data, resulting from unconstrained data acquisition protocols. He is an associate editor of the Image and Vision Computing Journal, the coordinating editor of the IEEE Biometrics Council Newsletter and the area editor (ocular biometrics) of the IEEE Biometrics Compendium Journal. Also, he is a member of the Editorial Board of the International Journal of Biometrics and served as a Guest Editor of special issues of the Pattern Recognition Letters, Image and Vision Computing and Signal, Image and Video Processing journals.
- 18. FOREWORD The problem of recognizing people using biometric techniques might be considered “solved” for certain instances of carefully controlled and con- strained environments. For a controlled and not-too-large population size, and with constrained acquisition of biometric samples, various biomet- ric techniques may be able to achieve acceptable accuracy for a specified level of security. However, the story of biometric research is that success in controlled and constrained scenarios only increases the desire for similar levels of success in more general and less-constrained ones. And so there is continual interest in pushing the limits of human recognition in more challenging conditions. Thus this book, Human Recognition in Outdoor Un- constrained Environments, edited by Maria De Marsico, Michele Nappi, and Hugo Proença, arrives at a good time and should find a broad and eager audience. The editors have assembled ten chapters, attracting contributions by eminent biometrics research groups from around the world. The con- tributed chapters cover a broad range of current topics consistent with the theme of recognition under less-constrained and outdoor conditions. The biometric modalities of face, ear, iris, fingerprint, gait, and eye tracking are all represented. The “soft biometric” of gender classification, a perennially popular research topic, is also represented. Neves and colleagues provide a solid overview chapter for general issues arising in biometric data acquisition in less-constrained scenarios. They dis- cuss the typical challenges of optical distortions, non-comprehensive view of the scene, out-of-focus images, and calibration of multi-camera systems, as a background for better understanding the complexity of biometric data acquisition in less-constrained scenarios. Each of these issues will arise again in the succeeding chapters. Chen and Chellappa discuss the topic of face recognition in the context of an active camera network located in an outdoor environment. This is, of course, a quite ambitious goal, combining the unconstrained nature of the outdoor environment, the unconstrained nature of subjects in a surveillance context, and the complexity of an active camera network. This chapter will be interesting to anyone wanting to better understand how to use camera networks effectively, especially in an outdoor environment. Nappi and colleagues exploit the relatively recent availability of low-cost 3D sensors on mobile devices to consider the possibility of ubiquitous 3D xiii
- 19. xiv Foreword face and ear recognition. Work on 3D face recognition has long been a part of the biometrics research community, and recognition based on 3D ear shape has also been a focus of substantial research. However, nearly all of this work was done with dedicated and relatively expensive 3D sensing devices. Nappi and colleagues explain that “we are very close to the point in which the imaging quality and the computing performance of mobile devices will be adequate to computer-vision tasks such as those required for 3D biometrics...”. This would certainly open up new technical possibilities for biometrics in less-constrained outdoor scenarios. Pasula and colleagues discuss the problem of handling varying pupil di- lation in iris recognition. There was a brief time early in the history of iris recognition when it was believed that the fact that people have differ- ent pupil dilation on different image acquisitions would have no effect on recognition accuracy. But it is now widely appreciated that an increased difference in pupil dilation between two images of the same iris leads to a degraded match score. And, of course, varied pupil dilation will only be more common in outdoor and less-constrained scenarios. There is already a good deal of research on handling varying pupil dilation. Pasula and col- leagues approach this problem through the use of iris codes obtained using multiple scale filters. This chapter will be of value to anyone who wants to better understand the problem of varying pupil dilation in iris matching. Sun and colleagues discuss the use of iris recognition on mobile phones. The complexities of this topic may not be immediately appreciated, espe- cially by those not working on iris recognition. Essentially all commercial iris recognition is done using images that are obtained with the sensor illu- minating the eye with near-infrared illumination. Visible-light illumination has been used for iris recognition, but the accuracy achieved is greatly re- duced. Mobile phone cameras use visible-light illumination by default. As Sun and colleagues argue, “it is necessary to mount additional near-infrared illuminators and cameras on the front panel of mobile devices”. They dis- cuss the use of super-resolution, fusion of different types of iris texture features, and fusion of iris and periocular features, as means to improve recognition accuracy. Malhotra and colleagues discuss the issues involved in acquiring finger- print images from mobile phones under varied environmental conditions that may be encountered outdoors. The problems involved with using smartphone photos for fingerprints include varying illumination, back- ground, and pose, and lower quality and resolution images than typically
- 20. Foreword xv used for fingerprinting. They have created the IIITD SmartPhone Fin- gerphoto Database, which also includes live-scan prints of the same peo- ple. Malhotra and colleagues present results of fingerphoto-to-fingerphoto matching and also fingerprint-to-fingerphoto matching. Castrillon and colleagues review work in the area of classifying the gen- der of a person from relatively unconstrained face images. They emphasize results obtained using various “in the wild” face image datasets. Anyone wanting to work in the area of gender classification from face images will want to read this chapter in order to get acquainted with the topic. Recognizing people by features associated with how they walk, or “gait recognition”, has been a topic of continued interest in the bio- metrics research community. With the widespread use of mobile phones having built-in sensors that record features associated with gait, interest in gait recognition expanded to non-vision-based scenarios. De Marsico and Mecca discuss non-vision-based gait recognition, focusing on what is termed the “wearable sensor approach”. This chapter provides a nice introduction to understanding the data and features involved in the “wear- able sensors” area. The factors that can affect gait, of course, include such things as shoe style (e.g., consider flip-flops versus high heels), clothing style (consider pants versus dresses), and walking surface (consider concrete walkway versus grassy surface after rain). Thus this is a quite ambitious area of research for extending recognition to outdoor, less-constrained environ- ments. Cantoni and colleagues present a survey of techniques related to eye- tracking biometrics. Eye tracking methods have long been used in research in human–computer interaction and in psychology. Researchers have re- cently begun to look at eye tracking as a source of data for biometric identification. The basic idea is that features extracted from observing the fixations and the scanpaths of a person’s eyes can be used to identify the person. This is typically more effective if the visual input is the same for different subjects, than if the visual input is allowed to vary between sub- jects. Cantoni and colleagues organize their survey into five major areas, based on the interaction principle and on the type of features used. Savastano closes the book with a chapter that discusses social, legal, and ethical issues arising in the use of non-cooperative biometrics across mul- tiple jurisdictions. Savastano points to the 2001 American football Super Bowl as the event where such concerns moved into the public conscious- ness in a big way. Face images of people attending this event were scanned against a “watch list” of wanted persons, and this fact was widely covered
- 21. xvi Foreword by the news media afterwards. Savastano also discusses the example of the Newham surveillance system in the United Kingdom. Naturally, as bio- metric methods become more widely used in less-constrained and outdoor scenarios, the legal, ethical, and social concerns will, at least at first, only increase. This collection of ten chapters provides an excellent overview of cur- rent research topics and issues of concern as the field of biometrics strives to bring today’s levels of accuracy on constrained and indoor biometric recog- nition to future applications that take place in less-constrained and outdoor applications. My congratulations to the editors and to all the chapter au- thors for bringing this volume to the research community. Kevin Bowyer University of Notre Dame October 2016
- 22. CHAPTER 1 Unconstrained Data Acquisition Frameworks and Protocols João C. Neves∗, Juan C. Moreno∗, Silvio Barra†, Fabio Narducci‡, Hugo Proença∗ ∗IT – Instituto de Telecomunicações, University of Beira Interior, Covilhã, Portugal †Department of Mathematic and Computer Science, University of Cagliari, Cagliari, Italy ‡BIPLab – University of Salerno, Fisciano, Italy Contents 1.1 Introduction 1 1.2 Unconstrained Biometric Data Acquisition Modalities 3 1.3 Typical Challenges 4 1.3.1 Optical Constraints 5 1.3.2 Non-comprehensive View of the Scene 5 1.3.3 Out-of-Focus 6 1.3.4 Calibration of Multi-camera Systems 7 1.4 Unconstrained Biometric Data Acquisition Systems 8 1.4.1 Low Resolutions Systems 9 1.4.1.1 Super-resolution Approach 9 1.4.1.2 Gait Recognition 10 1.4.2 PTZ-Based Systems 11 1.4.3 Face 15 1.4.3.1 Typical Design of Master–Slave Systems 17 1.4.3.2 Systems Based on Logical Alignment of the Cameras 19 1.4.3.3 QUIS–CAMPI System 21 1.4.3.4 Other Biometrics: Iris, Periocular, and Ear 23 1.5 Conclusions 25 References 26 1.1 INTRODUCTION The recognition of individuals either from physical or behavioral traits, usually denoted as biometric traits, is common practice in environments where the subjects cooperate with the acquisition system. In the last years, the focus has been placed on extending the robustness of recognition methods to address less constrained scenarios with non- cooperative subjects. Researchers have introduced different strategies to address high variability in the age [1], pose [2], illumination [3], expres- Human Recognition in Unconstrained Environments. DOI: http://dx.doi.org/10.1016/B978-0-08-100705-1.00001-4 © 2017 Elsevier Ltd. All rights reserved. 1
- 23. 2 Human Recognition in Unconstrained Environments sion [4] (A-PIE), and other confounding factors such as occlusion [5] and blur [6]. Also, these improvements have been evidenced by the performance advances reported in unconstrained biometric datasets, such as the Labeled Faces in the Wild (LFW) [7]. Despite these achievements, the recognition of humans in totally wild conditions observed in visual surveillance scenarios has not been achieved, yet. In this kind of setup, images are captured from large distances, and the acquired data have limited discriminative capabilities, even when using high-resolution cameras [8]. Considering that in unconstrained environ- ments data resolution may have a greater impact on the performance than A-PIE factors, several authors have been particularly devoted to extending the workability of biometric data acquisition frameworks to unconstrained scenarios in which human collaboration is not assumed. In this chapter, we review most of the relevant frameworks and proto- cols for acquiring biometric data in unconstrained scenarios. In Section 1.2, we provide a comparative analysis between the different acquisition modal- ities. The advantages of using magnifications devices, such as PTZ cameras, are evidenced by the difference in the resolution between the eyes (in- terpupillary distance). The resolution of biometric data acquired by PTZ cameras at the maximum zoom is five times higher than typical surveillance cameras. Also, the minimum resolution required to acquire high-resolution face images (interpupillary distance greater than 60 pixels) with a stand-off distance larger than 5 m can only be attained using PTZ devices. Section 1.3 discusses the advantages and drawbacks of the different ac- quisition modalities with special attention given to the use of magnification devices in unconstrained environments. The use of a highly narrow field of view introduces a multitude of challenges that restrict the workability of PTZ-based systems in outdoor scenarios (inter-camera calibration) and degrade image quality (e.g., out-of-focus, incorrect exposure). In Section 1.4, we present a comprehensive collection of the state- of-the-art systems for unconstrained scenarios. The systems are organized according to the modalities of unconstrained biometric data acquisition: (i) low-resolution systems; and (ii) PTZ-based systems. In the former, most approaches rely on soft biometrics (e.g., gait) for recognizing individuals in unconstrained scenarios, since the reduced discrimination of data inhibits the use of hard biometrics. The use of these traits is only feasible when relying on super-resolution. In the latter, the systems are grouped with re- spect to the biometric trait that they were designed to acquire. Despite the advantages of iris regarding recognition performance, its reduced size
- 24. Unconstrained Data Acquisition Frameworks and Protocols 3 curtails the maximum stand-off distance of these systems. Consequently, most approaches have introduced multiple strategies to acquire facial im- agery at large stand-off distances. The workability of these approaches in real unconstrained environments is discussed by analyzing their feasibility in surveillance scenarios, which are among of the most representative ex- amples of these environments. Among these systems, particular attention is given to the works of Park et al. [9] and Neves et al. [10], which are two representative examples of PTZ-based systems capable of acquiring high-resolution face images in surveillance scenarios. Finally, Section 1.5 concludes the chapter. 1.2 UNCONSTRAINED BIOMETRIC DATA ACQUISITION MODALITIES The acquisition of biometric data in unconstrained environments is gener- ally performed in two distinct manners: (i) using wide-view cameras and (ii) using magnification devices, such as PTZ cameras. The former strategy is usually associated with CCTV cameras, which have increased exponentially in number during the last years [11]. The large number of such devices deployed in outdoor scenarios and their re- duced cost are the major reasons for relying on CCTV surveillance systems. However, in wide open scenarios, the obtained resolution is not adequate to faithfully represent biometric data [12], restraining the recognition of humans at-a-distance. With the rise of high-resolution cameras, they have been considered as substitutes of old CCTV cameras and suggested as the solution for remote human recognition. Even though high-resolution cam- eras can be a practical solution for mid-term distances, they still cannot outperform PTZ-based systems. Fig. 1.1 illustrates the relation between the interpupillary distance (assumed to be 61 cm) and the stand-off distance (the distance between the front of the lens and the subject) when using different optical devices. In this comparison, the angle of view (AOV) of wide-view cameras was assumed to be 70◦ , while the AOV of PTZ cameras was as- sumed as 2.1◦ when set at the maximum zoom. This comparison shows that, apart from the notorious differences between the resolution of the data, the minimum number of pixels required to acquire high-resolution face images (interpupillary distance greater than 60 pixels) with a stand-off distance larger than 5 m can only be attained when using PTZ devices. The use of PTZ cameras has been pointed as the most efficient and practical way to acquire biometric data in unconstrained scenarios, since
- 25. 4 Human Recognition in Unconstrained Environments Figure 1.1 Relation between the interpupillary resolution and the stand-off distance when using different acquisition devices. The number of pixels between the eyes is determined with respect to the stand-off distance and four acquisition devices: (i) a typical surveillance camera (720p, 70◦); (ii) a high-resolution camera (4K, 70◦); (iii) a high-resolution camera (4K, 70◦); (iv) a PTZ camera with 15× zoom (1080p, 4.2◦); and (v) PTZ camera with 30× zoom (1080p, 2.1◦). Note the evident advantages of using PTZ cameras, the resolution of face traits is more than 5× the resolution of 8K cameras. the mechanical properties of these devices permit the acquisition of high resolution imagery at arbitrary scene locations. Although these devices can be used independently, they are usually exploited as a foveal sensor assisted by a wide-view camera. This strategy, known as a master–slave architecture, is regarded as the most efficient for acquiring biometrics at-a-distance in unconstrained scenarios, but at the same time it also introduces a multitude of challenges (refer to Section 1.3) that have been progressively addressed by different approaches (refer to Section 1.4). 1.3 TYPICAL CHALLENGES As discussed in Section 1.2, PTZ-based approaches are currently the best strategy for reliable acquisition of biometric data in outdoor environments. The zooming capabilities of such systems make it possible to image facial biometric traits that usually ask for dedicated hardware and user’s collabora- tion to be properly framed and processed at long distances. State-of-the-art PTZ cameras can achieve optical zoom magnifications up to 30× with an angle of view of about 2◦ . Despite these advantages, the use of a highly narrow field of view also entails additional challenges.
- 26. Unconstrained Data Acquisition Frameworks and Protocols 5 1.3.1 Optical Constraints The use of high zoom levels has a tremendous impact on the quality of the acquired images since optical magnification is achieved by increasing the focal distance of the camera (f ) and reducing its angle of view (AOV). As a consequence, the amount of light reaching the sensor is considerably less as the AOV decreases, which is particularly critical in outdoor scenarios where illumination is non-standard. To compensate for this effect, most cameras increase the aperture of the diaphragm (D) in the same proportion of f . The ratio between f and the aperture of the camera is denoted as the F-number (see Eq. (1.1)), and is commonly used in photography to maintain image brightness along different zoom magnifications. F-number = f D (1.1) However, its side effect is the reduction of the depth of field, which increases the chances of obtaining blurred images. As an alternative, it is possible to increase the exposure time (E) thus balancing the impact that extreme f values may have on the amount of light that reaches the sensor. However, higher values of E also increase the motion-blur level in the images. A more robust solution is to simultaneously adjust both D and E, which is, in general, the strategy adopted by PTZ devices. However, as illustrated in Fig. 1.2, the number of ideal configurations for (D,E) is greatly depen- dent on zoom magnification. 1.3.2 Non-comprehensive View of the Scene While zooming enables close inspection of narrow regions in the scene, it also inhibits scene monitoring. As a consequence, the detection and track- ing of individuals can be hardly attained when using extreme zoom levels. To mitigate this shortcoming, some systems alternate between different zoom levels, i.e., subject detection and tracking is performed at minimal zoom levels; close-up imaging of interesting subjects using maximum zoom levels is indeed used to process the details. However, zoom transition is the most time-consuming task of PTZ devices, which significantly restricts the efficiency of using a single PTZ camera for biometric recognition purposes. As an alternative, several authors have exploited a master–slave archi- tecture where the PTZ camera is assisted by a wide-view camera. The
- 27. 6 Human Recognition in Unconstrained Environments Figure 1.2 The spectrum of optical constraints with respect to the exposure time and the aperture and zoom level. The set of (E,D) combinations that produce non- degraded images decreases significantly as the focal distance increases. Besides, it is worth noting that the ideal set of (E,D) values (in white) varies with respect to the illu- mination conditions. wide-view camera stream, acting as the master, is used to globally moni- tor the scene and to detect and track the subjects. On the other hand, the PTZ camera is then used as the slave. It operates as a foveal sensor acquir- ing close-up shots from a set of interesting regions provided by the master camera. In this manner, pointing and zooming over a specific region allows acquiring a detailed view of detected subjects. It is important to ensure that the two cameras are strongly related to each other so that it is possible to find a proper correspondence of any point in 2D coordinates of PTZ camera to the wide-camera, and vice versa. Despite being the most effi- cient and practical architecture to acquire high-resolution biometric data in unconstrained environments, a master–slave system also entails additional challenges. 1.3.3 Out-of-Focus As previously discussed in Section 1.3.1, the use of extreme zoom levels significantly reduces the depth of field. To correctly adjust the focus dis- tance to the subject position in the scene, two different strategies can be exploited: (i) auto-focus and (ii) manual focus. In the former, the focus adjustment is guided by an image contrast maximization search. Despite being highly effective in wide-view cam- eras, this approach fails to provide focused images of moving subjects when
- 28. Unconstrained Data Acquisition Frameworks and Protocols 7 using extreme zoom magnifications. Firstly, the reduced field of view of the camera significantly reduces the amount of time the subject is imaged (approximately 1 s), and the auto-focus mechanism is not fast enough (ap- proximately 2 s) to seamlessly frame the subject over time. Secondly, motion also introduces blur in the image which may mislead the contrast adjust- ment scheme. As an alternative, the focus lens can be manually adjusted with respect to the distance of the subject to the camera. Given the 3D position of the subject, it is possible to infer its distance to the camera. Then, focus is dy- namically adjusted using a function relating the subject’s distance and the focus lens position. In this strategy, the estimation of a 3D subject position is regarded as the major bottleneck since it depends on the use of stereo reconstruction techniques. However, this issue has been progressively ad- dressed by the state-of-the-art methods since 3D information is critical for accurate PTZ-based systems. 1.3.4 Calibration of Multi-camera Systems Camera calibration typically refers to establishing the relationship between the world and camera coordinate systems. Several tools have been devel- oped to address this problem with effective results. However, when using more than one camera, the issues increase, thus turning the calibration into a harder problem. In a multi-camera system, the cameras are supposed, in general, to cooperate and share the acquisitions of the scene. Therefore, apart from calibrating each camera separately, in such systems a mapping function be- tween the camera streams must be defined that can turn a point in the coordinate system of a camera into that of another. It represents a non-trivial goal to achieve because of an important con- straint of multi-camera systems that is related to epipolar geometry. The epipolar geometry [13] is used to represent the geometric relations of two points onto 2D images that come from two cameras when pointing at the same location in the world coordinates (in a 3D space). Fig. 1.3 shows a typical example of epipolar geometry where a shared 3D point X is ob- served by both O1 and O2. We can see that, by changing the position of X (see dots along the view-axis of O1), its projection X1 remains the same but it changes in X2. Only if the relative position of the two cameras is known, it is possible to estimate the match between the two image planes and therefore obtain the exact measure for both cameras. Assuming that
- 29. 8 Human Recognition in Unconstrained Environments Figure 1.3 Epipolar geometry of a 3D point X over two image planes. Two cameras with their respective centers of projection points O1 and O2 observe a point X. The projection of X onto each of the image planes is denoted x1 and x2. Points e1 and e2 are the epipoles. O1 is the wide-view camera of a master–slave system and O2 is the PTZ camera, it would not be possible to determine the pan-tilt angle necessary to observe X by only using the information of its projection X1. Multi-camera video surveillance systems are particularly suited to be ex- ploited in person re-identification scenarios. Re-identification regards the task of assigning the same identifier to all the instances of the same ob- ject or, more specifically of the same person [14], by means of the visual aspect obtained from an image or a video. One of the most critical chal- lenges of person re-identification is to recognize the same person viewed by disjoint, possibly non-overlapping cameras, at different time instants and lo- cations [15]. Issues like tracking and indexing, camera–subject distance, and recognition-by-parts highly degrade the performance of re-identification. Relying on well calibrated cameras is therefore critical in order to have an efficient video surveillance system. The challenges and approaches of re-identification in surveillance is out of the scope of this study. Interested readers might find a useful source in [16]. 1.4 UNCONSTRAINED BIOMETRIC DATA ACQUISITION SYSTEMS This section presents a comprehensive collection of the state-of-the-art sys- tems for biometric data acquisition in unconstrained video surveillance scenarios. These frameworks can be broadly divided into two groups: (i) CCTV systems and (ii) PTZ-based systems. In the former, cameras are arranged using a maximum coverage strat- egy to monitor multiple subjects in a surveillance area. These systems are popular for their flexibility and reduced cost; however, the limited resolu-
- 30. Unconstrained Data Acquisition Frameworks and Protocols 9 tion of biometric data is regarded as their major drawback. The reduced discriminability of the data inhibits the use of hard biometrics for recogni- tion purposes. Consequently, two feasible approaches are commonly used to recognize individuals in low resolution data: (i) the use of soft biomet- ric traits (e.g., gait) and (ii) the use of super-resolution approaches to infer a higher level of details from poor acquisitions. A short overview of such systems is provided in Section 1.4.1. The second group comprises systems using PTZ cameras for acquiring high-resolution imagery of regions of interest in the scene. Challenges are numerous but it is commonly accepted that such systems (described in detail in Section 1.4.2) represent the most efficient and mature solution for acquiring biometric data at-a-distance (e.g., face, iris, periocular). 1.4.1 Low Resolutions Systems In surveillance scenarios, cameras are typically arranged in a way that max- imizes the coverage area, thus making the biometric data acquired hard to discriminate. Despite the vast number of factors affecting recognition per- formance, the low-resolution of data is one of the major causes for the hardness of human identification in surveillance environments. To over- come this limitation, methodologies like the super-resolution or gait-based recognition have been explored. In the following sections, some exam- ples of recent works of these two groups are discussed. We provide a short overview of such systems because we believe that current disadvantages of both make them still infeasible for video surveillance scenarios. 1.4.1.1 Super-resolution Approach Super-resolution approaches infer a high-resolution image from low- resolution data using a pre-learnt model that relates both representa- tions [17]. Even though the majority of the works focus on improving data quality, some approaches have also tried to boost biometric recogni- tion performance [18–20]. Ben-Ezra et al. [21] presented a Jitter Camera that exploits a micro actuator for enhancing the resolution provided by a low- resolution camera. Fig. 1.4 depicts the results attained for a low-resolution frame acquired in a surveillance scenario. Despite these improvements, it is commonly accepted that these approaches have to be extended to more realistic scenarios as described in [22]. This becomes particularly evident when trying to use such approaches to biometric recognition at-a-distance.
- 31. 10 Human Recognition in Unconstrained Environments Figure 1.4 Example of the effect of a super-resolution approach on a low resolution image. The resolution enhancement achieved by Ben-Ezra et al. [21] starting from a low- resolution acquisition in a surveillance scenario. 1.4.1.2 Gait Recognition The way humans walk can also be used for identification purposes and is usually known as gait recognition [23,24]. The advantages of gait can be summarized in the following: (i) it can be easily measured at-a-distance, (ii) it is difficult to disguise or occlude, and (iii) it is robust to low- resolution images. Moreover, a recent study about the covariate factors affecting recognition performance has found that gait is time-invariant in the short and medium term, thus gaining special attention among reliable biometric traits. On the other hand, gait strongly depends on the control over clothing and footwear [25], which negatively impacts its feasibility in surveillance scenarios. Notwithstanding, many methods have been introduced in the literature to optimize gait recognition systems. Ran et al. [26] used human walking to segment and label body parts that can be helpful to perform real-time recognition. Gait patterns were captured and stacked in a 3D data cube containing all possible deformations. The symmetries between the patterns were analyzed in order to measure all possible changes and correctly label different parts of the human body. Venkat and De Wilde [27] faced the problem of low-resolution in video data focusing on potential information from sub-gaits that can contribute to the recognition system. Moustakas et al. [28] exploited the height and stride length as soft biometric traits. These features were combined in a probabilistic framework to accurately perform gait recognition system. Conversely, Jung et al. [29] exploited gait to estimate the head pose in surveillance scenarios. In this approach, a 3D face model was also inferred to improve recognition performance.
- 32. Unconstrained Data Acquisition Frameworks and Protocols 11 Choudhury and Tjahjadi [30] exploited human silhouettes extracted from a gait system to perform recognition. By analyzing the shape of the contours, they were able to overcome some typical side effects introduced by the presence of noise in the gait recognition system. Considering that this strategy is highly dependent on clothing, the authors introduced an ex- tended version of the previous work [31] handling occlusion factors related to variations of view, subject’s clothing and the presence of a carried item. Nevertheless, the system requires the availability of a matching template for any possible view of interest. Kusakunniran [32] proposed a recognition model that directly extracts gait features by from raw video sequences on a spatio-temporal feature domain. They introduced the space-time interest points (STIPs) that represent a point of interest of a dominant walking pattern, which is used to represent the characteristics of each individual gait. The advantage of this method is that it does not require any pre-processing of the video stream (e.g., background subtraction, edge detection, human silhouettes, and so on). This makes the proposed method robust to partial occlusion caused by, among others, carrying items or hair/clothes/footwear variations over time. 1.4.2 PTZ-Based Systems In this section, a detailed description of PTZ-based system for uncon- strained biometric data acquisition is provided. Existing PTZ-based systems can be broadly divided into two groups: master–slave configuration and single-camera configuration. Single PTZ systems feature the advantage of trivial calibration issues. Due to the zooming capability of these acquisition systems, once the object of interest in the scene is detected, the pan-tilt motor can be easily managed to keep track of it (thus ensuring that it is seamlessly centered in the video frame). However, the engineering limitations of the pan-tilt engines should be considered in the design. PTZ-motor introduces a significant delay that negatively impacts the tracking performance. When using the maximum zoom of the camera, a strong or too fine change in pan-tilt angles may easily imply a tracking failure (details are explained in Section 1.3). Taking into account the limitations of single PTZ systems, the major- ity of works have focused on master–slave approaches. The typical design of this architecture is described in Section 1.4.3.1. In spite of the multiple advantages of the master–slave architecture, its feasibility is greatly depen- dent on the accurate inter-camera calibration (see Section 1.3.4). The lack
- 33. 12 Human Recognition in Unconstrained Environments of depth information poses the mapping between both devices as an ill- posed problem. To that end, several approximations have been proposed to minimize the inter-camera mapping error. Table 1.1 provides a comparison between PTZ-based systems for surveillance purposes. It must be noted that these systems were not de- signed specifically for acquiring biometric data. Notwithstanding, the per- formance and the control of the camera(s) makes them suitable to face the challenges of biometric detection/recognition at-a-distance. As already mentioned, single PTZ systems have no particular con- straints. They can be freely disposed in the working environments and do not pose any calibration issue. The work of Kumar et al. [45] and Varcheie and Bilodeau [46,48] are two examples of approaches using a single PTZ device in surveillance scenarios. Pan-tilt values are adjusted to keep the tracked subject in the central region of the camera view. In both proposals, the zoom feature is not implemented. Therefore, they could not be used for biometric recognition of traits like face or iris, but are indeed usable for gait recognition. Tracking methods based on traditional cameras or with a fixed zoom level have the drawback of providing a variable amount of details while an object moves far/close from/to the camera. When using PTZ cameras, the consequence is that the details of the target become unrecognizable at certain distances, and a larger zoom is then required. A reduced zoom is re- quired in the opposite condition, that is, when the high zoom level implies strong panning and tilting that might not ensure a continuous tracking. Yao et al. [47] proposed a vision-based tracking system that exploits a PTZ cam- era for real-time size preserving tracking. Therefore, the authors proposed to adjust, frame by frame, the zoom level of a PTZ so that the ratio of object’s pixels and background’s pixels is constant over time, thus preserv- ing the resolution at which the object is tracked. Challenges are numerous: (i) varying focal length implies a loop of parametrizations; (ii) practical implementation of the relation between the system’s focal length and the camera’s zoom control; (iii) feature extractions is affected by the differen- tiation between the target’s motion and the background motion caused by camera zooming. The authors exploited 3D affine shape methods for fast target feature separation/grouping and a target scale estimation algorithm based on a linear method of Structure From Motion (SFM) [49] with a detailed perspective projection model. Even though single PTZ systems impose few constraints for calibration and can be freely mounted everywhere in the environment (refer to column
- 34. Table 1.1 A list of PTZ-based video surveillance systems System Architecture Master camera Pan/tilt est. Cam. disp. Zoom Calib. marks Lu and Payandeh [33] Master–slave Wide Exact Arbitrary Yes Yes Xu and Song [34] Master–slave Wide Exact Arbitrary Yes No Bodor et al. [35] Master–slave Wide Approximated Specific No Yes Scotti et al. [36] Master–slave Catadioptric Exact Specific Yes Yes Tarhan and Altug [37] Master–slave Catadioptric Approximated Specific No No Chen et al. [38] Master–slave Omnidirectional Approximated Arbitrary No Yes Krahnstoever et al. [39] Master–slave PTZ (multiple) Exact Arbitrary No No Zhou et al. [40] Master–slave PTZ Exact Specific Yes No Yang et al. [41] Master–slave PTZ Exact Arbitrary No Yes Del Bimbo et al. [42] Master–slave PTZ Approximated Arbitrary Yes No Everts et al. [43] Master–slave PTZ Approximated Arbitrary No No Liao and Chen [44] Master–slave PTZ Approximated Specific Yes No Kumar et al. [45] Single PTZ – – – – – Varcheie and Bilodeau [46] Single PTZ – – – – – Yao et al. [47] Single PTZ – – – – – Varcheie and Bilodeau [48] Single PTZ – – – – –
- 35. 14 Human Recognition in Unconstrained Environments Camera Disposal in Table 1.1), they also have several limitations. Master– slave systems indeed represent the most appropriate solution to address the challenges of biometric recognition in video surveillance scenarios. As described in Table 1.1, there are diverse configurations for master–slave sys- tems. Most of them use two PTZ cameras where one acts like a master and the second one like a slave. The master camera is used as a wide-view cam- era, and therefore it is responsible for detecting and tracking objects in the scene. The slave receives the tracking information and tracks the objects of interest providing an alternative view of them (Yang et al. [41]). A very complex and effective calibration procedure is proposed by Del Bimbo et al. [42]. They exploited a pre-built map of visual 2D landmarks of the wide area to support multi-view image matching. The landmarks were extracted from a finite number of images taken from non-calibrated PTZ cameras. At run-time, the features that were detected in the current PTZ camera view are matched with those of the base set in the map. The matches were used to localize the camera with respect to the scene and hence estimate the position of the target body parts. Self-calibration is re- garded as the major advantage of this approach (see column Calib. Marks in Table 1.1). On the other hand, the dependency of stationary visual land- marks for calibration may be problematic in dynamic surveillance scenarios (in a crowded scene or in the presence of moving objects that significantly change the appearance of the scene). In [40] and [44], the authors implemented dual-PTZ systems with high resolution images of subjects obtained by exploiting the zooming capability of the PTZ cameras (refer to the Zoom column). No specific biometric traits were detected, but they could be reasonably used for face detection and tracking. Other approaches using different cameras for the wide view of the scene have been proposed in the literature. Omnidirectional (Chen et al. [38]) or catadioptric cameras1 (Scotti et al. [36,37]) have been exploited in surveillance scenarios. The added value of using an omni/catadioptric camera is that they make it feasible to seamlessly track a scene at about 360◦ . Biometric recognition at-a-distance in surveillance (unconstrained) sce- narios poses numerous challenges. Although the methods discussed are good candidates, none of them has been formally proved to be effective for human recognition. The following section explores the state-of-the-art 1 A catadioptric optical system is one where refraction and reflection are combined in an optical system, usually via lenses (dioptrics) and curved mirrors (catoptrics).
- 36. Unconstrained Data Acquisition Frameworks and Protocols 15 and presents a collection of notable systems that achieved significantly high level of accuracy in recognition of strong biometric traits, e.g., face and iris, thus proving the feasibility and the potentials of such a line of research. 1.4.3 Face By observing Table 1.2, it is evident that most systems opt to use face for recognizing individuals in surveillance. The robustness and detectability at long distances makes the human face the biometric trait of choice for the surveillance scenario. The work of Stillman et al. [51] represents one of the first attempts where multiple cameras were combined for biometric data acquisition in surveillance scenarios. Simple skin-color segmentation and color indexing methods were used to locate multiple people in a calibrated space. The proposed method demonstrated the feasibility of face detection in uncon- trolled environment exploiting a multi-camera system. As we can see in Table 1.2, the use of a wide-view as master camera is the most preferred op- tion. Wide-view cameras ensure a wide coverage area thus representing the most efficient solution in surveillance scenarios. Background subtraction is the approach that is typically adopted for people detection and tracking. Hampapur et al. [50] and Marchesotti et al. [53] used both background subtraction techniques to extract the people silhouettes from the scene and used face colors to detect and track people’s faces. Color-based techniques are in general computationally inexpensive but are also affected by several limitations related to illumination and occlusions. However, in surveillance scenarios with a wide-view camera, color-based detection techniques be- come almost the unique solution to adopt. Bernardin et al. [56] performed human detection using fuzzy rules to simulate the natural behavior of a human operator that allowed obtaining smoother camera handling. A KLT tracker [63] was used to track face’s features over the time. In any case, the detection phase of the proposed tracker relied on face colors. Mian [57] also proposed a single PTZ-camera system to detect and track faces over the video stream by exploiting the Camshift algorithm. As already discussed in previous sections, using a single camera for detection and tracking avoids the problems related to excessive calibration. However, especially when fac- ing with biometrics, multi-camera systems become necessary to deal with the problem of off-pose or occlusions. According to this perspective view of the problem, Amnuaykanjanasin et al. [55] used stereo-matching and triangulation between a pair of camera streams to estimate the 3D position
- 37. Table 1.2 A list of biometric video surveillance systems System Architecture Master camera Pan/tilt est. Cam. disp. I.Z.S. Calib. marks FACE Hampapur et al. [50] Master–slave Wide (multiple) Exact Arbitrary No Yes Stillman et al. [51] Master–slave Wide (multiple) Approximated Specific No No Neves et al. [10] Master–slave Wide Exact Arbitrary Yes No Wheeler et al. [52] Master–slave Wide Approximated Arbitrary No Yes Marchesotti et al. [53] Master–slave Wide Approximated Arbitrary Yes Yes Park et al. [9], [54] Master–slave Wide Exact Specific Yes No Amnuaykanjanasin et al. [55] Master–slave Wide Exact Specific No No Bernardin et al. [56] Single PTZ – – – – – Mian [57] Single PTZ – – – – – IRIS Wheeler et al. [58] Master–slave Wide (multiple) Exact Specific Yes No Yoon et al. [59] Master–slave Wide + light stripe Approximated Specific Yes Yes Bashir et al. [60] Master–slave Wide Exact Specific No No Venugopalan and Savvides [61] Single PTZ – – – – – PERIOCULAR Juefei-Xu and Savvides [62] Single PTZ – – – – –
- 38. Unconstrained Data Acquisition Frameworks and Protocols 17 of a person. The proposed method still relies on color information of the skin to detect the faces. On the other side, the depth information from stereo-matching ensures good estimation of the PTZ parameters to point the camera. Face recognition at-a-distance, although more explored than other hard biometrics, can be still considered as an unfulfilled and promising field of research in which improvements are expected in a recent future. In the following sections, the design of a typical master–slave biometric system for surveillance scenario is presented. Section 1.4.3.2 presents an inno- vative solution to face recognition for video surveillance while the last subsection (Section 1.4.3.3) discusses in more details a recent master–slave system, called QUIS–CAMPI, that exploits a novel calibration technique and automatic detection and tracking of people in-the-wild (outdoors) video surveillance scenario. 1.4.3.1 Typical Design of Master–Slave Systems In Fig. 1.5, an overview of a typical video surveillance system aimed at biometric recognition is depicted. Such a system is a generalization of the face recognition system proposed by Wheeler et al. [52], in which two cameras cooperate in a master–slave architecture for the tracking of an in- dividual and for the cropping of the face to achieve biometric recognition. In master–slave architectures, the hardware usually consists of: • A Wide Field of View camera (WFOV) that acts as a master. By pro- viding a wide view of the scene, such cameras allow actions like the tracking of objects/persons and detection of events of interest. • A Narrow Field of View camera (NFOV) that acts as a slave. This camera provides a narrowed view of the scene and allows focusing on a single element of the scene. If such cameras provide good resolution images, the acquisition of several biometric traits will be possible (face, ear, periocular area, in descending size order). The WFOV is responsible of providing the view of the whole scene in which the system will operate. Since it is a stationary device, a back- ground/foreground segmentation approach is applicable and thus detects moving objects in the scene. Intrinsic and extrinsic parameters of the cam- era have to be determined by means of a calibration procedure, so that a mapping with the real world coordinates is provided. As well as for the wide camera, a calibration procedure is also required for the NFOV camera. Firstly, it needs to be calibrated with the WFOV camera:
- 39. 18 Human Recognition in Unconstrained Environments Figure 1.5 Overview of a typical video surveillance system aimed at biometric recogni- tion. The system architecture shows a Wide-View camera and a PTZ camera operating in a master–slave configuration to detect, track and recognize biometric data in a surveil- lance scenario. • The pan, tilt, and zoom values of the NFOV camera are set such that it is in its home position; • A homography matrix is then estimated, by creating correspondences among the points in the wide scene and those in the narrow scene. A further calibration is usually applied to the NFOV camera, in order to determine how the pan, tilt, and zoom values affect the field of view of the camera. The zoom point is calibrated in order to reduce the offset be- tween several levels of zoom. The concept of a zoom point is introduced in [52] and indicates the pixel location that points at the same real world coordinates, even if the zoom factor is changing. Once the full calibration is accomplished, it is simple to determine the pan, tilt, and zoom values of the NFOV from the region of interest in the WFOV. Since multiple subjects/objects may be detected and tracked in the WFOV video, a Target Scheduler module is needed in order to keep trace of the position of the targets (Target Records) in the video and their current state. The scheduler passes the information regarding the position of the target to a PTZ controller that calculates the PTZ values and zooms-in on the detected target (the zoom value may vary depending on the resolution of the video and on the size of the biometric trait). Once the biometric trait is cropped, the Recognition Module handles the recognition activities
- 40. Unconstrained Data Acquisition Frameworks and Protocols 19 Figure 1.6 Schematic view of a multi-camera system using a beam splitter. The beamer splits the light into two so that the PTZ Camera and the Static Camera can share the same view of the scene. (segmentation of the trait, feature extraction, matching). If a trait matches one present in the gallery, an ID number is associated, and the Target Records dataset is updated. 1.4.3.2 Systems Based on Logical Alignment of the Cameras In Section 1.4.2, we described multiple surveillance systems designed to face the issues of the calibration between pairs of cameras. Ideally, if two identical cameras were mounted at the same point so that they could col- lect the same view of the scene, a calibration between them would not be necessary. Even though this configuration is not possible, the use of beam splitter2 can mimic this process and ease the calibration between the static and PTZ cameras. Solutions that use a beam splitter [64] are perfect ex- amples of how to approach the problem of low-calibration constraints in multi-camera systems. To better understand how the beam splitter works and how a multi-camera system can be configured, see the schematic view in Fig. 1.6. A particularly interesting approach that relaxes the constraints of calibra- tion was presented by Park et al. [9]. They proposed different multi-camera systems that indirectly solve the problem of sharing the same view between two cameras. In this approach, designated as coaxial–concentric configuration, the cameras are mounted in a way that they are all logically aligned along a shared view-axis. Therefore, it overcomes the problems related to the epipolar geometry (for details, refer to Section 1.3.4). A picture of the system proposed by Park et al. [9] is shown in Fig. 1.7. 2 A beam splitter is an optical device that splits a beam of light into two.
- 41. 20 Human Recognition in Unconstrained Environments Figure 1.7 A coaxial–concentric multi-camera system. It represents the solution pro- posed by Park et al. [9] that uses a beam splitter in combination with a wide-view camera and a PTZ camera to achieve a coaxial configuration. In the system of Park et al. [9] (Fig. 1.7), the multi-camera consisted of a hexahedral dark box with one of its sides tilted by 45 degrees and attached to a beam splitter. PTZ camera was configured inside the dark box and the static camera was placed outside the box. The incident beam was split at the beam splitter and captured by PTZ and static cameras to provide almost the same image to both of the cameras. All the camera axes were effectively parallel in this configuration (it enables the use of a single static camera to estimate the pan and tilt parameters of the PTZ camera). It is worth noting that such a system ensures a high level of matching between the two camera streams. However, the field of view does not completely overlap due to dif- ferent camera lens and optics. As such, the authors introduced a calibration method for estimating with minimum error the pan-tilt parameters of the PTZ camera after a user-assisted one-time parametrization. A similar solution was presented by Yoo et al. [65], where the wide- view and narrow-view cameras were combined with a beam splitter to simultaneously acquire facial and iris images. The authors combined two sensors (an image sensor for face and infra-red sensor for irises) with a beam splitter. The integrated dual-sensor system was therefore able to map rays to same position in both camera sensors, thus avoiding excessive calibration and the need of depth information. Compared to other camera systems proposed in the literature, the ap- proaches based on a logical alignment of the cameras feature interesting advantages: 1. World coordinates and their matching between pairs of camera streams are not involved in the calibration process; 2. Just a simple calibration, which mainly consists in a visual alignment between camera streams, is required; 3. The calibrated system can be easily deployed at a different location with no need of re-calibration.
- 42. Unconstrained Data Acquisition Frameworks and Protocols 21 Figure 1.8 Processing chain of the QUIS–CAMPI surveillance system. A master–slave ar- chitecture is adopted for the proposed surveillance system, where the master camera is responsible for monitoring a surveillance area and providing a set of regions of interest (in this case the location of subjects face) to the PTZ camera. Moreover, these approaches were already demonstrated to be feasible for human recognition at-a-distance (rank-1 face recognition accuracy of 91.5% in case of single person tracking with a probe set of 50 subjects against a notably larger gallery set of 10.050 subjects). However, the strict configuration required having the camera focal points aligned which might represent a limitation of the proposed approach in some video surveillance scenarios since the dimensions of the system inhibit its deployment in out- door scenarios. 1.4.3.3 QUIS–CAMPI System Recently, Neves et al. [10,66] have introduced an alternative solution to ex- tend PTZ-assisted facial recognition to surveillance scenarios. The authors proposed a novel calibration algorithm [10,67] capable of accurately esti- mating pan-tilt parameters without resorting to intermediate zoom states, multiple optical devices or highly stringent configurations. This approach exploits geometric cues, i.e., the vanishing points available in the scene, to automatically estimate subjects height (h) and thus determine their 3D position. Furthermore, the authors have built on the work of Lv et al. [68] to ensure robustness against human shape variability during walking. The proposed surveillance system is divided into five major mod- ules, broadly grouped into three main phases: (i) human motion analysis, (ii) inter-camera calibration, and (iii) camera scheduling. The workflow chart of the surveillance system used for acquiring the QUIS–CAMPI dataset is given in Fig. 1.8 and described in detail afterwards. The master camera is responsible for covering the whole surveillance area (about 650 m2) and for detecting and tracking subjects in the scene, so that it can provide to the PTZ camera a set of facial regions. In the calibra-
- 43. 22 Human Recognition in Unconstrained Environments tion phase, the coordinates (xi(t),yi(t)) of the ith subject in the scene need to be converted to the correspondent pan-tilt angle. However, 3D posi- tioning is required, which involves solving the following underdetermined equation: λ ⎛ ⎜ ⎝ xi yi 1 ⎞ ⎟ ⎠ = Km [Rm |Tm ] := Pm ⎛ ⎜ ⎜ ⎜ ⎝ X Y Z 1 ⎞ ⎟ ⎟ ⎟ ⎠ , (1.2) where Km and [Rm |Tm ] denote the intrinsic and extrinsic matrices of the master camera, whereas Pm represents the camera matrix. To address this ambiguity, the existing systems either relied on highly stringent camera disposals [52,54] or on multiple optical devices [9]. In contrast, the authors introduced a novel calibration algorithm [10] that ex- ploited geometric cues, i.e., the vanishing points available in the scene, to automatically estimate subjects’ height and thus determine their 3D posi- tion. By considering the ground as the XY plane, the Z coordinate equals the subject height, and therefore, Eq. (1.2) can be rearranged as λ ⎛ ⎜ ⎝ xi yi 1 ⎞ ⎟ ⎠ = [p1 p2 hp3 + p4] ⎛ ⎜ ⎝ X Y 1 ⎞ ⎟ ⎠, (1.3) where pi is the set of column vectors of the projection matrix Pm. The corresponding 3D position in the PTZ referential can then be calculated using the extrinsic parameters of the camera as ⎛ ⎜ ⎝ Xp Yp Zp ⎞ ⎟ ⎠ = [Rp |Tp ] ⎛ ⎜ ⎜ ⎜ ⎝ X Y Z 1 ⎞ ⎟ ⎟ ⎟ ⎠ . (1.4) The corresponding pan and tilt angles are given by θp = arctan X p Z p (1.5) and θt = arcsin ⎛ ⎜ ⎝ Y p (X p)2 + (Y p)2 + (Z p)2 ⎞ ⎟ ⎠. (1.6)
- 44. Unconstrained Data Acquisition Frameworks and Protocols 23 When multiple targets are available in the scene, a camera scheduling approach determines the sequence of observations that minimizes the cu- mulative transition time, in order to start the acquisition process as soon as possible and maximize the number of samples taken from the subjects in the scene. Considering that this problem has no known solution that runs in polynomial time, the authors have introduced a method capable of inferring an approximate solution in real-time [69]. 1.4.3.4 Other Biometrics: Iris, Periocular, and Ear Commercial iris recognition systems can identify subjects with extremely low error rates. However, they rely on highly restrictive capture volumes, reducing their workability in less constrained scenarios. In the last years, different works have attempted to relax the constraints of iris recognition systems by exploiting innovative strategies to increase both the capture vol- ume and the stand-off distance, i.e., the distance between the front of the lens and the subject. Successful identification of humans using iris is greatly dependent on the quality of the iris image. To be considered of acceptable quality, the standards recommend a resolution of 200 pixels across the iris (ISO/IEC 2004), and an in-focus image. Also, sufficient near infra-red (IR) illumination should be ensured (more than 2 mW/cm2) without harming human health (less than 10 mW/cm2 according to the international safety standard IEC-60852-1). The volume of space in front of the acquisition sys- tem where all these constraints are satisfied is denoted as the capture volume of the system. Considering all these constraints, the design of an acquisition framework capable of acquiring good quality iris images in unconstrained scenarios is extremely hard, particularly at large stand-off distances. This section reviews the most relevant works and acquisition protocols for iris and periocular recognition at-a-distance. In general, two strategies can be used to image iris in less constrained scenarios: (i) the use of typical cameras and (ii) the use of magnification de- vices. In the former, the Iris-on-the-Move system is notorious for having significantly decreased the cooperation in image acquisition. Iris images are acquired on-the-move while subjects walk through a portal equipped with NIR illuminators. Another example of a widely used commercial device is the LG IrisAccess4000. Image acquisition is performed at-a-distance; how- ever, the user has to be directed to an optimal position so that the system can acquire an in-focus iris image. The need for fine adjustment of the user position arises from the limited capture volume of the system.
- 45. 24 Human Recognition in Unconstrained Environments Considering the reduced size of periocular region and iris, several ap- proaches have exploited magnification devices, such as PTZ cameras, which permit extending the system stand-off distance while maintaining the nec- essary resolution for reliable iris recognition. Wheeler et al. [58] introduced a system to acquire iris at a resolution of 200 pixels from cooperative sub- jects at 1.5 m using a PTZ camera assisted by two wide view cameras. Dong et al. [70] also proposed a PTZ-based system, and due to a higher resolution of the camera they were capable of imaging iris at a distance of 3 m with more than 150 pixels. As an alternative, Yoon et al. [59] relied on a light stripe to determine 3D position, avoiding the use of an extra- wide camera. The eagle eye system [60] uses one wide-view camera and three close-view cameras for capturing the facial region and the two irises. This system uses multiple cameras with hierarchically-ordered field of view, a highly precise pan-tilt unit, and a long focal length zoom lens. It is one of a few example systems that can perform iris recognition at a large stand- off distance (3–6 m). Experimental tests show good acquisition quality for single stationary subjects of both face and irises. On the other hand, the av- erage acquisition time is 6.1 s which does not match with the requirements of real-time processing in non-cooperative scenarios. Regarding periocular recognition at-a-distance, few works have been developed. In general, the periocular region is significantly less depen- dent on face distortions (i.e., neutral expression, smiling expression, closed eyes, and facial occlusions) than the whole face for recognition across all kinds of unconstrained scenarios. The work by Juefei-Xu and Savvides [62] is considered the only notable proposal to perform periocular recogni- tion in highly unconstrained environments. The authors utilized the 3D generic elastic models (GEMs) [71] to correct the off-angle pose to rec- ognize non-cooperative subjects. To deal with illumination changes, they exploited a parallelized implementation of the anisotropic diffusion image preprocessing algorithm running on GPUs to achieve real-time process- ing time. In their experimental analysis, they reported a verification rate of 60.7% (in the presence of facial expression and occlusions) but, more notably, they attained a 16.9% performance boost over the full face ap- proach. Notwithstanding the encouraging results achieved, the periocular region at-a-distance still represents an unexplored field of research. The same holds for ear recognition. Ear is another interesting small biometric that has been proved relatively stable and has drawn researchers’ atten- tion recently [72]. However, like other similar biometrics (e.g., iris and
- 46. Unconstrained Data Acquisition Frameworks and Protocols 25 periocular), it is particularly hard to be managed in uncontrolled and non- cooperative environments. Currently, the recognition of human ears, with particular regard to challenges of at-a-distance scenarios, has not been faced yet, thus representing a promising and uncharted field of research which could reserve interesting opportunities and achievements in the recent fu- ture. 1.5 CONCLUSIONS Biometric recognition in-the-wild is a challenging topic with numerous open issues. However, it also represents a promising research field that is still unexplored nowadays. In this chapter, we reviewed the state-of-the-art in biometric recognition systems in unconstrained scenarios discussing the main challenges as well as the existing solutions. Despite the advances on biometric research, fully automated biomet- ric recognition systems are still at very early stages, particularly due to the limitations of the current acquisition hardware. As such, we discussed the typical problems related to optics distortions, out-of-focus, and calibration issues of multi-camera systems. Also, particular attention was given to the system stand-off distance, which is a sensible aspect of unconstrained sce- narios. The relation between the interpupillary resolution and the stand-off distance can vary significantly among different acquisition devices. Wide- field of view cameras do not represent feasible solutions for unconstrained biometric environments. Indeed, PTZ acquisition devices have been re- cently proven effective to improve the performance of surveillance systems supported by biometrics. We provided a comprehensive review of the state- of-the-art master–slave surveillance systems for acquiring biometric data at-a-distance in non-cooperative environments. In particular, we provided a comparison of the most representative works in the literature highlight- ing their strengths and weaknesses as well as their suitability to biometric recognition in unconstrained scenarios. We observed that face is the most mature and reliable biometric trait to be recognized at-a-distance. The detectability of this trait in challenging conditions as well as its robustness and identifiability justify the vast number of PTZ-based systems designed for acquiring face imagery in unconstrained scenarios. Simultaneously, the recognition of iris at-a-distance represents a new field of research that has gained significant attention. State-of-the-art ac-
- 47. 26 Human Recognition in Unconstrained Environments quisition frameworks are capable of collecting high-quality iris images up to 5 m. Despite all these achievements, biometric recognition in uncontrolled environments is still to be achieved. We hope that this review can contribute to advance this area, particularly the development of novel acquisition frameworks. REFERENCES 1. D. Gong, Z. Li, D. Lin, J. Liu, X. Tang, Hidden factor analysis for age invari- ant face recognition, in: IEEE International Conference on Computer Vision, 2013, pp. 2872–2879. 2. H. Li, G. Hua, Z. Lin, J. Brandt, J. Yang, Probabilistic elastic matching for pose variant face verification, in: IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 3499–3506. 3. F. Juefei-Xu, M. Savvides, Pokerface: partial order keeping and energy repressing method for extreme face illumination normalization, in: IEEE Conference on Bio- metrics Theory, Applications and Systems, 2015, pp. 1–8. 4. X. Zhu, Z. Lei, J. Yan, D. Yi, S. Li, High-fidelity pose and expression normalization for face recognition in the wild, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 787–796. 5. J. Wright, A. Yang, A. Ganesh, S. Sastry, Y. Ma, Robust face recognition via sparse representation, IEEE Trans. Pattern Anal. Mach. Intell. 31 (2) (2009) 210–227. 6. P. Vageeswaran, K. Mitra, R. Chellappa, Blur and illumination robust face recognition via set-theoretic characterization, IEEE Trans. Image Process. 22 (4) (2013) 1362–1372. 7. G.B.H.E. Learned-Miller, Labeled Faces in the Wild: Updates and New Reporting Procedures, Tech. Rep. UM-CS-2014-003, University of Massachusetts, Amherst, May 2014. 8. A.K. Jain, S.Z. Li, Handbook of Face Recognition, vol. 1, Springer, 2005. 9. U. Park, H.-C. Choi, A. Jain, S.-W. Lee, Face tracking and recognition at a distance: a coaxial and concentric PTZ camera system, IEEE Trans. Inf. Forensics Secur. 8 (10) (2013) 1665–1677. 10. J. Neves, J.C. Moreno, S. Barra, H. Proença, Acquiring high-resolution face images in outdoor environments: a master–slave calibration algorithm, in: IEEE 7th International Conference on Biometrics Theory, Applications and Systems, 2015, pp. 1–8. 11. M. McCahill, C. Norris, CCTV in Britain, Center for Criminology and Criminal Justice-University of Hull-United Kingdom, 2002, pp. 1–70. 12. J. Neves, F. Narducci, S. Barra, H. Proença, Biometric recognition in surveillance sce- narios: a survey, Artif. Intell. Rev. (2016) 1–27. 13. Z. Zhang, Determining the epipolar geometry and its uncertainty: a review, Int. J. Comput. Vis. 27 (2) (1998) 161–195. 14. R. Vezzani, D. Baltieri, R. Cucchiara, People reidentification in surveillance and foren- sics: a survey, ACM Comput. Surv. 46 (2) (2013) 29:1–29:37. 15. M. De Marsico, R. Distasi, S. Ricciardi, D. Riccio, A comparison of approaches for person re-identification, in: ICPRAM, 2014, pp. 189–198.
- 48. Unconstrained Data Acquisition Frameworks and Protocols 27 16. S. Gong, M. Cristani, S. Yan, C.C. Loy, Person Re-identification, vol. 1, Springer, 2014. 17. C. Liu, H.-Y. Shum, W. Freeman, Face hallucination: theory and practice, Int. J. Com- put. Vis. 75 (1) (2007) 115–134. 18. P. Hennings-Yeomans, S. Baker, B. Kumar, Simultaneous super-resolution and feature extraction for recognition of low-resolution faces, in: IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 1–8. 19. K. Jia, S. Gong, Multi-modal tensor face for simultaneous super-resolution and recog- nition, in: Tenth IEEE International Conference on Computer Vision.vol. 2, 2005, pp. 1683–1690. 20. O. Arandjelovic, R. Cipolla, A manifold approach to face recognition from low quality video across illumination and pose using implicit super-resolution, in: ICCV 2007: Proceedings of the International Conference on Computer Vision 2007, IEEE, 2007, pp. 1–8. 21. M. Ben-Ezra, A. Zomet, S.K. Nayar, Jitter camera: high resolution video from a low resolution detector, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, 2004, pp. 135–142. 22. S. Biswas, G. Aggarwal, P. Flynn, Pose-robust recognition of low-resolution face im- ages, in: IEEE Conference on Computer Vision and Pattern Recognition, 2011, pp. 601–608. 23. W. Gong, M. Sapienza, F. Cuzzolin, Fisher tensor decomposition for unconstrained gait recognition, in: Proceedings of Tensor Methods for Machine Learning, Workshop of the European Conference on Machine Learning, 2013. 24. A. Dantcheva, P. Elia, A. Ross, What else does your biometric data reveal? A survey on soft biometrics, IEEE Trans. Inf. Forensics Secur. 11 (3) (2015) 441–467. 25. D. Matovski, M. Nixon, S. Mahmoodi, J. Carter, The effect of time on gait recognition performance, IEEE Trans. Inf. Forensics Secur. 7 (2) (2012) 543–552. 26. Y. Ran, Q. Zhen, R. Chellappa, T.M. Strat, Applications of a simple characterization of human gait in surveillance, IEEE Trans. Syst. Man Cybern., Part B, Cybern. 40 (4) (2010) 1009–1020. 27. I. Venkat, P.D. Wilde, Robust gait recognition by learning and exploiting sub-gait char- acteristics, Int. J. Comput. Vis. 91 (1) (2010) 7–23. 28. K. Moustakas, D. Tzovaras, G. Stavropoulos, Gait recognition using geometric features and soft biometrics, IEEE Signal Process. Lett. 17 (4) (2010) 367–370. 29. S.-U. Jung, M. Nixon, On using gait to enhance frontal face extraction, IEEE Trans. Inf. Forensics Secur. 7 (6) (2012) 1802–1811. 30. S.D. Choudhury, T. Tjahjadi, Gait recognition based on shape and motion analysis of silhouette contours, Comput. Vis. Image Underst. 117 (12) (2013) 1770–1785. 31. S.D. Choudhury, T. Tjahjadi, Robust view-invariant multiscale gait recognition, Pattern Recognit. 48 (3) (2015) 798–811. 32. W. Kusakunniran, Recognize gaits on spatio-temporal feature domain, IEEE Trans. Inf. Forensics Secur. 9 (9) (2014) 1416–1423. 33. Y. Lu, S. Payandeh, Cooperative hybrid multi-camera tracking for people surveillance, Can. J. Electr. Comput. Eng. 33 (3/4) (2008) 145–152. 34. Y. Xu, D. Song, Systems and algorithms for autonomous and scalable crowd surveillance using robotic PTZ cameras assisted by a wide-angle camera, Auton. Robots 29 (1) (2010) 53–66.
- 49. 28 Human Recognition in Unconstrained Environments 35. R. Bodor, R. Morlok, N. Papanikolopoulos, Dual-camera system for multi-level ac- tivity recognition, in: IIEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 1, 2004, pp. 643–648. 36. G. Scotti, L. Marcenaro, C. Coelho, F. Selvaggi, C. Regazzoni, Dual camera intelligent sensor for high definition 360 degrees surveillance, IEE Proc., Vis. Image Signal Process. 152 (2) (2005) 250–257. 37. M. Tarhan, E. Atlug, A catadioptric and pan-tilt-zoom camera pair object tracking system for UAVs, J. Intell. Robot. Syst. 61 (1) (2011) 119–134. 38. C.-H. Chen, Y. Yao, D. Page, B. Abidi, A. Koschan, M. Abidi, Heterogeneous fusion of omnidirectional and PTZ cameras for multiple object tracking, IEEE Trans. Circuits Syst. Video Technol. 18 (8) (2008) 1052–1063. 39. N. Krahnstoever, T. Yu, S.-N. Lim, K. Patwardhan, P. Tu, Collaborative real-time con- trol of active cameras in large scale surveillance systems, in: Proceedings of the Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications, Mar- seille, France, 2008. 40. J. Zhou, D. Wan, Y. Wu, The chameleon-like vision system, IEEE Signal Process. Mag. 27 (5) (2010) 91–101. 41. C.-S. Yang, R.-H. Chen, C.-Y. Lee, S.-J. Lin, PTZ camera based position tracking in IP-surveillance system, in: 3rd International Conference on Sensing Technology, 2008, pp. 142–146. 42. A.D. Bimbo, F. Dini, G. Lisanti, F. Pernici, Exploiting distinctive visual landmark maps in pan-tilt-zoom camera networks, Comput. Vis. Image Underst. 114 (6) (2010) 611–623. 43. I. Everts, N. Sebe, G.A. Jones, Cooperative object tracking with multiple PTZ cameras, in: International Conference on Image Analysis and Processing, 2007, pp. 323–330. 44. H.-C. Liao, W.-Y. Chen, Eagle-eye: a dual-PTZ-camera system for target tracking in a large open area, Inf. Technol. Control 39 (3) (2015). 45. P. Kumar, A. Dick, T.S. Sheng, Real time target tracking with pan tilt zoom camera, in: Digital Image Computing: Techniques and Applications, 2009, pp. 492–497. 46. P. Varcheie, G.-A. Bilodeau, Active people tracking by a PTZ camera in IP surveillance system, in: IEEE International Workshop on Robotic and Sensors Environments, 2009, pp. 93–108. 47. Y. Yao, B. Abidi, M. Abidi, 3D target scale estimation and target feature separation for size preserving tracking in PTZ video, Int. J. Comput. Vis. 82 (3) (2009) 244–263. 48. P.D.Z. Varcheie, G.-A. Bilodeau, Adaptive fuzzy particle filter tracker for a PTZ camera in an IP surveillance system, IEEE Trans. Instrum. Meas. 60 (2) (2011) 1952–1961. 49. B. Tordoff, D. Murray, Reactive control of zoom while fixating using perspective and affine cameras, IEEE Trans. Pattern Anal. Mach. Intell. 26 (1) (2004) 98–112. 50. A. Hampapur, S. Pankanti, A. Senior, Y.-L. Tian, L. Brown, R. Bolle, Face cataloger: multi-scale imaging for relating identity to location, in: Proceedings of the IEEE Con- ference on Advance Video and Signal Based Surveillance, 2003, pp. 13–20. 51. S. Stillman, R. Tanawongsuwan, I. Essa, Tracking multiple people with multiple cam- eras, in: International Conference on Audio and Video-Based Biometric Person Au- thentication, 1999. 52. F. Wheeler, R. Weiss, P. Tu, Face recognition at a distance system for surveillance appli- cations, in: Proceedings of the Fourth IEEE International Conference on Biometrics: Theory Applications and Systems, 2010, pp. 1–8.
- 50. Unconstrained Data Acquisition Frameworks and Protocols 29 53. L. Marchesotti, S. Piva, A. Turolla, D. Minetti, C.S. Regazzoni, Cooperative multi- sensor system for real-time face detection and tracking in uncontrolled conditions, in: Electronic Imaging 2005, International Society for Optics and Photonics, 2005, pp. 100–114. 54. H.-C. Choi, U. Park, A. Jain, PTZ camera assisted face acquisition, tracking recog- nition, in: Proceedings of the Fourth IEEE International Conference on Biometrics: Theory Applications and Systems, 2010, pp. 1–6. 55. P. Amnuaykanjanasin, S. Aramvith, T.H. Chalidabhongse, Face tracking using two co- operative static and moving cameras, in: IEEE International Conference on Multimedia and Expo, 2005, pp. 1158–1161. 56. K. Bernardin, F. van de Camp, R. Stiefelhagen, Automatic person detection and track- ing using fuzzy controlled active cameras, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2007, pp. 1–8. 57. A. Mian, Realtime face detection and tracking using a single pan, tilt, zoom camera, in: 23rd International Conference in Image and Vision Computing New Zealand, 2008, pp. 1–6. 58. F.W. Wheeler, G. Abramovich, B. Yu, P.H. Tu, et al., Stand-off iris recognition sys- tem, in: 2nd IEEE International Conference on Biometrics: Theory, Applications and Systems, 2008, pp. 1–7. 59. S. Yoon, H.G. Jung, J.K. Suhr, J. Kim, Non-intrusive iris image capturing system using light stripe projection and pan-tilt-zoom camera, in: IEEE Conference on Computer Vision and Pattern Recognition, 2007, pp. 1–7. 60. F. Bashir, P. Casaverde, D. Usher, M. Friedman, Eagle-eyes: a system for iris recogni- tion at a distance, in: IEEE Conference on Technologies for Homeland Security, 2008, pp. 426–431. 61. S. Venugopalan, M. Savvides, Unconstrained iris acquisition and recognition using COTS PTZ camera, EURASIP J. Adv. Signal Process. 38 (2010). 62. F. Juefei-Xu, M. Savvides, Unconstrained periocular biometric acquisition and recog- nition using COTS PTZ camera for uncooperative and non-cooperative subjects, in: IEEE Workshop on Applications of Computer Vision, 2012, pp. 201–208. 63. B.D. Lucas, T. Kanade, An iterative image registration technique with an application to stereo vision, in: Proceedings of the 7th International Joint Conference on Artificial Intelligence, vol. 2, 1981, pp. 674–679. 64. J.Y. Han, K. Perlin, Measuring bidirectional texture reflectance with a kaleidoscope, in: Proceedings of SIGGRAPH, vol. 22, 2003, pp. 741–748. 65. J.-H. Yoo, B.J. Kang, A simply integrated dual-sensor based non-intrusive iris image acquisition system, in: IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015, pp. 113–117. 66. J. Neves, G. Santos, S. Filipe, E. Grancho, S. Barra, F. Narducci, H. Proença, Quis– campi: extending in the wild biometric recognition to surveillance environments, in: New Trends in Image Analysis and Processing – ICIAP 2015 Workshops, 2015, pp. 59–68. 67. J.C. Neves, J.C. Moreno, H. Proença, A master–slave calibration algorithm with fish-eye correction, in: Mathematical Problems in Engineering, Hindawi Publishing Corpora- tion, 2015. 68. F. Lv, T. Zhao, R. Nevatia, Self-calibration of a camera from video of a walking human, in: 16th International Conference on Pattern Recognition, vol. 1, 2002, pp. 562–567.
- 51. 30 Human Recognition in Unconstrained Environments 69. J.C. Neves, H. Proença, Dynamic camera scheduling for visual surveillance in crowded scenes using Markov random fields, in: 12th IEEE International Conference on Ad- vanced Video and Signal Based Surveillance, 2015, pp. 1–6. 70. W. Dong, Z. Sun, T. Tan, A design of iris recognition system at a distance, in: Chinese Conference on Pattern Recognition, CCPR, 2009, pp. 1–5. 71. U. Prabhu, J. Heo, M. Savvides, Unconstrained pose-invariant face recognition us- ing 3D generic elastic models, IEEE Trans. Pattern Anal. Mach. Intell. 33 (10) (2011) 1952–1961. 72. H. Chen, B. Bhanu, Human ear recognition in 3D, IEEE Trans. Pattern Anal. Mach. Intell. 29 (4) (2007) 718–737.
- 52. CHAPTER 2 Face Recognition Using an Outdoor Camera Network Ching-Hui Chen, Rama Chellappa Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, United States Contents 2.1 Introduction 31 2.2 Taxonomy of Camera Networks 34 2.2.1 Static Camera Networks 34 2.2.2 Active Camera Networks 35 2.2.3 Characteristics of Camera Networks 35 2.3 Face Association in Camera Networks 36 2.3.1 Face-to-Face Association 36 2.3.2 Face-to-Person Association 37 2.4 Face Recognition in Outdoor Environment 38 2.4.1 Robust Descriptors for Face Recognition 38 2.4.2 Video-Based Face Recognition 39 2.4.3 Multi-view and 3D Face Recognition 39 2.4.4 Face Recognition with Context Information 41 2.4.5 Incremental Learning of Face Recognition 41 2.5 Outdoor Camera Systems 42 2.5.1 Static Camera Approach 42 2.5.2 Single PTZ Camera Approach 43 2.5.3 Master and Slave Camera Approach 44 2.5.3.1 Camera Calibration 44 2.5.3.2 Camera Control 46 2.5.3.3 Face Recognition 47 2.5.4 Distributed Active Camera Networks 48 2.6 Remaining Challenges and Emerging Techniques 49 2.7 Conclusions 50 References 50 2.1 INTRODUCTION Outdoor camera networks have several applications in surveillance and scene understanding. Several prior works have investigated multiple person tracking [28,43,54], analysis of group behaviors [16,15], anomaly detec- Human Recognition in Unconstrained Environments. DOI: http://dx.doi.org/10.1016/B978-0-08-100705-1.00002-6 © 2017 Elsevier Ltd. All rights reserved. 31
- 53. 32 Human Recognition in Unconstrained Environments tion [49], person re-identification [17], and face recognition [29,18,19,8] in camera networks. Face recognition in outdoor camera networks is par- ticularly of interest in surveillance system for identifying persons of interest. Besides, the identities of subjects in the monitored area can be useful in- formation for high-level understanding and description of scenes [60]. As persons in the monitored area are non-cooperative, the face of a person is only visible to a subset of cameras. Hence, information collected from each camera should be jointly utilized to determine the identity of the subject. Unlike person re-identification, face recognition usually requires high-resolution images for extracting the detailed features of the face. As human faces possess a semi-rigid structure, this enables the face recognition method to develop 3D face models and multi-view descriptors for robust face representation. Camera networks can be categorized into static camera networks and active camera networks. In static camera networks, cameras are placed around the monitored area with preset field of views (FOVs). The ap- pearance of a face depends on the relative viewpoints observed from the camera sensors and the potential occlusion in the scene, which has direct impact on the performance of recognition algorithms. Hence, prior work in [64] has proposed a method for optimal placement of static cameras in the scene based on the visibility of objects. Active vision techniques have shown improvements for the task of low-level image understanding than conventional passive vision techniques [2] by allocating the resources based on current observations. Active camera networks usually comprise of a mixture of static cameras and pan-tilt-zoom (PTZ) cameras. During op- eration, PTZ cameras are continuously reconfigured such that the coverage, resolution (target coverage), informative view, and the risk of missing the target are properly managed to maximize the utility of the application [19, 18]. A recent research survey on active camera network is provided in [47], and the authors propose a high-level framework for dynamic reconfigura- tion of camera networks. This framework consists of local cameras, fusion unit, and a reconfiguration unit. The local cameras capture information in the environment and submit all the information to the fusion unit. The fusion unit abstracts the manipulation of information from local cam- eras in a centralized or distributed processing framework and outputs the fused information. The reconfiguration unit optimizes the reconfiguration parameters based on the fused information, resource constraints, and ob- jectives. In centralized processing frameworks, the information from each
- 54. Face Recognition Using an Outdoor Camera Network 33 camera is conveyed to a central node for predicting the states of the ob- servations and reconfiguring the local cameras. On the other hand, the distributed processing of the camera networks becomes desirable when the bandwidth and power resources are limited. In this scenario, each camera node receives information from its neighboring nodes and performs the tasks of prediction and reconfiguration locally. Face association across video frames is an important component in any face recognition algorithm that processes videos. When there are multiple faces appearing in a camera view, robust face-to-face association methods should track the multiple faces across the frames and avoid the potential of identity switching. Also, face images observed from multiple views should be properly associated for effective face recognition. When the cameras are calibrated, the correspondence of face images observed in multiple views can be established by geometric localization methods, e.g., triangulation. Nevertheless, geometric localization methods demand accurate calibration and synchronization among the cameras, and they usually require the target to be observed by at least two calibrated cameras. Hence, these methods are not suitable for associating the face images captured by a single PTZ camera operating at various zoom settings. Alternatively, the association between face images observed in multiple views can be established by utilizing the appearance of upper body [23,22,8]. This method is effective as the human body is more perceivable than the face. Besides, the visibility of human body is not restricted to certain view angle as the human face does. Based on this fact, a face-to-person technique has been developed in [8] to asso- ciate the face in the zoomed-in mode with the person in the zoomed-out mode. In order to effectively utilize all the captured face images for robust recognition, face-to-face and face-to-person associations become the fun- damental modules to ensure that the face images captured from different cameras and various FOVs are correctly associated. Face images captured by cameras in outdoor environments are often not as constrained as mug shots since persons in the scene are typically non- cooperative. Furthermore, the face images captured by cameras deployed in outdoor environments can be affected by illumination changes, pose variations, dynamic backgrounds, and occlusions. Moreover, the sudden changes in PTZ settings in active camera networks can introduce motion blur. Although constructing a 3D face model from face images enables syn- thesis of different views for pose-invariant recognition, it typically relies on accurate camera calibration, synchronization, and high-resolution images.
- 55. Random documents with unrelated content Scribd suggests to you:
- 56. CHAPTER XV DIVORCED AND MARRIED As if my troubles were not all-sufficient in themselves, Hasseena, in addition to the begging and other undesirable proclivities she had developed since the death of Makkieh, added that of thieving. She naturally devoted her talents in this direction to my friends, knowing that they would not, on my account, prosecute her. Numberless complaints came to me, and many a recommendation was made to get rid of her; but as she had been sent to me by the Khaleefa, I could not send her off without his sanction. The question also arose as to what excuse I might offer for divorcing her; to give the real reasons might end in her being stoned, mutilated, or imprisoned, and this I shrank from. I must admit, too, that, bad as she was then, I did not like the idea of throwing her over. Being in receipt of ten dollars a month, I sent word to my friends that I would save what I could to repay their losses, and do my best to break Hasseena of her bad habits. My friends warned me that if I was not careful I should find myself before the Kadi as Hasseena’s partner in crime; and the Kadi, being |186| no friend of mine, would certainly order me into prison again, which would put an end to all chances of escape. In the end Hasseena had to go. Nahoum Abbajee, my greatest friend, gave a feast at his house to celebrate the marriage of his son Yousef. Hasseena was one of the invited guests. She stole all the spoons and cutlery before the feast commenced, and also a number of articles of dress belonging to other guests, all of which she sold in the bazaar. Nahoum could overlook her stealing his property, but to
- 57. steal the property of guests under his roof was carrying matters too far. He sent word to me that I must get rid of her, and at once. Calling Hasseena to Khartoum, I was compelled to quarrel with her in such a way as to attract the attention of Hamad'na Allah, and on his asking me the reason for our constant squabbles, I told him that Hasseena was not acting as she should by me, and begged his intervention in obtaining through the Emir Yacoub the Khaleefa’s permission to divorce her. Abdullahi was “gracious,” permitted the divorce, and sent word that he would select another wife for me. This was just what I did not want. Always expecting the return of my guides, my not having a woman in the place lent probability to my having a whole night’s start upon my pursuers, for my absence might not be discovered until sunrise the following morning, at which time we went to work, and some hours more would be lost— and gained—by Hamad'na Allah and others making a thorough search for me before daring to tell the Khaleefa that I was missing. |187| Returning my thanks to Abdullahi, I asked to be left in single blessedness for a time; but to this he replied that “his heart was heavy at the loss of my child; that no man might be happy without children, and he wished me to be happy; he also wished me to have all the comforts of life, which did not exist where woman was not; that if I did not take another wife, he would believe I was not content with my life in the Soudan under his protection.” It was a long rigmarole of a message he sent, and it wound up by saying that as I had been ill for two months, he must send a wife to attend to me, and had selected for the purpose a daughter of Abd-el-Latif Terran. This was making matters worse than ever, for this girl, although brought up in the Soudan, and speaking only Arabic, was a French subject, being the granddaughter of Dr. Terran, an old employé of the Government. She was only nominally Mohammedan, and lived in the “Christian quarter.” When marriages took place in this quarter,
- 58. the Mohammedan form of marriage was gone through, and then Father Ohrwalder performed the Christian religious ceremony surreptitiously later in the day. I spoke to him about the Khaleefa’s intention, and as he knew I was already married, he advised me to try and get out of the proposed marriage by some means or another, as it would be considered binding. After casting about for excuses which I thought might appeal to the Khaleefa, I asked Hamad'na Allah to inform him that I thanked him for his selection of a wife, but as she was of European descent, had been brought up in a rich family where |188| the ladies are waited upon and never do any work, she would be no use to me, as I required some one to nurse me, do the cooking and house work, and go to the bazaar to buy food, all of which she had had servants to do for her; I therefore begged to be allowed to select a wife of the country. The latter part of my message evidently pleased the Khaleefa; it appeared to him as an earnest that I was “content,” but again he undertook the selection of the woman. When Abdullahi told any woman she was to be the wife of any one, she dare no more refuse to accept than the one she was sent to dare refuse to receive her. Fearing that he might send me some one from his hareem, I asked Nahoum and other friends to find me a wife—sharp. My object was to get her into the place before Abdullahi sent his “present,” whom, on arrival, I might send back on the plea that I was already married, and could not support two wives. Nahoum found me a wife, and sent me the following history of her.
- 59. UMM ES SHOLE AND TWO CHILDREN. Umm es Shole (the mother of Shole—Shole being the name she had given her first child) was an Abyssinian brought up from childhood in a Greek family settled in Khartoum. On reaching
- 60. womanhood, she was married to one of the sons of the family. On the fall of Khartoum, her husband, with seven male relatives, was butchered in the house in which they had taken refuge; Umm es Shole, with her three children, was taken as “property” to the Beit- el-Mal, where she was handed over as a concubine to the Emir of the Gawaamah tribe. Refusing this |189| man’s embraces, he in revenge tortured her children to death, upon which Umm es Shole escaped to Omdurman. Through Abd-el-Kader, the uncle of the Mahdi, she had her case brought before Mohammad Ahmed, who, after listening to the details, gave her a written document declaring that, as she had been married to and borne children to a free man, she was a free woman, but to make certain that she might never be claimed as a slave, the document also declared that she was “ateekh” (freed) by him. When Abdullahi succeeded the Mahdi, he ordered every woman without a husband, and every girl of a marriageable age, to be married at once. He was most particular that every one in the “Christian quarter” should be married. Umm es Shole married an old and decrepit Jew, whom she nursed until he died two years later. Returning to a woman relative of her husband’s, she supported the old woman and herself by cooking, preparing food for feasts, sewing, and general housework. This was the wife my friends had selected for me, and I accepted her thankfully; but when she was approached on the subject, she positively declined to be married again, and it was only upon her being told that I was ill, and might die, that she consented to the marriage. I had to appoint a “wakeel” (proxy, in this instance) to represent me at the marriage and the festivities; Nahoum prepared the feast at his house, the bride preparing the food and attending to the guests. At the conclusion of the few days’ ceremonies and feastings, Umm es Shole was escorted |190| to Khartoum—a married woman, and introduced for the first time to her husband. She set to
- 61. at once with her household duties and attendance upon me, and during a long and weary five months nursed me back to life. As can well be believed, Hasseena resented no less bitterly my projected marriage with Umm es Shole, or any one else, than she resented her divorce, and this she resented very bitterly indeed, for passing as the wife of a European and a presumed “General” to boot, gave her a certain social status in Omdurman, which she took advantage of when visiting in the various ways pointed out. On my saying to her, “You are divorced,” which is the only formula necessary in Mohammedan countries in such a momentous domestic affair, she promptly replied that she was again pregnant. A few words on the subject of divorce in the Soudan—and the rules are practically identical with those laid down in the Quoranic law—will assist towards an appreciation of the fix this declaration of Hasseena placed me in. If a woman, on being told “you are divorced,” declared herself with child, the husband was compelled to keep her until its birth; if it was a son, the divorce was null and void; if a daughter, the husband had to support the wife during two years of nursing, and provide for the child until her seventh year, when he might, if he chose to do so, claim her as his daughter. When a woman was divorced for the first time, she was not allowed to marry again without the consent of the husband; this was giving him a “first call” if he wanted her back, for divorce might be declared for |191| less trivial things than incompatibility of temper. If the husband took her back, and divorced her a second time, the woman was free to marry, but if the husband again wanted her, he had to pay her a marriage dowry as at her first marriage. Should he divorce her a third time, and again want her back, he would have to arrange for her to be married to—and divorced from—some one else first, when she was free to return to him. All this may sound very immoral to people in Europe, but one cannot help but admire the simplicity of the proceedings; and consider the amount of domestic
- 62. infelicity it prevented. There is no public examination of the parties concerned; no publication of interesting details in newspapers; some little thought is given to the woman who may have been the mother of your children, and should she have slipped in the path of virtue, you do not shout it from the housetops; the marriage was a private arrangement between you, so is the divorce, and the reasons for the latter are your affair and no one else’s. I have touched upon divorce in some detail, as many re-marriages under all the conditions given above occurred, and some family records became a hopeless tangle to all but those immediately concerned. When the new Soudan Government comes to settle up claims to properties, they will be confronted with a collection of “succession” puzzles to solve, for one woman might be the proud mother of the legitimate heirs of three or four different people, and being, as the widow and mother of the heritor, entitled to a fixed proportion of the properties, you |192| may be quite sure that she will fight to the death for her sons’ interests. Hasseena ought not to have been in the interesting state she declared she was, for we had been separated for a much longer period than that ordained by law. I was obliged to tell her that if she empanelled a jury, after the example of Idris es Saier, all the explanations they might offer would not convince me that I held any more relationship to the child than I did to Makkieh, and there was nothing now to induce me to claim the paternity,—indeed just the reverse. However, if Hasseena was with child, I should be bound to keep her for at least two years, and if the Khaleefa sent on his present, I should have two households to support on ten dollars a month. When making my plans for escape, Hasseena was included; she was to have got away on the same dromedary as myself. When my guides returned, they would find me with two wives, and having made arrangements for one only, they might demur at taking the two. The probabilities were they would abandon the thing altogether, fearing that one or the other might betray them, which meant
- 63. instant execution for them and imprisonment for me. If I kept Hasseena, she might steal from some stranger, as the houses of my friends were now closed to her, and then I should be sent back to the Saier; if I sent her away, she, knowing my guides and all my arrangements, would be the first to meet them on arrival in Omdurman, and would insist upon coming away with me under threats of disclosing the plot. It was a most awkward fix for me |193| to be placed in; but after considering the whole matter most carefully, I decided upon sending Hasseena off, and trusting to luck for the rest. I had hoped she might get married to some one in Omdurman, and then I should not have been afraid of her. But Hasseena returned in February, 1892, some months after my marriage with Umm es Shole, carrying a little bundle of male humanity, who had only been three or four months less tardy in arrival than Makkieh. Hasseena, doubtless, had for me the Soudan equivalent for what we understand as affection; she had saved my life when we were first captured; she had nursed me, as only a woman can nurse one, through my first attack of typhus fever, and had kept me from starvation during the famine. But while I could not forget all this, I could not forget also that she had become a source of great danger to me, and although my treatment of her in sending her away when I did, might to some appear harsh in the face of what she had done for me, it must not be forgotten that self-preservation is no less a law of nature in the Soudan than it is elsewhere. I supported Hasseena for nearly two years, when her child died. She then left Khartoum, where I was still a chained prisoner at large, and went utterly to the bad. I heard of her from time to time, and, on my release in September last, hearing that she was at Berber, I delayed there until I had hunted her out of the den of vice in which she was living, and provided for her elsewhere, only to receive a telegram a few weeks later to say that, |194| hankering for the life which she had led for a few years back, she had run off to return to it.
- 64. It was this action of mine, which probably gave rise to the legend that I had brought her to Cairo with me, where my wife arrived, “only to be confronted with a black wife after all her years of mental anxiety and sufferings.” Why facts should be so persistently misconstrued, I cannot understand. In making that last—and I do not say final—effort, to do something for the woman to whom, at one time, I owed so much, I feel I have nothing to be ashamed of. Those who think differently must remember that it takes one some little time to fall again into European ideas and thoughts after twelve years of chains and slavery amongst the people whom I was compelled to associate with; and no one in the Soudan was more out of the world than I was.
- 65. CHAPTER XVI HOPE AND DESPAIR While still a prisoner in the Saier, Mankarious Effendi, with Mohammad Fargoun and Selim Aly, engaged a man of the Ababdeh, Mohammad Ajjab, to make his way to Omdurman with a threefold object: he was to inquire if I was still alive; if so, to pay me a hundred dollars, and then to try and make arrangements for my escape. On arrival in Omdurman, Ajjab met two of his own people— Mohammad and Karrar Beshir—who recommended him, when he inquired about me, never to mention my name if he wished to keep his head on his shoulders. They could only tell him that I was still in prison, chained, and under sentence of death. Similar information and the same recommendation were given to him by people in the Muslimanieh quarter; but a Greek whom Ajjab knew only by his Mahdieh name of Abdallah, said that he would arrange for a meeting between him and my servant. Through Hasseena, Ajjab sent me word of the object of his coming to Omdurman. As the Greek offered to become my trustee, Ajjab handed him the hundred dollars, taking from him a receipt, and sending |196| the receipt to me concealed in a piece of bread, to be countersigned. Ajjab was to return to Assouan, let my friends know how matters stood, and tell them that I would try and communicate with them, if I ever got released from prison, as escape from the prison was an impossibility. Ajjab returned to Assouan, and handed over the receipt; but the tale he had to tell put an end, for the time being, to any attempts to assist me further.
- 66. When Father Ohrwalder escaped, bringing with him the two sisters and negress, Mankarious set about immediately to find some reliable messenger willing to undertake the journey to Omdurman with a view of ascertaining if my escape was at all possible. He argued that if Father Ohrwalder could escape with three women as an encumbrance to his flight, there was nothing, provided I was at liberty, to prevent my escaping; but those who knew the Soudan— and it was only such he might employ—argued that if the remainder of the captives were not already killed, they would be found chained in the prison awaiting their execution. Months slipped away before he could find any one to undertake the journey, and then an old but wiry desert Arab, El Haj Ahmad Abou Hawanein, came to terms with him. Hawanein was given two camels, some money, and a quantity of goods to sell and barter on his way up. Some time in June or July, 1894, Abou Kees, a man employed in the Mission gardens, came to me while I was working at the mounds of Khartoum, and whispered that a man who had news for me was |197| hiding in the gardens, and that I was to try and effect a meeting with him. The man was Hawanein. Always suspicious of traps laid for me by the Khaleefa, I asked the man what he wanted. He replied that he had come from friends to help me. He had brought no letters, but by questioning him my suspicions disappeared, and I was soon deep in the discussion of plans for my escape. The camels he had brought with him were, he said, not up to the work of a rapid flight, and he suggested that he should return to Assouan, procure two good trotting camels, and also the couple of revolvers I asked for, as it was more than likely I should have to use them in getting clear of Khartoum. Soon after Hawanein’s departure, the guide Abdallah, who brought away Rossignoli, put in his appearance. Ahmed Wad-el-Feki, employed in Marquet’s old garden, asked that I might be allowed to call and see a sick man at his house. On reaching the place, Feki introduced me to a young man, Abdallah, who, after a few words,
- 67. asked me to meet him the following day, when he would bring me a letter. I met my “patient” again, when he handed me a bit of paper on which faint marks were discernible; these, he said, would come out clear upon heating the paper, and, as cauterization is one of the favourite remedies in the Soudan, some live charcoal was procured without exciting any suspicion. The words, which appeared, proved that the man was no spy, but had really come from the Egyptian War Office; however, before we had time to drop into a discussion of plans, some men employed in the place |198| came near, and we had to adjourn to the following day, when I was again to meet my “patient.” On this occasion we were left undisturbed, and fully discussed and settled upon our plans. To escape along the western bank of the Nile was not to be thought of; this would necessitate our passing Omdurman, and to pass the town unobserved was very improbable. Abdallah, having left his camels and rifle at Berber, was to return there for them, and come up the eastern bank of the Nile, along which we were to travel when I escaped. During his absence I was to send Umm es Shole on weekly visits to her friends at Halfeyeh; as she was to escape with us, this arrangement was made for a twofold purpose. First, her visits would not excite suspicion at the critical moment, as the people both at Halfeyeh and Khartoum would have become accustomed to them; she was also to bring me the promised revolver concealed in her clothes, and then return to Halfeyeh for another visit. She and Abdallah would keep a watch on the banks of the Blue Nile for me and assist me in landing. My escape would have to be effected in my chains, and these, of course, would prevent my using my legs in swimming. I was to trust for support to the pieces of light wood on the banks, used by children and men when disporting themselves in the Nile, and to the current and whatever help I might get with my hands for landing on the opposite shore. Abdallah went off, but never came back. I kept to our agreement for months, for the plan formed with |199| Abdallah was similar to
- 68. that arranged with Hawanein. Besides this, Abdallah, in the event of not being able to find revolvers at Berber, was to continue his journey to the first military post, obtain them there, and exchange his camels for fast-trotting ones, as those he had left at Berber were of a poor race. In order to prove to any officer he met that he was really employed to effect my escape, I gave him two letters couched in such words that, should they fall into the hands of the Khaleefa or any of the Emirs, their contents would be a sort of puzzle to them. Each day during those months I looked forward eagerly to a sign from any one of the people entrusted with my escape. For various reasons I considered it advisable to interview Abdallah after my release, and did so; but to make certain of his explanations, I also arranged that others should question him on the subject of Rossignoli’s flight and his reasons for not keeping his engagement with me, and this is what he says. On leaving Cairo, he was given a sort of double mission; he was promised three hundred pounds if he brought me away safely, and a hundred pounds if he brought away any of the other captives. Seeing the difficulties to be encountered in effecting my escape, and appreciating the risks, unless we had revolvers and swift camels, he decided upon “working out the other plan,” as he expresses it, viz. the escape of Rossignoli, as “he was at liberty and could go anywhere he pleased,” whilst I was shackled and constantly under the eyes of my guards. Instead of returning |200| for the camels, Abdallah arranged for Rossignoli to escape on a donkey as far as Berber. When some distance from Omdurman, Rossignoli got off his donkey, squatted on the ground, and refused to budge, saying he was tired. Abdallah tried to persuade him to continue the journey, but Rossignoli refused, said Abdallah was only leading him to his death, and demanded to be taken back to Omdurman. For a few moments Abdallah admits that he was startled and frightened. To go back to Omdurman was madness and suicide for him; to leave Rossignoli squatting in the desert made Cairo almost as dangerous
- 69. for him as Omdurman, for who would believe his tale there? He felt sure he would be accused of having deserted the man, and there was also the chance of Rossignoli being discovered by pursuers, when a hue and cry would be set up for Abdallah. One cannot help but admire Abdallah’s solution of the difficulty. There was a tree growing close by; he selected from it a good thick branch, and with this flogged Rossignoli either into his right senses or into obedience to orders; then placing him on the camel behind him, he made his way to Berber. Here Rossignoli, instead of keeping in hiding, wandered into the town, was recognized by some people, and, when spoken to, told them that Abdallah was leading him to Egypt, but that he preferred to return to Omdurman. Fortunately native cupidity saved Abdallah; he baksheeshed the people into a few hours of silence, with great difficulty got his charge clear of the town, and with still greater difficulty |201| hammered and “bullydamned” him into Egypt and safety. This is Abdallah’s own tale. He assures me, and I believe him, that it was his intention, as soon as he had handed over Rossignoli safe, to have asked for the revolvers and started back to try and effect my escape, risky as he knew it to be; but as Rossignoli had betrayed his name in Berber, he knew well that the Khaleefa would have men waiting for him from Omdurman to the frontier, and he showed no better sense in flogging Rossignoli, than he showed in settling down with his well- earned hundred pounds rather than attempting to make it into four hundred by passing the frontier. Rossignoli’s absence was not noticed for a little time, and fortunately, for a donkey leaves better tracks to follow than a camel. The Khaleefa was not particularly angry about the affair, although he imprisoned for a day Mr. Cocorombo, the husband of Sister Grigolini, the former superioress of Father Ohrwalder’s Mission, and Rossignoli’s lay companion, Beppo; but the latter, after Slatin’s escape, became my fellow-prisoner in the Saier.
- 70. One would be inclined to believe that either myself or some dramatist had purposely invented the series of accidents, which cropped up to frustrate every one of my plans for escape. On February 28, 1895, without a word of warning, I was so heavily loaded with chains that I was unable to move, and I was placed under a double guard in the house of Shereef Hamadan, the Mahdist Governor of Khartoum. At first I surmised that either Abdallah or Hawanein |202| had been suspected and imprisoned, or had confessed, or that our plots had been divulged in some way, so that it was with no little surprise I heard the questions put to me concerning the escape of Slatin. I denied all knowledge of the escape, or any arrangement connected with it. I pointed out that I had not seen, spoken to, or heard of Slatin directly for eight years, as my gaolers and guards could prove. It was from no sense of justice to me, but to prove that he had not neglected his duty in keeping a strict watch upon me, that Hamadan took my part in the inquiry. I might have been again released, had Hawanein not put in his appearance a few days after the escape of Slatin was discovered. Slatin’s absence from his usual post had not been reported to the Khaleefa until three days after his escape; he was supposed to be ill. On the third day, Hajji Zobheir, the head of the Khaleefa’s bodyguard, sent to his house to inquire about him. Not being satisfied with the reply he received, he informed the Khaleefa, who ordered an immediate search. A letter from Slatin to the Khaleefa was found sticking in the muzzle of a rifle, and was taken to Abdullahi. After the usual string of compliments and blessings, the letter continues― “For ten years I have sat at your gate; your goodness and grace has been great to me, but all men have a love of family and country; I have gone to see them; but in going I still hold to the true religion. I shall never betray your bread and salt, even should I die; I was wrong to leave without your permission; every one, myself included, acknowledges your great power and influence; forgive me; your desires are mine; I shall never betray you, |203| whether I reach my destination or
- 71. die upon the road; forgive me; I am your kinsman and of your religion; extend to me your clemency.* * This letter was found on the fall of Omdurman, and came into the hands of people who, probably on the ground of its contents differing from those given by Slatin after his escape, published it in such a manner as to lead people to believe that the protestations of loyalty it contained were sincere. In my opinion the letter should be looked upon as a clever composition to humbug Abdullahi, so that, in the event of Slatin being retaken, the protestation of loyalty would at least save him from the hands of the Khaleefa’s mutilator or executioner.
- 72. SAID BEY GUMAA. Abdullahi, on first realizing that Slatin had actually escaped, and had had about three days’ start of any pursuers he might send after him, was furious; losing his temper, he anathematized him in the
- 73. presence of the assembled Emirs, Kadis, and bodyguard. He reminded them that when Slatin first tendered his submission, he had been received with honours because he had openly professed the Mohammedan faith and had been circumcised while still the “Turk” Governor-General of Darfur; he reminded them also how Slatin had been allowed to bring into the camp his household, bodyguard, and servants, and had been attached to the Mahdi’s personal suite, of which he, Abdullahi, was chief; how, with Zoghal, his former subordinate, he had been entrusted with the subjugation of Said Gumaa, who had refused to surrender El Fasher when ordered by him to do so; how he himself had treated him as his son and his confidant, never taking any step without his advice and guidance; but, suddenly pulling himself up, seeing the mistake he had made in showing how much he had been dependent on him, he broke off short to say what he would do to Slatin if he ever laid hands on him, and promised a similar punishment to any one else who returned him ingratitude for his favours. Reading |204| out aloud Slatin’s letter to him, he calmed down on reaching the protestations of loyalty, and ordered the letter to be read in the mosque and the different quarters of Omdurman. Abdullahi has been considered as an ignorant brutal savage, devoid of all mental acumen, and but little removed from the brute creation. As I may be able to show later, such an expression of opinion either carries a denial with it, or it is paying a very poor compliment to those who, once governors of towns and provinces, or high officials, should have bowed down, kissed hands, and so far prostrated themselves as to kiss the feet of the representatives of this “ignorant brute,” by whom for years they had been dominated. Since Abdullahi respected me, as a man, by keeping me constantly in chains, I respect him for the intellectual powers he displayed, and which apparently paralyzed those of others who submitted to him. Slatin, having given a good account of himself in his many fights, was, after his submission, looked up to as the military genius of the
- 74. Mahdist army; he could not, as I did, play any pranks with the work he was entrusted with; the map he had drawn of Egypt, showing the principal towns and routes, and upon which the former telegraph- clerk, Mohammad Sirri, had been instructed to write the Arabic names, had given some the idea that no expedition might be planned without the aid of Slatin and this map. Abdullahi’s object in having the letter publicly read will be divined; first, it would assure the dervishes themselves that there was no fear of |205| Slatin, after his protestations of loyalty, returning at the head of the Government troops to overthrow the rule of the Mahdi, and without help from the exterior the wavering Mahdists could not hope to throw off the yoke of Abdullahi. Moreover, the reading of the letter to the Christian captives would confirm the opinion formed by many, that Slatin was at heart with the present Soudan dynasty, and that they could not expect any help as a result of his escape. There is another incident, which must be here mentioned, to show how acute Abdullahi really was. Slatin had publicly proclaimed his conversion to Mahommedanism before his submission to the Mahdi, so that, when he did submit, he was accepted as one of the faithful, and treated as one of themselves. The remainder of the captives— those taken before and after the fall of Khartoum—had not, up to the time of the escape of Rossignoli, been actually accepted as Muslims. At the suggestion of Youssef Mansour, on January 25, 1895, the Khaleefa was gracious enough to take all into his fold as real converts to the faith, and, on the anniversary of Gordon’s death, all the Muslimanieh (Christians) were ordered to be circumcised, the only two people not being operated upon being, I believe, Beppo, who was overlooked while in prison, and an old Italian mason, who pleaded old age as an excuse for not undergoing the operation. The Christian quarter was, therefore, at the time of Slatin’s escape, considered as a Muslim community, and the practical immunity they had |206| enjoyed from a rigorous application of the Mahdieh laws was thereby put an end to.
- 75. Consequently, when Slatin escaped, leaving behind him such protestations of loyalty, the safest card the Khaleefa could play was to read to them his letter. The reading of it caused some little consternation and comment, no doubt, but I have already expressed my opinion as to the light in which this letter should be considered. It was a clever move of Abdullahi; the public reading of the letter blasted all hopes on the part of the discontented Soudanese of any assistance from Slatin in crumbling to dust the kingdom of the Khaleefa, and put an end to all hopes on the part of the former Muslimanieh captives of release, for the small proportion of old Government employés who had, up to then, firmly believed that Slatin was acting, as they express it, “politeeka” in all his dealings, now joined the ranks of those who believed differently. But in this they were, of course, mistaken. After the public reading of the letter, the Khaleefa sent for the officials of the Beit-el-Mal and ordered them to take possession of Slatin’s house, wives, servants, slaves, land, and cattle, at the same time giving them strict instructions, in the presence of all, that the household were to be treated gently, as being the property of a true Muslim. His Darfurian wife, Hassanieh, whom he had married when Governor-General of Darfur, was claimed from the Beit-el-Mal by Dood (Sultan) Benga as of a royal family, and was by him married to another of the Darfurian royal |207| family. Desta, his Abyssinian wife, was within a few days of her confinement, and either, as a result of fright at the ransacking of the house and her reduction to the position of a common slave, or as a result of what would be to her, in her then delicate condition, rough handling, gave birth to a baby boy, who survived but a few weeks. It was while the Khaleefa was awaiting the return of the scouts sent out to recapture Slatin that Hawanein put in his appearance at Omdurman. He was at once seized, accused of assisting in the escape of Slatin, and also of having returned to effect mine. Pleading ignorance of myself and Slatin, he was not believed; he was first
- 76. sent into the Saier, and then, as he refused to confess, he was taken out and publicly flogged. Even this did not extort a confession; the Khaleefa, not being satisfied, ordered another flogging, but the Bisharas interceded for Hawanein, and succeeded in obtaining his release. As my would-be deliverer passed through the portals of the Saier, I passed in (March 26, 1895). Hawanein lost no time in returning to Assouan, where the relation of his experiences, with his torn back and unhealed wounds to bear him out, put an end finally to all attempts in that quarter to assist me in any way whatever. It might be as well that I should not attempt to describe my mental condition on finding myself again in the Saier. I have a faint idea of what my state must have been; despair cannot describe it; insanity at blasted hopes might. Yes, I must have been insane; but I was mentally sound, if such a contradiction |208| of terms is permissible. I remember that for days I shuffled about, refusing to look at or speak to any one. Perhaps what brought me round was that, in my perambulations, I came near the Saier anvil and heard a man crying. It was Ibrahim Pasha Fauzi, Gordon’s old favourite, who was being shackled. My expostulations on his acting as a child and bullying him into a sense of manhood, again prevented that slender thread between reason and insanity snapping. It must, in some way, have calmed and comforted me to be brought to the knowledge that others were suffering as much as I was; and just as a child, which requires care and attention itself, gives all its affection and sympathy to a limbless doll, so must I have given my sympathy to Fauzi, and in so doing taken a step back from the abyss of insanity, which I was certainly approaching.
- 77. CHAPTER XVII A NEW OCCUPATION When Said Abdel Wohatt was transferred from the Khartoum to the Alti saltpetre works, his father-in-law, Ali Khaater, the storekeeper of the Omdurman arsenal, considered that he was no longer under the obligation of risking his neck by mixing the Khartoum product with the Fellati’s, or substituting it with good saltpetre in stock. A consignment of mine was consequently sent direct to the powder factory, and was used in making what Abd es Semmieh and Hosny, the directors, believed would be a good explosive. The result, while being eminently satisfactory to myself, was just the reverse for the people responsible for making the powder. Not being certain where the fault actually lay, they mixed this powder with a quantity of really good powder made from the Fellati’s product, only to succeed in spoiling the whole bulk. When my next consignment was sent in they carried out some experiments, and, discovering where the fault lay, sent me an intimation that if our works did not turn out saltpetre equal in quality to that formerly supplied by us, I should be reported to the Khaleefa. Nahoum Abbajee, hearing of the affair, came to me in |210| a state of excitement, and pointed out the danger I was running into, and as he was then trying to think out an invention for coining money, he suggested that he should apply to the Khaleefa for my services in assisting him. This request Abdullahi was only too glad at the time to accede to; saltpetre was coming in in large quantities, and he was in great trouble about his monetary system.
- 78. As Khaleefa, he was entitled to one-fifth of all loot, property, taxes, and goods coming to the Beit-el-Mal; and as all property of whatever description was considered to belong primarily to this administration, it followed that Abdullahi was entitled to one-fifth of the property in the Soudan; but as he had not much use for hides, skins, gum, ivory, and such-like, he took his proportion in coin—after putting his own valuation upon his share. As the money he took from the Beit-el-Mal was hoarded and never came into circulation again, a sort of specie famine set in. Attempts had been made in the early days of Abdullahi’s rule to produce a dollar with a fair modicum of silver; but Nur-el-Garfawi, Adlan’s successor at the Beit-el-Mal, came to the conclusion, evidently, that a coin was but a token, and that it was immaterial what it was made of, provided it carried some impression upon it. The quantity of silver in his dollars grew less and less, and then was only represented by a light plating which wore off in a few weeks’ time. When people grumbled, he unblushingly issued copper dollars pure and simple. All the dollars were issued from the Beit-el-Mal as being of equivalent value to |211| the silver dollar, and when these coins were refused, the Khaleefa decreed that all future offenders should be punished by the confiscation of their property and the loss of a hand and foot. The merchants, though, were equal to the occasion; when an intending purchaser inquired about the price of an article, the vendor asked him in what coinage he intended to pay; the merchant then knew what price to ask. As the silver dollars gradually disappeared, the few remaining went up enormously in value, until in the end they were valued at fifty to sixty of the Beit-el-Mal coins, so that an article which could be bought for a silver dollar could not be purchased under fifty to sixty copper dollars. Although a rate of exchange was forbidden, the Beit-el-Mal took advantage of the state of affairs by buying in the copper dollars, melting them up, recasting, and striking from a different die. These coins would be again issued at the value of a silver dollar, and the remaining copper dollars in the town were put
- 79. out of circulation by the Beit-el-Mal’s refusal to receive them. To make matters worse, the die cutters cut dies for themselves and their friends, and it was worth the while of the false (?) coiners to make a dollar of better metal than the Beit-el-Mal did, and these we re-accepted at a premium. The false coinage business flourished until Elias el Kurdi, one of the best of the die cutters, was permanently incapacitated by losing his right hand and left foot; and this punishment, for a time at least, acted as a deterrent upon others, leaving the Beit-el-Mal the entire monopoly of coinage. |212| Sovereigns might at any time be bought for a dollar, for their possessors were glad to get rid of them. Being in possession of a gold coin denoted wealth, and many people who attempted to change a gold coin returned only to find their hut in the hands of the Beit-el-Mal officials, searching for the remainder of the presumed gold hoard. Failing to find it, they confiscated the goods and chattels. The trade with the Egyptian frontier, Suakin and Abyssinia, was carried on through the medium of barter and the Austrian (Maria Theresa) trade dollar. It was while the currency question was at its height that Abbajee came forward with his scheme for a coining press; and, in order that I might assist him, I was transferred to the Khartoum arsenal. I was obliged to give up my quarters in the Mission buildings, and live with the bodyguard of thirty Baggaras in the house of Hamadan, the Mahdist governor of Khartoum. The arsenal was presided over by Khaleel Hassanein, at one time a clerk under Roversi, in the department for the repression of the slave trade. Although ten years had elapsed since the fall of Khartoum, the arsenal must have been in as perfect working order as when Gordon made it into a model Woolwich workshop. Power was obtained from a traction-engine, which drove lathes, a rolling-mill, drills, etc., while punches, iron scissors, and smaller machinery were worked by hand. In the shops proper were three engines and boilers complete, ready to be fitted into Nile steamers, and duplicates and triplicates of all parts of the
- 80. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com