![Chapter 1
Introduction
Abstract Human behavior sensing and understanding has been a popular research
area during the past decades, which plays an important role in human-computer
interaction and public security. It aims to understand what people are doing by
sensing and recognizing their activities and their environments. However, accurate
detection and recognition of human behavior is still a big challenge that attracts a lot
of research efforts. In this chapter, we aim to present an overview of human behavior
sensing and understanding techniques, including from vision-based to sensor-based
and device-free behavior sensing, from individual to group and community behavior
understanding, and from pattern-based to model-based behavior understanding.
Behavior recognition technology has been applied to a number of key applications,
including pervasive and mobile computing [1, 2], surveillance-based security [3–
10], context-aware computing [11–13], and ambient assistive living [14–18]. Human
behavior sensing also serves as the key component in several consumer products.
For example, game consoles such as the Nintendo Wii and the Microsoft Kinect rely
on the recognition of gestures or even full-body movements to deliver outstanding
game interaction with the users. While originally developed for the entertainment
sector, these systems have found additional applications, such as for personal fitness
training and rehabilitation, and also stimulated new behavior recognition research
[19]. Finally, some sports products such as the Philips DirectLife or the Nike+
running shoes integrate motion sensors and offer both amateur and professional
athletes feedback on their performance. Moreover, monitoring daily behavior to
support medical diagnosis, for rehabilitation, or to assist patients with chronic
impairments has been shown to provide key enhancements to traditional medical
methods. Some early assistance to encourage humans to adopt a healthy lifestyle was
regarded as another important goal, leading to a vast exploration of related human
Part of this chapter is based on a previous work: L. Chen, J. Hoey, C. D. Nugent, D. J. Cook and
Z. Yu, “Sensor-Based Activity Recognition,” in IEEE Transactions on Systems, Man, and
Cybernetics, Part C (Applications and Reviews), vol. 42, no. 6, pp. 790–808, Nov. 2012. DOI:
https://doi.org/10.1109/TSMCC.2012.2198883
© Springer Nature Singapore Pte Ltd. 2020
Z. Yu, Z. Wang, Human Behavior Analysis: Sensing and Understanding,
https://doi.org/10.1007/978-981-15-2109-6_1
1](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-15-320.jpg)
![behaviors, for example, brushing teeth [20] or handwashing, food [21, 22] and
medication intake [23, 24], or transportation routines [25].
1.1 From Vision-Based to Sensor-Based and Device-Free
Behavior Sensing
1.1.1 Vision-Based Human Behavior Sensing
and Recognition
Using vision to sense and understand the behavior of human beings is a very
important topic. Vision-based human behavior recognition is the process of labeling
image sequences with behavior labels. It is based on the use of visual sensing
facilities, such as video cameras, to monitor user behaviors and environmental
changes. The generated sensor data are video sequences or digitized visual data.
The approaches exploit computer vision techniques, including feature extraction,
structural modeling, movement segmentation, action extraction, and movement
tracking, to analyze visual observations for pattern recognition.
Specifically, the vision-based behavior recognition methods mainly consist of
two steps: representation and classification. Generally, for representation
approaches, related literatures follow a research trajectory of global representations,
local representations, and recent depth-based representations. Earlier studies
attempted to model the whole images or silhouettes and represent human activities
in a global manner. The approach in ref. 26 is an example of global representation in
which space-time shapes are generated as the image descriptors. Then, the emer-
gence of space-time interest points (STIPs) [27] triggered significant attention to a
new local representation view that focuses on the informative interest points.
Meanwhile, local descriptors such as histogram of oriented gradients (HOG) and
histogram of optical flow (HOF) oriented from object recognition are widely used or
extended to 3D in human behavior recognition area. With the upgrades of camera
devices, especially the launch of RGBD cameras in the year 2010, depth image-
based representations have been a new research topic and have drawn growing
concern in recent years. For example, Jalal et al. [28] propose novel multi-fused
features for online human behavior recognition system that recognizes human
behaviors from continuous sequences of depth map. Yang et al. [29] propose a
general scheme of super normal vector (SNV) to aggregate the low-level
polynormals into a discriminative representation, which can be viewed as a simpli-
fied version of the Fisher kernel representation.
On the other hand, classification techniques keep developing in step with machine
learning methods. In fact, lots of classification methods were not originally designed
for human behavior recognition. Generally speaking, most behavior classification
algorithms can be divided into three categories, namely template-based approaches,
generative models, and discriminative models. Template-based approach is a
2 1 Introduction](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-16-320.jpg)
![relatively simple and well-accepted approach; however, it can be sometimes com-
putationally expensive. Generative models learn a model of the joint probability P
(X, Y) of the inputs X and the label Y, and then P(Y|X) is calculated using
algorithms (e.g., the Bayes classifier) to pick the most likely label Y [30]. In contrast,
discriminative models determine the result label directly. Typical algorithms of
generative models are hidden Markov model (HMM) and dynamic Bayesian net-
work (DBN), while support vector machine (SVM), relevance vector machine
(RVM), and artificial neural network (ANN) are typical discriminative models. In
recent years, deep learning methods are developed for a large amount of image
classification and have achieved good performance. Basically, the deep learning
architectures can be categorized into four groups, namely deep neural networks
(DNNs), convolutional neural networks (ConvNets or CNNs), recurrent neural
networks (RNNs), and some emergent architectures [31]. The ConvNets is the
most widely used one among the mentioned deep learning architectures. Compared
with traditional machine learning method and their handcrafted features, the
ConvNets can learn some representational features automatically [32]. Unlike
ConvNets, DNNs still use handcrafted features instead of automatically learning
features by deep networks from raw data. RNNs are designed for sequential infor-
mation. Activity itself is a kind of time-series data and it is a natural thought to use
RNNs for activity recognition.
However, it is worth mentioning that while visual monitoring is intuitive and
information rich, the vision-based approaches can only work with line-of-sight
(LoS) coverage in rich-lighting environments, and also may cause privacy concerns.
1.1.2 Sensor-Based Human Behavior Sensing
and Recognition
Sensor-based behavior recognition is based on the use of emerging sensor network
technologies for activity sensing and understanding [33–35]. The generated sensor
data are mainly time series of state changes and/or various parameter values that are
usually processed through data fusion, probabilistic, or statistical analysis methods
and formal knowledge technologies for activity recognition. Generally, sensors can
be attached to an actor under observation, namely wearable sensors or smartphones,
or objects that constitute the activity environment, namely dense sensing.
Preliminary studies have shown that commodity smartphones equipped with
accelerometers can also be used for behavior recognition, which are widely available
to the general public [36]. Wearable sensors often use inertial measurement units or
radio frequency identification (RFID) tags to gather behavioral information. This
approach is effective to recognize physical movements such as exercises. In contrast,
dense sensing infers activities by monitoring human–object interactions through the
usage of multimodal miniaturized sensors. These sensors are different in types,
purposes, output signals, underpinning theoretical principles, and technical
1.1 From Vision-Based to Sensor-Based and Device-Free Behavior Sensing 3](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-17-320.jpg)
![infrastructure, and can be classified into two main categories in terms of the way they
are deployed in activity monitoring applications, i.e., wearable sensors and dense
sensors.
Wearable sensors, generally, refer to sensors that are positioned directly or
indirectly in human body. They generate signals when the user performs activities.
As a result, they can monitor features that are descriptive of the person’s physiolog-
ical state or movement. Wearable sensors can be embedded into clothes, eyeglasses,
belts, shoes, wristwatches, and mobile devices or positioned directly in the body.
They can be used to collect information such as body position and movement, pulse,
and skin temperature. However, wearable sensors are not suitable for monitoring
activities that involve complex physical motions and/or multiple interactions with
the environment.
In some cases, sensor observations from wearable sensors alone are not sufficient
to differentiate activities involving simple physical movements (e.g., making tea and
making coffee). As a result, dense sensing-based activity monitoring has emerged.
Dense sensing-based activity monitoring refers to the practice that sensors are
attached to objects and activities are monitored by detecting user–object interactions.
The approach is based on real-world observations that activities are characterized by
the objects that are manipulated during their performance. A simple indication of an
object being used can often provide useful clues about the behavior being under-
taken. As such, it is assumed that activities can be recognized from sensor data that
monitors human interactions with objects in the environment. By dense sensing, we
refer to the way and scale with which sensors are used. Using dense sensing, a large
number of sensors, normally of low cost low power, and miniaturized, are deployed
in a range of objects or locations within an environment for the purpose of moni-
toring movements and behaviors.
1.1.3 Device-Free Human Behavior Sensing and Recognition
Compared with sensor-based behavior sensing, device-free behavior sensing is also
an important way to sense the activity of human beings. It can sense the behavior of
human beings without the need of carrying any tags or devices. It has the additional
advantage of being unobtrusive while offering good privacy protection. Over the
past decades, researchers have studied ways of tracking device-free human subjects
using different techniques such as camera [37], capacitance [38], pressure [39],
infrared [40], and ultrasonic [41]. However, most of these approaches suffer from
serious limitations such as occlusion [37, 40], high deployment cost [38, 39], or short
range [41].
Radio frequency (RF)-based techniques have the advantages of long range, low
cost, and ability to work through nonconducting walls and obstacles. Therefore,
RF-based device-free systems become popular to detect, locate, and track human
behaviors. Specifically, RF devices include ZigBee, Wi-Fi, RFID, etc. The basic idea
of RF-based device-free sensing system is as follows: when located in indoor
4 1 Introduction](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-18-320.jpg)
![environment with such a system, the movement of human bodies will affect the
wireless signals and change the multipath profile of the system. Based on this
principle, we are able to recognize human behaviors by exploring the changes of
wireless signals caused by user movements. Especially, Wi-Fi signal-based human
activity recognition technologies can be applied at commercial off-the-shelf (COTS)
devices. By monitoring the wireless channel state, the receiver can measure small
signal changes caused by human movements and use these changes to recognize the
surrounding human activities [42].
Device-free behavior sensing technology can be useful in many practical appli-
cations including intrusion detection and tracking for home and office applications,
which could enhance the safety of law-enforcement personnel, and low-cost long-
range asset protection, e.g., border protection or protection of railroad tracks. In
addition, device-free behavior sensing technology can be used to enhance traditional
security systems, such as motion detection and video surveillance by providing
non-line-of-sight (NLoS) detection and lower deployment cost.
1.2 From Individual to Group and Community Behavior
Recognition
In the past decades, numerous research efforts have been made to model and
recognize the behavior of individuals. However, group and community behavior
recognition has attracted much attention recently, as it has many practical applica-
tions such as abnormal group detection, group affective computing, video surveil-
lance, and public security [43]. Specifically, on the one hand, the large deployment
of sensor network in public facilities, private buildings and outdoors environments,
and digital traces left by people while interacting with cyber-physical spaces are
accumulating at an unprecedented breadth, depth, and scale. On the other hand, the
recent explosion of sensor-rich smartphone market and the phenomenal growth of
geo-tagged data (e.g., Twitter messages, Foursquare check-ins) have enabled the
analysis of new dimensions of contexts that involve the social and urban context. We
call all those traces left by people the “digital footprints.” Leveraging the capacity to
collect and analyze the “digital footprints” at group or community scale, a new
research field called “social and community intelligence (SCI)” [44] is emerging that
aims at revealing the patterns of individual, group, and societal behaviors.
The scale and heterogeneity of the multimodal, mixed data sources present us an
opportunity to compile the digital footprints into a comprehensive picture of indi-
vidual’s daily life facets, and radically change the way we build computational
models of human behaviors. Numerous innovative services will be enabled, includ-
ing human health, public safety, urban planning, environment monitoring, and so
on. The development of social and community intelligence will greatly expand the
scale and depth of context-aware computing, from merely personal awareness to the
understanding of social interactions (e.g., social relations, community structures) and
1.2 From Individual to Group and Community Behavior Recognition 5](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-19-320.jpg)
![urban dynamics (e.g., traffic jams, hot spots in cities). For example, Su et al. [45]
explored coherent LSTM to model the nonlinear characteristics in crowd behaviors.
Zhuang et al. [46] proposed an end-to-end deep architecture, differential recurrent
convolutional neural networks (DRCNN), for group activity recognition.
Specifically, people constantly participate in social activities to interact with
others and form various communities. Social activities such as making new friends,
forming an interest group to exchange ideas, and sharing knowledge with others are
constantly taking place in human society. The analysis of social interactions has been
studied by social scientists and physicists for a couple of decades [47]. An excellent
introduction to the concepts and the mathematical tools for social networks analysis
can be referred [48]. In the early stage, efforts on social network analysis are mostly
based on the relational data obtained by survey. During the last two decades, we
have observed an explosive growth of social applications such as chatting, shopping,
and experience sharing. These applications, along with traditional email and instant
messaging, have changed the way we communicate with each other and form social
communities. Corresponding to this trend, a large amount of work on social network
analysis and knowledge discovery springs up, including email communication
networks [49], scientific collaboration, and co-authorship network [50].
More recently, as the Internet stepped into the era of Web 2.0, which advocates
that users interact with each other as contributors to the websites’ content,
researchers turned their attention to the online social utilities, such as Facebook,
Twitter, and Blogs. For example, ArterMiner [51] seeks to harvest personal profile
information from a user’s homepage. AmitSheth’s research group has done much
work on summarization of event information like space, time, and theme from social
web resources for building public services [52]. Twitter has been reported to support
real-time mining of natural disasters such as earthquakes [53] and the moods of
citizens [54]. Understanding human movement in urban environments has direct
implications for the design of public transport systems (e.g., more precise bus
scheduling, improved services for public transport users), traffic forecasting (e.g.,
hotspot prediction), and route recommendation (e.g., for transit-oriented urban
development). A number of studies have extracted citywide human mobility patterns
using large-scale data from smart vehicles, mobile phones, and smart cards used in
public transportation systems. The Real Time Rome project of MIT uses aggregated
data from buses and taxies to better understand urban dynamics in real time [55]. The
learned human mobility patterns are also useful for urban planning. For example,
Nicholson and Noble [56] have studied how to leverage the learned human move-
ment dynamics to improve the distribution of cell infrastructure (e.g., to have a better
load balance among cell towers) in wireless communication networks. Zhang et al.
[57] has put forward a novel cross-modal representation learning method that
uncovers urban dynamics with massive geo-tagged social media data.
Specifically, we will mainly focus on individual, group, and community behavior
sensing and understanding in Chaps. 5–7, respectively.
6 1 Introduction](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-20-320.jpg)
![1.3 From Pattern-Based to Model-Based Behavior
Recognition
Although primitive sensor data can be obtained through behavior monitoring,
recognition models are critical to interpret the sensor data to understand human
behaviors. In particular, the mechanisms through which behaviors are recognized are
closely related to the nature and representation of behavior recognition models.
Generally speaking, recognition models can be built using two different approaches,
i.e., pattern based and model based [58–60].
1.3.1 Pattern-Based Behavior Recognition
The first one is to learn recognition models from existing large-scale datasets of
human behaviors based on data mining and machine learning techniques. This
method involves the creation of probabilistic or statistical behavior recognition
models, followed by training and learning processes. To build a human behavior
sensing and recognition system, the connection between signal variations and human
activities must be established. If the signal variation patterns have unique and
consistent relations with certain human activities, it is possible for a pattern-based
(or learning-based) method to recognize human behaviors accurately from signal
patterns. As this method is driven by data, and the behavior inference is based on
probabilistic or statistical classification, it is often referred to as data-driven or
pattern-based approaches.
The key to designing pattern-based approaches is to observe and find discrimi-
native patterns to construct features and differentiate different human behaviors of
interest. The features can be very simple or sophisticated, depending on the com-
plexity of the recognition task and the required granularity. For simple sensing tasks,
feature selection is often based on intuition or direct observation. When the number
of behaviors that need to be distinguished is small, it is often easy to find regular but
differentiable patterns. In this case, one or two features may be enough to distinguish
among behaviors. As both the number of human behaviors and the sensing granu-
larity increase, it becomes challenging to find one-to-one mappings between behav-
iors and feature patterns. One or two simple features are not enough for this task
anymore. In this case, more features are needed to increase the dimension of feature
space, and more powerful classifiers are needed.
The advantages of data-driven approaches are the capabilities of handling uncer-
tainty and temporal information. However, this method requires large datasets for
training and learning, and suffers from the data scarcity or the “cold start” problem. It
is also difficult to apply learnt recognition models from one person to another. As
such, this method suffers from the problems of scalability and reusability. Despite
these drawbacks, pattern-based approaches have been very popular and successful in
human behavior sensing and understanding applications because they are not only
1.3 From Pattern-Based to Model-Based Behavior Recognition 7](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-21-320.jpg)
![34. H. Alemdar and C. Ersoy, “Wireless sensor networks for healthcare: A survey,” Comput. Netw.,
vol. 54, no. 15, pp. 2688–2710, 2010.
35. D. Ding, R. A. Cooper, P. F. Pasquina, and L. Fici-Pasquina, “Sensor technology for smart
homes,” Maturitas, vol. 69, no. 2, pp. 131–136, 2011.
36. Y. Lu, Y. Wei, L. Liu, et al. Towards unsupervised physical activity recognition using
smartphone accelerometers. Multimedia Tools and Applications, 2017, 76(8): 10701–10719.
37. J. Krumm, S. Harris, B. Meyers, B. Brumitt, M. Hale, and S. Shafer. Multi-camera multi-person
tracking for easy living. In Proceedings of Third IEEE International Workshop on Visual
Surveillance, IWVS’ 00, pages 3–10, 2000.
38. M. Valtonen, J. Maentausta, and J. Vanhala. Tiletrack: Capacitive human tracking using floor
tiles. In IEEE International Conference on Pervasive Computing and Communications,
PerCom’09, pages 1–10, 2009.
39. R. J. Orr and G. D. Abowd. The smart floor: a mechanism for natural user identification and
tracking. In CHI ‘00 extended abstracts on Human factors in computing systems, CHI EA ‘00,
pages 275–276, 2000.
40. D. De, W. Song, M. Xu, C. Wang, D. Cook, and X. Huo. Findinghumo: Real-time tracking of
motion trajectories from anonymous binary sensing in smart environments. In Proceedings of
the 32nd IEEE International Conference on Distributed Computing Systems, ICDCS’12, pages
163–172, 2012.
41. T.W. Hnat, E. Griffiths, R. Dawson, and K. Whitehouse. Doorjamb: unobtrusive room-level
tracking of people in homes using doorway sensors. In Proceedings of the 10th ACM Confer-
ence on Embedded Network Sensor Systems, SenSys’12, pages 309–322, 2012.
42. Wang W, Liu A X, Shahzad M, et al. Device-free human activity recognition using commercial
WiFi devices[J]. IEEE Journal on Selected Areas in Communications, 2017, 35(5): 1118–1131.
43. Li T, Chang H, Wang M, et al. Crowded scene analysis: A survey IEEE transactions on circuits
and systems for video technology, 2015, 25(3): 367–386.
44. Zhang, D., Guo, B., & Yu, Z. (2011). The emergence of social and community intelligence.
IEEE Computer, 44(7), 21–28.
45. Su H, Dong Y, Zhu J, et al. Crowd Scene Understanding with Coherent Recurrent Neural
Networks[C]. IJCAI. 2016, 1: 2.
46. Zhuang N, Yusufu T, Ye J, et al. Group activity recognition with differential recurrent
convolutional neural networks[C]. Automatic Face & Gesture Recognition (FG 2017), 2017
12th IEEE International Conference on. IEEE, 2017: 526–531.
47. Freeman, L. C. (2004). The development of social network analysis: A study in the sociology of
science. Empirical Press.
48. Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications.
Cambridge, UK: Cambridge University Press.
49. McCallum, A., Wang, X., & Corrada-Emmanuel, A. (2007). Topic and role discovery in social
networks with experiments on Enron and academic email. Journal of Artificial Intelligence
Research. 30(1), 249–272.
50. Barabasi, A. L., Jeong, H., Neda, Z., Ravasz, E., Schubert, A., & Vicsek, T. (2002). Evolution
of the social network of scientific collaborations. Statistical Mechanics and its Applications,
311 (3–4), 590–614.
51. Tang, J., Jin, R. M., & Zhang, J. (2008). A topic modeling approach and its integration into the
random walk framework for academic search. Paper presented at the Meeting of 2008 IEEE
International Conference on the Meeting of 2008 IEEE International Conference on Data
Mining. Pisa, Italy.
52. Sheth, A. (2010). Computing for human experience – Semantics-empowered sensors, services,
and social computing on the ubiquitous web. IEEE Internet Computing, 14 (1), 88–97.
53. Sakaki, T., Okazaki, M., & Matsuo, Y. (2010). Earthquake shakes Twitter users: Real-time
event detection by social sensors. Paper presented at the Meeting of WWW 2010 Conference.
Raleigh, NC.
10 1 Introduction](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-24-320.jpg)
![54. Bollen, J., Pepe, A., & Mao, H. (2009). Modeling public mood and emotion: Twitter sentiment
and socio-economic phenomena. Paper presented at the meeting of WWW 2009 Conference.
Madrid, Spain.
55. Calabrese, F., & Ratti, C. (2006). Real time Rome. Networks and Communications Studies, 20
(3–4), 247–258.
56. Nicholson, J., & Noble, B. D. (2008). Bread crumbs: Forecasting mobile connectivity. Paper
presented at the Meeting of Mobile Computing and Networking. San Francisco, CA.
57. Zhang C, Zhang K, Yuan Q, et al. Regions, periods, activities: Uncovering urban dynamics via
cross-modal representation learning[C]//Proceedings of the 26th International Conference on
World Wide Web. International World Wide Web Conferences Steering Committee, 2017:
361–370.
58. Wang Z, Guo B, Yu Z, et al. Wi-Fi CSI-Based Behavior Recognition: From Signals and Actions
to Activities. IEEE Communications Magazine, 2018, 56(5): 109–115.
59. Wu D, Zhang D, Xu C, Wang H, Li X (2017) Device-free Wi-Fi human sensing: from pattern-
based to model-based approaches. IEEE Communications Magazine, 55(10): 91–97.
60. Chen L, Hoey J, Nugent C D, et al. Sensor-Based Activity Recognition, IEEE Transactions on
Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2012, 42(6):790–808.
References 11](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-25-320.jpg)
![As a matter of fact, the operating systems running on each device have
transformed the hardware sensors into software APIs, and similarly the online social
network platforms also provide programming interfaces for developers to access
different types of data. Therefore, it is straightforward to collect data from those two
distinct domains by applying the following common procedures: researchers or
developers first write programs leveraging those available APIs, after that they run
the programs in a backend manner, and finally the behavior data are collected and
stored for further analysis. For example, web crawler programming could be applied
for collecting online human behavior data by leveraging APIs like Twitter APIs [1]
and Foursquare APIs [2]. Besides, operating systems like Android and iOS also
provide APIs [3, 4] that allow developers to directly access embedded sensor
readings in smart devices. Other than this, researchers usually need to recruit
volunteers or make the data collection program publicly available, so that they can
gather human behavior data through physical devices. However, collecting data
from social network platforms does not require such procedures; researchers only
have to query corresponding platforms using some indexes, such as a location, a
specific word, or a target user ID, to obtain the desired human activity information.
By applying the above procedures, one can collect sensory data of human behavior
not only from physical devices, but also from online social network platforms.
2.2 Data Preprocessing
The original sensory data, especially from physical sensors, is often incomplete,
inconsistent, and likely to contain many errors. For instance, due to the affection on
signals from crowd buildings, there might be huge differences between consecutive
GPS locations than it should have in real world. Therefore, data preprocessing
becomes one sufficient procedure before any insightful analysis can be taken onto
the collected data.
In general, the following steps are considered: data anonymization, data cleaning,
data filtering, and data transformation. Firstly, data anonymization is required since
human behavior is personal sensitive and is usually related to privacy issue, and
MD5 hashing [5] and k-anonymity [6] are two popular data anonymization algo-
rithms. After that, data cleaning is applied to deal with problems like data inconsis-
tency and noisy data influence. Specifically, data inconsistency usually appears in
data collected from online platforms, where user profile information may contain
Fig. 2.1 Main steps of human behavior sensing and understanding
14 2 Main Steps of Human Behavior Sensing and Understanding](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-27-320.jpg)
![conflict records. Meanwhile, noisy influence often exists in physical sensory data
that sensor readings are affected by other signals or environmental changes, which
could cause inaccuracy when monitoring human activities.
Afterwards, researchers have to choose the most effective or meaningful parts for
further analysis, which refers to data filtering. In fact, real-world sensory data are
usually incomplete, such that human behavior is often recorded as many fragments
and only some of them, which contain enough records in terms of timescale or other
criterions, are valuable to be analyzed. Hence, data filtering is performed to filter out
sensory data by different criterions that could contribute to discovering pattern from
those filtered data.
Finally, data transformation is applied to conduct data normalization or aggrega-
tion, and data representation before eventually feeding the data into learning models.
In particular, different learning models require different data types or formats, such
as vector, matrix, and time-series sequences. Therefore, researchers have to trans-
form original data formats into model-orientated formats and normalize the data
when needed. After conducting all these above procedures, the raw data is prepared
to be imported into learning algorithms or models for further analysis.
2.3 Feature Extraction
Apart from monitoring human behaviors and activities based on physical sensors or
cyber information, one significant task is to understand the relationship between
activities and sensory data, which aims to discover correlations that could lead to
better human interpretations.
Generally, each instance of sensory data consists of several attributes, and feature
extraction manages to derive values intended to be informative and nonredundant,
which can be used to facilitate the subsequent learning steps. Specifically, since there
are various types of sensors, a human activity could be captured from different
aspects that result in high-dimensional sensory data. Therefore, some data-driven-
based dimensionality reduction methods (e.g., principal component analysis [7]) are
employed to extract meaningful features from the original data. Besides, researchers
can also define their own features according to expert knowledge. After that,
correlation analysis is conducted to discover insightful relationships between
extracted features and human activities. Usually, some metrics like Pearson correla-
tion coefficient [8] are applied to measure the relationship between features and
activities, while some other metrics like Information Gain [9] are used to determine
the significance of different features. By conducting the above procedures, not only
features are extracted from the original sensory data, but also some correlations are
learned with respect to different tasks and models.
2.3 Feature Extraction 15](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-28-320.jpg)
![2.4 Human Behavior Modeling and Classification
Many methodologies are proposed to address human behavior modeling and classi-
fication problems, and they can be roughly divided into two directions of
approaches: logical reasoning and probabilistic reasoning [10]. Specifically, logical
reasoning is the process of using a rational, systematic series of steps based on sound
mathematical procedures and given statements to arrive at a conclusion. Traditional
human activity recognition and modeling usually prefer this approach to infer human
behaviors based on deployed sensors’ readings, and this kind of method is more like
an expert knowledge-based approach. Logical reasoning models are easy to be
interpreted and computationally efficient that could be applied in real-time
situations.
Compared with logical reasoning, probabilistic reasoning is a pattern-based
approach, and is commonly classified into three types: supervised, unsupervised,
and semi-supervised learning models. The supervised learning models depend on the
labeled dataset by which the parameters could be learned from each category. On the
contrast, unsupervised learning models try to find hidden structure in unlabeled data,
such as clustering algorithms. Semi-supervised learning falls between the supervised
and unsupervised learning, which mainly makes use of both labeled and unlabeled
data for training—typically for the sparse of labeled data. Different from logical
reasoning, probabilistic reasoning is more robust when facing problems like noisy,
uncertain, and incomplete sensory data. However, since it is data-driven-based
approach, it is more computationally expensive and difficult to achieve the goal of
real-time modeling.
References
1. https://dev.twitter.com/overview/api, visited: 24 Jul 2018.
2. https://developer.foursquare.com/docs/, visited: 24 Jul 2018.
3. https://developer.android.com/reference/, visited: 24 Jul 2018.
4. https://developer.apple.com/documentation/, visited: 24 Jul 2018.
5. https://en.wikipedia.org/wiki/md5, visited: 24 Jul 2018.
6. Sweeney, Latanya. "k-anonymity: A model for protecting privacy." International Journal of
Uncertainty, Fuzziness and Knowledge-Based Systems 10.05 (2002): 557–570.
7. Jolliffe, Ian. “Principal component analysis.” International encyclopedia of statistical science.
Springer, Berlin, Heidelberg, 2011. 1094–1096.
8. Benesty, Jacob, et al. "Pearson correlation coefficient." Noise reduction in speech processing,
Springer, Berlin, Heidelberg, 2009. 1–4.
9. Kent, John T. "Information gain and a general measure of correlation." Biometrika 70.1 (1983):
163–173.
10. Liang, Yunji, et al. "Activity Recognition Using Ubiquitous Sensors: An Overview." Wearable
Technologies: Concepts, Methodologies, Tools, and Applications. IGI Global, 2018. 199–230.
16 2 Main Steps of Human Behavior Sensing and Understanding](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-29-320.jpg)
![Chapter 3
Sensor-Based Behavior Recognition
Abstract In this monograph, sensor-based behavior recognition mainly refers to the
use of emerging sensor network technologies for behavior monitoring and under-
standing. The generated sensor data from sensor-based monitoring are mainly time
series of state changes and/or various parameter values that are usually processed
through data fusion, probabilistic, or statistical analysis methods and formal knowl-
edge technologies for behavior recognition. Specifically, sensors can be attached to
an actor under observation, namely wearable sensors or smartphones, or objects that
constitute the environment, namely dense sensing. Wearable sensors often use
inertial measurement units and radio frequency identification (RFID) tags to gather
a user’s behavioral information. This approach is effective to recognize physical
movements such as physical exercises. In contrast, dense sensing infers behaviors by
monitoring human–object interactions through the usage of multiple multimodal
miniaturized sensors. In this chapter, we first give a brief introduction to the
historical evolution of sensor-based behavior recognition. Afterwards, we present
the mobile device-enabled behavior recognition approach, which is a typical type of
sensor-based behavior recognition, followed by a discussion on the key issues of
developing behavior recognition systems using mobile devices.
3.1 Sensor-Based Behavior Recognition Evolution
The idea of using sensors for behavior monitoring and recognition has been existent
since the late 1990s. It was initially pioneered and experimented by the work of the
Neural Network house [1] in the context of home automation, and a number of
location-based applications aiming to adapt systems to users’ whereabouts [2–
4]. The approach was soon found to be more useful and suitable in the area of
ubiquitous and mobile computing—an emerging area in the late 1990s—due to its
Part of this chapter is based on a previous work: L. Chen, J. Hoey, C. D. Nugent, D. J. Cook and
Z. Yu, “Sensor-Based Activity Recognition,” in IEEE Transactions on Systems, Man, and
Cybernetics, Part C (Applications and Reviews), vol. 42, no. 6, pp. 790–808, Nov. 2012. DOI:
https://doi.org/10.1109/TSMCC.2012.2198883
© Springer Nature Singapore Pte Ltd. 2020
Z. Yu, Z. Wang, Human Behavior Analysis: Sensing and Understanding,
https://doi.org/10.1007/978-981-15-2109-6_3
17](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-30-320.jpg)
![easy deployment. As such, extensive research has been undertaken to investigate the
use of sensors in various application scenarios of ubiquitous and mobile computing,
leading to considerable work on context awareness [5–7], smart appliances [8, 9],
and activity recognition [10–14]. Most research at that time made use of wearable
sensors, either dedicated sensors attached to human bodies or portable devices like
mobile phones, with application to ubiquitous computing scenarios such as provid-
ing context-aware services. Activities being monitored in these researches are
mainly physical activities like motion, walking, and running. These early works
lay a solid foundation for wearable computing and still inspire and influence today’s
research.
In the early 2000s, a new sensor-based approach that uses sensors attached to
objects to monitor human activities appeared. This approach, which was later
dubbed as the “dense sensing” approach, performs activity recognition through the
inference of user–object interactions [15, 16]. The approach is particularly suitable to
deal with activities that involve a number of objects within an environment, or
instrumental activities of daily living (ADL) [17, 18]. Research on this approach
has been heavily driven by the intensive research interests and huge research effort
on smart home-based assistive living, such as the EU’s AAL program [19]. In
particular, sensor-based activity recognition can better address sensitive issues in
assistive living such as privacy, ethics, and obtrusiveness than conventional vision-
based approaches. This combination of application needs and technological advan-
tages has stimulated considerable research activities in a global scale, which gave
rise to a large number of research projects, including the House_n [20], CASAS [21],
Gator-Tech [22], inHaus [23], AwareHome [24], DOMUS [25], and iDorm [26]
projects, to name but a few. As a result of the wave of intensive investigation, there
have seen a plethora of impressive works on sensor-based behavior recognition in
the past several years [27–40].
While substantial research has been undertaken, and significant progress has been
made, the two main approaches, wearable sensor-based and dense sensing-based
behavior recognition, are currently still focuses of the research community [41–
45]. The former is mainly driven by the ever-popular pervasive and mobile com-
puting, while the latter is predominantly driven by smart environment applications
such as AAL. Interests in various novel applications are still increasing and appli-
cation domains are rapidly expanding.
3.2 Behavior Recognition Based on Mobile Devices
During the past decade, smart mobile devices become more and more popular, which
are usually embedded with various sensors, such as accelerometer and GPS. These
sensors generate signals when the user performs behaviors. As a result, sensor-
enriched mobile devices can monitor features that are descriptive of the person’s
physiological state or movement, and can be used to collect information such as
body position and movement, pulse, and skin temperature. Researchers have found
18 3 Sensor-Based Behavior Recognition](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-31-320.jpg)
![that different types of sensor information are effective to classify different types of
activities. For example, accelerometer embedded in smartphones is capable of
characterizing human’s movements, e.g., standing, walking, and running [46–
48]. Similarly, by collecting audio information from the phone’s microphone, it is
possible to recognize a user’s activity, e.g., listening to music, speaking, and
sleeping [49, 50], and even monitor running rhythm [51], as well as the sound-
related respiratory symptoms of a user, such as sneeze or cough [52]. With various
sensor data, including Wi-Fi, accelerometer, compass, and GPS, the user’s emotion
can also be inferred [53–55].
Generally, mobile device-based behavior recognition systems may operate at
multiple scales, enabling applications from personal sensing and group sensing to
community sensing [56]. Meanwhile, another key issue is about how much the user
(i.e., the person carrying the mobile device) should be actively involved during the
sensing activity (e.g., taking the device out of the pocket to collect a sound sample or
take a picture). In other words, should the user actively participate, known as
participatory sensing, or, alternatively, passively participate, known as opportunistic
sensing [57]? Each of these sensing paradigms presents important trade-offs. In this
section, we discuss different sensing and understanding scales and paradigms of
mobile device-enabled behavior recognition systems.
3.2.1 Behavior Sensing and Understanding Scales
Personal behavior sensing and understanding systems are designed for a single
individual. Typical scenarios include tracking the user’s exercise routines or auto-
mating diary collection. Typically, personal behavior sensing and understanding
systems generate data for the sole consumption of the user and are not shared with
others. An exception is healthcare systems where limited sharing with medical
professionals is common (e.g., primary care giver or specialist).
Individuals who use behavior sensing and understanding systems that share a
common goal, concern, or interest collectively represent a group. These group
behavior recognition systems are likely to be popular and reflect the growing interest
in social networks or connected groups (e.g., at work, in the neighborhood, friends)
who may want to share information freely or with privacy protection. There is an
element of trust in group behavior sensing and understanding systems that simplify
otherwise difficult problems, such as attesting that the collected sensor data is correct
or reducing the degree to which aggregated data must protect the individual.
Common use cases include assessing neighborhood safety, sensor-driven mobile
social networks, and forms of citizen science.
Most examples of community behavior sensing and understanding only become
useful once they have a large number of people participating, e.g., tracking the
spread of disease across a city, congestion patterns across city roads, or a noise map
of a city. These systems represent large-scale data collection and analysis for the
good of the community. To achieve scale implicitly requires the cooperation of
3.2 Behavior Recognition Based on Mobile Devices 19](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-32-320.jpg)
![strangers who will not trust each other. This increases the need for systems with
strong privacy protection and low commitment levels from users.
3.2.2 Behavior Sensing and Understanding Paradigms
One issue common to the different types of sensing and understanding scale is to
what extent the user is actively involved in the system [57]. We discuss two points in
the design space: participatory sensing, where the user actively engages in the data
collection activity (i.e., the user manually determines how, when, what, and where to
sample), and opportunistic sensing, where the data collection stage is fully auto-
mated with no user involvement.
The benefit of opportunistic sensing is that it lowers the burden placed on the
user, allowing overall participation by a population of users to remain high even if
the system is not that personally appealing. This is particularly useful for community
behavior sensing and understanding. One of the main challenges of using opportu-
nistic sensing is the context problem of mobile devices, e.g., the system wants to
only take a sound sample for a citywide noise map when the device is out of the
pocket or bag. These types of context issues can be solved by using the embedded
sensors; for example, the accelerometer or light sensors can determine if the device is
out of the pocket.
Participatory sensing, which is gaining interest in the mobile crowd sensing
community [57], places a higher burden or cost on the user, e.g., manually selecting
data to collect (e.g., lowest petrol prices) and then sampling it (e.g., taking a picture).
An advantage is that complex operations can be supported by leveraging the
intelligence of the person in the loop who can solve the context problem in an
efficient manner. For example, a person who wants to participate in collecting a
noise or air quality map of their neighborhood simply takes out his/her mobile device
to solve the context problem. One drawback of participatory sensing is that the
quality of data is dependent on participant enthusiasm to reliably collect sensing data
and the compatibility of a person’s mobility patterns to the intended goals of the
system (e.g., collect pollution samples around schools).
Clearly, opportunistic and participatory represent extreme points in the design
space. Each approach has pros and cons, and there is a need to develop models to
best understand the usability and performance issues of these schemes. In addition, it
is likely that many systems will emerge that represent a hybrid of both these sensing
paradigms.
20 3 Sensor-Based Behavior Recognition](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-33-320.jpg)
![3.3 Energy-Efficient Behavior Recognition Using
Ubiquitous Sensors
With the popularity of mobile devices equipped with unprecedented sensing capa-
bilities, context-aware applications on mobile devices are going flourishing. How-
ever, the long-term sensing with the full working load of sensors is energy
consuming. The battery capacity of mobile devices is a major bottleneck of
context-aware applications. For example, the battery lifetime of Samsung i909
reaches up to over 30 h when all applications and sensors are turned off. But that
declines to 5.5 h (50 Hz) and 8 h (20 Hz), respectively, when the single 3D
accelerometer is monitored with different sampling frequencies.
In the wireless networks, the use of energy harvesting techniques offers a way of
supplying sensor systems without the need for batteries and maintenance
[58, 59]. For ubiquitous sensors, existing solutions extend the battery life by the
collaboration of multiple sensors and the reduction of sensor active time [60]. Wang
et al. [61] designed a scalable framework of energy-efficient mobile sensing system
for automatic user state recognition. The core component is a sensor management
scheme which defines user states and defines transition rules by an XML configu-
ration. The sensor management scheme allocates the minimum set of sensors and
invokes new sensors when state transitions happen. Zappi et al. [62] selected the
minimum set of sensors according to their contributions to classification accuracy as
assessed during data training process and tested this solution by recognizing manip-
ulative activities of assembly-line workers in a car production environment. Li et al.
[63] applied machine learning technologies to infer the status of heavy-duty sensors
for energy-efficient context sensing. They try to infer the status of high-energy-
consuming sensors according to the outputs of lightweight sensors.
Towards the energy efficiency of behavior recognition based on a single sensor
(e.g., accelerometer) in the mobile device, it is intuitive to reduce the working time of
sensors by adopting a low sampling frequency. Lower sampling frequency means
less work time for the heavy-duty sensor. However, whether the low sample
frequency is feasible for detecting human behaviors is still an open question. It is
claimed that the sampling rate to assess daily physical activities should be no less
than 20 Hz [64–66]. Kahatapitiya et al. [67] utilized the harvesting signal to estimate
the step count, completely removing the requirement of accelerometer sampling. On
the other hand, low sampling frequency may result in the loss of sampling data,
reducing the recognition rate with low-resolution sensory data [68]. So there is a
trade-off between energy consumption and recognition rate. Furthermore, many
classification algorithms are heavyweight and time consuming for mobile devices.
The size of sliding window in most classification algorithms is constant, which not
only reduces the ability to detect short-duration movements, but also occupies lots of
resources with the consumption of battery power.
To overcome above issues, two factors (i.e., sampling frequency and computa-
tional load) should be considered in the design of the behavior recognition algo-
rithm. For example, we propose an energy-efficient method to recognize user
3.3 Energy-Efficient Behavior Recognition Using Ubiquitous Sensors 21](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-34-320.jpg)
![activities based on a single triaxial accelerometer embedded in smartphones, where a
hierarchical recognition scheme is adopted to reduce the probability of time-
consuming frequency-domain features for lower computational complexity and
adjust the size of sliding window to enhance the recognition accuracy [69].
References
1. M. C. Mozer, “The neural network house: An environment that adapts to its inhabitants,” in
Proc. AAAI Spring Symp. Intell. Environ., 1998, pp. 110–114.
2. U. Leonhardt and J. Magee, “Multi-sensor location tracking,” in Proc. 4th ACM/IEEE Int. Conf.
Mobile Comput. Netw., 1998, pp. 203–214
3. A. R. Golding and N. Lesh, “Indoor navigation using a diverse set of cheap, wearable sensors,”
in Proc. 3rd Int. Symp. Wearable Comput., Oct. 1999, pp. 29–36.
4. A. Ward and A. Hopper, “A new location technique for the active office,” IEEE Personal
Commun., vol. 4, no. 5, pp. 42–47, Oct. 1997.
5. A. Schmidt, M. Beigl, and H. Gellersen, “There is more to context than location,” Comput.
Graph., vol. 23, no. 6, pp. 893–901, 1999.
6. C. Randell and H. L. Muller, “Context awareness by analyzing accelerometer data,” in Proc. 4th
Int. Symp. Wearable Comput., 2000, pp. 175–176.
7. H. W. Gellersen, A. Schmidt, and M. Beigl, “Multi-sensor context awareness in mobile devices
and smart artifacts,” Mobile Netw. Appl., vol. 7, no. 5, pp. 341–351, Oct. 2002.
8. A. Schmidt and K. Van Laerhoven, “How to build smart appliances,” IEEE Pers. Commun.,
vol. 8, no. 4, pp. 66–71, Aug. 2001.
9. K. Van Laerhoven and K. A. Aidoo, “Teaching context to applications,” J. Pers. Ubiquitous
Comput., vol. 5, no. 1, pp. 46–49, 2001.
10. K. Van Laerhoven, K. Aidoo, and S. Lowette, “Real-time analysis of data from many sensors
with neural networks,” in Proc. 5th Int. Symp. Wearable Comput., 2001, pp. 115–123.
11. K. Van Laerhoven and O. Cakmakci, “What shall we teach our pants?” in Proc. 4th Int. Symp.
Wearable Comput., 2000, pp. 77–84.
12. F. Foerster and J. Fahrenberg, “Motion pattern and posture: Correctly assessed by calibrated
accelerometers,” Behav. Res. Methods Instrum. Comput., vol. 32, no. 3, pp. 450–457, 2000.
13. K. Van Laerhoven and H. W. Gellersen, “Spine versus Porcupine: A study in distributed
wearable activity recognition,” in Proc. 8th Int. Symp. Wearable Comput., 2004, pp. 142–150.
14. S. W. Lee and K. Mase, “Activity and location recognition using wearable sensors,” IEEE
Pervasive Comput., vol. 1, no. 3, pp. 24–32, Jul.–Sep. 2002.
15. L. Bao and S. Intille, “Activity recognition from user-annotated acceleration data,” in Proc.
Pervasive, 2004, vol. 3001, pp. 1–17.
16. D. J. Patterson, L. Liao, D. Fox, and H. Kautz, “Inferring high-level behavior from low-level
sensors,” in Proc. 5th Conf. Ubiquitous Comput., 2003, pp. 73–89.
17. M. Chan, D. Esteve, C. Escriba, and E. Campo, “A review of smart homes—present state and
future challenges,” Comput. Methods Programs Biomed., vol. 91, no. 1, pp. 55–81, 2008.
18. C. D. Nugent, “Experiences in the development of a smart lab,” Int. J. Biomed. Eng. Technol.,
vol. 2, no. 4, pp. 319–331, 2010.
19. The ambient assisted living joint programme [Online]. Available: www.aal-europe.eu
20. The house of the future [Online]. Available: http://architecture.mit.edu/house_n
21. P. Rashidi and D. Cook, “Activity knowledge transfer in smart environments,” J. Pervasive
Mobile Comput., vol. 7, no. 3, pp. 331–343, 2011.
22. The gator-tech smart house [Online]. Available: http://www.icta.ufl.edu/gt.htm
23. The inHaus project in Germany [Online]. Available: http://www.inhaus.fraunhofer.de
24. The aware home [Online]. Available: http://awarehome.imtc.gatech.edu
22 3 Sensor-Based Behavior Recognition](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-35-320.jpg)
![25. The DOMUS laboratory [Online]. Available: http://domus.usherbrooke.ca
26. The iDorm project [Online]. Available: http://cswww.essex.ac.uk/iieg/idorm.htm
27. N. Kern, B. Schiele, H. Junker, P. Lukowicz, and G. Troster, “Wearable sensing to annotate
meeting recordings,” Pers. Ubiquitous Comput., vol. 7, no. 5, pp. 263–274, Oct. 2003.
28. P. Lukowicz, J. A. Ward, H. Junker, and T. Starner, “Recognizing workshop activity using body
worn microphones and accelerometers,” in Proc. Pervasive Comput., Apr. 2004, pp. 18–23.
29. J. Mantyjarvi, J. Himberg, and T. Seppanen, “Recognizing human motion with multiple
acceleration sensors,” in Proc. IEEE Int. Conf. Syst., Man, Cybern., 2001, vol. 2, pp. 747–752.
30. D. Ashbrook and T. Starner, “Using GPS to learn significant locations and predict movement
across multiple users,” Pers. Ubiquitous Comput., vol. 7, no. 5, pp. 275–286, Oct. 2003.
31. M. Philipose, K. P. Fishkin, M. Perkowitz, D. J. Patterson, D. Fox, H. Kautz, and D. Hahnel,
“Inferring activities from interactions with objects,” IEEE Pervasive Comput., vol. 3, no. 4, pp.
50–57, Oct./Dec. 2004.
32. K. P. Fishkin, M. Philipose, and A. Rea, “Hands-on RFID: Wireless wearables for detecting use
of objects,” in Proc. 9th IEEE Int. Symp. Wearable Comput., 2005, pp. 38–43.
33. D. J. Patterson, D. Fox, H. Kautz, and M. Philipose, “Fine-grained activity recognition by
aggregating abstract object usage,” in Proc. 9th IEEE Int. Symp. Wearable Comput., 2005,
pp. 44–51.
34. M. R. Hodges and M. E. Pollack, “An object-use fingerprint: The use of electronic sensors for
human identification,” in Proc. 9th Int. Conf. Ubiquitous Comput., 2007, pp. 289–303.
35. M. Buettner, R. Prasad, M. Philipose, and D. Wetherall, “Recognizing daily activities with
RFID-based sensors,” in Proc. 11th Int. Conf. Ubiquitous Comput., 2009, pp. 51–60.
36. D. Wilson and C. Atkeson, “Simultaneous tracking and activity recognition (STAR) using
many anonymous, binary sensor,” in Proc. 3rd Int. Conf. Pervasive Comput., 2005, pp. 62–79.
37. C. R. Wren and E. M. Tapia, “Toward scalable activity recognition for sensor networks,” in
Proc. 2nd Int. Workshop Location Context Awareness, 2006, pp. 168–185.
38. R. Aipperspach, E. Cohen, and J. Canny, “Modeling human behavior from simple sensors in the
home,” in Proc. Pervasive, 2006, vol. 3968, pp. 337–348.
39. T. Gu, Z. Wu, X. Tao, H. K. Pung, and J. Lu, “epSICAR: An emerging patterns based approach
to sequential, interleaved and concurrent activity recognition,” in Proc. 7th IEEE Int. Conf.
Pervasive Comput. Commun., 2009, pp. 1–9.
40. L. Liao, D. J. Patterson, D. Fox, and H. Kautz, “Learning and inferring transportation routines,”
Artif. Intell., vol. 171, no. 5–6, pp. 311–331, 2007.
41. A. Montanari, C. Mascolo, K. Sailer, and S. Nawaz, “Detecting Emerging Activity-Based
Working Traits through Wearable Technology,” Proceedings of the ACM on Interactive,
Mobile, Wearable and Ubiquitous Technologies, vol. 1, no. 3, pp. 86, 2017.
42. Y. Zhang, M. Haghdan, and K. S. Xu, “Unsupervised motion artifact detection in wrist-
measured electrodermal activity data,” Proceedings of the ACM on Interactive, Mobile, Wear-
able and Ubiquitous Technologies, 2017.
43. J. M. Echterhoff, J. Haladjian, and B. Brügge, “Gait and jump classification in modern
equestrian sports,” In Proceedings of the 2018 ACM International Symposium on Wearable
Computers, pp. 88–91, 2018.
44. S. A. Elkader, M. Barlow, and E. Lakshika. “Wearable sensors for recognizing individuals
undertaking daily activities.” Proceedings of the 2018 ACM International Symposium on
Wearable Computers. ACM, 2018.
45. V. Becker, P. Oldrati, L. Barrios, and G. Sörös, “Touchsense: classifying finger touches and
measuring their force with an electromyography armband,” In Proceedings of the 2018 ACM
International Symposium on Wearable Computers, pp. 1–8, 2018.
46. P. Siirtola and J. Röning, “Recognizing human activities user independently on smartphones
based on accelerometer data,” Int. J. Interact. Multimedia Artif. Intell., vol. 1, no. 5, pp. 38–45,
2012.
References 23](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-36-320.jpg)
![47. A. M. Khan, A. Tufail, A. M. Khattak, and T. H. Laine, “Activity recognition on smartphones
via sensor-fusion and KDA-based SVMS,” Int. J. Distrib. Sensor Netw., vol. 10, no. 5, pp.
1–14, 2014.
48. M. Shoaib, S. Bosch, Ö. D. Incel, H. Scholten, and P. J. M. Havinga, “Fusion of smartphone
motion sensors for physical activity recognition,” Sensors, vol. 14, no. 6, pp. 10146–10176,
2014.
49. H. Lu, W. Pan, N. D. Lane, T. Choudhury, and A. T. Campbell, “Soundsense: Scalable sound
sensing for people-centric applications on mobile phones,” in Proc. MobiSys, 2009,
pp. 165–178.
50. H. Du, Z. Yu, F. Yi, Z. Wang et al., “Recognition of group mobility level and group structure
with mobile devices,” IEEE Transactions on Mobile Computing, vol. 17, no. 4, pp. 884–897,
2018.
51. T. Hao, G. Xing, and G. Zhou, “RunBuddy: A smartphone system for running rhythm
monitoring,” in Proc. UBICOMP 2015, pp. 133–144.
52. X. Sun, Z. Lu, W. Hu, and G. Cao, “Symdetector: Detecting sound-related respiratory symp-
toms using smartphones,” in Proc. UbiComp 2015, pp. 97–108.
53. A. Mottelson and K. Hornbæk, “An affect detection technique using mobile commodity sensors
in the wild,” In Proceedings of the 2016 ACM International Joint Conference on Pervasive and
Ubiquitous Computing, pp. 781–792, 2016.
54. B. Cao, L. Zheng, C. Zhang, P. S. Yu, A. Piscitello, et al. “Deepmood: modeling mobile phone
typing dynamics for mood detection,” In Proceedings of the 23rd ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining,” pp. 747–755, 2017.
55. X. Zhang, W. Li, X. Chen, and S. Lu. MoodExplorer: Towards Compound Emotion Detection
via Smartphone Sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and
Ubiquitous Technologies, vol. 1, no. 4, pp. 176, 2018.
56. Lane N D, Miluzzo E, Lu H, et al. A survey of mobile phone sensing[J]. IEEE Communications
Magazine, 2010, 48(9):140–150.
57. B. Guo, Z. Wang, Z. Yu, Y. Wang, N.Y. Yen, R. Huang, and X. Zhou. Mobile Crowd Sensing
and Computing: The Review of an Emerging Human-Powered Sensing Paradigm. ACM
Comput. Surv. 48, 1, Article 7 (August 2015), 31 pages.
58. K. Ylli, D. Hoffmann, A. Willmann, P. Becker, B. Folkmer, and Y. Manoli. Energy harvesting
from human motion: exploiting swing and shock excitations. Smart Materials and Structures,
vol. 24, no. 2, pp. 025–029, 2015.
59. K. Li, C. Yuen, B. Kusy, R. Jurdak, A. Ignatovic, S. Kanhere, and S. K. Jha. Fair scheduling for
data collection in mobile sensor networks with energy harvesting. IEEE Transactions on Mobile
Computing, 2018.
60. D. Liaqat, S. Jingoi, E. de Lara, A. Goel, W. To, et al. Sidewinder: An energy efficient and
developer friendly heterogeneous architecture for continuous mobile sensing. ACM SIGARCH
Computer Architecture News, vol. 44, no. 2, pp. 205–215, 2016.
61. Wang Y, Lin J, Annavaram M, Quinn JA, Jason H, Bhaskar K, Sadeh N (2009) A framework of
energy efficient mobile sensing for automatic user state recognition. In: Proceedings of the 7th
ACM international conference on mobile systems, applications, and services, New York, pp
179–192.
62. Zappi P, Lombriser C, Stiefmeier T, Farella E, Roggen D, Benini L, Troster G (2008) Activity
recognition from on-body sensors: accuracy-power trade-off by dynamic sensor selection.
Wirel Sens Netw Lect Notes Comput Sci 943: 17–33.
63. Li X, Cao H, Chen E, Tian J (2012) Learning to infer the status of heavy-duty sensors for energy
efficient context-sensing. ACM Trans Intell Syst Technol 3(2):1–23.
64. Bouten C, Koekkoek K, Verduin M, Kodde R, Janssen JD (1997) A triaxial accelerometer and
portable data processing unit for the assessment of daily physical activity. IEEE Trans Biomed
Eng 44:136–147.
24 3 Sensor-Based Behavior Recognition](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-37-320.jpg)
![4.2 Wi-Fi CSI-Based Behavior Sensing and Recognition1
Human behavior understanding plays an important role in human-computer interac-
tion and public security. Researchers have developed many methods using cameras,
radars, or wearable sensors [1]. However, they also suffer from certain shortcomings
(e.g., user privacy, sensing coverage range). For example, the vision-based
approaches only work with line-of-sight (LoS) coverage and rich-lighting environ-
ments, which also causes privacy concerns. The low-cost 60 GHz radar solutions can
only offer an operation range of tens of centimeters, and such radar devices are not
widely deployed in our daily living environments. Wearable sensor-based
approaches require people to wear some extra devices.
With the recent advances in wireless communications, behavior recognition
based on Wi-Fi has been attracting more and more attention due to its ubiquitous
availability in indoor areas. Moreover, Wi-Fi-based behavior recognition approach
is able to overcome the aforementioned shortcomings of traditional approaches, as it
only leverages the wireless communication feature and does not need any physical
sensor.
A typical Wi-Fi-based behavior recognition system consists of a Wi-Fi access
point (AP) and one or several Wi-Fi-enabled devices in the environment. When
located in indoor environment with such a system, the movement of human bodies
will affect the wireless signals and change the multipath profile of the system. Based
on this principle, we are able to recognize human behaviors by exploring the changes
of wireless signals caused by user movements.
Recently, numerous studies have devoted to pervasive sensing using Wi-Fi
devices, such as indoor localization [2] and gesture recognition [3]. These studies
are mainly based on received signal strength indication (RSSI). However, the RSSI
can fluctuate dramatically even at a stationary link, which makes these detection
results unreliable. Quite recently, channel state information (CSI), i.e., the fine-
grained information regarding Wi-Fi communication, becomes available [4]. Specif-
ically, CSI describes how the signal propagates from the transmitter to the receiver
and reflects the combined effects of the surrounding objects (e.g., scattering, fading,
and power decay with distance). Meanwhile, there are a set of subcarriers in CSI,
each of which contains the information of attenuation and phase shift. Therefore,
CSI contains rich information and is more sensitive to environmental variances
caused by moving objects [5].
CSI is a metric which estimates the channel by representing the channel proper-
ties of a wireless communication link. In the frequency domain, the wireless channel
can be described as Y ¼ H X + N, where X and Y correspond to the transmitted and
1
Part of this section is based on a previous work: Z. Wang, B. Guo, Z. Yu and X. Zhou, “Wi-Fi
CSI-Based Behavior Recognition: From Signals and Actions to Activities,” in IEEE Communica-
tions Magazine, vol. 56, no. 5, pp. 109-115, May 2018. DOI: https://doi.org/10.1109/MCOM.2018.
1700144
4.2 Wi-Fi CSI-Based Behavior Sensing and Recognition 29](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-41-320.jpg)
![received signal vectors, H is the channel matrix presented in the format of CSI, and
N is the additive white Gaussian noise vector.
In the IEEE 802.11n standard, CSI is measured and reported at the scale of
orthogonal frequency division modulation (OFDM) subcarriers, where each
CSIi ¼ j CSIi j exp {j(∠CSIi)} depicts the amplitude response (i.e., jCSIij) and
phase response (i.e., ∠CSIi) of one subcarrier. Specifically, each entry in matrix H
corresponds to the channel frequency response (CFR) value between a pair of
antennas at a certain OFDM subcarrier frequency at a particular time, and the time
series of CFR values for a given pair of antennas and OFDM subcarrier is called a
CSI stream. In other words, while CFR describes the combined effects of fading,
scattering, and attenuation of a specific subcarrier, CSI is the union of these CFRs.
Specifically, 802.11n specifications have provisions for reporting quantized CSI
field per packet using various subcarrier grouping options as per clause. However,
different manufacturers may choose to implement a subset of the subcarrier grouping
options. For example, the Intel 5300 wireless network interface card (NIC) imple-
ments an OFDM system with 56 subcarriers of a 20 MHz channel or 114 subcarriers
of a 40 MHz channel, 30 out of which can be read for CSI information via the device
driver. Thereby, a time series of CSI values includes 30 NumTx NumRx CSI
streams, where NumTx and NumRx stand for the amount of transmitting and receiv-
ing antennas, respectively.
Given an indoor environment with two wireless nodes, as shown in Fig. 4.1, the
wireless signal will propagate in a multipath manner, and the wireless channel will
be relatively stable as long as there are no people or no motion. However, once a
person moves, the scattered signals will change (the red line in Fig. 4.1), which
causes channel disturbances, involving both amplitude attenuation and phase dis-
tortion. In other words, different multipath effects can be obtained if a person is
moving, which results in different CSI streams at the receiver, and can be used to
recognize different behaviors by correlating them with the corresponding channel
distortion patterns.
Fig. 4.1 Wi-Fi signal propagation in indoor environments [6]
30 4 Device-Free Behavior Recognition](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-42-320.jpg)
![Wi-Fi CSI-enabled behavior recognition approaches can be categorized into two
groups, i.e., pattern based [7–17] and model based [18–21]. The pattern-based
approaches aim to classify behaviors by exploring different features of CSI mea-
surements, while the model-based approaches implement recognition by modeling
the relationship between signal space and behavior space. A general architecture of
Wi-Fi CSI-based behavior recognition approaches is shown in Fig. 4.2. Though the
middle part (i.e., CSI data collection and preprocessing) is common to both pattern-
based and model-based approaches, the left and right parts illustrate the key differ-
ence of these two approaches.
Most existing CSI-based behavior recognition studies adopt the pattern-based
approach. The intuition is that different behaviors have distinct impacts on the
received CSI streams, which can be leveraged to mine patterns or construct profiles
for predefined behaviors, as shown in the left part of Fig. 4.2. Afterwards, each
behavior can be classified as one of the predefined types based on profile matching or
pattern recognition. The key benefit of the pattern-based recognition approaches is
that they do not require intensive deployments and can work with even a single AP,
which ensures the low hardware cost and maintenance and has no obstruction to
human’s normal life. However, pattern-based approaches usually require a learning
process to construct profiles or classifiers, which restricts them to identify only a
limited set of predefined behaviors.
Model-based approaches are based on the characterization of mathematical
relationships between human behaviors and received signals. In the case of Wi-Fi
CSI-based behavior recognition, the aim of modeling is to relate the signal space to
the physical space including human and environment, and characterize the physical
law through mathematical relationship between the received CSI signals and the
sensing target, as shown in the right part of Fig. 4.2. Since the model-based
Fig. 4.2 A general architecture of Wi-Fi CSI-based behavior recognition approaches [6]
4.2 Wi-Fi CSI-Based Behavior Sensing and Recognition 31](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-43-320.jpg)
![approaches do not need predefined behavior profiles, they can track an arbitrary set
of human behaviors, which enables wider ranges of real-time applications. Cur-
rently, there have been several model-based behavior recognition works, such as the
angle-of-arrival (AoA) model [18], the CSI-speed model [19], and the Fresnel zone
model [20, 21].
4.3 Acoustic-Based Behavior Sensing and Recognition
Acoustic-based sensing is a method for estimating the state of objects by transmitting
acoustic signals and analyzing the response. A typical acoustic-based sensing system
consists of a pair of speaker and microphone, which forms an audio transceiver
system. The speaker is usually programmed to transmit acoustic signals continu-
ously, while the microphone is applied to receive the echo with a certain sampling
rate (e.g., 48 kHz) and send the received signal to computers for data processing and
behavior recognition.
Most of the acoustic-based device-free behavior sensing and recognition systems
adopt similar ideas from RF-based (e.g., Wi-Fi) approaches, either exploring the
Doppler effect when human approaching or away from the microphone, or decoding
the echo of frequency-modulated continuous wave (FMCW) of the acoustic signal to
measure the human body, or utilizing the OFDM to achieve real-time behavior
tracking, as shown in Fig. 4.3.
The first line of acoustic-based human behavior sensing and recognition systems
is on the basis of the Doppler effect of the signal reflected by users [23–26]. Such
systems do not have tracking capability and can only recognize predefined behav-
iors, as Doppler shift can only provide coarse-grained measurements of the speed or
direction of user movements due to the limited frequency measurement precision.
The second line of acoustic-based systems is on the basis of decoding the echo of
FMCW sound wave to measure the human body. Specifically, FMCW [27] indi-
rectly estimates the propagation delay based on the frequency shift of the chirp
signal, based on which we can measure the movement displacement of the target
user, and a number of behaviors can be recognized [28]. However, the distance
estimation resolution of FMCW is restricted by the sweep bandwidth; it is difficult to
achieve very high distance estimation resolution (e.g., detecting human respiration
with chest movement displacement of less than 1 cm) with narrowband commodity
Fig. 4.3 A general framework of acoustic-based behavior recognition approaches [22]
32 4 Device-Free Behavior Recognition](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-44-320.jpg)
![acoustic device. To tackle this issue, a correlation-based FMCW (C-FMCW)
approach [22] is proposed to achieve more accurate distance estimation. Unlike
traditional FMCW, C-FMCW estimates the round-trip propagation time of acoustic
signals by discovering the maximum correlation between transmitted signal and
received signal. For digital FMCW signal, the round-trip propagation time can be
measured by detecting the number of samples corresponding to the maximum
correlation. As such, the distance estimation resolution of C-FMCW is limited
only by the acoustic signal sampling rate. With current prevailing audio systems
of 48 kHz, C-FMCW can achieve a ranging resolution of around 0.4 cm. The
ranging resolution can be further improved if higher sampling rates (e.g., 96 kHz)
are supported by the acoustic systems.
The third line of research uses OFDM pulses or CW signals to detect phase
changes and facilitate real-time behavior tracking [29, 30]. For instance, by leverag-
ing the OFDM pulses, the FingerIO system [29] achieves a finger location tracking
accuracy of 8 mm and also allows 2D drawing in the air using COTS mobile devices.
Similarly, the LLAP system [30] uses CW signals, which is less noisy due to the
narrower bandwidth compared with OFDM pulses, allowing it to achieve better
tracking accuracy.
In general, compared with other kinds of wireless signals (e.g., RF signal, optical
signal), sensing human behavior using acoustic signals has the following advan-
tages. (1) Short wavelength means more accurate behavior sensing. Due to relatively
slow propagation velocity, even with 20 kHz transmitting frequency, the wavelength
of transmitting acoustic signal is only about 1.7 cm. (2) It is easy to control the
process of acoustic transmitting and receiving. For example, using commodity
acoustic devices, we can directly control acoustic transmitting and receiving process
by calling system audio interface. (3) Compared with the raw data (e.g., RSSI or
CSI) of RF-based sensing systems, the data that we can obtain from speaker and
microphone is the original transmitting and receiving signal, which contains more
abundant information.
The disadvantages of using acoustic signal to sense and recognize human behav-
ior can be summarized as follows. (1) Acoustic signal is one kind of mechanical
wave, which attenuates severely when propagating in the air. This characteristic
limits its sensing range. (2) Compared with RF signals, the acoustic signal has poor
penetration ability.
From the above discussion, we know that acoustic is suitable for sensing fine-
grained behavior in short range, such as human vital signal sensing, gesture recog-
nition, and hand tracking.
References
1. Nickel, C., Busch, C., Rangarajan, S., Möbius, M. (2011). Using Hidden Markov Models for
accelerometer-based biometric gait recognition. IEEE, International Colloquium on Signal
Processing and ITS Applications, pp 58–63.
References 33](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-45-320.jpg)
![Chapter 5
Individual Behavior Recognition
Abstract In this chapter, we present some of our recent research advances on
individual behavior sensing and recognition. Specifically, in Sections 5.1 and 5.2,
we present two human mobility-related works (i.e., mobility prediction and disori-
entation detection) by leveraging GPS trajectories. Afterwards, we discuss how to
recognize human behaviors by using smartphones in Sections 5.3 and 5.4 (i.e.,
human-computer operation recognition and human localization), followed by two
device-free sensing-based behavior analysis practices in Sections 5.5 and 5.6 (i.e.,
human identity recognition and respiration detection).
5.1 Human Mobility Prediction by Exploring History
Trajectories1
5.1.1 Introduction
With the rising popularity of mobile sensors and portable devices, “serendipitous”
social interaction is emerging and becoming pervasive, where interaction is triggered
when two devices (i.e., two users) are located closely in a mobile peer-to-peer
environment [1], or wireless ATM networks [2]. Studies have been conducted to
support unexpected interaction utilizing users’ mobile wireless devices (e.g., mobile
phone, vehicle PDA, and wearable devices). In these studies, human interaction is
facilitated by means of peer-to-peer data sharing between matching devices [3] (i.e.,
users), and the “matching” is defined by similarity computed from user profiles (e.g.,
music, pictures, and tagged preferences) on the phone. However, these mechanisms
merely provide services for people who have common interests and thus cannot
enhance human social relations in the physical world.
1
Part of this section is based on a previous work: Z. Yu, H. Wang, B. Guo, T. Gu and T. Mei,
“Supporting Serendipitous Social Interaction Using Human Mobility Prediction,” in IEEE Trans-
actions on Human-Machine Systems, vol. 45, no. 6, pp. 811–818, Dec. 2015. DOI: https://doi.org/
10.1109/THMS.2015.2451515
© Springer Nature Singapore Pte Ltd. 2020
Z. Yu, Z. Wang, Human Behavior Analysis: Sensing and Understanding,
https://doi.org/10.1007/978-981-15-2109-6_5
37](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-48-320.jpg)
![The term “serendipity” was coined by the novelist Horace Walpole in the
eighteenth century to describe unexpected and fortunate discoveries; “serendipity”
was originally used to refer to making accidental discoveries when looking for one
thing and finding another [4]. We, therefore, characterize the serendipitous social
interaction as unplanned, not affecting users’ schedules, bringing convenience, and
creating positive emotion (e.g., happiness) in serving as an activity user, and users
can choose to participate or not. For example, take two college students who are
friends and who have not met for weeks. Both of them often visit the same library at
the same time, but they do not recognize that. In such a scenario, if a mobile
application helps them to capture and learn the serendipitous opportunities to meet
and have a chat, the friendship between these two students will be enhanced.
To support such social interactions, we need first to discover the serendipitous
interaction opportunities [5], similar to the communication channel of mobile inter-
mediate nodes in an opportunistic network [6]. With the captured unplanned and
transitory opportunities, serendipitous interaction can be used to make our lives
easier. For example, if a user is aware that his/her roommate is passing by a grocery,
he/she may ask him/her to buy something for him/her. However, this kind of
serendipitous opportunity will disappear in a matter of seconds as he/she walks
away. In this scenario, prediction, rather than instant behavior detection or notifica-
tion, would be better. Mobility prediction typically leverages human trajectory data
(e.g., GPS data, check-in records, and intercell [7]). The majority of mobility
prediction methods focus on predicting a user’s future mobility status (e.g., where
and when a user arrives at the next venue, and duration). In this chapter, we predict
users’ future temporal and spatial contexts such as venue, arrival time, and user
encounter. As user mobility status (e.g., location and time) may change in a very
short period of time in the physical world, coarse-grained (e.g., hour-level temporal
prediction error) and low-accuracy (e.g., accuracy of next venue judgment) mobility
prediction cannot satisfy the needs of practical applications. Aiming to achieve high
accuracy and low error, we discover the strong spatial and temporal regularities in
GPS trajectories and use supervised learning algorithms to train historical mobility
instances (generated by crowd users).
Few applications have been proposed, especially any combining spatial and
temporal prediction information. In this chapter, we aim to facilitate serendipitous
social interactions and the real participation activities in the physical world, which is
different from simple information sharing mechanisms in existing traditional appli-
cations. Leveraging the overlap and regularity in collected GPS trajectories, we
deploy supervised learning algorithms and achieve an accuracy of over 90% for
predicting a user’s next venue, and minute-level (i.e., an average of about 5 min)
prediction error for arrival time.
By leveraging mobility prediction to forecast the occurrence of serendipitous
interaction opportunities, we propose a three-layer framework to support serendip-
itous social interaction, develop two applications for use on a university campus, and
conduct a survey about the application. The main contributions of this chapter are
summarized as follows.
38 5 Individual Behavior Recognition](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-49-320.jpg)
![• We propose a system framework for supporting social interaction by means of
facilitating users to participate in interaction activities in the physical world.
Under this framework, mobility prediction is introduced to capture serendipitous
interaction opportunities.
• We leverage mobility prediction to discover serendipitous social interaction
opportunities. Based on the spatiotemporal mobility regularities in users’ trajec-
tories, we first predict where and when a user will arrive, and then, we can
determine if some users may soon encounter (i.e., have the same destination).
• We develop the prototype of the proposed framework and build two applications
based on the framework, HelpBuy and EaTogether, to support serendipitous
social interaction on campus.
5.1.2 Related Work
Previous studies attempted to facilitate online social interactions in social networks.
Under the premise that users who have the same interests are more willing to interact
with each other (i.e., user homogeneity), studies try to help users find interaction
with friends through personal preference similarity calculation in social networks, in
which social interactions are facilitated in the form of learning [8], date [1, 9], and
travel [10]. These kinds of applications only enhance social interaction within an
online social network, rather than establish social connectivity in the physical world.
On the other hand, researchers focus on supporting face-to-face human interac-
tion, in which social interactions are facilitated by leveraging serendipitous commu-
nication opportunities between mobile devices (i.e., users). BlueFriend [11] is an
application that leverages Bluetooth to find friends among nearby users. Bluedating
[9] provides localized dating services to help users find desired partners. Paradiso
et al. [12] developed a badge system, which is equipped with wireless infrared and
radio frequency networking, to facilitate social interaction between wearers. Law-
rence et al. [3] developed three applications by exploiting the “co-presence” interac-
tions (i.e., incidental interactions) between mobile devices. In the mobile peer-to-peer
environment, as Yang et al. [10] proposed, information sharing and social interaction
are facilitated by capturing serendipitous interaction opportunities. However, inter-
action opportunities are always discovered by device detection. Devices are the actual
participant in these applications, rather than the humans themselves. In this chapter,
we predict serendipitous interaction opportunities leveraging mobility prediction
using users’ current mobility status and present mobile applications that facilitate
people actively participating and interacting in the physical world. Mobility predic-
tion is the main supporting technology of our system. It aims to discover serendip-
itous interaction opportunities.
Mobility prediction has been studied to perceive human future mobility status
(e.g., next venue, arrival time, and duration) from different perspectives. Xiong et al.
[13] took advantage of the similarity between people’s trajectories and proposed
collective behavioral patterns for improving prediction accuracy. Do and Gatica-
5.1 Human Mobility Prediction by Exploring History Trajectories 39](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-50-320.jpg)
![Perez [14] adopted a factorized conditional model according to the extracted fea-
tures, which can reduce the size of the parameter space as compared to conditional
model. Cho et al. [15] discussed the contribution of a location-based social network
and an individual user’s periodic movement pattern in mobility prediction and
developed a model that combines the periodic day-to-day movement patterns with
the social movement effects coming from the friendship network. Noulas et al. [16]
used check-in data in Foursquare and proposed a set of mobility prediction features
to capture the factor that drives users’ movement, and achieved around 90%
prediction accuracy. Baumann et al. [17] analyzed the influence of temporal and
spatial features in mobility prediction and predicted transitions. McInerney et al. [18]
presented a Bayesian model of population mobility to tackle the data sparsity
problem in mobility prediction. Song et al. [19] utilized the Order-2 Markov
predictor with fallback and obtained a median accuracy of about 72% for users
with long trace lengths. Song et al. [19] provided evidence that the prediction
accuracy of an individual’s next location had an upper bound of 93%. Lin et al.
[20] reported on a study of GPS data-based mobility prediction accuracy and
suggested that a predictability upper bound of 90% is able to support ubiquitous
applications. Considering the sparsity of trajectory data, such as Call Detail Records
[21] and taxi GPS data, researchers also made much progress. Wang et al. [22]
proposed a mobility gradient decent approach and predicted user’s destination, by
using sparse taxi trajectories. Recently, some works explore mobility prediction
using deep learning [23, 24]. By discovering region of interest (ROI) in the city,
Jiang et al. [24] proposed a deep ROI-based modeling approach for effectively
predicting human mobility. In this chapter, we predict not only an individual
user’s next venue, but also the arrival time and multiple users’ encounters based
on users’ current mobility features (CMFs). We achieved a high prediction accuracy
of around 90%, which enables discovering opportunities to support serendipitous
live interactions.
5.1.3 Serendipitous Social Interactions Supporting System
5.1.3.1 Framework
The framework consists of three layers, as shown in Fig. 5.1.
Data Layer: In the data layer, a GPS dataset of trajectories is collected from users
by the GPS data logger. Each GPS point has several spatial and temporal attributes,
such as date, time, latitude, longitude, and speed, and trajectories with thousands of
GPS points are recorded. These mobility data reflect the regularity of users’ daily
visits (i.e., the GPS point sequences in some venues) and the movement (i.e., the
GPS point sequences between two venues). After data denoising, historical mobility
instances are extracted from the GPS dataset for mobility prediction.
Mobility Prediction Layer: Repeatable behaviors and patterns exist in the GPS
trajectories. For example, at a campus crossroad during the morning, if a student is
40 5 Individual Behavior Recognition](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-51-320.jpg)
![walking westward, then we predict that he/she will go to the gym; if he/she is
walking towards the east, then we predict that he/she will arrive at the library.
Moreover, people living in the same environment (e.g., students in a dormitory)
may have the same regularity, and thus, users’ mobility trajectories can be utilized as
historical instances to predict the next venue and arrival time.
Human GPS trajectory contains many temporal (i.e., morning) and spatial (i.e., at
a crossroad) context features. It is important to select the appropriate features to
achieve high prediction accuracy and low complexity. As trajectories have regular-
ity, we apply learning algorithms to discover patterns. Then, after perceiving indi-
vidual users’ CMFs (e.g., time, latitude, and longitude), the inference models can
predict users’ future mobility features (FMFs) (e.g., next venue and arrival time). At
the individual level, we can predict a user’s future status. Then, at the crowd level,
we can predict multiple users’ occasional encounters at a next venue (e.g., two users
will meet at a restaurant in 10 min), which is similar to predicting data source
location in [25].
Application Layer: In this layer, we design and develop applications to facilitate
serendipitous interactions based on the prediction results of user’s FMFs from the
mobility prediction layer. For example, we can develop more efficient participatory
sensing [26] applications with perceiving users’ future positions. Furthermore, we
can discover unexpected opportunities with user encounter prediction results. In this
chapter, we implement two applications that facilitate social interactions in specific
scenarios on a campus.
Fig. 5.1 System framework
5.1 Human Mobility Prediction by Exploring History Trajectories 41](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-52-320.jpg)
![5.1.3.2 Mobility Prediction
We observe repeatable behavior and patterns in the GPS trajectories. For instance, a
person is going to the same next venue when he/she is at the same position at the
same time on different days (e.g., at noon, students at the same place are likely to go
to restaurant; in the 5 min before class, students on different paths are all moving
towards the classrooms). These real phenomena imply that students’ trajectories
contain strong spatial and temporal regularities. We explore these regularities for
mobility prediction and apply supervised learning algorithms to train the inference
model.
In this chapter, we predict users’ FMFs at two granularities: individual and crowd.
At the individual level, we predict where a user is going to (i.e., next venue) and
when the user will arrive there (i.e., arrival time). At the crowd level, we predict the
intersection of multiple users’ trajectories (i.e., user encounter) based on the indi-
vidual user mobility prediction.
Figure 5.2 shows the detailed mobility prediction procedure. First, historical
mobility instances are extracted from the historical GPS dataset to form the training
dataset. Second, since supporting serendipitous social interaction requires high
prediction accuracy both spatially and temporally, we adopt learning algorithms to
train using the historical dataset. The prediction task contains discrete (next venue)
and continuous (arrival time) output; therefore, we use different learning algorithms
to train inference models to predict each FMF. According to ref. 17, more CMFs are
not necessary for higher prediction accuracy. We find that we need to select the
suitable CMF set to achieve high accuracy and low complexity. Experiments are
conducted by selecting different CMF sets from Table 5.1 and comparing their
performance. Finally, after predicting results of the user’s FMFs (i.e., next venue
and arrival time) with the inference models, we can predict user encounter at the next
venue.
Taking every GPS point in the trajectories as a mobility instance, we extract fine-
grained attributes (e.g., latitude, longitude, time, direction, and speed of a GPS point)
as the mobility instance’s features. Leveraging the overlap and regularity in these
trajectories, the mobility instances’ features are highly cohesive and exhibit low
coupling.
Essentially, each mobility point is a GPS point recorded on the path between two
venues with a timestamp and GPS coordinate. However, in this chapter, a mobility
point in fact has eight features (see Table 5.2). The former five features are all CMFs,
which are originally recorded by GPS sensors, and the latter three features are all FMFs,
@NV, @AT, and @UE, which are acquired. Specifically, the historical mobility
points’ FMF values are assigned using feature extraction steps to form training
instances, while the mobile phone users’ FMF values are to be predicted through
inference models in practical applications. The CMFs are mutually independent.
None of the mobility features are related to personal information as limited by the
data scale of individual users. Inspired by the collective behavior pattern [13] and the
population modeling mechanism [25], we are able to achieve high prediction
accuracy to meet the demands of application scenarios.
42 5 Individual Behavior Recognition](https://image.slidesharecdn.com/23639119-250517045814-ea801179/85/Human-Behavior-Analysis-Sensing-And-Understanding-Zhiwen-Yu-53-320.jpg)
Human Behavior Analysis Sensing And Understanding Zhiwen Yu
- 1. Human Behavior Analysis Sensing And Understanding Zhiwen Yu download https://ebookbell.com/product/human-behavior-analysis-sensing- and-understanding-zhiwen-yu-47278238 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Explainable Ai Applications For Human Behavior Analysis Paramasivan P https://ebookbell.com/product/explainable-ai-applications-for-human- behavior-analysis-paramasivan-p-58491564 Explainable Ai Applications For Human Behavior Analysis https://ebookbell.com/product/explainable-ai-applications-for-human- behavior-analysis-58491550 Exploring Human Behavior Through Isotope Analysis Applications In Archaeological Research Melanie M Beasley https://ebookbell.com/product/exploring-human-behavior-through- isotope-analysis-applications-in-archaeological-research-melanie-m- beasley-50590798 Computer Analysis Of Human Behavior 1st Edition Gwenn Englebienne Auth https://ebookbell.com/product/computer-analysis-of-human-behavior-1st- edition-gwenn-englebienne-auth-2350562
- 3. The Ellipsis Manual Analysis And Engineering Of Human Behavior Chase Hughes https://ebookbell.com/product/the-ellipsis-manual-analysis-and- engineering-of-human-behavior-chase-hughes-23829898 Semantic Analysis And Understanding Of Human Behavior In Video Streaming Alberto Amato https://ebookbell.com/product/semantic-analysis-and-understanding-of- human-behavior-in-video-streaming-alberto-amato-38378652 How To Rethink Human Behavior A Practical Guide To Social Contextual Analysis Reprint Bernard Guerin https://ebookbell.com/product/how-to-rethink-human-behavior-a- practical-guide-to-social-contextual-analysis-reprint-bernard- guerin-5480872 Machine Learning And Python For Human Behavior Emotion And Health Status Analysis 1st Edition Md Zia Uddin https://ebookbell.com/product/machine-learning-and-python-for-human- behavior-emotion-and-health-status-analysis-1st-edition-md-zia- uddin-58442568 Machine Learning And Python For Human Behavior Emotion And Health Status Analysis Md Zia Uddin https://ebookbell.com/product/machine-learning-and-python-for-human- behavior-emotion-and-health-status-analysis-md-zia-uddin-58466560
- 6. Human Behavior Analysis: Sensing and Understanding
- 7. Zhiwen Yu • Zhu Wang Human Behavior Analysis: Sensing and Understanding
- 8. Zhiwen Yu School of Computer Science Northwestern Polytechnical University Xi’an, China Zhu Wang School of Computer Science Northwestern Polytechnical University Xi’an, China ISBN 978-981-15-2108-9 ISBN 978-981-15-2109-6 (eBook) https://doi.org/10.1007/978-981-15-2109-6 © Springer Nature Singapore Pte Ltd. 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore
- 9. Preface In recent years, human behavior sensing and understanding attracts a lot of interests due to various societal needs, including security, natural interfaces, gaming, affec- tive computing, and assisted living. However, accurate detection and recognition of human behavior is still a big challenge that attracts a lot of research efforts. Traditionally, to identify human behaviors, we first need to continuously collect the sensory data from physical sensing devices (e.g., camera, GPS, and RFID), which can be either worn by humans, attached on objects, or deployed in environ- ments. Afterwards, based on recognition algorithms or classification models, the behavior types can be identified so as to facilitate upper layer applications. Although such traditional behavior identification approaches perform well and are widely adopted, most of them are intrusive and require specific sensing devices, raising issues such as privacy and deployment cost. In this monograph, we aim to provide an overview of recent research progress on noninvasive human behavior sensing and understanding. Specifically, this mono- graph differs from existing literature in the following aspects. On the one hand, we mainly focus on human behavior understanding approaches that are based on noninvasive sensing technologies, including both sensor-based and device-free approaches. On the other hand, while most existing studies are about individual behaviors, we will systematically elaborate how to understand human behaviors of different granularities, including not only individual-level behaviors but also group- level and community-level behaviors. The book includes four parts. In Part I (Chaps. 1 and 2), we introduce and analyze the design, implementation, and development of a typical human behavior sensing and understanding system and then give the main steps of such a system. Part II (Chaps. 3 and 4) mainly focuses on two noninvasive (i.e., sensor-based and device-free) behavior sensing approaches. In Part III (Chaps. 5–7), we elaborate our studies on the understanding of different granularity human behaviors, from individual level to group level and community level. Finally, in Part IV (Chap. 8), we discuss the open issues and possible solutions involved in human behavior sensing and understanding, followed by a conclusion to the whole monograph. Specifically, v
- 10. some of the contents in this monograph might be of particular interest to readers, including noninvasive human behavior sensing approaches (i.e., sensor-based and device-free), as well as the understanding of different granularity human behaviors (i.e., individual level, group level, and community level). We would like to thank Prof. Daqing Zhang at the Software Engineering Institute of Peking University, Beijing, China; Prof. Liming Chen at the School of Computer Science and Informatics of De Montfort University, Leicester, UK; Prof. Xingshe Zhou at the School of Computer Science of Northwestern Polytechnical University, Xi’an, China; and Prof. Bin Guo at the School of Computer Science of Northwestern Polytechnical University, Xi’an, China. We would like to thank all of the members of Ubiquitous Computing group of Northwestern Polytechnical University, China, for their valuable discussions, insights, and helpful comments. We would also like to thank the staff at Springer, Ms. Celine Chang and Ms. Jane Li, for their kind help throughout the publication and preparation processes of the monograph. Xi’an, China Zhiwen Yu Xi’an, China Zhu Wang vi Preface
- 11. Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 From Vision-Based to Sensor-Based and Device-Free Behavior Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.1 Vision-Based Human Behavior Sensing and Recognition . . . . 2 1.1.2 Sensor-Based Human Behavior Sensing and Recognition . . . . 3 1.1.3 Device-Free Human Behavior Sensing and Recognition . . . 4 1.2 From Individual to Group and Community Behavior Recognition . . . 5 1.3 From Pattern-Based to Model-Based Behavior Recognition . . . . . . 7 1.3.1 Pattern-Based Behavior Recognition . . . . . . . . . . . . . . . . . 7 1.3.2 Model-Based Behavior Recognition . . . . . . . . . . . . . . . . . 8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2 Main Steps of Human Behavior Sensing and Understanding . . . . . . 13 2.1 Sensory Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.4 Human Behavior Modeling and Classification . . . . . . . . . . . . . . . 16 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Sensor-Based Behavior Recognition . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.1 Sensor-Based Behavior Recognition Evolution . . . . . . . . . . . . . . . 17 3.2 Behavior Recognition Based on Mobile Devices . . . . . . . . . . . . . . 18 3.2.1 Behavior Sensing and Understanding Scales . . . . . . . . . . . 19 3.2.2 Behavior Sensing and Understanding Paradigms . . . . . . . . 20 3.3 Energy-Efficient Behavior Recognition Using Ubiquitous Sensors . . . 21 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4 Device-Free Behavior Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.1 The Basic Concept of Device-Free Behavior Sensing and Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.1.1 General Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 vii
- 12. 4.1.2 Typical Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.2 Wi-Fi CSI-Based Behavior Sensing and Recognition . . . . . . . . . . 29 4.3 Acoustic-Based Behavior Sensing and Recognition . . . . . . . . . . . . 32 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 5 Individual Behavior Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.1 Human Mobility Prediction by Exploring History Trajectories . . . . 37 5.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.3 Serendipitous Social Interactions Supporting System . . . . . 40 5.1.4 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 5.1.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.2 Disorientation Detection by Mining GPS Trajectories . . . . . . . . . . 48 5.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 5.2.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 5.2.3 Disorientation Detection Problem Formulation . . . . . . . . . . 53 5.2.4 iBDD: Isolation-Based Disorientation Detection . . . . . . . . 56 5.2.5 Disorientation Trajectory Detection Algorithm . . . . . . . . . . 61 5.2.6 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 65 5.3 Human Computer Operation Recognition Based on Smartphone . . . . 70 5.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.3.3 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.3.4 Keystroke Identification . . . . . . . . . . . . . . . . . . . . . . . . . . 74 5.3.5 Word Correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.3.6 Human-Computer Operation Recognition . . . . . . . . . . . . . 80 5.3.7 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.4 Swimmer Localization Based on Smartphone . . . . . . . . . . . . . . . . 85 5.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 5.4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.4.3 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.4.4 Swimming Behavior Recognition . . . . . . . . . . . . . . . . . . . 88 5.4.5 Swimmer Locating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.4.6 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.5 Human Identity Recognition Based on Wi-Fi Signals . . . . . . . . . . 98 5.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.5.3 Problem Analysis and System Framework . . . . . . . . . . . . . 101 5.5.4 Detailed Design of Human Identification . . . . . . . . . . . . . . 103 5.5.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.6 C-FMCW-Based Contactless Respiration Detection Using Acoustic Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 5.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 5.6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 5.6.3 C-FMCW: A High-Resolution Distance Estimation Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 viii Contents
- 13. 5.6.4 Contactless Respiration Detection Using C-FMCW with Commodity Acoustic Devices . . . . . . . . . . . . . . . . . . 119 5.6.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 124 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6 Group Behavior Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.1 Recognition of Group Mobility Level and Group Structure with Mobile Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 6.1.3 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 6.1.4 Group Mobility Classification . . . . . . . . . . . . . . . . . . . . . . 144 6.1.5 Group Structure Recognition . . . . . . . . . . . . . . . . . . . . . . 145 6.1.6 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 152 6.2 Recognition of Group Semantic Interactions . . . . . . . . . . . . . . . . . 154 6.2.1 Social Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 6.2.2 Gathering Multimodal Meeting Content . . . . . . . . . . . . . . 156 6.2.3 Recognizing the Social Semantics . . . . . . . . . . . . . . . . . . . 156 6.2.4 Mining Social Semantics . . . . . . . . . . . . . . . . . . . . . . . . . 158 6.2.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 6.3 Recognition of Group Interaction Patterns . . . . . . . . . . . . . . . . . . 165 6.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 6.3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 6.3.3 Human Semantic Interaction . . . . . . . . . . . . . . . . . . . . . . . 168 6.3.4 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 6.3.5 Collaborative Interaction Capture . . . . . . . . . . . . . . . . . . . 170 6.3.6 Multimodal Interaction Recognition . . . . . . . . . . . . . . . . . 171 6.3.7 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 173 6.4 Group Activity Organization and Suggestion with Mobile Crowd Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 6.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 6.4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 6.4.3 Group Activity Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 177 6.4.4 MobiGroup Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.4.5 Planned Group Activity Preparation . . . . . . . . . . . . . . . . . 182 6.4.6 Running Activity Recognition and Suggestion . . . . . . . . . . 188 6.4.7 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 192 6.5 Predicting Activity Attendance in Mobile Social Networks . . . . . . 194 6.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 6.5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 6.5.3 Problem Statement and System Overview . . . . . . . . . . . . . 197 6.5.4 Feature Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 6.5.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 207 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Contents ix
- 14. 7 Community Behavior Understanding . . . . . . . . . . . . . . . . . . . . . . . . 219 7.1 Discovering Communities in Mobile Social Networks . . . . . . . . . . 219 7.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 7.1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 7.1.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 7.1.4 Multimode Multi-Attribute Edge-Centric Co-clustering Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 7.1.5 Empirical Study Based on Foursquare . . . . . . . . . . . . . . . . 232 7.1.6 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 234 7.2 Understanding Social Relationship Evolution by Using Real-World Sensing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 7.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 7.2.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 7.2.3 Friendship Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 7.2.4 Social Relationship Evolution . . . . . . . . . . . . . . . . . . . . . . 241 7.2.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 242 7.3 Interlinking Off-Line and Online Communities . . . . . . . . . . . . . . . 245 7.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 7.3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 7.3.3 An Overview of HSN . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 7.3.4 Detailed Design of HSN . . . . . . . . . . . . . . . . . . . . . . . . . . 251 7.3.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 254 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 8 Open Issues and Emerging Trends . . . . . . . . . . . . . . . . . . . . . . . . . . 261 8.1 Research Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 8.1.1 Challenges from Human Behavior Itself . . . . . . . . . . . . . . 261 8.1.2 Challenges from the Data . . . . . . . . . . . . . . . . . . . . . . . . . 262 8.1.3 Challenges from Modeling and Evaluation . . . . . . . . . . . . 262 8.1.4 Ten Most Important Problems . . . . . . . . . . . . . . . . . . . . . 263 8.2 Emerging Trends and Directions . . . . . . . . . . . . . . . . . . . . . . . . . 266 8.2.1 Complex Behavior Recognition . . . . . . . . . . . . . . . . . . . . 266 8.2.2 Multilevel Behavior Modeling for Scalability and Reusability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 8.2.3 Abnormal Behavior Recognition . . . . . . . . . . . . . . . . . . . . 268 8.2.4 Intent or Goal Recognition . . . . . . . . . . . . . . . . . . . . . . . . 268 8.2.5 Sensor Data Reuse and Repurposing . . . . . . . . . . . . . . . . . 269 8.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 x Contents
- 15. Chapter 1 Introduction Abstract Human behavior sensing and understanding has been a popular research area during the past decades, which plays an important role in human-computer interaction and public security. It aims to understand what people are doing by sensing and recognizing their activities and their environments. However, accurate detection and recognition of human behavior is still a big challenge that attracts a lot of research efforts. In this chapter, we aim to present an overview of human behavior sensing and understanding techniques, including from vision-based to sensor-based and device-free behavior sensing, from individual to group and community behavior understanding, and from pattern-based to model-based behavior understanding. Behavior recognition technology has been applied to a number of key applications, including pervasive and mobile computing [1, 2], surveillance-based security [3– 10], context-aware computing [11–13], and ambient assistive living [14–18]. Human behavior sensing also serves as the key component in several consumer products. For example, game consoles such as the Nintendo Wii and the Microsoft Kinect rely on the recognition of gestures or even full-body movements to deliver outstanding game interaction with the users. While originally developed for the entertainment sector, these systems have found additional applications, such as for personal fitness training and rehabilitation, and also stimulated new behavior recognition research [19]. Finally, some sports products such as the Philips DirectLife or the Nike+ running shoes integrate motion sensors and offer both amateur and professional athletes feedback on their performance. Moreover, monitoring daily behavior to support medical diagnosis, for rehabilitation, or to assist patients with chronic impairments has been shown to provide key enhancements to traditional medical methods. Some early assistance to encourage humans to adopt a healthy lifestyle was regarded as another important goal, leading to a vast exploration of related human Part of this chapter is based on a previous work: L. Chen, J. Hoey, C. D. Nugent, D. J. Cook and Z. Yu, “Sensor-Based Activity Recognition,” in IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 42, no. 6, pp. 790–808, Nov. 2012. DOI: https://doi.org/10.1109/TSMCC.2012.2198883 © Springer Nature Singapore Pte Ltd. 2020 Z. Yu, Z. Wang, Human Behavior Analysis: Sensing and Understanding, https://doi.org/10.1007/978-981-15-2109-6_1 1
- 16. behaviors, for example, brushing teeth [20] or handwashing, food [21, 22] and medication intake [23, 24], or transportation routines [25]. 1.1 From Vision-Based to Sensor-Based and Device-Free Behavior Sensing 1.1.1 Vision-Based Human Behavior Sensing and Recognition Using vision to sense and understand the behavior of human beings is a very important topic. Vision-based human behavior recognition is the process of labeling image sequences with behavior labels. It is based on the use of visual sensing facilities, such as video cameras, to monitor user behaviors and environmental changes. The generated sensor data are video sequences or digitized visual data. The approaches exploit computer vision techniques, including feature extraction, structural modeling, movement segmentation, action extraction, and movement tracking, to analyze visual observations for pattern recognition. Specifically, the vision-based behavior recognition methods mainly consist of two steps: representation and classification. Generally, for representation approaches, related literatures follow a research trajectory of global representations, local representations, and recent depth-based representations. Earlier studies attempted to model the whole images or silhouettes and represent human activities in a global manner. The approach in ref. 26 is an example of global representation in which space-time shapes are generated as the image descriptors. Then, the emer- gence of space-time interest points (STIPs) [27] triggered significant attention to a new local representation view that focuses on the informative interest points. Meanwhile, local descriptors such as histogram of oriented gradients (HOG) and histogram of optical flow (HOF) oriented from object recognition are widely used or extended to 3D in human behavior recognition area. With the upgrades of camera devices, especially the launch of RGBD cameras in the year 2010, depth image- based representations have been a new research topic and have drawn growing concern in recent years. For example, Jalal et al. [28] propose novel multi-fused features for online human behavior recognition system that recognizes human behaviors from continuous sequences of depth map. Yang et al. [29] propose a general scheme of super normal vector (SNV) to aggregate the low-level polynormals into a discriminative representation, which can be viewed as a simpli- fied version of the Fisher kernel representation. On the other hand, classification techniques keep developing in step with machine learning methods. In fact, lots of classification methods were not originally designed for human behavior recognition. Generally speaking, most behavior classification algorithms can be divided into three categories, namely template-based approaches, generative models, and discriminative models. Template-based approach is a 2 1 Introduction
- 17. relatively simple and well-accepted approach; however, it can be sometimes com- putationally expensive. Generative models learn a model of the joint probability P (X, Y) of the inputs X and the label Y, and then P(Y|X) is calculated using algorithms (e.g., the Bayes classifier) to pick the most likely label Y [30]. In contrast, discriminative models determine the result label directly. Typical algorithms of generative models are hidden Markov model (HMM) and dynamic Bayesian net- work (DBN), while support vector machine (SVM), relevance vector machine (RVM), and artificial neural network (ANN) are typical discriminative models. In recent years, deep learning methods are developed for a large amount of image classification and have achieved good performance. Basically, the deep learning architectures can be categorized into four groups, namely deep neural networks (DNNs), convolutional neural networks (ConvNets or CNNs), recurrent neural networks (RNNs), and some emergent architectures [31]. The ConvNets is the most widely used one among the mentioned deep learning architectures. Compared with traditional machine learning method and their handcrafted features, the ConvNets can learn some representational features automatically [32]. Unlike ConvNets, DNNs still use handcrafted features instead of automatically learning features by deep networks from raw data. RNNs are designed for sequential infor- mation. Activity itself is a kind of time-series data and it is a natural thought to use RNNs for activity recognition. However, it is worth mentioning that while visual monitoring is intuitive and information rich, the vision-based approaches can only work with line-of-sight (LoS) coverage in rich-lighting environments, and also may cause privacy concerns. 1.1.2 Sensor-Based Human Behavior Sensing and Recognition Sensor-based behavior recognition is based on the use of emerging sensor network technologies for activity sensing and understanding [33–35]. The generated sensor data are mainly time series of state changes and/or various parameter values that are usually processed through data fusion, probabilistic, or statistical analysis methods and formal knowledge technologies for activity recognition. Generally, sensors can be attached to an actor under observation, namely wearable sensors or smartphones, or objects that constitute the activity environment, namely dense sensing. Preliminary studies have shown that commodity smartphones equipped with accelerometers can also be used for behavior recognition, which are widely available to the general public [36]. Wearable sensors often use inertial measurement units or radio frequency identification (RFID) tags to gather behavioral information. This approach is effective to recognize physical movements such as exercises. In contrast, dense sensing infers activities by monitoring human–object interactions through the usage of multimodal miniaturized sensors. These sensors are different in types, purposes, output signals, underpinning theoretical principles, and technical 1.1 From Vision-Based to Sensor-Based and Device-Free Behavior Sensing 3
- 18. infrastructure, and can be classified into two main categories in terms of the way they are deployed in activity monitoring applications, i.e., wearable sensors and dense sensors. Wearable sensors, generally, refer to sensors that are positioned directly or indirectly in human body. They generate signals when the user performs activities. As a result, they can monitor features that are descriptive of the person’s physiolog- ical state or movement. Wearable sensors can be embedded into clothes, eyeglasses, belts, shoes, wristwatches, and mobile devices or positioned directly in the body. They can be used to collect information such as body position and movement, pulse, and skin temperature. However, wearable sensors are not suitable for monitoring activities that involve complex physical motions and/or multiple interactions with the environment. In some cases, sensor observations from wearable sensors alone are not sufficient to differentiate activities involving simple physical movements (e.g., making tea and making coffee). As a result, dense sensing-based activity monitoring has emerged. Dense sensing-based activity monitoring refers to the practice that sensors are attached to objects and activities are monitored by detecting user–object interactions. The approach is based on real-world observations that activities are characterized by the objects that are manipulated during their performance. A simple indication of an object being used can often provide useful clues about the behavior being under- taken. As such, it is assumed that activities can be recognized from sensor data that monitors human interactions with objects in the environment. By dense sensing, we refer to the way and scale with which sensors are used. Using dense sensing, a large number of sensors, normally of low cost low power, and miniaturized, are deployed in a range of objects or locations within an environment for the purpose of moni- toring movements and behaviors. 1.1.3 Device-Free Human Behavior Sensing and Recognition Compared with sensor-based behavior sensing, device-free behavior sensing is also an important way to sense the activity of human beings. It can sense the behavior of human beings without the need of carrying any tags or devices. It has the additional advantage of being unobtrusive while offering good privacy protection. Over the past decades, researchers have studied ways of tracking device-free human subjects using different techniques such as camera [37], capacitance [38], pressure [39], infrared [40], and ultrasonic [41]. However, most of these approaches suffer from serious limitations such as occlusion [37, 40], high deployment cost [38, 39], or short range [41]. Radio frequency (RF)-based techniques have the advantages of long range, low cost, and ability to work through nonconducting walls and obstacles. Therefore, RF-based device-free systems become popular to detect, locate, and track human behaviors. Specifically, RF devices include ZigBee, Wi-Fi, RFID, etc. The basic idea of RF-based device-free sensing system is as follows: when located in indoor 4 1 Introduction
- 19. environment with such a system, the movement of human bodies will affect the wireless signals and change the multipath profile of the system. Based on this principle, we are able to recognize human behaviors by exploring the changes of wireless signals caused by user movements. Especially, Wi-Fi signal-based human activity recognition technologies can be applied at commercial off-the-shelf (COTS) devices. By monitoring the wireless channel state, the receiver can measure small signal changes caused by human movements and use these changes to recognize the surrounding human activities [42]. Device-free behavior sensing technology can be useful in many practical appli- cations including intrusion detection and tracking for home and office applications, which could enhance the safety of law-enforcement personnel, and low-cost long- range asset protection, e.g., border protection or protection of railroad tracks. In addition, device-free behavior sensing technology can be used to enhance traditional security systems, such as motion detection and video surveillance by providing non-line-of-sight (NLoS) detection and lower deployment cost. 1.2 From Individual to Group and Community Behavior Recognition In the past decades, numerous research efforts have been made to model and recognize the behavior of individuals. However, group and community behavior recognition has attracted much attention recently, as it has many practical applica- tions such as abnormal group detection, group affective computing, video surveil- lance, and public security [43]. Specifically, on the one hand, the large deployment of sensor network in public facilities, private buildings and outdoors environments, and digital traces left by people while interacting with cyber-physical spaces are accumulating at an unprecedented breadth, depth, and scale. On the other hand, the recent explosion of sensor-rich smartphone market and the phenomenal growth of geo-tagged data (e.g., Twitter messages, Foursquare check-ins) have enabled the analysis of new dimensions of contexts that involve the social and urban context. We call all those traces left by people the “digital footprints.” Leveraging the capacity to collect and analyze the “digital footprints” at group or community scale, a new research field called “social and community intelligence (SCI)” [44] is emerging that aims at revealing the patterns of individual, group, and societal behaviors. The scale and heterogeneity of the multimodal, mixed data sources present us an opportunity to compile the digital footprints into a comprehensive picture of indi- vidual’s daily life facets, and radically change the way we build computational models of human behaviors. Numerous innovative services will be enabled, includ- ing human health, public safety, urban planning, environment monitoring, and so on. The development of social and community intelligence will greatly expand the scale and depth of context-aware computing, from merely personal awareness to the understanding of social interactions (e.g., social relations, community structures) and 1.2 From Individual to Group and Community Behavior Recognition 5
- 20. urban dynamics (e.g., traffic jams, hot spots in cities). For example, Su et al. [45] explored coherent LSTM to model the nonlinear characteristics in crowd behaviors. Zhuang et al. [46] proposed an end-to-end deep architecture, differential recurrent convolutional neural networks (DRCNN), for group activity recognition. Specifically, people constantly participate in social activities to interact with others and form various communities. Social activities such as making new friends, forming an interest group to exchange ideas, and sharing knowledge with others are constantly taking place in human society. The analysis of social interactions has been studied by social scientists and physicists for a couple of decades [47]. An excellent introduction to the concepts and the mathematical tools for social networks analysis can be referred [48]. In the early stage, efforts on social network analysis are mostly based on the relational data obtained by survey. During the last two decades, we have observed an explosive growth of social applications such as chatting, shopping, and experience sharing. These applications, along with traditional email and instant messaging, have changed the way we communicate with each other and form social communities. Corresponding to this trend, a large amount of work on social network analysis and knowledge discovery springs up, including email communication networks [49], scientific collaboration, and co-authorship network [50]. More recently, as the Internet stepped into the era of Web 2.0, which advocates that users interact with each other as contributors to the websites’ content, researchers turned their attention to the online social utilities, such as Facebook, Twitter, and Blogs. For example, ArterMiner [51] seeks to harvest personal profile information from a user’s homepage. AmitSheth’s research group has done much work on summarization of event information like space, time, and theme from social web resources for building public services [52]. Twitter has been reported to support real-time mining of natural disasters such as earthquakes [53] and the moods of citizens [54]. Understanding human movement in urban environments has direct implications for the design of public transport systems (e.g., more precise bus scheduling, improved services for public transport users), traffic forecasting (e.g., hotspot prediction), and route recommendation (e.g., for transit-oriented urban development). A number of studies have extracted citywide human mobility patterns using large-scale data from smart vehicles, mobile phones, and smart cards used in public transportation systems. The Real Time Rome project of MIT uses aggregated data from buses and taxies to better understand urban dynamics in real time [55]. The learned human mobility patterns are also useful for urban planning. For example, Nicholson and Noble [56] have studied how to leverage the learned human move- ment dynamics to improve the distribution of cell infrastructure (e.g., to have a better load balance among cell towers) in wireless communication networks. Zhang et al. [57] has put forward a novel cross-modal representation learning method that uncovers urban dynamics with massive geo-tagged social media data. Specifically, we will mainly focus on individual, group, and community behavior sensing and understanding in Chaps. 5–7, respectively. 6 1 Introduction
- 21. 1.3 From Pattern-Based to Model-Based Behavior Recognition Although primitive sensor data can be obtained through behavior monitoring, recognition models are critical to interpret the sensor data to understand human behaviors. In particular, the mechanisms through which behaviors are recognized are closely related to the nature and representation of behavior recognition models. Generally speaking, recognition models can be built using two different approaches, i.e., pattern based and model based [58–60]. 1.3.1 Pattern-Based Behavior Recognition The first one is to learn recognition models from existing large-scale datasets of human behaviors based on data mining and machine learning techniques. This method involves the creation of probabilistic or statistical behavior recognition models, followed by training and learning processes. To build a human behavior sensing and recognition system, the connection between signal variations and human activities must be established. If the signal variation patterns have unique and consistent relations with certain human activities, it is possible for a pattern-based (or learning-based) method to recognize human behaviors accurately from signal patterns. As this method is driven by data, and the behavior inference is based on probabilistic or statistical classification, it is often referred to as data-driven or pattern-based approaches. The key to designing pattern-based approaches is to observe and find discrimi- native patterns to construct features and differentiate different human behaviors of interest. The features can be very simple or sophisticated, depending on the com- plexity of the recognition task and the required granularity. For simple sensing tasks, feature selection is often based on intuition or direct observation. When the number of behaviors that need to be distinguished is small, it is often easy to find regular but differentiable patterns. In this case, one or two features may be enough to distinguish among behaviors. As both the number of human behaviors and the sensing granu- larity increase, it becomes challenging to find one-to-one mappings between behav- iors and feature patterns. One or two simple features are not enough for this task anymore. In this case, more features are needed to increase the dimension of feature space, and more powerful classifiers are needed. The advantages of data-driven approaches are the capabilities of handling uncer- tainty and temporal information. However, this method requires large datasets for training and learning, and suffers from the data scarcity or the “cold start” problem. It is also difficult to apply learnt recognition models from one person to another. As such, this method suffers from the problems of scalability and reusability. Despite these drawbacks, pattern-based approaches have been very popular and successful in human behavior sensing and understanding applications because they are not only 1.3 From Pattern-Based to Model-Based Behavior Recognition 7
- 22. conceptually intuitive but also relatively simple to design, for both data collection and algorithm development. 1.3.2 Model-Based Behavior Recognition Different from pattern-based approaches, which often involve nontrivial training effort and could only recognize a limited set of pre-defined behaviors, model-based recognition approaches are based on the understanding and abstraction of a mathe- matical relationship among human behaviors and the sensory data. Model-based approaches usually exploit rich prior knowledge in the domain of interest to con- struct models directly using knowledge engineering and management technologies, which involves knowledge acquisition, formal modeling, and representation. Rec- ognition models generated based on this method are normally used for behavior recognition or prediction through formal logical reasoning, e.g., deduction, induc- tion, or abduction. As such, this method is referred to as knowledge-driven or model- based approach. Knowledge-driven approaches have the advantages of being semantically clear, logically elegant, and easy to get started. However, they are weak in handling uncertainty and temporal information and the models could be viewed as static and incomplete. References 1. M. Weiser, “The computer for the twenty-first century,” Sci Am., vol. 265, no. 3, pp. 94–104, 1991. 2. T. Choudhury, S. Consolvo, and B. Harrison, “The mobile sensing platform: An embedded activity recognition system,” IEEE Pervasive Comput., vol. 7, no. 2, pp. 32–41, Apr./Jun. 2008. 3. J.K. Aggarwal and Q. Cai, “Human motion analysis: A review,” Comput. Vis. Image Underst., vol. 73, no. 3, pp. 428–440, 1999. 4. C. Cedras and M. Shah, “Motion-based recognition: A survey,” Image Vis. Comput., vol. 13, no. 2, pp. 129–155, 1995. 5. D.M. Gavrila, “The visual analysis of human movement: A survey,” Comput. Vis. Image Underst., vol. 73, no. 1, pp. 82–98, 1999. 6. R. Poppe, “A survey on vision-based human action recognition,” Image Vis. Comput., vol. 28, no. 6, pp. 976–990, 2010. 7. T.B. Moeslund, A. Hilton, and V. Kruger, “A survey of advances in vision-based human motion capture and analysis,” Comput. Vis. Image Underst., vol. 104, no. 2, pp. 90–126, 2006. 8. A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Comput. Surv., vol. 38, no. 4, pp. 1–45, 2006. 9. P. Turaga, R. Chellappa, and O. Udrea, “Machine recognition of human activities: A survey,” IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 11, pp. 1473–1488, Nov. 2008. 10. D. Weinland, R. Ronfard, and E. Boyer, “A survey of vision-based methods for action representation, segmentation and recognition,” Comput. Vis. Image Underst., vol. 115, no. 2, pp. 224–241, 2011. 8 1 Introduction
- 23. 11. K. Laerhoven and K. A. Aidoo, “Teaching context to applications,” J. Pers. Ubiquitous Comput., vol. 5, no. 1, pp. 46–49, 2001. 12. C. R. Wren and E. M. Tapia, “Toward scalable activity recognition for sensor networks,” in Proc. 2nd Int. Workshop Location Context Awareness, 2006, pp. 168–185. 13. M. Stikicand and B. Schiele, “Activity recognition from sparsely labeled data using multi- instance learning,” in Proc. Location Context Awareness, 2009, vol. 5561, pp. 156–173. 14. M. Philipose, K.P. Fishkin, M. Perkowitz, D.J. Patterson, D. Fox, H. Kautz, and D. Hahnel, “Inferring activities from interactions with objects,” IEEE Pervasive Comput., vol. 3, no. 4, pp. 50–57, Oct./Dec. 2004. 15. D. Cook and M. Schmitter-Edgecombe, “Assessing the quality of activities in a smart environ- ment,” Methods Inf. Med., vol. 48, no. 5, pp. 480–485, 2009. 16. T. Kasterenand and B. Krose, “Bayesian activity recognition in residence for elders,” in Proc. Int. Conf. Intell. Environ., Feb. 2008, pp. 209–212. 17. L. Chen and C. D. Nugent, “Ontology-based activity recognition in intelligent pervasive environments,” Int. J. Web Inf. Syst., vol. 5, no. 4, pp. 410–430, 2009. 18. G. Singla, D. Cook, and M. Schmitter-Edgecombe, “Recognizing independent and joint activities among multiple residents in smart environments,” J. Ambient Intell. Humaniz Comput., vol. 1, no. 1, pp. 57–63, 2010. 19. J. Sung, C. Ponce, B. Selman, and A. Saxena. 2011. Human activity detection from RGBD images. In Proceedings of the AAAI Workshop on Plan, Activity, and Intent Recognition. 20. Jonathan Lester, Tanzeem Choudhury, and Gaetano Borriello. 2006. A practical approach to recognizing physical activities. In Proceedings of the International Conference on Pervasive Computing. 1–16. 21. Oliver Amft, Martin Kusserow, and Gerhard Troster. 2007. Probabilistic parsing of dietary activity events. In Proceedings of BSN. Springer, 242–247. 22. G. Pirkl, K. Stockinger, K. Kunze, and P. Lukowicz. 2008. Adapting magnetic resonant coupling based relative positioning technology for wearable activity recognition. In Proceed- ings of ISWC. 47–54. 23. D. Wan. 1999. Magic medicine cabinet: A situated portal for consumer healthcare. In Handheld and Ubiquitous Computing. Springer, 352–355. 24. R. de Oliveira, M. Cherubini, and N. Oliver. 2010. MoviPill: Improving medication compliance for elders using a mobile persuasive social game. In Proceedings of UbiComp, Vol. 1001. 36. 25. J. Krumm and E. Horvitz. 2006. Predestination: Inferring destinations from partial trajectories. In Proceedings of UbiComp. 243–260. 26. M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri, “Actions as space-time shapes,” in Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, pp. 1395–1402, Beijing, China, 2005. 27. I. Laptev and T. Lindeberg, “Space-time interest points,” in Proceedings Ninth IEEE Interna- tional Conference on Computer Vision, vol. 1, pp. 432–439, Nice, France, 2003. 28. A. Jalal, Y. Kim, Y. Kim, et al. Robust human activity recognition from depth video using spatiotemporal multi-fused features. Pattern recognition, 2017, 61: 295–308. 29. X. Yang and Y. Tian. Super normal vector for human activity recognition with depth cameras. IEEE transactions on pattern analysis and machine intelligence, 2017, 39(5): 1028–1039. 30. A. Jordan, “On discriminative vs. generative classifiers: a comparison of logistic regression and naive Bayes,” Advances in neural information processing systems, vol. 14, p. 841, 2002. 31. S. Min, B. Lee, and S. Yoon, “Deep learning in bioinformatics,” Briefings in Bioinformatics, vol. 17, 2016. 32. G. Luo, S. Dong, K. Wang, and H. Zhang, “Cardiac left ventricular volumes prediction method based on atlas location and deep learning,” in 2016 IEEE International Conference on Bioin- formatics and Biomedicine (BIBM), pp. 1604–1610, Shenzhen, China, 2016 33. A. Pantelopoulos and N. G. Bourbakis, “A survey on wearable sensor-based systems for health monitoring and prognosis,” IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 40, no. 1, pp. 1–12, Jan. 2010. References 9
- 24. 34. H. Alemdar and C. Ersoy, “Wireless sensor networks for healthcare: A survey,” Comput. Netw., vol. 54, no. 15, pp. 2688–2710, 2010. 35. D. Ding, R. A. Cooper, P. F. Pasquina, and L. Fici-Pasquina, “Sensor technology for smart homes,” Maturitas, vol. 69, no. 2, pp. 131–136, 2011. 36. Y. Lu, Y. Wei, L. Liu, et al. Towards unsupervised physical activity recognition using smartphone accelerometers. Multimedia Tools and Applications, 2017, 76(8): 10701–10719. 37. J. Krumm, S. Harris, B. Meyers, B. Brumitt, M. Hale, and S. Shafer. Multi-camera multi-person tracking for easy living. In Proceedings of Third IEEE International Workshop on Visual Surveillance, IWVS’ 00, pages 3–10, 2000. 38. M. Valtonen, J. Maentausta, and J. Vanhala. Tiletrack: Capacitive human tracking using floor tiles. In IEEE International Conference on Pervasive Computing and Communications, PerCom’09, pages 1–10, 2009. 39. R. J. Orr and G. D. Abowd. The smart floor: a mechanism for natural user identification and tracking. In CHI ‘00 extended abstracts on Human factors in computing systems, CHI EA ‘00, pages 275–276, 2000. 40. D. De, W. Song, M. Xu, C. Wang, D. Cook, and X. Huo. Findinghumo: Real-time tracking of motion trajectories from anonymous binary sensing in smart environments. In Proceedings of the 32nd IEEE International Conference on Distributed Computing Systems, ICDCS’12, pages 163–172, 2012. 41. T.W. Hnat, E. Griffiths, R. Dawson, and K. Whitehouse. Doorjamb: unobtrusive room-level tracking of people in homes using doorway sensors. In Proceedings of the 10th ACM Confer- ence on Embedded Network Sensor Systems, SenSys’12, pages 309–322, 2012. 42. Wang W, Liu A X, Shahzad M, et al. Device-free human activity recognition using commercial WiFi devices[J]. IEEE Journal on Selected Areas in Communications, 2017, 35(5): 1118–1131. 43. Li T, Chang H, Wang M, et al. Crowded scene analysis: A survey IEEE transactions on circuits and systems for video technology, 2015, 25(3): 367–386. 44. Zhang, D., Guo, B., & Yu, Z. (2011). The emergence of social and community intelligence. IEEE Computer, 44(7), 21–28. 45. Su H, Dong Y, Zhu J, et al. Crowd Scene Understanding with Coherent Recurrent Neural Networks[C]. IJCAI. 2016, 1: 2. 46. Zhuang N, Yusufu T, Ye J, et al. Group activity recognition with differential recurrent convolutional neural networks[C]. Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference on. IEEE, 2017: 526–531. 47. Freeman, L. C. (2004). The development of social network analysis: A study in the sociology of science. Empirical Press. 48. Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. Cambridge, UK: Cambridge University Press. 49. McCallum, A., Wang, X., & Corrada-Emmanuel, A. (2007). Topic and role discovery in social networks with experiments on Enron and academic email. Journal of Artificial Intelligence Research. 30(1), 249–272. 50. Barabasi, A. L., Jeong, H., Neda, Z., Ravasz, E., Schubert, A., & Vicsek, T. (2002). Evolution of the social network of scientific collaborations. Statistical Mechanics and its Applications, 311 (3–4), 590–614. 51. Tang, J., Jin, R. M., & Zhang, J. (2008). A topic modeling approach and its integration into the random walk framework for academic search. Paper presented at the Meeting of 2008 IEEE International Conference on the Meeting of 2008 IEEE International Conference on Data Mining. Pisa, Italy. 52. Sheth, A. (2010). Computing for human experience – Semantics-empowered sensors, services, and social computing on the ubiquitous web. IEEE Internet Computing, 14 (1), 88–97. 53. Sakaki, T., Okazaki, M., & Matsuo, Y. (2010). Earthquake shakes Twitter users: Real-time event detection by social sensors. Paper presented at the Meeting of WWW 2010 Conference. Raleigh, NC. 10 1 Introduction
- 25. 54. Bollen, J., Pepe, A., & Mao, H. (2009). Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. Paper presented at the meeting of WWW 2009 Conference. Madrid, Spain. 55. Calabrese, F., & Ratti, C. (2006). Real time Rome. Networks and Communications Studies, 20 (3–4), 247–258. 56. Nicholson, J., & Noble, B. D. (2008). Bread crumbs: Forecasting mobile connectivity. Paper presented at the Meeting of Mobile Computing and Networking. San Francisco, CA. 57. Zhang C, Zhang K, Yuan Q, et al. Regions, periods, activities: Uncovering urban dynamics via cross-modal representation learning[C]//Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017: 361–370. 58. Wang Z, Guo B, Yu Z, et al. Wi-Fi CSI-Based Behavior Recognition: From Signals and Actions to Activities. IEEE Communications Magazine, 2018, 56(5): 109–115. 59. Wu D, Zhang D, Xu C, Wang H, Li X (2017) Device-free Wi-Fi human sensing: from pattern- based to model-based approaches. IEEE Communications Magazine, 55(10): 91–97. 60. Chen L, Hoey J, Nugent C D, et al. Sensor-Based Activity Recognition, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2012, 42(6):790–808. References 11
- 26. Chapter 2 Main Steps of Human Behavior Sensing and Understanding Abstract Human behavior sensing and understanding is a complex process that can be roughly characterized by four main steps, including the following: (1) to choose appropriate sensors to monitor and capture a user’s behavior along with the state change of the environment; (2) to collect, store, and process collected information through data analysis techniques and/or knowledge representation formalisms at appropriate levels of abstraction; (3) to extract features in a way that different behaviors can be distinguished accordingly; and (4) to select or develop reasoning/ classification algorithms to infer behaviors from sensor data. For each individual task, a number of methods, technologies, and tools are available for use. It is often the case that the selection of a method used for one task is dependent on the method of another task. We present the comprehensive methods used for each of these steps in the following sections. 2.1 Sensory Data Collection With the development of sensing technologies and the advanced mobile computing techniques, it becomes a reality that human behavior data not only could be obtained from device-based sensors, but also is accessible from online social networks, as shown in Fig. 2.1. Traditionally, a single device like a GPS tracker is employed to record human trajectories, while it is more convenient and efficient to obtain different aspects of human behavior data since sensors are now integrated and embedded in every portable or wearable device. Specifically, a commercial smartphone now contains a wide range of sensors including accelerometers, gyro- scopes, and magnetometers, to name a few, which could be employed to collect human activity data. Besides, the prevalence uses of social network applications, such as Twitter and Facebook, have revealed millions of human behavior records (e.g., check-in trajectories) that could be leveraged to understand human behavior patterns. As a result, gathering behavior data through physical devices and collecting activity trajectories from online social networks are considered as two major domains for human behavior sensing; both domains could provide effective infor- mation for research applications. © Springer Nature Singapore Pte Ltd. 2020 Z. Yu, Z. Wang, Human Behavior Analysis: Sensing and Understanding, https://doi.org/10.1007/978-981-15-2109-6_2 13
- 27. As a matter of fact, the operating systems running on each device have transformed the hardware sensors into software APIs, and similarly the online social network platforms also provide programming interfaces for developers to access different types of data. Therefore, it is straightforward to collect data from those two distinct domains by applying the following common procedures: researchers or developers first write programs leveraging those available APIs, after that they run the programs in a backend manner, and finally the behavior data are collected and stored for further analysis. For example, web crawler programming could be applied for collecting online human behavior data by leveraging APIs like Twitter APIs [1] and Foursquare APIs [2]. Besides, operating systems like Android and iOS also provide APIs [3, 4] that allow developers to directly access embedded sensor readings in smart devices. Other than this, researchers usually need to recruit volunteers or make the data collection program publicly available, so that they can gather human behavior data through physical devices. However, collecting data from social network platforms does not require such procedures; researchers only have to query corresponding platforms using some indexes, such as a location, a specific word, or a target user ID, to obtain the desired human activity information. By applying the above procedures, one can collect sensory data of human behavior not only from physical devices, but also from online social network platforms. 2.2 Data Preprocessing The original sensory data, especially from physical sensors, is often incomplete, inconsistent, and likely to contain many errors. For instance, due to the affection on signals from crowd buildings, there might be huge differences between consecutive GPS locations than it should have in real world. Therefore, data preprocessing becomes one sufficient procedure before any insightful analysis can be taken onto the collected data. In general, the following steps are considered: data anonymization, data cleaning, data filtering, and data transformation. Firstly, data anonymization is required since human behavior is personal sensitive and is usually related to privacy issue, and MD5 hashing [5] and k-anonymity [6] are two popular data anonymization algo- rithms. After that, data cleaning is applied to deal with problems like data inconsis- tency and noisy data influence. Specifically, data inconsistency usually appears in data collected from online platforms, where user profile information may contain Fig. 2.1 Main steps of human behavior sensing and understanding 14 2 Main Steps of Human Behavior Sensing and Understanding
- 28. conflict records. Meanwhile, noisy influence often exists in physical sensory data that sensor readings are affected by other signals or environmental changes, which could cause inaccuracy when monitoring human activities. Afterwards, researchers have to choose the most effective or meaningful parts for further analysis, which refers to data filtering. In fact, real-world sensory data are usually incomplete, such that human behavior is often recorded as many fragments and only some of them, which contain enough records in terms of timescale or other criterions, are valuable to be analyzed. Hence, data filtering is performed to filter out sensory data by different criterions that could contribute to discovering pattern from those filtered data. Finally, data transformation is applied to conduct data normalization or aggrega- tion, and data representation before eventually feeding the data into learning models. In particular, different learning models require different data types or formats, such as vector, matrix, and time-series sequences. Therefore, researchers have to trans- form original data formats into model-orientated formats and normalize the data when needed. After conducting all these above procedures, the raw data is prepared to be imported into learning algorithms or models for further analysis. 2.3 Feature Extraction Apart from monitoring human behaviors and activities based on physical sensors or cyber information, one significant task is to understand the relationship between activities and sensory data, which aims to discover correlations that could lead to better human interpretations. Generally, each instance of sensory data consists of several attributes, and feature extraction manages to derive values intended to be informative and nonredundant, which can be used to facilitate the subsequent learning steps. Specifically, since there are various types of sensors, a human activity could be captured from different aspects that result in high-dimensional sensory data. Therefore, some data-driven- based dimensionality reduction methods (e.g., principal component analysis [7]) are employed to extract meaningful features from the original data. Besides, researchers can also define their own features according to expert knowledge. After that, correlation analysis is conducted to discover insightful relationships between extracted features and human activities. Usually, some metrics like Pearson correla- tion coefficient [8] are applied to measure the relationship between features and activities, while some other metrics like Information Gain [9] are used to determine the significance of different features. By conducting the above procedures, not only features are extracted from the original sensory data, but also some correlations are learned with respect to different tasks and models. 2.3 Feature Extraction 15
- 29. 2.4 Human Behavior Modeling and Classification Many methodologies are proposed to address human behavior modeling and classi- fication problems, and they can be roughly divided into two directions of approaches: logical reasoning and probabilistic reasoning [10]. Specifically, logical reasoning is the process of using a rational, systematic series of steps based on sound mathematical procedures and given statements to arrive at a conclusion. Traditional human activity recognition and modeling usually prefer this approach to infer human behaviors based on deployed sensors’ readings, and this kind of method is more like an expert knowledge-based approach. Logical reasoning models are easy to be interpreted and computationally efficient that could be applied in real-time situations. Compared with logical reasoning, probabilistic reasoning is a pattern-based approach, and is commonly classified into three types: supervised, unsupervised, and semi-supervised learning models. The supervised learning models depend on the labeled dataset by which the parameters could be learned from each category. On the contrast, unsupervised learning models try to find hidden structure in unlabeled data, such as clustering algorithms. Semi-supervised learning falls between the supervised and unsupervised learning, which mainly makes use of both labeled and unlabeled data for training—typically for the sparse of labeled data. Different from logical reasoning, probabilistic reasoning is more robust when facing problems like noisy, uncertain, and incomplete sensory data. However, since it is data-driven-based approach, it is more computationally expensive and difficult to achieve the goal of real-time modeling. References 1. https://dev.twitter.com/overview/api, visited: 24 Jul 2018. 2. https://developer.foursquare.com/docs/, visited: 24 Jul 2018. 3. https://developer.android.com/reference/, visited: 24 Jul 2018. 4. https://developer.apple.com/documentation/, visited: 24 Jul 2018. 5. https://en.wikipedia.org/wiki/md5, visited: 24 Jul 2018. 6. Sweeney, Latanya. "k-anonymity: A model for protecting privacy." International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 10.05 (2002): 557–570. 7. Jolliffe, Ian. “Principal component analysis.” International encyclopedia of statistical science. Springer, Berlin, Heidelberg, 2011. 1094–1096. 8. Benesty, Jacob, et al. "Pearson correlation coefficient." Noise reduction in speech processing, Springer, Berlin, Heidelberg, 2009. 1–4. 9. Kent, John T. "Information gain and a general measure of correlation." Biometrika 70.1 (1983): 163–173. 10. Liang, Yunji, et al. "Activity Recognition Using Ubiquitous Sensors: An Overview." Wearable Technologies: Concepts, Methodologies, Tools, and Applications. IGI Global, 2018. 199–230. 16 2 Main Steps of Human Behavior Sensing and Understanding
- 30. Chapter 3 Sensor-Based Behavior Recognition Abstract In this monograph, sensor-based behavior recognition mainly refers to the use of emerging sensor network technologies for behavior monitoring and under- standing. The generated sensor data from sensor-based monitoring are mainly time series of state changes and/or various parameter values that are usually processed through data fusion, probabilistic, or statistical analysis methods and formal knowl- edge technologies for behavior recognition. Specifically, sensors can be attached to an actor under observation, namely wearable sensors or smartphones, or objects that constitute the environment, namely dense sensing. Wearable sensors often use inertial measurement units and radio frequency identification (RFID) tags to gather a user’s behavioral information. This approach is effective to recognize physical movements such as physical exercises. In contrast, dense sensing infers behaviors by monitoring human–object interactions through the usage of multiple multimodal miniaturized sensors. In this chapter, we first give a brief introduction to the historical evolution of sensor-based behavior recognition. Afterwards, we present the mobile device-enabled behavior recognition approach, which is a typical type of sensor-based behavior recognition, followed by a discussion on the key issues of developing behavior recognition systems using mobile devices. 3.1 Sensor-Based Behavior Recognition Evolution The idea of using sensors for behavior monitoring and recognition has been existent since the late 1990s. It was initially pioneered and experimented by the work of the Neural Network house [1] in the context of home automation, and a number of location-based applications aiming to adapt systems to users’ whereabouts [2– 4]. The approach was soon found to be more useful and suitable in the area of ubiquitous and mobile computing—an emerging area in the late 1990s—due to its Part of this chapter is based on a previous work: L. Chen, J. Hoey, C. D. Nugent, D. J. Cook and Z. Yu, “Sensor-Based Activity Recognition,” in IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 42, no. 6, pp. 790–808, Nov. 2012. DOI: https://doi.org/10.1109/TSMCC.2012.2198883 © Springer Nature Singapore Pte Ltd. 2020 Z. Yu, Z. Wang, Human Behavior Analysis: Sensing and Understanding, https://doi.org/10.1007/978-981-15-2109-6_3 17
- 31. easy deployment. As such, extensive research has been undertaken to investigate the use of sensors in various application scenarios of ubiquitous and mobile computing, leading to considerable work on context awareness [5–7], smart appliances [8, 9], and activity recognition [10–14]. Most research at that time made use of wearable sensors, either dedicated sensors attached to human bodies or portable devices like mobile phones, with application to ubiquitous computing scenarios such as provid- ing context-aware services. Activities being monitored in these researches are mainly physical activities like motion, walking, and running. These early works lay a solid foundation for wearable computing and still inspire and influence today’s research. In the early 2000s, a new sensor-based approach that uses sensors attached to objects to monitor human activities appeared. This approach, which was later dubbed as the “dense sensing” approach, performs activity recognition through the inference of user–object interactions [15, 16]. The approach is particularly suitable to deal with activities that involve a number of objects within an environment, or instrumental activities of daily living (ADL) [17, 18]. Research on this approach has been heavily driven by the intensive research interests and huge research effort on smart home-based assistive living, such as the EU’s AAL program [19]. In particular, sensor-based activity recognition can better address sensitive issues in assistive living such as privacy, ethics, and obtrusiveness than conventional vision- based approaches. This combination of application needs and technological advan- tages has stimulated considerable research activities in a global scale, which gave rise to a large number of research projects, including the House_n [20], CASAS [21], Gator-Tech [22], inHaus [23], AwareHome [24], DOMUS [25], and iDorm [26] projects, to name but a few. As a result of the wave of intensive investigation, there have seen a plethora of impressive works on sensor-based behavior recognition in the past several years [27–40]. While substantial research has been undertaken, and significant progress has been made, the two main approaches, wearable sensor-based and dense sensing-based behavior recognition, are currently still focuses of the research community [41– 45]. The former is mainly driven by the ever-popular pervasive and mobile com- puting, while the latter is predominantly driven by smart environment applications such as AAL. Interests in various novel applications are still increasing and appli- cation domains are rapidly expanding. 3.2 Behavior Recognition Based on Mobile Devices During the past decade, smart mobile devices become more and more popular, which are usually embedded with various sensors, such as accelerometer and GPS. These sensors generate signals when the user performs behaviors. As a result, sensor- enriched mobile devices can monitor features that are descriptive of the person’s physiological state or movement, and can be used to collect information such as body position and movement, pulse, and skin temperature. Researchers have found 18 3 Sensor-Based Behavior Recognition
- 32. that different types of sensor information are effective to classify different types of activities. For example, accelerometer embedded in smartphones is capable of characterizing human’s movements, e.g., standing, walking, and running [46– 48]. Similarly, by collecting audio information from the phone’s microphone, it is possible to recognize a user’s activity, e.g., listening to music, speaking, and sleeping [49, 50], and even monitor running rhythm [51], as well as the sound- related respiratory symptoms of a user, such as sneeze or cough [52]. With various sensor data, including Wi-Fi, accelerometer, compass, and GPS, the user’s emotion can also be inferred [53–55]. Generally, mobile device-based behavior recognition systems may operate at multiple scales, enabling applications from personal sensing and group sensing to community sensing [56]. Meanwhile, another key issue is about how much the user (i.e., the person carrying the mobile device) should be actively involved during the sensing activity (e.g., taking the device out of the pocket to collect a sound sample or take a picture). In other words, should the user actively participate, known as participatory sensing, or, alternatively, passively participate, known as opportunistic sensing [57]? Each of these sensing paradigms presents important trade-offs. In this section, we discuss different sensing and understanding scales and paradigms of mobile device-enabled behavior recognition systems. 3.2.1 Behavior Sensing and Understanding Scales Personal behavior sensing and understanding systems are designed for a single individual. Typical scenarios include tracking the user’s exercise routines or auto- mating diary collection. Typically, personal behavior sensing and understanding systems generate data for the sole consumption of the user and are not shared with others. An exception is healthcare systems where limited sharing with medical professionals is common (e.g., primary care giver or specialist). Individuals who use behavior sensing and understanding systems that share a common goal, concern, or interest collectively represent a group. These group behavior recognition systems are likely to be popular and reflect the growing interest in social networks or connected groups (e.g., at work, in the neighborhood, friends) who may want to share information freely or with privacy protection. There is an element of trust in group behavior sensing and understanding systems that simplify otherwise difficult problems, such as attesting that the collected sensor data is correct or reducing the degree to which aggregated data must protect the individual. Common use cases include assessing neighborhood safety, sensor-driven mobile social networks, and forms of citizen science. Most examples of community behavior sensing and understanding only become useful once they have a large number of people participating, e.g., tracking the spread of disease across a city, congestion patterns across city roads, or a noise map of a city. These systems represent large-scale data collection and analysis for the good of the community. To achieve scale implicitly requires the cooperation of 3.2 Behavior Recognition Based on Mobile Devices 19
- 33. strangers who will not trust each other. This increases the need for systems with strong privacy protection and low commitment levels from users. 3.2.2 Behavior Sensing and Understanding Paradigms One issue common to the different types of sensing and understanding scale is to what extent the user is actively involved in the system [57]. We discuss two points in the design space: participatory sensing, where the user actively engages in the data collection activity (i.e., the user manually determines how, when, what, and where to sample), and opportunistic sensing, where the data collection stage is fully auto- mated with no user involvement. The benefit of opportunistic sensing is that it lowers the burden placed on the user, allowing overall participation by a population of users to remain high even if the system is not that personally appealing. This is particularly useful for community behavior sensing and understanding. One of the main challenges of using opportu- nistic sensing is the context problem of mobile devices, e.g., the system wants to only take a sound sample for a citywide noise map when the device is out of the pocket or bag. These types of context issues can be solved by using the embedded sensors; for example, the accelerometer or light sensors can determine if the device is out of the pocket. Participatory sensing, which is gaining interest in the mobile crowd sensing community [57], places a higher burden or cost on the user, e.g., manually selecting data to collect (e.g., lowest petrol prices) and then sampling it (e.g., taking a picture). An advantage is that complex operations can be supported by leveraging the intelligence of the person in the loop who can solve the context problem in an efficient manner. For example, a person who wants to participate in collecting a noise or air quality map of their neighborhood simply takes out his/her mobile device to solve the context problem. One drawback of participatory sensing is that the quality of data is dependent on participant enthusiasm to reliably collect sensing data and the compatibility of a person’s mobility patterns to the intended goals of the system (e.g., collect pollution samples around schools). Clearly, opportunistic and participatory represent extreme points in the design space. Each approach has pros and cons, and there is a need to develop models to best understand the usability and performance issues of these schemes. In addition, it is likely that many systems will emerge that represent a hybrid of both these sensing paradigms. 20 3 Sensor-Based Behavior Recognition
- 34. 3.3 Energy-Efficient Behavior Recognition Using Ubiquitous Sensors With the popularity of mobile devices equipped with unprecedented sensing capa- bilities, context-aware applications on mobile devices are going flourishing. How- ever, the long-term sensing with the full working load of sensors is energy consuming. The battery capacity of mobile devices is a major bottleneck of context-aware applications. For example, the battery lifetime of Samsung i909 reaches up to over 30 h when all applications and sensors are turned off. But that declines to 5.5 h (50 Hz) and 8 h (20 Hz), respectively, when the single 3D accelerometer is monitored with different sampling frequencies. In the wireless networks, the use of energy harvesting techniques offers a way of supplying sensor systems without the need for batteries and maintenance [58, 59]. For ubiquitous sensors, existing solutions extend the battery life by the collaboration of multiple sensors and the reduction of sensor active time [60]. Wang et al. [61] designed a scalable framework of energy-efficient mobile sensing system for automatic user state recognition. The core component is a sensor management scheme which defines user states and defines transition rules by an XML configu- ration. The sensor management scheme allocates the minimum set of sensors and invokes new sensors when state transitions happen. Zappi et al. [62] selected the minimum set of sensors according to their contributions to classification accuracy as assessed during data training process and tested this solution by recognizing manip- ulative activities of assembly-line workers in a car production environment. Li et al. [63] applied machine learning technologies to infer the status of heavy-duty sensors for energy-efficient context sensing. They try to infer the status of high-energy- consuming sensors according to the outputs of lightweight sensors. Towards the energy efficiency of behavior recognition based on a single sensor (e.g., accelerometer) in the mobile device, it is intuitive to reduce the working time of sensors by adopting a low sampling frequency. Lower sampling frequency means less work time for the heavy-duty sensor. However, whether the low sample frequency is feasible for detecting human behaviors is still an open question. It is claimed that the sampling rate to assess daily physical activities should be no less than 20 Hz [64–66]. Kahatapitiya et al. [67] utilized the harvesting signal to estimate the step count, completely removing the requirement of accelerometer sampling. On the other hand, low sampling frequency may result in the loss of sampling data, reducing the recognition rate with low-resolution sensory data [68]. So there is a trade-off between energy consumption and recognition rate. Furthermore, many classification algorithms are heavyweight and time consuming for mobile devices. The size of sliding window in most classification algorithms is constant, which not only reduces the ability to detect short-duration movements, but also occupies lots of resources with the consumption of battery power. To overcome above issues, two factors (i.e., sampling frequency and computa- tional load) should be considered in the design of the behavior recognition algo- rithm. For example, we propose an energy-efficient method to recognize user 3.3 Energy-Efficient Behavior Recognition Using Ubiquitous Sensors 21
- 35. activities based on a single triaxial accelerometer embedded in smartphones, where a hierarchical recognition scheme is adopted to reduce the probability of time- consuming frequency-domain features for lower computational complexity and adjust the size of sliding window to enhance the recognition accuracy [69]. References 1. M. C. Mozer, “The neural network house: An environment that adapts to its inhabitants,” in Proc. AAAI Spring Symp. Intell. Environ., 1998, pp. 110–114. 2. U. Leonhardt and J. Magee, “Multi-sensor location tracking,” in Proc. 4th ACM/IEEE Int. Conf. Mobile Comput. Netw., 1998, pp. 203–214 3. A. R. Golding and N. Lesh, “Indoor navigation using a diverse set of cheap, wearable sensors,” in Proc. 3rd Int. Symp. Wearable Comput., Oct. 1999, pp. 29–36. 4. A. Ward and A. Hopper, “A new location technique for the active office,” IEEE Personal Commun., vol. 4, no. 5, pp. 42–47, Oct. 1997. 5. A. Schmidt, M. Beigl, and H. Gellersen, “There is more to context than location,” Comput. Graph., vol. 23, no. 6, pp. 893–901, 1999. 6. C. Randell and H. L. Muller, “Context awareness by analyzing accelerometer data,” in Proc. 4th Int. Symp. Wearable Comput., 2000, pp. 175–176. 7. H. W. Gellersen, A. Schmidt, and M. Beigl, “Multi-sensor context awareness in mobile devices and smart artifacts,” Mobile Netw. Appl., vol. 7, no. 5, pp. 341–351, Oct. 2002. 8. A. Schmidt and K. Van Laerhoven, “How to build smart appliances,” IEEE Pers. Commun., vol. 8, no. 4, pp. 66–71, Aug. 2001. 9. K. Van Laerhoven and K. A. Aidoo, “Teaching context to applications,” J. Pers. Ubiquitous Comput., vol. 5, no. 1, pp. 46–49, 2001. 10. K. Van Laerhoven, K. Aidoo, and S. Lowette, “Real-time analysis of data from many sensors with neural networks,” in Proc. 5th Int. Symp. Wearable Comput., 2001, pp. 115–123. 11. K. Van Laerhoven and O. Cakmakci, “What shall we teach our pants?” in Proc. 4th Int. Symp. Wearable Comput., 2000, pp. 77–84. 12. F. Foerster and J. Fahrenberg, “Motion pattern and posture: Correctly assessed by calibrated accelerometers,” Behav. Res. Methods Instrum. Comput., vol. 32, no. 3, pp. 450–457, 2000. 13. K. Van Laerhoven and H. W. Gellersen, “Spine versus Porcupine: A study in distributed wearable activity recognition,” in Proc. 8th Int. Symp. Wearable Comput., 2004, pp. 142–150. 14. S. W. Lee and K. Mase, “Activity and location recognition using wearable sensors,” IEEE Pervasive Comput., vol. 1, no. 3, pp. 24–32, Jul.–Sep. 2002. 15. L. Bao and S. Intille, “Activity recognition from user-annotated acceleration data,” in Proc. Pervasive, 2004, vol. 3001, pp. 1–17. 16. D. J. Patterson, L. Liao, D. Fox, and H. Kautz, “Inferring high-level behavior from low-level sensors,” in Proc. 5th Conf. Ubiquitous Comput., 2003, pp. 73–89. 17. M. Chan, D. Esteve, C. Escriba, and E. Campo, “A review of smart homes—present state and future challenges,” Comput. Methods Programs Biomed., vol. 91, no. 1, pp. 55–81, 2008. 18. C. D. Nugent, “Experiences in the development of a smart lab,” Int. J. Biomed. Eng. Technol., vol. 2, no. 4, pp. 319–331, 2010. 19. The ambient assisted living joint programme [Online]. Available: www.aal-europe.eu 20. The house of the future [Online]. Available: http://architecture.mit.edu/house_n 21. P. Rashidi and D. Cook, “Activity knowledge transfer in smart environments,” J. Pervasive Mobile Comput., vol. 7, no. 3, pp. 331–343, 2011. 22. The gator-tech smart house [Online]. Available: http://www.icta.ufl.edu/gt.htm 23. The inHaus project in Germany [Online]. Available: http://www.inhaus.fraunhofer.de 24. The aware home [Online]. Available: http://awarehome.imtc.gatech.edu 22 3 Sensor-Based Behavior Recognition
- 36. 25. The DOMUS laboratory [Online]. Available: http://domus.usherbrooke.ca 26. The iDorm project [Online]. Available: http://cswww.essex.ac.uk/iieg/idorm.htm 27. N. Kern, B. Schiele, H. Junker, P. Lukowicz, and G. Troster, “Wearable sensing to annotate meeting recordings,” Pers. Ubiquitous Comput., vol. 7, no. 5, pp. 263–274, Oct. 2003. 28. P. Lukowicz, J. A. Ward, H. Junker, and T. Starner, “Recognizing workshop activity using body worn microphones and accelerometers,” in Proc. Pervasive Comput., Apr. 2004, pp. 18–23. 29. J. Mantyjarvi, J. Himberg, and T. Seppanen, “Recognizing human motion with multiple acceleration sensors,” in Proc. IEEE Int. Conf. Syst., Man, Cybern., 2001, vol. 2, pp. 747–752. 30. D. Ashbrook and T. Starner, “Using GPS to learn significant locations and predict movement across multiple users,” Pers. Ubiquitous Comput., vol. 7, no. 5, pp. 275–286, Oct. 2003. 31. M. Philipose, K. P. Fishkin, M. Perkowitz, D. J. Patterson, D. Fox, H. Kautz, and D. Hahnel, “Inferring activities from interactions with objects,” IEEE Pervasive Comput., vol. 3, no. 4, pp. 50–57, Oct./Dec. 2004. 32. K. P. Fishkin, M. Philipose, and A. Rea, “Hands-on RFID: Wireless wearables for detecting use of objects,” in Proc. 9th IEEE Int. Symp. Wearable Comput., 2005, pp. 38–43. 33. D. J. Patterson, D. Fox, H. Kautz, and M. Philipose, “Fine-grained activity recognition by aggregating abstract object usage,” in Proc. 9th IEEE Int. Symp. Wearable Comput., 2005, pp. 44–51. 34. M. R. Hodges and M. E. Pollack, “An object-use fingerprint: The use of electronic sensors for human identification,” in Proc. 9th Int. Conf. Ubiquitous Comput., 2007, pp. 289–303. 35. M. Buettner, R. Prasad, M. Philipose, and D. Wetherall, “Recognizing daily activities with RFID-based sensors,” in Proc. 11th Int. Conf. Ubiquitous Comput., 2009, pp. 51–60. 36. D. Wilson and C. Atkeson, “Simultaneous tracking and activity recognition (STAR) using many anonymous, binary sensor,” in Proc. 3rd Int. Conf. Pervasive Comput., 2005, pp. 62–79. 37. C. R. Wren and E. M. Tapia, “Toward scalable activity recognition for sensor networks,” in Proc. 2nd Int. Workshop Location Context Awareness, 2006, pp. 168–185. 38. R. Aipperspach, E. Cohen, and J. Canny, “Modeling human behavior from simple sensors in the home,” in Proc. Pervasive, 2006, vol. 3968, pp. 337–348. 39. T. Gu, Z. Wu, X. Tao, H. K. Pung, and J. Lu, “epSICAR: An emerging patterns based approach to sequential, interleaved and concurrent activity recognition,” in Proc. 7th IEEE Int. Conf. Pervasive Comput. Commun., 2009, pp. 1–9. 40. L. Liao, D. J. Patterson, D. Fox, and H. Kautz, “Learning and inferring transportation routines,” Artif. Intell., vol. 171, no. 5–6, pp. 311–331, 2007. 41. A. Montanari, C. Mascolo, K. Sailer, and S. Nawaz, “Detecting Emerging Activity-Based Working Traits through Wearable Technology,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 1, no. 3, pp. 86, 2017. 42. Y. Zhang, M. Haghdan, and K. S. Xu, “Unsupervised motion artifact detection in wrist- measured electrodermal activity data,” Proceedings of the ACM on Interactive, Mobile, Wear- able and Ubiquitous Technologies, 2017. 43. J. M. Echterhoff, J. Haladjian, and B. Brügge, “Gait and jump classification in modern equestrian sports,” In Proceedings of the 2018 ACM International Symposium on Wearable Computers, pp. 88–91, 2018. 44. S. A. Elkader, M. Barlow, and E. Lakshika. “Wearable sensors for recognizing individuals undertaking daily activities.” Proceedings of the 2018 ACM International Symposium on Wearable Computers. ACM, 2018. 45. V. Becker, P. Oldrati, L. Barrios, and G. Sörös, “Touchsense: classifying finger touches and measuring their force with an electromyography armband,” In Proceedings of the 2018 ACM International Symposium on Wearable Computers, pp. 1–8, 2018. 46. P. Siirtola and J. Röning, “Recognizing human activities user independently on smartphones based on accelerometer data,” Int. J. Interact. Multimedia Artif. Intell., vol. 1, no. 5, pp. 38–45, 2012. References 23
- 37. 47. A. M. Khan, A. Tufail, A. M. Khattak, and T. H. Laine, “Activity recognition on smartphones via sensor-fusion and KDA-based SVMS,” Int. J. Distrib. Sensor Netw., vol. 10, no. 5, pp. 1–14, 2014. 48. M. Shoaib, S. Bosch, Ö. D. Incel, H. Scholten, and P. J. M. Havinga, “Fusion of smartphone motion sensors for physical activity recognition,” Sensors, vol. 14, no. 6, pp. 10146–10176, 2014. 49. H. Lu, W. Pan, N. D. Lane, T. Choudhury, and A. T. Campbell, “Soundsense: Scalable sound sensing for people-centric applications on mobile phones,” in Proc. MobiSys, 2009, pp. 165–178. 50. H. Du, Z. Yu, F. Yi, Z. Wang et al., “Recognition of group mobility level and group structure with mobile devices,” IEEE Transactions on Mobile Computing, vol. 17, no. 4, pp. 884–897, 2018. 51. T. Hao, G. Xing, and G. Zhou, “RunBuddy: A smartphone system for running rhythm monitoring,” in Proc. UBICOMP 2015, pp. 133–144. 52. X. Sun, Z. Lu, W. Hu, and G. Cao, “Symdetector: Detecting sound-related respiratory symp- toms using smartphones,” in Proc. UbiComp 2015, pp. 97–108. 53. A. Mottelson and K. Hornbæk, “An affect detection technique using mobile commodity sensors in the wild,” In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 781–792, 2016. 54. B. Cao, L. Zheng, C. Zhang, P. S. Yu, A. Piscitello, et al. “Deepmood: modeling mobile phone typing dynamics for mood detection,” In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,” pp. 747–755, 2017. 55. X. Zhang, W. Li, X. Chen, and S. Lu. MoodExplorer: Towards Compound Emotion Detection via Smartphone Sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 1, no. 4, pp. 176, 2018. 56. Lane N D, Miluzzo E, Lu H, et al. A survey of mobile phone sensing[J]. IEEE Communications Magazine, 2010, 48(9):140–150. 57. B. Guo, Z. Wang, Z. Yu, Y. Wang, N.Y. Yen, R. Huang, and X. Zhou. Mobile Crowd Sensing and Computing: The Review of an Emerging Human-Powered Sensing Paradigm. ACM Comput. Surv. 48, 1, Article 7 (August 2015), 31 pages. 58. K. Ylli, D. Hoffmann, A. Willmann, P. Becker, B. Folkmer, and Y. Manoli. Energy harvesting from human motion: exploiting swing and shock excitations. Smart Materials and Structures, vol. 24, no. 2, pp. 025–029, 2015. 59. K. Li, C. Yuen, B. Kusy, R. Jurdak, A. Ignatovic, S. Kanhere, and S. K. Jha. Fair scheduling for data collection in mobile sensor networks with energy harvesting. IEEE Transactions on Mobile Computing, 2018. 60. D. Liaqat, S. Jingoi, E. de Lara, A. Goel, W. To, et al. Sidewinder: An energy efficient and developer friendly heterogeneous architecture for continuous mobile sensing. ACM SIGARCH Computer Architecture News, vol. 44, no. 2, pp. 205–215, 2016. 61. Wang Y, Lin J, Annavaram M, Quinn JA, Jason H, Bhaskar K, Sadeh N (2009) A framework of energy efficient mobile sensing for automatic user state recognition. In: Proceedings of the 7th ACM international conference on mobile systems, applications, and services, New York, pp 179–192. 62. Zappi P, Lombriser C, Stiefmeier T, Farella E, Roggen D, Benini L, Troster G (2008) Activity recognition from on-body sensors: accuracy-power trade-off by dynamic sensor selection. Wirel Sens Netw Lect Notes Comput Sci 943: 17–33. 63. Li X, Cao H, Chen E, Tian J (2012) Learning to infer the status of heavy-duty sensors for energy efficient context-sensing. ACM Trans Intell Syst Technol 3(2):1–23. 64. Bouten C, Koekkoek K, Verduin M, Kodde R, Janssen JD (1997) A triaxial accelerometer and portable data processing unit for the assessment of daily physical activity. IEEE Trans Biomed Eng 44:136–147. 24 3 Sensor-Based Behavior Recognition
- 38. 65. Khan AM, Lee Y, Lee SY, Kim T (2010) A triaxial accelerometer-based physical-activity recognition via augmented-signal features and a hierarchical recognizer. IEEE Trans Inf Technol Biomed 14:1166–1172. 66. Kwapisz JR, Weiss GM, Moore SA (2010) Activity recognition using cell phone accelerom- eters. ACM SIGKDD Explorations Newsletter 12. 67. K. Kahatapitiya, C. Weerasinghe, J. Jayawardhana, H. Kuruppu, K. Thilakarathna, and D. Días, Low-power step counting paired with electromagnetic energy harvesting for wearables. In Proceedings of the 2018 ACM International Symposium on Wearable Computers. pp. 218–219, 2018. 68. Maurer U, Smailagic A, Siewiorek DP, Deisher M (2006) Activity recognition and monitoring using multiple sensors on different body positions. In: Proceedings of international workshop on wearable and implantable body sensor networks, Cambridge, pp 113–116. 69. Liang Y, Zhou X, Yu Z, et al. Energy-Efficient Motion Related Activity Recognition on Mobile Devices for Pervasive Healthcare. Mobile Networks & Applications, 2014, 19(3):303–317. References 25
- 39. Chapter 4 Device-Free Behavior Recognition Abstract Traditional methods to sense and recognize human behavior include using wearable devices, cameras, and devices embedded in the environment. Recently, a new kind of behavior sensing approach, device-free behavior sensing, attracts a great amount of interests as it holds the promise to provide a ubiquitous sensing solution by using the pervasive signal (including RF signal, acoustic signal, optical signal, etc). In this chapter, we first introduce the basic concept of device-free behavior sensing and understanding, and then present two typical device-free behav- ior sensing approaches, i.e., Wi-Fi based and acoustic based. 4.1 The Basic Concept of Device-Free Behavior Sensing and Recognition The basic idea of device-free sensing can be summarized as follows: human behav- ior in the range where wireless signal that passes will incur signal reflection, scattering, and diffraction, and finally results in changes of the received signal. By exploiting these changes, we can infer specific human behaviors. Generally speaking, there are mainly three different kind of wireless signals that have been applied to sense human behavior. The first kind of wireless signal is the RF signal, and Wi-Fi sensing is the most attractive RF-based wireless sensing technology. It provides a ubiquitous sensing solution by only using existing Wi-Fi infrastructure. Another kind of wireless signal is the acoustic signal, which can be transmitted and received by both commodity audio device and customized audio devices. Compared with the RF signal, the wavelength of acoustic signal is relatively shorter, making it capable of providing more accurate sensing results. The third kind of wireless signal is the optical signal (e.g., visible light). Generally, transmitting and receiving the optical signal require specific and relatively expensive devices. In addition, the optical signal is usually affected by the ambient lighting conditions. © Springer Nature Singapore Pte Ltd. 2020 Z. Yu, Z. Wang, Human Behavior Analysis: Sensing and Understanding, https://doi.org/10.1007/978-981-15-2109-6_4 27
- 40. 4.1.1 General Methodology There are two basic methods for wireless human behavior sensing: velocity estima- tion and distance estimation. Velocity estimation: Velocity is a key characteristic of human behavior. Different behavior shows different velocity change. Thus, we can sense and recognize human behavior by exploiting the velocity variation over time. Generally, we can estimate the velocity of moving objects using Doppler effect. Specifically, radial velocity estimation requires at least one wireless signal transceiver, 2-dimension velocity estimation requires at least two wireless signal transceivers, and 3-dimension veloc- ity estimation requires at least three wireless signal transceivers. The transceivers have to be kept steady when estimating absolute velocity. Distance estimation: Real-time distance measurement is another way to sense human behavior. The variation of distances from object to transceivers indicates the trace of object movement. Traditional distance estimation methods include pulse ranging, FMCW ranging, and phase ranging. 4.1.2 Typical Applications Wireless sensing is capable of enabling different kinds of human behavior sensing and recognition applications, including health monitoring, indoor localization, secu- rity monitoring, etc. • Health monitoring. Wireless sensing technology can be applied to monitor human health status. For example, wireless sensing technology can be used for contactless human respiration and heartbeat monitoring during sleeping. In addi- tion, it can also be applied to detect falling and other abnormal behaviors without interfering daily activity. • Indoor localization. Indoor localization is the basis of multiple interesting appli- cations, such as indoor navigation and advertising push. Wireless sensing tech- nique can achieve high-accuracy indoor localization without any wearable devices, and the state-of-the-art positioning accuracy is decimeter scale. • Security monitoring. Existing security monitoring systems are implemented using traditional camera and infrared devices. These light-based systems have two major drawbacks: (1) we have to deploy many devices to cover the target area and (2) it may raise privacy concerns. Wireless sensing technique is capable of fixing the above two problems. Therefore, the security system based on wireless sensing technique is promising to be a satisfactory solution in specific environments. 28 4 Device-Free Behavior Recognition
- 41. 4.2 Wi-Fi CSI-Based Behavior Sensing and Recognition1 Human behavior understanding plays an important role in human-computer interac- tion and public security. Researchers have developed many methods using cameras, radars, or wearable sensors [1]. However, they also suffer from certain shortcomings (e.g., user privacy, sensing coverage range). For example, the vision-based approaches only work with line-of-sight (LoS) coverage and rich-lighting environ- ments, which also causes privacy concerns. The low-cost 60 GHz radar solutions can only offer an operation range of tens of centimeters, and such radar devices are not widely deployed in our daily living environments. Wearable sensor-based approaches require people to wear some extra devices. With the recent advances in wireless communications, behavior recognition based on Wi-Fi has been attracting more and more attention due to its ubiquitous availability in indoor areas. Moreover, Wi-Fi-based behavior recognition approach is able to overcome the aforementioned shortcomings of traditional approaches, as it only leverages the wireless communication feature and does not need any physical sensor. A typical Wi-Fi-based behavior recognition system consists of a Wi-Fi access point (AP) and one or several Wi-Fi-enabled devices in the environment. When located in indoor environment with such a system, the movement of human bodies will affect the wireless signals and change the multipath profile of the system. Based on this principle, we are able to recognize human behaviors by exploring the changes of wireless signals caused by user movements. Recently, numerous studies have devoted to pervasive sensing using Wi-Fi devices, such as indoor localization [2] and gesture recognition [3]. These studies are mainly based on received signal strength indication (RSSI). However, the RSSI can fluctuate dramatically even at a stationary link, which makes these detection results unreliable. Quite recently, channel state information (CSI), i.e., the fine- grained information regarding Wi-Fi communication, becomes available [4]. Specif- ically, CSI describes how the signal propagates from the transmitter to the receiver and reflects the combined effects of the surrounding objects (e.g., scattering, fading, and power decay with distance). Meanwhile, there are a set of subcarriers in CSI, each of which contains the information of attenuation and phase shift. Therefore, CSI contains rich information and is more sensitive to environmental variances caused by moving objects [5]. CSI is a metric which estimates the channel by representing the channel proper- ties of a wireless communication link. In the frequency domain, the wireless channel can be described as Y ¼ H X + N, where X and Y correspond to the transmitted and 1 Part of this section is based on a previous work: Z. Wang, B. Guo, Z. Yu and X. Zhou, “Wi-Fi CSI-Based Behavior Recognition: From Signals and Actions to Activities,” in IEEE Communica- tions Magazine, vol. 56, no. 5, pp. 109-115, May 2018. DOI: https://doi.org/10.1109/MCOM.2018. 1700144 4.2 Wi-Fi CSI-Based Behavior Sensing and Recognition 29
- 42. received signal vectors, H is the channel matrix presented in the format of CSI, and N is the additive white Gaussian noise vector. In the IEEE 802.11n standard, CSI is measured and reported at the scale of orthogonal frequency division modulation (OFDM) subcarriers, where each CSIi ¼ j CSIi j exp {j(∠CSIi)} depicts the amplitude response (i.e., jCSIij) and phase response (i.e., ∠CSIi) of one subcarrier. Specifically, each entry in matrix H corresponds to the channel frequency response (CFR) value between a pair of antennas at a certain OFDM subcarrier frequency at a particular time, and the time series of CFR values for a given pair of antennas and OFDM subcarrier is called a CSI stream. In other words, while CFR describes the combined effects of fading, scattering, and attenuation of a specific subcarrier, CSI is the union of these CFRs. Specifically, 802.11n specifications have provisions for reporting quantized CSI field per packet using various subcarrier grouping options as per clause. However, different manufacturers may choose to implement a subset of the subcarrier grouping options. For example, the Intel 5300 wireless network interface card (NIC) imple- ments an OFDM system with 56 subcarriers of a 20 MHz channel or 114 subcarriers of a 40 MHz channel, 30 out of which can be read for CSI information via the device driver. Thereby, a time series of CSI values includes 30 NumTx NumRx CSI streams, where NumTx and NumRx stand for the amount of transmitting and receiv- ing antennas, respectively. Given an indoor environment with two wireless nodes, as shown in Fig. 4.1, the wireless signal will propagate in a multipath manner, and the wireless channel will be relatively stable as long as there are no people or no motion. However, once a person moves, the scattered signals will change (the red line in Fig. 4.1), which causes channel disturbances, involving both amplitude attenuation and phase dis- tortion. In other words, different multipath effects can be obtained if a person is moving, which results in different CSI streams at the receiver, and can be used to recognize different behaviors by correlating them with the corresponding channel distortion patterns. Fig. 4.1 Wi-Fi signal propagation in indoor environments [6] 30 4 Device-Free Behavior Recognition
- 43. Wi-Fi CSI-enabled behavior recognition approaches can be categorized into two groups, i.e., pattern based [7–17] and model based [18–21]. The pattern-based approaches aim to classify behaviors by exploring different features of CSI mea- surements, while the model-based approaches implement recognition by modeling the relationship between signal space and behavior space. A general architecture of Wi-Fi CSI-based behavior recognition approaches is shown in Fig. 4.2. Though the middle part (i.e., CSI data collection and preprocessing) is common to both pattern- based and model-based approaches, the left and right parts illustrate the key differ- ence of these two approaches. Most existing CSI-based behavior recognition studies adopt the pattern-based approach. The intuition is that different behaviors have distinct impacts on the received CSI streams, which can be leveraged to mine patterns or construct profiles for predefined behaviors, as shown in the left part of Fig. 4.2. Afterwards, each behavior can be classified as one of the predefined types based on profile matching or pattern recognition. The key benefit of the pattern-based recognition approaches is that they do not require intensive deployments and can work with even a single AP, which ensures the low hardware cost and maintenance and has no obstruction to human’s normal life. However, pattern-based approaches usually require a learning process to construct profiles or classifiers, which restricts them to identify only a limited set of predefined behaviors. Model-based approaches are based on the characterization of mathematical relationships between human behaviors and received signals. In the case of Wi-Fi CSI-based behavior recognition, the aim of modeling is to relate the signal space to the physical space including human and environment, and characterize the physical law through mathematical relationship between the received CSI signals and the sensing target, as shown in the right part of Fig. 4.2. Since the model-based Fig. 4.2 A general architecture of Wi-Fi CSI-based behavior recognition approaches [6] 4.2 Wi-Fi CSI-Based Behavior Sensing and Recognition 31
- 44. approaches do not need predefined behavior profiles, they can track an arbitrary set of human behaviors, which enables wider ranges of real-time applications. Cur- rently, there have been several model-based behavior recognition works, such as the angle-of-arrival (AoA) model [18], the CSI-speed model [19], and the Fresnel zone model [20, 21]. 4.3 Acoustic-Based Behavior Sensing and Recognition Acoustic-based sensing is a method for estimating the state of objects by transmitting acoustic signals and analyzing the response. A typical acoustic-based sensing system consists of a pair of speaker and microphone, which forms an audio transceiver system. The speaker is usually programmed to transmit acoustic signals continu- ously, while the microphone is applied to receive the echo with a certain sampling rate (e.g., 48 kHz) and send the received signal to computers for data processing and behavior recognition. Most of the acoustic-based device-free behavior sensing and recognition systems adopt similar ideas from RF-based (e.g., Wi-Fi) approaches, either exploring the Doppler effect when human approaching or away from the microphone, or decoding the echo of frequency-modulated continuous wave (FMCW) of the acoustic signal to measure the human body, or utilizing the OFDM to achieve real-time behavior tracking, as shown in Fig. 4.3. The first line of acoustic-based human behavior sensing and recognition systems is on the basis of the Doppler effect of the signal reflected by users [23–26]. Such systems do not have tracking capability and can only recognize predefined behav- iors, as Doppler shift can only provide coarse-grained measurements of the speed or direction of user movements due to the limited frequency measurement precision. The second line of acoustic-based systems is on the basis of decoding the echo of FMCW sound wave to measure the human body. Specifically, FMCW [27] indi- rectly estimates the propagation delay based on the frequency shift of the chirp signal, based on which we can measure the movement displacement of the target user, and a number of behaviors can be recognized [28]. However, the distance estimation resolution of FMCW is restricted by the sweep bandwidth; it is difficult to achieve very high distance estimation resolution (e.g., detecting human respiration with chest movement displacement of less than 1 cm) with narrowband commodity Fig. 4.3 A general framework of acoustic-based behavior recognition approaches [22] 32 4 Device-Free Behavior Recognition
- 45. acoustic device. To tackle this issue, a correlation-based FMCW (C-FMCW) approach [22] is proposed to achieve more accurate distance estimation. Unlike traditional FMCW, C-FMCW estimates the round-trip propagation time of acoustic signals by discovering the maximum correlation between transmitted signal and received signal. For digital FMCW signal, the round-trip propagation time can be measured by detecting the number of samples corresponding to the maximum correlation. As such, the distance estimation resolution of C-FMCW is limited only by the acoustic signal sampling rate. With current prevailing audio systems of 48 kHz, C-FMCW can achieve a ranging resolution of around 0.4 cm. The ranging resolution can be further improved if higher sampling rates (e.g., 96 kHz) are supported by the acoustic systems. The third line of research uses OFDM pulses or CW signals to detect phase changes and facilitate real-time behavior tracking [29, 30]. For instance, by leverag- ing the OFDM pulses, the FingerIO system [29] achieves a finger location tracking accuracy of 8 mm and also allows 2D drawing in the air using COTS mobile devices. Similarly, the LLAP system [30] uses CW signals, which is less noisy due to the narrower bandwidth compared with OFDM pulses, allowing it to achieve better tracking accuracy. In general, compared with other kinds of wireless signals (e.g., RF signal, optical signal), sensing human behavior using acoustic signals has the following advan- tages. (1) Short wavelength means more accurate behavior sensing. Due to relatively slow propagation velocity, even with 20 kHz transmitting frequency, the wavelength of transmitting acoustic signal is only about 1.7 cm. (2) It is easy to control the process of acoustic transmitting and receiving. For example, using commodity acoustic devices, we can directly control acoustic transmitting and receiving process by calling system audio interface. (3) Compared with the raw data (e.g., RSSI or CSI) of RF-based sensing systems, the data that we can obtain from speaker and microphone is the original transmitting and receiving signal, which contains more abundant information. The disadvantages of using acoustic signal to sense and recognize human behav- ior can be summarized as follows. (1) Acoustic signal is one kind of mechanical wave, which attenuates severely when propagating in the air. This characteristic limits its sensing range. (2) Compared with RF signals, the acoustic signal has poor penetration ability. From the above discussion, we know that acoustic is suitable for sensing fine- grained behavior in short range, such as human vital signal sensing, gesture recog- nition, and hand tracking. References 1. Nickel, C., Busch, C., Rangarajan, S., Möbius, M. (2011). Using Hidden Markov Models for accelerometer-based biometric gait recognition. IEEE, International Colloquium on Signal Processing and ITS Applications, pp 58–63. References 33
- 46. 2. Zanca, G., Zorzi, F., Zanella, A., Zorzi, M. (2008). Experimental comparison of RSSI-based localization algorithms for indoor wireless sensor networks. The Workshop on Real-World Wireless Sensor Networks, pp 1–5. 3. Huang, Y. F., Yao, T. Y., Yang, H. J. (2015). Performance of Hand Gesture Recognition Based on Received Signal Strength with Weighting Signaling in Wireless Communications. International Conference on Network-Based Information Systems, pp 596–600. 4. Wang, G., Zou, Y., Zhou, Z., Wu, K., Ni, L. M. (2014). We can hear you with Wi-Fi!. International Conference on Mobile Computing and NETWORKING, pp 593–604. 5. Xi, W., Zhao, J., Li, X. Y., Zhao, K. (2014). Electronic frog eye: Counting crowd using WiFi. INFOCOM, 2014 Proceedings IEEE, pp 361–369. 6. Z. Wang, B. Guo, Z. Yu and X. Zhou, Wi-Fi CSI-Based Behavior Recognition: From Signals and Actions to Activities, in IEEE Communications Magazine, vol. 56, no. 5, pp. 109-115, May 2018. DOI: https://doi.org/10.1109/MCOM.2018.1700144 7. X. Liu, J. Cao, S. Tang, J. Wen and P. Guo. Contactless Respiration Monitoring via Off-the- shelf WiFi Devices. IEEE Transactions on Mobile Computing, 15(10): 2466-2479, 2016. 8. J. Liu, Y. Wang, Y. Chen, J. Yang, X. Chen, and J. Cheng. Tracking Vital Signs during Sleep Leveraging Off-the-shelf WiFi. ACM MobiHoc 2015, 267–276. 9. H. Li, W. Yang, J. Wang, Y. Xu, L. Huang. WiFinger: Talk to Your Smart Devices with Finger- grained Gesture. ACM UbiComp 2016, 250–261. 10. G. Wang, Y. Zou, Z. Zhou, K. Wu, and L. M. Ni. We Can Hear You with Wi-Fi. ACM MobiCom 2014, 593–604. 11. K. Ali, X. Liu, W. Wang, and M. Shahzad. Keystroke Recognition Using WiFi Signals. ACM MobiCom 2015, 90–102. 12. Y. Wang, J. Liu, Y. Chen, M. Gruteser, J. Yang, and H. Liu. E-eyes: device-free location- oriented activity identification using fine-grained Wi-Fi signatures: ACM MobiCom 2014, 617–628. 13. W. Wang, X. Liu, and M. Shahzad. Gait Recognition Using WiFi Signals. ACM UbiComp 2016, 363–373. 14. Y. Zeng, P. Patha, and P. Mohapatra. WiWho: WiFi-based Person Identification in Smart Spaces. IEEE IPSN 2016, 1–12. 15. H. Wang, D. Zhang, Y. Wang, and J. Ma. RT-Fall: A Real-time and Contactless Fall Detection System with Commodity WiFi Devices. IEEE Transactions on Mobile Computing, 16(2): 511-526, 2017. 16. J. Zhang, B. Wei, W. Hu, and S. Kanhere. WiFi-ID: Human Identification using WiFi signal. IEEE DCOSS 2016, 75–82. 17. T. Xin, B. Guo, Z. Wang, M. Li, Z. Yu, and X. Zhou. FreeSense: Indoor Human Identification with Wi-Fi Signals. IEEE GlobeCom 2016, pp. 1–6. 18. L. Sun, S. Sen, D.S Koutsonikolas, and K. Kim. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. ACM MobiCom 2015, 77–89. 19. W. Wang, X. Liu, M. Shahzad, et al. Understanding and modeling of Wi-Fi signal based human activity recognition. ACM MobiCom 2015, 65–76. 20. H. Wang, D. Zhang, J. Ma, Y. Wang, Y. Wang, D. Wu, T. Gu, and B. Xie. Human Respiration Detection with Commodity WiFi Devices: Do User Location and Body Orientation Matter. ACM UbiComp 2016, 363–373. 21. D. Zhang, H. Wang, and D. Wu. Toward Centimeter-Scale Human Activity Sensing with Wi-Fi Signals. IEEE Computer, 50(1): 48-57, January, 2017. 22. Tianben Wang, Daqing Zhang, Yuanqing Zheng, Tao Gu, Xingshe Zhou, Bernadette Dorizzi. C-FMCW Based Contactless Respiration Detection Using Acoustic Signal. ACM UbiComp 2018. 23. Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. Sound wave: using the Doppler effect to sense gestures. In Proc. ACM CHI, 2012. 24. Md Tanvir Islam Aumi, Sidhant Gupta, Mayank Goel, Eric Larson, and Shwetak Patel. Doplink: Using the Doppler effect for multi-device interaction. In Proc. ACM UbiComp, 2013. 34 4 Device-Free Behavior Recognition
- 47. 25. Ke-Yu Chen, Daniel Ashbrook, Mayank Goel, Sung-Hyuck Lee, and Shwetak Patel. Airlink: sharing files between multiple devices using in-air gestures. In Proc. ACM UbiComp, 2014. 26. Tianben Wang, Daqing Zhang, Leye Wang, Xin Qi, Bernadette Dorizzi, Xingshe Zhou. Contactless Respiration Monitoring using Acoustic Signal with Off-the-shelf Audio Devices. IEEE Internet of Things Journal, DOI:https://doi.org/10.1109/JIOT.2018.2877607. 27. A. G. Stove. (1992). Linear FMCW radar techniques. Radar Signal Processing IEEE Pro- ceedings F , 139, 5:343-350. 28. Rajalakshmi Nandakumar, Shyamnath Gollakota, Nathaniel Watson. Contactless Sleep Apnea Detection on Smartphones. The 13th ACM Annual International Conference on Mobile Sys- tems, Applications, and Services, 2015, 45–57. 29. Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. FingerIO: Using active sonar for fine-grained finger tracking. In Proc. ACM CHI, 2016. 30. Wei Wang, Alex X. Liu, and Ke Sun. Device-Free Gesture Tracking Using Acoustic Signals. In Proc. ACM MobiCom, 2016. References 35
- 48. Chapter 5 Individual Behavior Recognition Abstract In this chapter, we present some of our recent research advances on individual behavior sensing and recognition. Specifically, in Sections 5.1 and 5.2, we present two human mobility-related works (i.e., mobility prediction and disori- entation detection) by leveraging GPS trajectories. Afterwards, we discuss how to recognize human behaviors by using smartphones in Sections 5.3 and 5.4 (i.e., human-computer operation recognition and human localization), followed by two device-free sensing-based behavior analysis practices in Sections 5.5 and 5.6 (i.e., human identity recognition and respiration detection). 5.1 Human Mobility Prediction by Exploring History Trajectories1 5.1.1 Introduction With the rising popularity of mobile sensors and portable devices, “serendipitous” social interaction is emerging and becoming pervasive, where interaction is triggered when two devices (i.e., two users) are located closely in a mobile peer-to-peer environment [1], or wireless ATM networks [2]. Studies have been conducted to support unexpected interaction utilizing users’ mobile wireless devices (e.g., mobile phone, vehicle PDA, and wearable devices). In these studies, human interaction is facilitated by means of peer-to-peer data sharing between matching devices [3] (i.e., users), and the “matching” is defined by similarity computed from user profiles (e.g., music, pictures, and tagged preferences) on the phone. However, these mechanisms merely provide services for people who have common interests and thus cannot enhance human social relations in the physical world. 1 Part of this section is based on a previous work: Z. Yu, H. Wang, B. Guo, T. Gu and T. Mei, “Supporting Serendipitous Social Interaction Using Human Mobility Prediction,” in IEEE Trans- actions on Human-Machine Systems, vol. 45, no. 6, pp. 811–818, Dec. 2015. DOI: https://doi.org/ 10.1109/THMS.2015.2451515 © Springer Nature Singapore Pte Ltd. 2020 Z. Yu, Z. Wang, Human Behavior Analysis: Sensing and Understanding, https://doi.org/10.1007/978-981-15-2109-6_5 37
- 49. The term “serendipity” was coined by the novelist Horace Walpole in the eighteenth century to describe unexpected and fortunate discoveries; “serendipity” was originally used to refer to making accidental discoveries when looking for one thing and finding another [4]. We, therefore, characterize the serendipitous social interaction as unplanned, not affecting users’ schedules, bringing convenience, and creating positive emotion (e.g., happiness) in serving as an activity user, and users can choose to participate or not. For example, take two college students who are friends and who have not met for weeks. Both of them often visit the same library at the same time, but they do not recognize that. In such a scenario, if a mobile application helps them to capture and learn the serendipitous opportunities to meet and have a chat, the friendship between these two students will be enhanced. To support such social interactions, we need first to discover the serendipitous interaction opportunities [5], similar to the communication channel of mobile inter- mediate nodes in an opportunistic network [6]. With the captured unplanned and transitory opportunities, serendipitous interaction can be used to make our lives easier. For example, if a user is aware that his/her roommate is passing by a grocery, he/she may ask him/her to buy something for him/her. However, this kind of serendipitous opportunity will disappear in a matter of seconds as he/she walks away. In this scenario, prediction, rather than instant behavior detection or notifica- tion, would be better. Mobility prediction typically leverages human trajectory data (e.g., GPS data, check-in records, and intercell [7]). The majority of mobility prediction methods focus on predicting a user’s future mobility status (e.g., where and when a user arrives at the next venue, and duration). In this chapter, we predict users’ future temporal and spatial contexts such as venue, arrival time, and user encounter. As user mobility status (e.g., location and time) may change in a very short period of time in the physical world, coarse-grained (e.g., hour-level temporal prediction error) and low-accuracy (e.g., accuracy of next venue judgment) mobility prediction cannot satisfy the needs of practical applications. Aiming to achieve high accuracy and low error, we discover the strong spatial and temporal regularities in GPS trajectories and use supervised learning algorithms to train historical mobility instances (generated by crowd users). Few applications have been proposed, especially any combining spatial and temporal prediction information. In this chapter, we aim to facilitate serendipitous social interactions and the real participation activities in the physical world, which is different from simple information sharing mechanisms in existing traditional appli- cations. Leveraging the overlap and regularity in collected GPS trajectories, we deploy supervised learning algorithms and achieve an accuracy of over 90% for predicting a user’s next venue, and minute-level (i.e., an average of about 5 min) prediction error for arrival time. By leveraging mobility prediction to forecast the occurrence of serendipitous interaction opportunities, we propose a three-layer framework to support serendip- itous social interaction, develop two applications for use on a university campus, and conduct a survey about the application. The main contributions of this chapter are summarized as follows. 38 5 Individual Behavior Recognition
- 50. • We propose a system framework for supporting social interaction by means of facilitating users to participate in interaction activities in the physical world. Under this framework, mobility prediction is introduced to capture serendipitous interaction opportunities. • We leverage mobility prediction to discover serendipitous social interaction opportunities. Based on the spatiotemporal mobility regularities in users’ trajec- tories, we first predict where and when a user will arrive, and then, we can determine if some users may soon encounter (i.e., have the same destination). • We develop the prototype of the proposed framework and build two applications based on the framework, HelpBuy and EaTogether, to support serendipitous social interaction on campus. 5.1.2 Related Work Previous studies attempted to facilitate online social interactions in social networks. Under the premise that users who have the same interests are more willing to interact with each other (i.e., user homogeneity), studies try to help users find interaction with friends through personal preference similarity calculation in social networks, in which social interactions are facilitated in the form of learning [8], date [1, 9], and travel [10]. These kinds of applications only enhance social interaction within an online social network, rather than establish social connectivity in the physical world. On the other hand, researchers focus on supporting face-to-face human interac- tion, in which social interactions are facilitated by leveraging serendipitous commu- nication opportunities between mobile devices (i.e., users). BlueFriend [11] is an application that leverages Bluetooth to find friends among nearby users. Bluedating [9] provides localized dating services to help users find desired partners. Paradiso et al. [12] developed a badge system, which is equipped with wireless infrared and radio frequency networking, to facilitate social interaction between wearers. Law- rence et al. [3] developed three applications by exploiting the “co-presence” interac- tions (i.e., incidental interactions) between mobile devices. In the mobile peer-to-peer environment, as Yang et al. [10] proposed, information sharing and social interaction are facilitated by capturing serendipitous interaction opportunities. However, inter- action opportunities are always discovered by device detection. Devices are the actual participant in these applications, rather than the humans themselves. In this chapter, we predict serendipitous interaction opportunities leveraging mobility prediction using users’ current mobility status and present mobile applications that facilitate people actively participating and interacting in the physical world. Mobility predic- tion is the main supporting technology of our system. It aims to discover serendip- itous interaction opportunities. Mobility prediction has been studied to perceive human future mobility status (e.g., next venue, arrival time, and duration) from different perspectives. Xiong et al. [13] took advantage of the similarity between people’s trajectories and proposed collective behavioral patterns for improving prediction accuracy. Do and Gatica- 5.1 Human Mobility Prediction by Exploring History Trajectories 39
- 51. Perez [14] adopted a factorized conditional model according to the extracted fea- tures, which can reduce the size of the parameter space as compared to conditional model. Cho et al. [15] discussed the contribution of a location-based social network and an individual user’s periodic movement pattern in mobility prediction and developed a model that combines the periodic day-to-day movement patterns with the social movement effects coming from the friendship network. Noulas et al. [16] used check-in data in Foursquare and proposed a set of mobility prediction features to capture the factor that drives users’ movement, and achieved around 90% prediction accuracy. Baumann et al. [17] analyzed the influence of temporal and spatial features in mobility prediction and predicted transitions. McInerney et al. [18] presented a Bayesian model of population mobility to tackle the data sparsity problem in mobility prediction. Song et al. [19] utilized the Order-2 Markov predictor with fallback and obtained a median accuracy of about 72% for users with long trace lengths. Song et al. [19] provided evidence that the prediction accuracy of an individual’s next location had an upper bound of 93%. Lin et al. [20] reported on a study of GPS data-based mobility prediction accuracy and suggested that a predictability upper bound of 90% is able to support ubiquitous applications. Considering the sparsity of trajectory data, such as Call Detail Records [21] and taxi GPS data, researchers also made much progress. Wang et al. [22] proposed a mobility gradient decent approach and predicted user’s destination, by using sparse taxi trajectories. Recently, some works explore mobility prediction using deep learning [23, 24]. By discovering region of interest (ROI) in the city, Jiang et al. [24] proposed a deep ROI-based modeling approach for effectively predicting human mobility. In this chapter, we predict not only an individual user’s next venue, but also the arrival time and multiple users’ encounters based on users’ current mobility features (CMFs). We achieved a high prediction accuracy of around 90%, which enables discovering opportunities to support serendipitous live interactions. 5.1.3 Serendipitous Social Interactions Supporting System 5.1.3.1 Framework The framework consists of three layers, as shown in Fig. 5.1. Data Layer: In the data layer, a GPS dataset of trajectories is collected from users by the GPS data logger. Each GPS point has several spatial and temporal attributes, such as date, time, latitude, longitude, and speed, and trajectories with thousands of GPS points are recorded. These mobility data reflect the regularity of users’ daily visits (i.e., the GPS point sequences in some venues) and the movement (i.e., the GPS point sequences between two venues). After data denoising, historical mobility instances are extracted from the GPS dataset for mobility prediction. Mobility Prediction Layer: Repeatable behaviors and patterns exist in the GPS trajectories. For example, at a campus crossroad during the morning, if a student is 40 5 Individual Behavior Recognition
- 52. walking westward, then we predict that he/she will go to the gym; if he/she is walking towards the east, then we predict that he/she will arrive at the library. Moreover, people living in the same environment (e.g., students in a dormitory) may have the same regularity, and thus, users’ mobility trajectories can be utilized as historical instances to predict the next venue and arrival time. Human GPS trajectory contains many temporal (i.e., morning) and spatial (i.e., at a crossroad) context features. It is important to select the appropriate features to achieve high prediction accuracy and low complexity. As trajectories have regular- ity, we apply learning algorithms to discover patterns. Then, after perceiving indi- vidual users’ CMFs (e.g., time, latitude, and longitude), the inference models can predict users’ future mobility features (FMFs) (e.g., next venue and arrival time). At the individual level, we can predict a user’s future status. Then, at the crowd level, we can predict multiple users’ occasional encounters at a next venue (e.g., two users will meet at a restaurant in 10 min), which is similar to predicting data source location in [25]. Application Layer: In this layer, we design and develop applications to facilitate serendipitous interactions based on the prediction results of user’s FMFs from the mobility prediction layer. For example, we can develop more efficient participatory sensing [26] applications with perceiving users’ future positions. Furthermore, we can discover unexpected opportunities with user encounter prediction results. In this chapter, we implement two applications that facilitate social interactions in specific scenarios on a campus. Fig. 5.1 System framework 5.1 Human Mobility Prediction by Exploring History Trajectories 41
- 53. 5.1.3.2 Mobility Prediction We observe repeatable behavior and patterns in the GPS trajectories. For instance, a person is going to the same next venue when he/she is at the same position at the same time on different days (e.g., at noon, students at the same place are likely to go to restaurant; in the 5 min before class, students on different paths are all moving towards the classrooms). These real phenomena imply that students’ trajectories contain strong spatial and temporal regularities. We explore these regularities for mobility prediction and apply supervised learning algorithms to train the inference model. In this chapter, we predict users’ FMFs at two granularities: individual and crowd. At the individual level, we predict where a user is going to (i.e., next venue) and when the user will arrive there (i.e., arrival time). At the crowd level, we predict the intersection of multiple users’ trajectories (i.e., user encounter) based on the indi- vidual user mobility prediction. Figure 5.2 shows the detailed mobility prediction procedure. First, historical mobility instances are extracted from the historical GPS dataset to form the training dataset. Second, since supporting serendipitous social interaction requires high prediction accuracy both spatially and temporally, we adopt learning algorithms to train using the historical dataset. The prediction task contains discrete (next venue) and continuous (arrival time) output; therefore, we use different learning algorithms to train inference models to predict each FMF. According to ref. 17, more CMFs are not necessary for higher prediction accuracy. We find that we need to select the suitable CMF set to achieve high accuracy and low complexity. Experiments are conducted by selecting different CMF sets from Table 5.1 and comparing their performance. Finally, after predicting results of the user’s FMFs (i.e., next venue and arrival time) with the inference models, we can predict user encounter at the next venue. Taking every GPS point in the trajectories as a mobility instance, we extract fine- grained attributes (e.g., latitude, longitude, time, direction, and speed of a GPS point) as the mobility instance’s features. Leveraging the overlap and regularity in these trajectories, the mobility instances’ features are highly cohesive and exhibit low coupling. Essentially, each mobility point is a GPS point recorded on the path between two venues with a timestamp and GPS coordinate. However, in this chapter, a mobility point in fact has eight features (see Table 5.2). The former five features are all CMFs, which are originally recorded by GPS sensors, and the latter three features are all FMFs, @NV, @AT, and @UE, which are acquired. Specifically, the historical mobility points’ FMF values are assigned using feature extraction steps to form training instances, while the mobile phone users’ FMF values are to be predicted through inference models in practical applications. The CMFs are mutually independent. None of the mobility features are related to personal information as limited by the data scale of individual users. Inspired by the collective behavior pattern [13] and the population modeling mechanism [25], we are able to achieve high prediction accuracy to meet the demands of application scenarios. 42 5 Individual Behavior Recognition
- 54. Predicting Next Venue: People’s daily life generates almost the same life track every day, and their GPS trajectories exhibit high spatial and temporal overlapping and regularities. Specifically, a user’s next venue is predominantly determined by his/her current spatial and temporal context (e.g., time and location), and people are likely to make the same decision when they are in the same situation (e.g., same Fig. 5.2 Mobility prediction procedure Table 5.1 Mobility features Feature Description Example @WE Weekday or weekend Weekday/weekend @Time Current time 18:06:08 @LatLng Latitude and longitude 34.0356, 108.7600 @Di (direction) N(north)/S(south), E(east)/W(west) N, E @Sp (speed) Discretized into 4 levels, separated by 2, 5 and 10 (km/h) Stroll(0–2)/walk(2–5)/scurry (5–10)/trot(10) @NV Next venue Library @AT Arrival time of next venue 18:14:34 @UE Who, where, and when will meet Tom, restaurant, 11:56:32 5.1 Human Mobility Prediction by Exploring History Trajectories 43
- 55. Random documents with unrelated content Scribd suggests to you:
- 56. JUSTICE Depuis onze mois bientôt, j'ai quitté la France. Pendant onze mois, je me suis imposé l'exil le plus total, la retraite la plus ignorée, le silence le plus absolu. J'étais comme le mort volontaire, couché au secret tombeau, dans l'attente de la vérité et de la justice. Et, aujourd'hui, la vérité ayant vaincu, la justice régnant enfin, je renais, je rentre et reprends ma place sur la terre française. Le 18 juillet 1898 restera, dans ma vie, la date affreuse, celle où j'ai saigné tout mon sang. C'est le 18 juillet que, cédant à des nécessités de tactique, écoutant les frères d'armes qui menaient avec moi la même bataille, pour l'honneur de la France, j'ai dû m'arracher à tout ce que j'aimais, à toutes mes habitudes de cœur et d'esprit. Et, depuis tant de jours qu'on me menace et qu'on m'abreuve d'injures, ce brusque départ a été sûrement le plus cruel sacrifice qu'on eût exigé de moi, ma suprême immolation à la cause. Les âmes basses et sottes, qui se sont imaginé, qui ont répété que je fuyais la prison, ont fait preuve d'autant de vilenie que d'inintelligence. La prison, grand Dieu! mais je n'ai jamais demandé que la prison! mais je suis prêt encore à m'y rendre, s'il est nécessaire! Il faut, pour m'accuser de la fuir, avoir oublié toute cette histoire, et le procès que j'ai voulu, dans l'unique désir qu'il fût le champ où pousserait la moisson de vérité, et le complet sacrifice que j'avais fait de mon repos, de ma liberté, m'offrant en holocauste, acceptant à l'avance ma ruine, si la justice triomphait. N'est-il pas d'une évidence éclatante, aujourd'hui, que notre longue campagne, à mes conseils, à mes amis et à moi, n'a été qu'une lutte désintéressée pour faire jaillir des faits le plus de lumière possible? Si nous avons voulu gagner du temps, si nous avons opposé procédure à procédure, c'est que nous avions charge de vérité, comme on a charge d'âme, c'est que nous ne voulions pas laisser éteindre entre
- 57. nos mains la faible lueur, qui chaque jour grandissait. C'était comme la petite lampe sacrée, qu'on porte par un grand vent, et qu'il faut défendre contre les colères de la foule, affolée de mensonges. Nous n'avions qu'une tactique, rester les maîtres de notre affaire, la prolonger autant que nous le pourrions, pour qu'elle provoquât les événements, tirer d'elle enfin ce que nous nous étions promis de preuves décisives. Et nous n'avons jamais songé à nous, nous n'avons jamais agi que pour le triomphe du droit, prêts à le payer de notre liberté et de notre vie. Qu'on se souvienne de la situation qui m'était faite, en juillet, à Versailles. C'était l'étranglement sans phrases. Et je ne voulais pas être étranglé ainsi, cela ne me convenait pas qu'on m'exécutât pendant l'absence du Parlement, au milieu des passions de la rue. Notre volonté était d'atteindre octobre, dans l'espoir que la vérité aurait marché encore, que la justice alors s'imposerait. D'autre part, il ne faut pas oublier tout le travail sourd qui se faisait à chaque heure, tout ce que nous pouvions attendre des instructions, ouvertes contre le commandant Esterhazy et contre le colonel Picquart. L'un et l'autre étaient en prison, nous n'ignorions pas que des clartés vives jailliraient forcément des enquêtes ouvertes, si elles étaient menées loyalement; et, sans prévoir pourtant l'aveu, puis le suicide du colonel Henry, nous comptions sur l'inévitable événement, qui, d'un jour à l'autre, devait éclater, éclairant toute la monstrueuse affaire de sa vraie et sinistre lueur. Dès lors, est-ce que notre désir de gagner du temps ne s'explique pas? est-ce que nous n'avions pas raison d'user de tous les moyens légaux pour choisir notre heure, au mieux des intérêts de la justice? est-ce que temporiser n'était pas vaincre, dans la plus douloureuse et la plus sainte des luttes? A n'importe quel prix, il fallait attendre, car tout ce que nous savions, tout ce que nous espérions, nous permettait de donner, pour l'automne, rendez-vous à la victoire. Encore une fois, nous autres nous ne comptions pas, il s'agissait uniquement de sauver un innocent, d'éviter à la patrie le plus effroyable désastre moral dont elle eût jamais couru le danger. Et ces raisons avaient une telle force que je partis, résigné, en annonçant mon retour pour octobre, avec
- 58. la certitude d'être ainsi un bon ouvrier de la cause et d'assurer le triomphe. Mais ce que je ne dis pas aujourd'hui, ce que je dirai un jour, ce fut l'arrachement, l'amertume de ce sacrifice. On oublie que je ne suis ni un polémiste, ni un homme politique, tirant bénéfice des bagarres. Je suis un libre écrivain qui n'a eu qu'une passion dans sa vie, celle de la vérité, qui s'est battu pour elle sur tous les champs de bataille. Depuis quarante ans bientôt, j'ai servi mon pays par la plume, de tout mon courage, de toute ma force de travail et de bonne foi. Et je vous jure qu'il y a une affreuse douleur, à s'en aller seul, par une nuit sombre, à voir s'effacer au loin les lumières de France, lorsqu'on a simplement voulu son honneur, sa grandeur de justicière parmi les peuples. Moi! moi qui l'ai chantée par plus de quarante œuvres déjà! Moi dont la vie n'a été qu'un long effort pour porter son nom aux quatre coins du monde! Moi, partir ainsi, fuir ainsi, avec cette meute de misérables et de fous galopant derrière mes talons, me poursuivant de menaces et d'outrages! Ce sont là des heures atroces, dont l'âme sort trempée, invulnérable désormais aux blessures iniques. Et, plus tard, pendant les longs mois d'exil qui ont suivi, s'imagine-t-on cette torture d'être supprimé des vivants, dans l'attente quotidienne d'un réveil de la justice, que chaque jour, attarde? Je ne souhaite pas au pire des criminels la souffrance que, depuis onze mois, m'a causée chaque matin la lecture des dépêches de France, sur cette terre étrangère, où elles prenaient un effrayant écho de folie et de désastre. Il faut avoir promené ce tourment pendant de longues heures solitaires, il faut avoir revécu au loin, et seul toujours, la crise où s'effondrait la patrie, pour savoir ce qu'est l'exil, dans les conditions tragiques où je viens de le connaître. Et ceux qui pensent que je suis parti pour fuir la prison, et pour faire sans doute la fête à l'étranger avec l'or juif, sont de tristes gens qui m'inspirent un peu de dégoût et beaucoup de pitié.
- 59. Je devais revenir en octobre. Nous avions résolu de temporiser jusqu'à la rentrée des Chambres, tout en comptant sur l'événement imprévu, qui était pour nous, au courant des choses, l'événement certain. Et voilà que cet événement imprévu n'attendit pas octobre, il éclata dès la fin d'août, avec l'aveu et le suicide du colonel Henry. Dès le lendemain, je voulus rentrer. Pour moi, la révision s'imposait, l'innocence de Dreyfus allait être immédiatement reconnue. D'ailleurs, je n'avais jamais demandé que la révision, mon rôle devait forcément finir, dès que la Cour de cassation serait saisie, et j'étais prêt à m'effacer. Quant à mon procès, il n'était plus, à mes yeux, qu'une formalité pure, puisque la pièce produite par les généraux de Pellieux, Gonse et de Boisdeffre, et sur laquelle le jury m'avait condamné, était un faux dont l'auteur venait de se réfugier dans la mort. Et je me préparais donc au retour, lorsque mes amis de Paris, mes conseils, tous ceux qui étaient restés dans la bataille, m'écrivirent des lettres pleines d'inquiétude. La situation restait grave. Loin d'être résolue, la révision semblait encore incertaine. M. Brisson, le chef du cabinet, se heurtait à des obstacles sans cesse renaissants, trahi par tous, ne disposant pas lui-même d'un simple commissaire de police. De sorte que mon retour, au milieu des passions surchauffées, apparaissait comme un prétexte à des violences nouvelles, un danger pour la cause, un embarras de plus pour le ministère, dans sa tâche déjà si difficile. Et, désireux de ne pas compliquer la situation, je dus m'incliner, je consentis à patienter encore. Quand la Chambre criminelle fut enfin saisie, je voulus rentrer. Je le répète, je n'avais jamais demandé que la révision, je considérais mon rôle comme terminé, du moment que l'affaire était portée devant la juridiction suprême, instituée par la loi. Mais de nouvelles lettres m'arrivèrent, me suppliant d'attendre, de ne rien hâter. La situation, qui me semblait si simple, était au contraire, me disait-on, pleine d'obscurité et de péril. Mon nom, ma personnalité ne pouvait être qu'une torche, qui rallumerait l'incendie. C'est pourquoi mes amis, mes conseils faisaient appel à mes sentiments de bon citoyen, en me parlant de l'apaisement nécessaire, en me disant que je
- 60. devais attendre le retour fatal de l'opinion, pour éviter de rejeter notre pauvre pays dans une agitation néfaste. L'affaire était en bonne voie, mais rien n'était fini, quel serait mon regret, si une impatience de ma part attardait la vérité triomphante! Et je m'inclinai une fois de plus, je restai dans le tourment de ma solitude et de mon silence. Quand la Chambre criminelle, admettant la demande de révision, décida d'ouvrir une vaste enquête, je voulus rentrer. Cette fois, je l'avoue, j'étais à bout de courage, je comprenais bien que cette enquête durerait de longs mois, je pressentais l'angoisse continue où elle devait me faire vivre. Puis, vraiment, est-ce qu'assez de lumière n'était pas faite, est-ce que le rapport du conseiller Bard, le réquisitoire du procureur général Manau, la plaidoirie de l'avocat Mornard n'avaient pas établi assez de vérité, pour que je pusse revenir le front haut? Toutes les accusations que j'avais portées, dans ma Lettre au Président de la République, se trouvaient confirmées. Mon rôle était rempli, je n'avais qu'à rentrer dans le rang. Et ce fut pour moi un grand chagrin, une révolte indignée, d'abord, lorsque je trouvai, chez mes amis, la même résistance à mon retour. Ils étaient toujours en pleine bataille, ils m'écrivaient que je ne pouvais juger la situation comme eux, que ce serait une dangereuse faute de laisser recommencer mon procès parallèlement à l'enquête de la Chambre criminelle. Le nouveau ministère, hostile à la révision, trouverait peut-être dans ce procès la diversion voulue, l'occasion cherchée de nouveaux troubles. En tout cas, la Cour avait besoin d'une paix absolue, j'aurais mal agi en venant l'embarrasser d'une émotion populaire, qu'on exploiterait sûrement contre nous. J'ai lutté, j'ai voulu même tomber à Paris, un beau soir, contre tous ces conseils, sans prévenir personne. Et la sagesse seulement m'a vaincu, je me suis résigné encore à de longs mois de torture. Voilà pourquoi, depuis onze mois bientôt, je ne suis pas rentré. En me tenant à l'écart, je n'ai agi, comme le jour où je me suis mis en avant, qu'en soldat de la vérité et de la justice. Je n'ai été que le bon citoyen qui se dévoue jusqu'à l'exil, jusqu'à la totale disparition, qui consent à n'être plus, pour l'apaisement du pays, pour ne pas
- 61. passionner inutilement les débats de la monstrueuse affaire. Et je dois dire aussi que, dans la certitude de la victoire, je gardais mon procès comme la ressource suprême, la petite lampe sacrée, dont nous rallumerions la clarté, si les puissances mauvaises venaient à éteindre le soleil. Mon abnégation, je l'ai poussée jusqu'au silence complet. J'ai voulu non seulement être un mort, mais un mort qui ne parle pas. La frontière passée, j'ai su me taire. On ne doit parler que lorsqu'on est là, pour prendre la responsabilité de ce qu'on dit. Personne ne m'a entendu, personne ne m'a vu. Je le répète, j'étais au tombeau, dans une retraite inviolable, que pas un étranger n'a pu connaître. Les quelques journalistes qui ont laissé entendre qu'ils m'avaient approché, ont menti. Je n'en ai reçu aucun, j'ai vécu au désert, ignoré de tous. Et je me demande ce que mon pays, si dur pour moi, me reproche, depuis les onze mois de bannissement volontaire que je souffre pour lui rendre la paix, dans la dignité et dans le patriotisme de mon silence. Et c'est fini, et je rentre, puisque la vérité éclate, puisque la justice est rendue. Je désire rentrer en silence, dans la sérénité de la victoire, sans que mon retour puisse donner lieu au moindre trouble, à la moindre agitation de la rue. Cela serait indigne de moi qu'on pût me confondre un instant avec les bas exploiteurs des manifestations populaires. De même que j'ai su me taire au dehors, je saurai reprendre ma place au foyer national en bon citoyen paisible, qui entend ne déranger personne et se remettre discrètement à sa tâche accoutumée, sans qu'on s'occupe de lui davantage. Maintenant que la bonne œuvre est faite, je ne veux ni applaudissements ni récompense, même si l'on estime que j'ai pu en être un des utiles ouvriers. Je n'ai eu aucun mérite, la cause était si belle, si humaine! C'est la vérité qui a vaincu, et il ne pouvait en être autrement. Dès la première heure, j'en ai eu la certitude, j'ai marché à coup sûr, ce qui diminue mon courage. Cela était tout simple. Je veux bien qu'on dise de moi, comme unique hommage, que je n'ai
- 62. été ni une bête ni un méchant. D'ailleurs, je l'ai déjà, ma récompense, celle de songer à l'innocent que j'aurai aidé à tirer du tombeau, où, vivant, depuis quatre années, il agonisait. Ah! j'avoue que l'idée de son retour, la pensée de le voir libre, de lui serrer les mains, me bouleverse d'une émotion extraordinaire, qui m'emplit les yeux de larmes heureuses. Cette minute suffira à payer tous mes soucis. Mes amis et moi, nous aurons fait là une bonne action, dont les braves cœurs de France nous garderont quelque gratitude. Et que voulez-vous de plus, une famille qui nous aimera, une femme et des enfants qui nous béniront, un homme qui nous devra d'avoir incarné en lui le triomphe du droit et de la solidarité humaine! Mais, cependant, si la lutte actuelle est finie pour moi, si je ne désire tirer de la victoire aucune curée, ni mandat politique, ni place, ni honneurs, si mon ambition unique est de continuer mon combat de vérité par la plume, tant que ma main la pourra tenir, je voudrais bien faire remarquer, avant de passer à d'autres luttes, quelle a été ma prudence, ma modération dans la bataille. Se souvient-on des abominables clameurs qui accueillirent ma Lettre au Président de la République? J'étais un insulteur de l'armée, un vendu, un sans- patrie. Des amis littéraires à moi, consternés, épouvantés, s'écartaient, m'abandonnaient, dans l'horreur de mon crime. Il y eut des articles écrits, qui désormais pèseront lourd sur la conscience des signataires. Enfin, jamais écrivain brutal, fou, malade d'orgueil, n'avait adressé à un chef d'État une Lettre plus grossière, plus mensongère, plus criminelle. Et, maintenant, qu'on la relise, ma pauvre Lettre. J'en suis devenu un peu honteux, je l'avoue, honteux de sa discrétion, de son opportunisme, je dirais presque de sa lâcheté. Car, puisque je me confesse, je puis bien reconnaître que j'avais beaucoup adouci les choses, que j'en avais même beaucoup passé sous silence, de celles qui sont connues, avérées aujourd'hui, et dont je voulais douter encore, tellement elles me semblaient monstrueuses et déraisonnables. Oui, je soupçonnais Henry déjà, mais sans preuve, à ce point que je crus sage de ne pas même le mettre en cause. Je devinais bien des histoires, certaines confidences étaient venues à moi, si terribles, que je ne me sentis
- 63. pas le droit de les risquer, dans leurs effroyables conséquences. Et voilà qu'elles sont révélées, qu'elles sont devenues la vérité banale d'aujourd'hui! Et voilà que ma pauvre Lettre n'est plus au point, apparaît comme tout à fait enfantine, une simple berquinade, une invention de romancier timide, à côté de la superbe et farouche réalité! Je répète que je n'ai ni le désir ni le besoin de triompher. Mais, pourtant, je dois bien constater que les événements ont, à cette heure, fait la preuve de toutes mes accusations. Il n'est pas un des hommes accusés par moi dont la culpabilité ne soit démontrée, à la lumière aveuglante de l'enquête. Ce que j'ai annoncé, ce que j'ai prévu, est là debout, éclatant. Et ce dont je suis plus doucement fier encore, c'est que ma Lettre était sans violence, indignée, mais digne de moi: on n'y trouvera pas un outrage, pas même un mot excessif, rien que la hautaine douleur d'un citoyen qui demande justice au chef de l'État. Telle a été l'éternelle histoire de mes œuvres, je n'ai jamais pu écrire un livre, une page, sans être abreuvé de mensonges et d'injures, quitte à ce qu'on soit forcé, le lendemain, de me donner raison. J'ai donc l'âme sereine, sans colère ni rancune. Si je n'écoutais que la faiblesse de mon cœur, d'accord avec le dédain de mon intelligence, je serais même pour le grand pardon, je laisserais les malfaiteurs sous le seul châtiment de l'éternel mépris public. Mais il est, je crois, des sanctions pénales nécessaires, et l'argument décisif est que, si quelque redoutable exemple n'est pas fait, si la justice ne frappe pas les hauts coupables, jamais le petit peuple ne croira à l'immensité du crime. Il faut un pilori dressé pour que la foule sache enfin. Je laisse donc la Némésis achever son œuvre vengeresse, je ne l'aiderai pas. Et, dans mon indulgence de poète, pleinement satisfait du triomphe de l'idéal, il ne reste qu'une révolte exaspérée, la pensée affreuse que le colonel Picquart est encore sous les verrous. Pas un jour ne s'est passé, sans que, de mon exil, ma douleur fraternelle ne soit allée à lui, dans sa prison. Que Picquart ait pu être arrêté, que depuis un an bientôt on le tienne dans une geôle, comme un malfaiteur, qu'on ait prolongé sa torture par la plus
- 64. infâme des comédies judiciaires, c'est là un fait monstrueux qui affole la raison. La tache restera ineffaçable sur tous ceux qui ont trempé dans cette iniquité suprême. Et, si demain Picquart n'est pas libre, c'est la France tout entière qui ne se lavera jamais de l'inexplicable folie d'avoir laissé aux mains criminelles des bourreaux, des menteurs, des faussaires, le plus noble, le plus héroïque et le plus glorieux de ses enfants. Alors seulement l'œuvre sera complète. Et ce n'est pas une moisson de haine, c'est une moisson de bonté, d'équité, d'espérance infinie, que nous avons semée. Il faut qu'elle pousse. Aujourd'hui, on ne peut encore qu'en prévoir la richesse. Tous les partis politiques ont sombré, le pays s'est partagé en deux camps: d'une part, les forces réactionnaires du passé; de l'autre, les esprits d'examen, de vérité et de droiture, en marche vers l'avenir. Ces postes de combat sont les seuls logiques, nous devons les garder pour les conquêtes de demain. A l'œuvre donc, par la plume, par la parole, par l'action! à l'œuvre de progrès et de délivrance! Ce sera l'achèvement de 89, la révolution pacifique des intelligences et des cœurs, la démocratie solidaire, libérée des puissances mauvaises, fondée enfin sur la loi du travail, qui permettra l'équitable répartition des richesses. Dès lors, la France libre, la France justicière, annonciatrice de la juste société du prochain siècle, se retrouvera souveraine parmi les nations. Il n'est pas d'empire si bardé de fer, qui ne croulera, quand elle aura donné la justice au monde, comme elle lui a déjà donné la liberté. Je ne vois plus pour elle d'autre rôle historique, et elle n'a pas connu encore un tel resplendissement de gloire. Je suis chez moi. Monsieur le procureur général peut donc, quand il lui plaira, me faire signifier l'arrêt de la cour d'assises de Versailles, qui m'a condamné, par défaut, à un an de prison et à trois mille francs d'amende. Et nous nous retrouverons devant le jury.
- 65. En me faisant poursuivre, je n'ai voulu que la vérité et la justice. Elles sont aujourd'hui. Mon procès n'est plus utile, et il ne m'intéresse plus. La justice devra simplement dire s'il y a crime à vouloir la vérité.
- 66. LE CINQUIÈME ACTE Ces pages ont paru dans l'Aurore, le 12 septembre 1899. J'avais fait opposition à l'arrêt de la Cour d'assises de Versailles et au jugement de la Cour d'appel de Paris, pour les experts, tous les deux rendus par défaut, et j'attendais. La justice n'avait d'ailleurs plus de hâte, elle désirait connaître le résultat du nouveau procès Dreyfus, à Rennes. Le ministère Dupuy, tombé le 12 juin 1899, venait d'être remplacé par le ministère Waldeck-Rousseau, le 22 juin. Ce fut le 1er juillet que Dreyfus débarqua en France, par une nuit de tempête, le 8 août que commença son nouveau procès, et le 9 septembre qu'un conseil de guerre le condamna une seconde fois. J'écrivis cet article, le lendemain. LE CINQUIÈME ACTE Je suis dans l'épouvante. Et ce n'est plus la colère, l'indignation vengeresse, le besoin de crier le crime, d'en demander le châtiment, au nom de la vérité et de la justice; c'est l'épouvante, la terreur sacrée de l'homme qui voit l'impossible se réaliser, les fleuves remonter vers leurs sources, la terre culbuter sous le soleil. Et ce que je crie, c'est la détresse de notre généreuse et noble France, c'est l'effroi de l'abîme où elle roule. Nous nous étions imaginé que le procès de Rennes était le cinquième acte de la terrible tragédie que nous vivons depuis bientôt deux ans. Toutes les péripéties dangereuses nous semblaient épuisées, on croyait aller vers un dénouement d'apaisement et de concorde. Après la douloureuse bataille, la victoire du droit devenait inévitable, la pièce devait se terminer heureusement par le triomphe
- 67. classique de l'innocent. Et voilà que nous nous sommes trompés, une péripétie nouvelle se déclare, la plus inattendue, la plus affreuse de toutes, assombrissant encore le drame, le prolongeant et le lançant vers une fin ignorée, devant laquelle notre raison se trouble et défaille. Le procès de Rennes n'était décidément que le quatrième acte. Et, grand Dieu! quel sera donc le cinquième? de quelles douleurs et de quelles souffrances nouvelles va-t-il donc être fait, à quelle expiation suprême va-t-il jeter la nation? Car, n'est-ce pas? il est bien certain que l'innocent ne peut pas être condamné deux fois et qu'un tel dénouement éteindrait le soleil et soulèverait les peuples! Ah! ce quatrième acte, ce procès de Rennes, dans quelle agonie morale je l'ai vécu; au fond de la complète solitude où je m'étais réfugié, pour disparaître de la scène en bon citoyen, désireux de n'être plus une occasion de passion et de trouble! Avec quel serrement de cœur j'attendais les nouvelles, les lettres, les journaux, et quelles révoltes, quelles douleurs à les lire! Les journées de cet admirable mois d'août en devenaient noires, et jamais je n'ai senti l'ombre et le froid d'un deuil si affreux, sous des cieux plus éclatants. Certes depuis deux ans, les souffrances ne m'ont pas manqué. J'ai entendu les foules hurler à la mort sur mes talons, j'ai vu passer à mes pieds un immonde débordement d'outrages et de menaces, j'ai connu pendant onze mois les désespérances de l'exil. Et il y a eu aussi mes deux procès, des spectacles lamentables de vilenie et d'iniquité. Mais que sont mes procès à côté du procès de Rennes? des idylles, des scènes rafraîchissantes, où fleurit l'espoir. Nous avions bien assisté à des monstruosités, les poursuites contre le colonel Picquart, l'enquête sur la Chambre criminelle, la loi de dessaisissement qui en est résultée. Seulement, tout cela n'est plus qu'enfantillage, l'inévitable progression a suivi son cours, le procès
- 68. de Rennes s'épanouit au sommet, énorme, comme la fleur abominable de tous les fumiers entassés. On aura vu là le plus extraordinaire ensemble d'attentats contre la vérité et contre la justice. Une bande de témoins dirigeant les débats, se concertant chaque soir pour le louche guet-apens du lendemain, requérant à coups de mensonges au lieu et place du ministère public, terrorisant et insultant leurs contradicteurs, s'imposant par l'insolence de leurs galons et de leurs panaches. Un tribunal en proie à cette invasion des chefs, souffrant visiblement de les voir en criminelle posture, obéissant à toute une mentalité spéciale, qu'il faudrait démonter longuement pour juger les juges. Un ministère public grotesque, reculant les limites de l'imbécillité, laissant aux historiens de demain un réquisitoire dont le néant stupide et meurtrier sera une éternelle stupeur, d'une telle cruauté sénile et têtue, qu'elle apparaît inconsciente, née d'un animal humain inclassé encore. Une défense qu'on tente d'abord d'assassiner, puis qu'on fait asseoir chaque fois qu'elle devient gênante, à laquelle on refuse de laisser apporter la preuve décisive, lorsqu'elle réclame les seuls témoins qui savent. Et, pendant un mois, l'abomination a duré devant l'innocent, ce pitoyable Dreyfus, dont la pauvre loque humaine ferait pleurer les pierres, et ses anciens camarades sont venus lui donner un coup de pied encore, et ses anciens chefs sont venus l'écraser de leurs grades, pour se sauver eux-mêmes du bagne, et il n'y a pas eu un cri de pitié, un frisson de générosité, dans ces vilaines âmes. Et c'est notre douce France qui a donné ce spectacle au monde. Quand on aura publié le compte rendu in extenso du procès de Rennes, il n'existera pas un monument plus exécrable de l'infamie humaine. Cela dépasse tout, jamais document plus scélérat n'aura encore été fourni à l'histoire. L'ignorance, la sottise, la folie, la cruauté, le mensonge, le crime, s'y étalent avec une impudence telle, que les générations de demain en frémiront de honte; Il y a là dedans des aveux de notre bassesse dont l'humanité entière rougira. Et c'est bien cela qui fait mon épouvante, car pour qu'un tel procès
- 69. ait pu se produire dans une nation, pour qu'une nation livre au monde civilisé une telle consultation sur son état moral et intellectuel, il faut qu'elle traverse une horrible crise. Est-ce donc la mort prochaine? et quel bain de bonté, de pureté, d'équité nous sauvera de la boue empoisonnée où nous agonisons? Comme je l'écrivais dans ma Lettre au Président de la République, après le scandaleux acquittement d'Esterhazy, il est impossible qu'un conseil de guerre défasse ce qu'a fait un conseil de guerre. Cela est contraire à la discipline. Et l'arrêt du conseil de guerre de Rennes, dans son embarras jésuitique, cet arrêt qui n'a pas le courage de dire oui ou non, est la preuve éclatante que la justice militaire est impuissante à être juste, puisqu'elle n'est pas libre, puisqu'elle se refuse à l'évidence, jusqu'à condamner de nouveau un innocent, plutôt que de mettre en doute son infaillibilité. Elle n'apparaît plus que comme une arme d'exécution, dans la main des chefs. Désormais, elle ne saurait être qu'une justice expéditive, en temps de guerre. Elle doit disparaître en temps de paix, du moment qu'elle est incapable d'équité, de simple logique et de bon sens. Elle-même s'est condamnée. Songe-t-on à cette situation atroce qui nous est faite, parmi les nations civilisées? Un premier conseil de guerre, trompé dans son ignorance des lois, dans sa maladresse à juger, condamne un innocent. Un second conseil de guerre, qui a pu être trompé encore par le plus impudent complot de mensonges et de fraudes, acquitte un coupable. Un troisième conseil de guerre, quand la lumière est faite, quand la plus haute magistrature du pays veut lui laisser la gloire de réparer l'erreur, ose nier le plein jour et de nouveau condamne l'innocent. C'est l'irréparable, le crime suprême a été commis. On n'avait condamné Jésus qu'une fois. Mais que tout croule, que la France soit en proie aux factions, que la patrie en feu s'abîme dans les décombres, que l'armée elle-même y laisse son honneur, plutôt que de confesser que des camarades se sont
- 70. trompés et que des chefs ont pu être des menteurs et des faussaires! L'idée sera crucifiée, le sabre doit rester roi. Et nous voilà, devant l'Europe, devant le monde, dans cette belle situation. Le monde entier est convaincu de l'innocence de Dreyfus. Si un doute était resté chez quelque peuple lointain, l'éclat aveuglant du procès de Rennes aurait achevé d'y porter la lumière. Toutes les cours des grandes puissances nos voisines sont renseignées, connaissent les documents, ont la preuve de l'indignité de trois ou quatre de nos généraux et de la paralysie honteuse de notre justice militaire. Notre Sedan moral est perdu, cent fois plus désastreux que l'autre, celui où il n'y a eu que du sang versé. Et, je le répète, ce qui m'épouvante, c'est que cette défaite de notre honneur semble irréparable, car comment casser les jugements de trois conseils de guerre, où trouverons-nous l'héroïsme de confesser la faute, pour marcher encore le front haut? Où est le gouvernement de courage et de salut public, où sont les Chambres qui comprendront, qui agiront, avant l'inévitable effondrement final? Le pis est que nous voici arrivés à une échéance de gloire. La France a voulu fêter son siècle de travail, de science, de luttes pour la liberté, pour la vérité et la justice. Il n'y a pas eu de siècle d'un effort plus superbe, on te verra plus tard. Et la France a donné rendez- vous chez elle à tous les peuples pour glorifier sa victoire, la liberté conquise, la vérité et la justice promises à la terre. Alors, dans quelques mois, les peuples vont venir, et ce qu'ils trouveront, ce sera l'innocent condamné deux fois, la vérité souffletée, la justice assassinée. Nous sommes tombés dans leur mépris, et ils viendront godailler chez nous, ils boiront nos vins, ils embrasseront nos servantes, comme on fait dans l'auberge louche où l'on consent à s'encanailler. Est-ce possible cela, est-ce que nous allons accepter que notre Exposition soit le mauvais lieu méprisé où le monde entier voudra bien faire la fête? Non, non! il nous faut tout de suite le cinquième acte de la monstrueuse tragédie, dussions-nous y laisser encore de notre chair. Il nous faut notre honneur, avant que nous saluions les peuples, dans une France guérie et régénérée.
- 71. Ce cinquième acte, il me hante, et je reviens toujours à lui, je le cherche, je l'imagine. A-t-on remarqué que cette affaire Dreyfus, ce drame géant qui remue l'univers, semble mis en scène par quelque dramaturge sublime, désireux d'en faire un chef-d'œuvre incomparable? Je ne rappelle pas les extraordinaires péripéties qui ont bouleversé toutes les âmes. A chaque acte nouveau, la passion a grandi, l'horreur a éclaté plus intense. Dans cette œuvre vivante, c'est le destin qui a du génie, il est quelque part poussant les personnages, déterminant les faits, sous la tempête qu'il déchaîne. Et il veut sûrement que le chef-d'œuvre soit complet, et il nous prépare quelque cinquième acte surhumain qui refera la France glorieuse, à la tête des nations. Car, soyez-en convaincus, c'est lui qui a voulu le crime suprême, l'innocent condamné une deuxième fois. Il fallait que le crime fût commis, pour la grandeur tragique, pour la beauté souveraine, pour l'expiation peut-être, qui permettra l'apothéose. Et, maintenant, puisqu'on a touché le fond de l'horreur, j'attends le cinquième acte qui terminera le drame, en nous délivrant, en nous refaisant une santé et une jeunesse nouvelles. Mon épouvante, je la dirai nettement aujourd'hui. Elle a toujours été, comme je l'ai laissé entendre, à diverses reprises, que la vérité, la preuve décisive, accablante, ne nous vienne de l'Allemagne. L'heure n'est plus de faire le silence sur ce mortel danger. Trop de lumière rayonne, il faut envisager courageusement le cas où ce serait l'Allemagne qui, dans un coup de tonnerre, apporterait le cinquième acte. Voici ma confession. Avant mon procès, dans le courant de janvier 1898, je sus de la façon la plus certaine qu'Esterhazy était «le traître», qu'il avait fourni à M. de Schwartzkoppen un nombre considérable de documents, que beaucoup de ces documents étaient de son écriture, et que la collection complète se trouvait à Berlin, au ministère de la guerre. Je ne fais point métier d'être patriote, mais j'avoue que les certitudes qui me furent données me bouleversèrent;
- 72. et, depuis ce temps, mon angoisse de bon Français n'a point cessé, j'ai vécu dans la terreur que l'Allemagne, notre ennemie de demain peut-être, ne nous souffletât avec les preuves qui sont en sa possession. Eh quoi! le conseil de guerre de 1894 condamne Dreyfus innocent, le conseil de guerre de 1898 acquitte Esterhazy coupable, et notre ennemie détient les preuves de la double erreur de notre justice militaire, et tranquillement la France s'entête dans cette erreur, accepte l'effroyable danger dont elle est menacée! On dit que l'Allemagne ne peut user de documents qu'elle tient de l'espionnage. Qu'en sait-on? Que la guerre éclate demain, ne commencera-t-elle pas peut-être par perdre notre armée d'honneur devant l'Europe, en publiant les pièces, en montrant l'iniquité abominable où se sont obstinés certains chefs? Est-ce qu'une telle pensée est tolérable, est- ce que la France jouira d'un instant de repos, tant qu'elle saura aux mains de l'étranger les preuves de son déshonneur? Moi, je n'en ai plus dormi, je le dis simplement. Alors, avec Labori, j'ai décidé de citer comme témoins les attachés militaires étrangers, nous doutant bien que nous ne les amènerions pas à la barre, mais voulant faire entendre au gouvernement que nous savions la vérité, espérant qu'il agirait. On a fait la sourde oreille, on a plaisanté, laissant l'arme aux mains de l'Allemagne. Et les choses sont restées en l'état, jusqu'au procès de Rennes. Dès ma rentrée en France, j'ai couru chez Labori, j'ai insisté désespérément pour que des démarches fussent faites auprès du ministère en lui signalant la terrifiante situation, en lui demandant s'il n'allait pas intervenir, afin qu'on nous donnât les documents, grâce à son entremise. Certes, rien n'était plus délicat, puis il y avait ce malheureux Dreyfus qu'on voulait sauver, de sorte qu'on était prêt à toutes les concessions, par crainte d'irriter l'opinion publique affolée. D'ailleurs, si le conseil de guerre acquittait Dreyfus, il ôtait par là même tout virus nuisible aux documents, il brisait entre les mains de l'Allemagne l'arme dont elle pourrait se servir. Dreyfus acquitté, c'était l'erreur reconnue, réparée. L'honneur redevenait sauf.
- 73. Et mon tourment patriotique a recommencé, plus intolérable, lorsque j'ai senti qu'un conseil de guerre allait aggraver le péril, en condamnant de nouveau l'innocent, celui dont la publication des documents de Berlin criera un jour l'innocence. C'est pourquoi je n'ai cessé d'agir, suppliant Labori de réclamer les documents, de citer en témoignage M. de Schwartzkoppen, qui seul peut faire la pleine lumière. Et le jour où Labori, ce héros frappé d'une balle sur le champ de bataille, a profité d'une occasion que lui offraient les accusateurs, en poussant à la barre un étranger indigne, le jour où il s'est levé pour demander qu'on entendît l'homme dont un mot devait terminer l'affaire, il a rempli tout son devoir, il a été la voix héroïque que rien ne fera taire, dont la demande survit au procès; et doit fatalement, à l'heure voulue, le recommencer pour le finir par la seule solution possible, l'acquittement de l'innocent. La demande des documents est posée, je défie que les documents ne soient pas produits. Voyez dans quel péril accru, intolérable, nous a mis le président du conseil de guerre de Rennes, en usant de son pouvoir discrétionnaire pour empêcher la production des documents. Rien de plus brutal, pas de porte plus volontairement fermée à la vérité. «Nous ne voulons pas qu'on nous apporte l'évidence, car nous voulons condamner.» Et un troisième conseil de guerre s'est joint aux deux autres, dans l'erreur aveugle, de sorte que le démenti venu de l'Allemagne frapperait maintenant trois sentences iniques. N'est-ce pas de la démence pure, n'est-ce pas à crier de révolte et d'inquiétude? Le ministère que ses agents ont trahi, qui a eu la faiblesse de laisser les grands enfants de mentalité obscure jouer avec les allumettes et les couteaux, le ministère qui a oublié que gouverner c'est prévoir, n'a qu'à se hâter d'agir, s'il ne veut pas abandonner au bon plaisir, de l'Allemagne le cinquième acte, le dénouement devant lequel tout Français devrait trembler. C'est lui, le gouvernement, qui a la charge de jouer ce cinquième acte au plus tôt, pour empêcher qu'il ne nous vienne de l'étranger. Il peut se procurer les documents, la diplomatie a résolu des difficultés plus grandes. Le jour où il saura demander
- 74. les documents énumérés au bordereau, on les lui donnera. Et ce sera là le fait nouveau, qui nécessitera une seconde révision devant la Cour de cassation, instruite cette fois, je l'espère, et cassant sans renvoi, dans la plénitude de sa souveraine magistrature. Mais, si le gouvernement reculait encore, les défenseurs de la vérité et de la justice feront le nécessaire. Pas un de nous ne désertera son poste. La preuve, la preuve invincible, nous finirons bien par l'avoir. Le 23 novembre, nous serons à Versailles. Mon procès recommencera, puisqu'on veut qu'il recommence dans toute son ampleur. Si d'ici là justice n'est pas faite, nous aiderons encore à la faire. Mon cher, mon vaillant Labori, dont l'honneur n'a fait que grandir, prononcera donc à Versailles la plaidoirie qu'il n'a pu prononcer à Rennes; et c'est bien simple, rien ne sera perdu. Moi, je ne le ferai pas taire. Il n'aura qu'à dire la vérité, sans craindre de me nuire, car je suis prêt à la payer de ma liberté et de mon sang. Devant la cour d'assises de la Seine, j'ai juré l'innocence de Dreyfus. Je la jure devant le monde entier, qui maintenant la crie avec moi. Et je le répète, la vérité est en marche, rien ne l'arrêtera. A Rennes, elle vient de faire un pas de géant. Je n'ai plus que l'épouvante de la voir arriver, dans un coup de foudre de la Némésis vengeresse, saccageant la patrie, si nous ne nous hâtons pas de la faire resplendir nous-mêmes, sous notre clair soleil de France.
- 75. LETTRE A MADAME ALFRED DREYFUS Ces pages ont paru dans l'Aurore, le 29 septembre 1899. Je les écrivis, lorsque M. le président Loubet eut signé la grâce d'Alfred Dreyfus, le 19 septembre, et que l'innocent, condamné deux fois, fut rendu aux siens. J'étais décidé à garder le silence, tant que mon procès ne serait pas revenu devant la Cour d'assises de Versailles; et là seulement j'aurais parlé. Mais il était des circonstances où je ne pouvais rester muet. LETTRE A MADAME ALFRED DREYFUS Madame, On vous rend l'innocent, le martyr, on rend à sa femme, à son fils, à sa fille, le mari et le père, et ma première pensée va vers la famille réunie enfin, consolée, heureuse. Quel que soit encore mon deuil de citoyen, malgré la douleur indignée, la révolte où continuent à s'angoisser les âmes justes, je vis avec vous cette minute délicieuse, trempée de bonnes larmes, la minute où vous avez serré dans vos bras le mort ressuscité, sorti vivant et libre du tombeau. Et, quand même, ce jour est un grand jour de victoire et de fête. Je m'imagine la première soirée, sous la lampe, dans l'intimité familiale, lorsque les portes sont fermées et que toutes les abominations de la rue meurent au seuil domestique. Les deux enfants sont là, le père est revenu du lointain voyage, si long, si
- 76. obscur. Ils le baisent, ils attendent de lui le récit qu'il leur fera plus tard. Et quelle paix confiante, quel espoir d'un avenir réparateur, tandis que la mère s'empresse doucement, ayant encore, après tant d'héroïsme, une tâche héroïque à remplir, celle d'achever par ses soins et par sa tendresse le salut du crucifié, du pauvre être qu'on lui rend. Une douceur endort la maison close, une infinie bonté baigne de toutes parts la chambre discrète où sourit la famille, et nous sommes là dans l'ombre, muets, récompensés, nous tous qui avons voulu cela, qui luttons depuis tant de mois pour cette minute de bonheur. Quant à moi, je le confesse, mon œuvre n'a été d'abord qu'une œuvre de solidarité humaine, de pitié et d'amour. Un innocent souffrait le plus effroyable des supplices, je n'ai vu que cela, je ne me suis mis en campagne que pour le délivrer de ses maux. Dès que son innocence me fut prouvée, il y eut en moi une hantise affreuse, cette pensée de tout ce que le misérable avait souffert, de tout ce qu'il souffrait encore dans le cachot muré où il agonisait, sous la fatalité monstrueuse dont il ne pouvait même déchiffrer l'énigme. Quelle tempête sous ce crâne, quelle attente dévorante, ramenée par chaque aurore! Et je n'ai plus vécu, et mon courage n'a été fait que de ma pitié, et mon but unique a été de mettre fin à la torture, de soulever la pierre pour que le supplicié revînt à la clarté du jour, fût rendu aux siens, qui panseraient ses plaies. Affaire de sentiment, comme disent les politiques, avec un léger haussement d'épaules. Mon Dieu! oui, mon cœur seul était pris, j'allais au secours d'un homme en détresse, fût-il juif, catholique ou mahométan. Je croyais alors à une simple erreur judiciaire, j'ignorais la grandeur du crime qui tenait cet homme enchaîné, écrasé au fond de la fosse scélérate, où l'on guettait son agonie. Aussi étais-je sans colère contre les coupables, inconnus encore. Simple écrivain, arraché par la compassion à sa besogne coutumière, je ne poursuivais aucun but politique, je ne travaillais pour aucun parti. Mon parti, à moi, dès ce début de la campagne, ce n'était que l'humanité à servir.
- 77. Et ce que je compris, ensuite, ce fut la terrible difficulté de notre tâche. A mesure que la bataille se déroulait, s'étendait, je sentais que la délivrance de l'innocent demanderait des efforts surhumains. Toutes les puissances sociales se liguaient contre nous, et nous n'avions pour nous que la force de la vérité. Il nous faudrait faire un miracle, pour ressusciter l'enseveli. Que de fois, pendant ces deux cruelles années, j'ai désespéré de l'avoir, de le rendre vivant à sa famille! Il était toujours là-bas, dans sa tombe, et nous avions beau nous mettre à cent, à mille, à vingt mille, la pierre était si lourde des iniquités entassées, que je craignais de voir nos bras s'user, avant le suprême effort. Jamais, jamais plus! Peut-être un jour, dans longtemps, ferions-nous la vérité, obtiendrions-nous la justice. Mais lui, le malheureux serait mort, jamais sa femme, jamais ses enfants ne lui auraient donné le baiser triomphant du retour. Aujourd'hui, madame, voilà que nous avons fait le miracle. Deux années de luttes géantes ont réalisé l'impossible, notre rêve est accompli, puisque le supplicié est descendu de sa croix, puisque l'innocent est libre, puisque votre mari vous est rendu. Il ne souffrira plus, la souffrance de nos cœurs est donc finie, l'image intolérable cesse de troubler notre sommeil. Et c'est pourquoi, je le répète, c'est aujourd'hui jour de grande fête, de grande victoire. Discrètement, tous nos cœurs communient avec le vôtre, il n'est pas une femme, pas une mère, qui n'ait senti son cœur se fondre, en songeant à cette première soirée intime, sous la lampe, dans l'affectueuse émotion du monde entier, dont la sympathie vous entoure. Sans doute, madame, cette grâce est amère. Est-il possible qu'une telle torture morale soit imposée après tant de tortures physiques? et quelle révolte à se dire qu'on obtient de la pitié ce qu'on ne devrait tenir que de la justice! Le pis est que tout semble avoir été concerté pour aboutir à cette iniquité dernière. Les juges ont voulu cela, frapper encore l'innocent,
- 78. pour sauver les coupables, quittes à se réfugier dans l'hypocrisie affreuse d'une apparence de miséricorde. «Tu veux l'honneur, nous ne te ferons que l'aumône de la liberté, pour que ton déshonneur légal couvre les crimes de tes bourreaux.» Et il n'est pas, dans la longue série des ignominies commises, un attentat plus abominable contre la dignité humaine. Cela dépasse tout, faire mentir la divine pitié, en faire l'instrument du mensonge, en souffleter l'innocence pour que le meurtre se promène au soleil, galonné et empanaché! Et quelle tristesse, en outre, que le gouvernement d'un grand pays se résigne, par une faiblesse désastreuse, à être miséricordieux, quand il devrait être juste! Trembler devant l'arrogance d'une faction, croire qu'on va faire de l'apaisement avec de l'iniquité, rêver je ne sais quelle embrassade menteuse et empoisonnée, est le comble de l'aveuglement volontaire. Est-ce que le gouvernement, au lendemain de l'arrêt scandaleux de Rennes, ne devait pas le déférer à la Cour de cassation, cette juridiction suprême qu'il bafoue d'une si insolente façon? Est-ce que le salut du pays n'était pas dans cet acte d'énergie nécessaire, qui sauvait notre honneur aux yeux du monde, qui rétablissait chez nous le règne de la loi? Il n'y a d'apaisement définitif que dans la justice, toute lâcheté ne sera qu'une cause de fièvre nouvelle, et ce qui nous a manqué jusqu'ici, c'est un gouvernement de bravoure qui veuille bien aller jusqu'au bout de son devoir, pour remettre dans le droit chemin la nation égarée, affolée de mensonges. Mais notre déchéance est telle, que nous en sommes réduits à féliciter le gouvernement de s'être montré pitoyable. Il a osé être bon, grand Dieu! Quelle audace folle, quelle extraordinaire vaillance, qui l'expose aux morsures des fauves, dont les bandes sauvages, sorties de la forêt ancestrale, rôdent parmi nous! Être bon quand on ne peut peut pas être fort, c'est déjà méritoire. Et, d'ailleurs, madame, cette réhabilitation qui aurait dû être immédiate, pour la juste gloire du pays lui-même, votre mari peut l'attendre, le front haut, car il n'est pas d'innocent qui soit plus innocent, devant tous les peuples de la terre.
- 79. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com