![5
IF Technology Capabilities for Modern Security Problems
reformat the data from some native form to a usable one. In spite of some a priori mapping of how
CI influences or constrains the way in which situational inferences or estimates can be developed,
which may serve certain environments, the defense and security type applications, with their various
dynamic and uncertain types of CI, demand a more adaptive approach. Given a nominated situational
hypothesis Hf from a fusion process or “engine,” the first question is: What CI type information is
relevant to this hypothesis? Relevant CI is only that information that influences our interpretation or
understanding of Hf. Presuming a “relevancy filter” can be crafted, a search function would explore
the available CI and make this CI available to an “a posteriori” reasoning engine. That reasoning
engine would then use (1) a CI-guided subset of Domain Knowledge and (2) the retrieved CI to reason
over Hf to first determine consistency of Hf with the relevant CI. If it is inconsistent, then some type
of adjudication logic will need to be applied to reconcile this inconsistency between (1) the fusion
process that produced Hf and (2) the a posteriori reasoning process that judges it as inconsistent. If,
however, Hf is judged as consistent with the additional CI, an expanded interpretation of Hf could be
developed, providing a deeper situational understanding. This overall process, which can be consid
ered a “Process Refinement” operation, would be a so-called “Level 4” process in the context of the
Joint Directors of Laboratories (JDL) Data Fusion Process Model [1], that is, as an adaptive operation
for fusion process enhancement. The overall ideas discussed here are elaborated in Ref. [2].
1.1.4
Ontological Data
IF processes and algorithms historically have been developed in a framework that has assumed the
a priori availability of a reliable body of procedural and dynamic knowledge about the problem
domain, that is, knowledge that supports a more direct approach to temporal reasoning about the
unfolding patterns of interest in the problem domain. In COIN and other complex problems, such a
priori and reliable knowledge is most often not available—the Tactics, Techniques, and Procedures
of modern-day adversaries are highly adaptive and extremely hard to model with confidence. The
US DARPA COMPOEX Program [3] attempted to develop such models but achieved only partial
success, experiencing gaps in the overall modeling space of such desired behavioral models. We
label these types of problems as “weak knowledge” problems, implying that only fragmentary a
priori behavioral model type knowledge is available to aid in IF-based reasoning, inferencing, and
estimation.
Ontological information, however, that does not attempt to form such comprehensive behavioral
and temporal models overtly but does include temporal primitives along with structural/syntactic
relations among entities can be specified a priori with reasonably good confidence, and thus pro
vides a declarative knowledge base to support IF reasoning and estimation. Note that such knowl
edge is also represented in language and is available as digital text, in the same way as data from
messages, documents, Twitter, and so forth. The use of ontological information in IF systems can be
varied; ontological information can augment observed data and can aid in asserting possible rela
tionships, directing search and also sensor management (to acquire expected information based on
ontological relations), and yet in other ways. Importantly, specified ontologies can also serve as pro
viding consistent and grounded semantic terminology for any given system. In our current research,
we employ ontologies primarily for augmenting observational data with asserted ontological data
whose relevance is algorithmically determined using “spreading activation” and then integrated to
enrich the evidential basis for reasoning [4]. The broader implications of ontologies for intelligence
analysis are described in Ref. [5], which come from the University of Buffalo’s National Center for
Ontological Research (see http://ncorwiki.buffalo.edu/index.php/Main_Page).
1.1.5 Learned Information
Finally, there is the class of information that could be learned (online) from all of the aforemen
tioned sources if the IF process is designed with a Data Mining/Inductive or Abductive Learning
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-33-320.jpg)
![6 Multisensor Data Fusion
functional component. Very little research and prototyping of such dual-process type IF systems
has been done although the conceptualization of such IF schemes and architectures was put forward
some time ago (e.g., Ref. [6]), as shown in Figure 1.1. The runtime integration of learned informa
tion raises a number of both algorithmic issues as well as architectural issues. For example, if mean
ingful patterns of behavior can be learned and can be measured/judged as persistent or enduring,
such patterns could be incorporated in a dynamically modifiable knowledge base to be reused (as
a Level 4 Process refinement function). Such learning processes will also not be perfect and have
some uncertainty that also needs to be factored into the traditional Common Referencing and Data
Association functions of the target fusion process.
1.2
COMMON REFERENCING AND DATA ASSOCIATION
Common Referencing (CR) is that traditional IF system function that is sometimes called
“Alignment” and is the function that normalizes these input sources for any given fusion applica
tion or design. CR addresses such issues as coordinate system normalization, temporal alignment,
and uncertainty alignment across the input streams, among others. With the highly disparate
input streams described earlier, the design of required CR techniques is a nontrivial challenge.
There are at least two major CR issues that these heterogeneous data represent temporal align
ment and uncertainty alignment. Consider a textual input message whose free text, in just a few
lines, could have past–present–future tense expressions, for example, “3 days ago I saw…”, “past
precedents lead me to believe that tomorrow I should see…” and so forth. Other sources can also
have varied temporal structures regarding their input. Such data lead to the issue of what the IF
community has called “OOSM: out-of-sequence measurements” for hard/sensor data but the issue
carries over to all sources as well and requires complex temporal alignment techniques for CR;
it also raises the issue of retrospective fusion processing operations to correct for delayed inputs
(if warranted; this is a design choice). Temporal alignment methods we have used for soft data
are described in Ref. [7].
Data mining Data fusion
Discovery modeling
Data
mining
search
operations
Object base
Knowledge
Information
understood and
explained
Data transform
Data cleansing
Data warehouse
Operational process data storage
Sensor
1
Sensor
2
Sensor
3
Data
Observations
and
measurements
Information
Data, organized
and placed in
context
Model
Common user
visualization Visualization, management
Situation
Level 3
Impact
Level 2
Situation refinement
Object base
Level 1
Object refinement
Level 0
Signal data refinement
Situations
Impacts
Objects
Level
4
Resource
refinement
Visualization, validation
FIGURE 1.1 Notional fusion process architecture combining data mining and data fusion. (From ISCAS
‘98—Proceedings of the 1998 IEEE International Symposium on Circuits and Systems, Ann Arbor, 1998.)
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-34-320.jpg)
![7
IF Technology Capabilities for Modern Security Problems
The uncertainty alignment requirement evolves as a result of the high likelihood that any uncer
tainty in the widely various sources will be represented in disparate forms. Consider the basic
differences between the uncertainty in sensor (hard) data and textual (soft) data; sensor data uncer
tainty is sensibly always expressed in probabilistic form whereas, owing to the problem of imprecise
adjectives and adverbs in language, linguistic uncertainty is often expressed in possibilistic (fuzzy)
terms. It can be expected that uncontrolled open source or social media data may use yet other
uncertainty formalisms to express or tag inputs. Transformation and normalization of disparate
forms of uncertainty is a specialized topic in the uncertainty/statistical literature (e.g., Ref. [8]),
and is among the high-priority issues in the IF community [9]. It should be noted that such trans
formations largely can be developed only by invoking some statistical type qualities that are pre
served across the transform, such as some form of total uncertainty; that is, the transform does
not create an “equivalent” value of a probability in say a possibilistic space; seminal papers on the
probability–
possibility transformation issue are in Refs. [10–12]. In our research, we have addressed
the probabilistic–
possibilistic
transformation issue in an approach that satisfies the consistency and
preference preservation principles [13], while resulting in the most specific distribution for a speci
fied portion of a probabilistic representation, resulting in the use of a truncated triangular transfor
mation in our case [14].
Regarding the Data Association (DA) function, which some consider the heart of a fusion
process, these varied data raise the level of DA complexity in significant ways. The soft data
category, which inherently reports about Entities and (judged) Relationships, and is inherently
in semantic format (language/words), raises the important issue of how to measure semantic
similarity of such elements as reported in these various input streams. Such scores are needed
in the Hypothesis Evaluation step of the DA process (see Ref. [15] on these DA subfunctions).
But there are further DA complications that arise due to the soft data: linguistic phrases have
verbs that reflect inter-Entity (noun) relationships; also of note is that the Natural Language
Processing (NLP) community has employed graphical methods for the representation of linguis
tic structures. As a result, the DA process now involves interassociation of both Entities (nouns)
and Relations (verbs), that is, of graphical structures. This requirement extends to the hard data
as well because that data need to be cast in a semantic framework to enable the overall DA
process for the combined hard and soft data. Developing DA methods for graphical structures
represents an entirely new challenge for the DA function. In such approaches for these applica
tions, a scoring approach also needs to be developed to assess Relational similarity as well as
Entity similarity, and a composite association scheme for these graphical substructures needs
to be evolved. Historical approaches to DA have often employed solution methods drawn from
assignment problems in operations research. When association is required between many non
graphical data sources, this can be handled by the multidimensional assignment problem [16,17].
The main difference between the multidimensional assignment problem and graph association
is how topological information from the graphs is used. Our research center has attacked this
problem and has developed research prototype algorithms, as described in Ref. [18], where the
graph association problem is formulated as a binary linear program and a heuristic for solving
the multiple graph association is developed using a Lagrangian relaxation approach to address
issues with a between-graph transitivity requirement.
1.3 SEMANTICS
The introduction of linguistic information, as well as the transformation of sensor + algorithm
estimation process outputs into a semantic frame, also adds to the complexity of IF process design
and development. Semantic complexity is also added by the very nature of modern intelligence
and security problems wherein the situations of interest relate to both military operations and also
sociopolitical behaviors and entities. Clear meanings of such notions of interest as “patterns of life,”
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-35-320.jpg)
![8 Multisensor Data Fusion
“rhythm of the city,” and “radicalization” as patterns or situations of interest—to be estimated by
IF systems—have proven difficult to specify in clear semantic terms. Although the use of ontolo
gies helps in this regard, standardization issues remain when considering networked and distributed
systems, which are typical in the modern era. For example, in distributed intelligence or military
systems there is typically no single point of architectural authority that can mandate a single onto
logical framework for the network. For large-scale real systems there is also the problem of large
legacy systems that were never designed with ontological formalisms in mind; this creates a “retro
fit” problem of adjusting the semantic framework of that system to some new ontological standard,
a costly and complex operation.
It must also be noted that the way in which all textual/linguistic information gets into an IF
system is through processing in some type of NLP or text extraction system. Such systems serve as
a front-end filter for the admission of fundamental entity and relationship data, the raw soft data of
the system, and so any imperfections in such extractions bound the capture of evidential informa
tion for the subsequent reasoning and estimation processes. Whereas errors in hard sensor data are
typically known with reasonable accuracy because of sensor calibrations, the errors in text extrac
tion and NLP systems are either weakly known or unknown, sometimes as a result of proprietary
constraints.
Other strategies to deal with the complexities of semantics involve the use of controlled languages,
to bound the grammatical structures and also the extent of the vocabulary that has to be dealt with.
A good example for military/intelligence applications is the Battle Management Language (BML)
[19] that has been under development since about 2003 for both Command and Control simulation
studies but also for IF applications (e.g., Refs. [20,21]).
There is a corresponding need to understand better the nature of semantic (and syntactic) com
plexity in language, and also to develop measures and metrics that aid in developing better NLP
processes and controlled languages. There is a reasonably rich literature on these topics (e.g., Ref. [22])
that should be exploited in regard to the integrated design of IF systems that today have to deal with
a wide range of semantic difficulties.
1.4
GRAPHICAL REPRESENTATIONS AND METHODS
There are a number of reasons that, for COIN and asymmetric warfare-type problems, graphs are
becoming a dominant representational form for the information in and the processes involved in IF
systems. In the information domain, many of the components discussed in Section 1.1 are textual/
linguistic and to capture this information in digital form, graphs are the representational form of
choice. The problem domain is also described in the ontologies that are also typically couched in
graphical forms. Note that ontologies describe inter-Entity relations of various types. Note too that
the inferences and estimates of interest in these problems are of the higher level type in the sense
of the JDL Model of Information Fusion, that is, estimates of situations and threat states. These
higher level states—the conditions of interest for intelligence and security applications—are also
best described as graphs, as situations can in the most abstract sense be considered as a graph of
entities and relations.
As a result, it is not unexpected to see that the core functions of IF, such as DA as previously
described, are employing graphical methods in these fusion function operations. The US Army’s
primary intelligence support system, the Distributed Common Ground Station-Army (DCGS-A),
employs a global graph approach to capture all of the evidentiary information that supports IF and
other intelligence analysis operations; see Ref. [23] and Figure 1.2, which shows the top-level struc
ture of this graphical concept.
Developing a comprehensive understanding of these problems thus involves a logical synthesis
of the many situational substructures or subgraphs in these problem domains. The subgraphs are
somewhat thematic and can be thought of as revolving about the Political, Military, Economic,
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-36-320.jpg)
![9
IF Technology Capabilities for Modern Security Problems
Social, Infrastructure, and Information (PMESII) notion of the heterogeneity of the classes of
information of interest in such problems. Thus, it is not surprising to see social network analysis
tools—which are by the way graph-theoretic and graph-centric—employed in support of intelli
gence analysis, here with the focus on the social and infrastructure patterns and subgraphs of the
problem space.
In our own work for such problems, we considered that it would be broadly helpful in analysis
to enable a subgraph-querying capability as a generalized analysis tool. In such an approach, the
analyst forms a query in text that can be transformed to a graph (we call these template graphs in
that they are subgraph structures of interest) that is then searched for in the associated-evidence
graph that is formed by the DA process. This search operation is in effect an stochastic inexact
graph-matching problem, as the nodes and arcs of the evidential (or perhaps the template graph)
have uncertainty values associated with them, and also because what is sought is the best match to
the query, not an exact match, as there may be no exact match in such unpredictable problem situa
tions. Other complexities arise in trying to realize such capability, such as executing such operations
incrementally for streaming data, and also doing them in a computationally efficient way because
the graphs can get quite large. As a consequence of several PhD efforts, we have realized today a
rather mature graph-matching capability for intelligence analysis that is implemented in a cloud-
based process; see Refs. [24–26], among other of our works.
1.5
OVERALL SYSTEM ARCHITECTURES AND ANALYSIS FRAMEWORKS
It can perhaps be appreciated from the preceding discussion that the major challenge for intel
ligence analysis in these modern problems is the synthesis of a total situational picture. In the face
of highly heterogeneous data of varying uncertainty and of a problem domain that has many sub
structures and relations and entities of interest, and has a varying temporal operational tempo, these
problems—
even with state of the art automated support/analysis systems such as IF systems—
create a cognitive challenge even for the best analysts.
Global graph
“object”
Relationship
or verb
Action Person
Organization
Account
Task
Event
Society
Place
Region
Feature Material Reference
Equipment
Consumable
(Docs, media,
etc.)
• 12 primary types of objects
• An object is a “thing” whose existence is
not predicated on the existence of some
other “thing”
Facility
FIGURE 1.2 US Army’s “global graph” concept for DCGS-A. (From Walsh, D. Relooking the JDL Model
for fusion on a global graph. In National Symposium on Sensor and Data Fusion, Las Vegas, July 2010.)
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-37-320.jpg)
![10 Multisensor Data Fusion
Automated methods for aiding such synthesis are in research and are just now being experi
mented with (e.g., Complex Event Processing [CEP], Probabilistic Argumentation, Graph-Based
Relational Learning, and other methods). At the moment, intelligence support systems comprise
suites of disparate tools and hopefully some agile visualization schemes that aid analysts in the
hypothesis-synthesis challenge.
Our research prototype system, supported by the US Army Research Office, has addressed a
number of the issues discussed here (as commented on within the chapter) and is just now entering
the phase where the user end of the system is being designed and developed. The current Tool Suite
comprises Dynamic Social Network Analysis Tools (uses a random graph approach), the Graph
Matching Tools mentioned previously, a Link Analysis Tool (finds a wide variety of inter-Entity
connections), Named Entity Recognizer (part of the NLP system), Entity and Activity Recognizers
(uses automated semantic labeling methods from imagery and video), and an early prototype
Abductive Reasoner. Together, these form the Community of Interest Service Layer or analyst
layer in our architecture, which is basically a service-oriented architecture. That layer is shown in
Figure 1.3, and comprises three main services: Evidence and Entity-estimate Foraging Service (this
includes the CR and DA functions previously described), a Sensemaking Service where the above
tools reside, and an Analytic Support Service that includes Visualization support, Pedigree Service,
and other processes. Note that the Hypothesis Composition Service is still in conceptualization;
at present we are exploring a CEP approach. CEP addresses the challenge of combining multiple
heterogeneous data streams into a hierarchical structure that can represent higher order events and
semantic meaning through the application of rules and filters at multiple levels of information (e.g.,
Ref. [27]).
The Core Enterprise Services that do all of the front-end data processing and conditioning are
shown in Figure 1.4; this figure does not show much of the detail but the flavor of these operations
can be appreciated. Multiple hard data streams (in our case these are LIDAR, EO/IR, and Visible
imagery, video sources, and acoustic devices) are processed individually to the point where seman
tic information is developed from various estimation algorithms. Multiple soft message streams as
arising from multiple soldier reports enter the NLP-based soft processing stream and the primary
Hypothesis
composition
services
Data
association
services
Graph
matching
tool
Common ref
services
Dynamic
social net
tool
Group
activity
tool
Link
analysis
tool
Intelligence
analysts
Metadata and
pedigree
services
Workflow
services
Alert
services
Visualization
services
Enterprise service bus
Analytic support
services
Sensemaking
services
Evidence and entity-estimate
foraging services
FIGURE 1.3 Analyst layer in our service-oriented architecture for counterinsurgency analysis support.
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-38-320.jpg)
![11
IF Technology Capabilities for Modern Security Problems
entities and relations are extracted. All of this semantic information flows to the Enterprise Bus,
where it is accessible to the Community of Interest (Analysts) Service Layer for inferencing and
estimation operations.
Space permits us to describe only one of our tools and we choose to show our Activity
Recognizer Tool system; this is developed by our colleagues at Tennessee State University and is
described in Ref. [28]. This tool focuses on human–vehicle interactions and activities, a type of
activity that is critically important for the problem of improvised explosive devices (IEDs). It is
assumed that video and acoustic sensing of the human–vehicle settings is feasible. An approach
that is based on human–vehicle activity ontology is used, wherein discrete activities (e.g., door
opening as detected by acoustics, human entering vehicle as detected by video) are detected
by each sensing modality. Spatiotemporal and semantic association of these discrete activities,
along with the activity-class ontology, allows fusion-based inferencing of aggregated activity
classes of interest. Further, as previously described, these inferences are framed into “pseudo-
messages” that are sent to the Enterprise Bus for access by the hard + soft DA service, to allow
combination with soft message data on the same activities. These operations are depicted in
Figure 1.5.
Single-sensor-based evidence
Single-sensor-based evidence
Single-sensor-based evidence
Single-sensor-based evidence
Single-sensor-based evidence
Single-sensor-based evidence
State
estimation
D
C
Entity-level
evidence
formation
Entity-level
evidence
formation
Entity-level
evidence
formation
Hb
Ha
State
estimation
State
estimation
Common
referencing
Common
referencing
Data
association
Common
referencing
Data
association
Data
association
Direct human inference
Direct human inference
Sensor-specific
preprocessing
Sensor-specific
preprocessing
Sensor-specific
preprocessing
Enterprise
service
bus
A
B
FIGURE 1.4 Core enterprise services in our service-oriented architecture for counterinsurgency analysis
support.
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-39-320.jpg)
![12 Multisensor Data Fusion
1.6
CAPABILITY SHORTFALLS AND RESEARCH NEEDS
Intelligence and security problems of the type discussed here are very likely to continue for the
foreseeable future, although the prospects for conventional nation-state conflict still remain as well,
and will drive yet other technology requirements, and it should be noted that there are some over
laps in such requirements. Although the IF community is reacting to the COIN and asymmetric/
irregular warfare and stability operations needs of the type described here, there are very few well-
tested capabilities that have been transitioned to operational systems. The IF community also has to
make judgments about research investments that are peculiar to IF process needs and those that are
supportive of IF processes; a good example is in NLP and text extraction—this is a core capability
supportive of IF but that is actively being matured by the NLP community. For IF processing in
particular, there are needs to improve CR methods for temporal and uncertainty alignment, and to
improve capabilities for retrospective and reliable temporal estimation when meaningful amounts
of input are continually shifting in time. Improvements in DA techniques are similarly required,
both for new ideas in measures for semantic similarity and scoring in support of DA, but equally
in graphical or other methods that in addition exhibit computational efficiency, as DA is typically
the computational bottleneck in IF systems. Regarding state estimation, we see the major chal
lenge being in the creation of automated methods to support synthesis of the disparate hypotheses
emanating from data and theme-specific inferencing/estimation tools, to aid intelligence analysts
in forming more comprehensive assessments of the “story” of interest suggested by the evidence.
New studies and implementations of information foraging theory [29] are needed as one means to
achieve more informative and efficient examinations of both associated evidence and inferences.
Deeper examinations are also needed of the various Sensemaking paradigms [30,31] and ways that
IF technologies can support them.
Event alignment
and fusion
Acoustic event
detection and analysis
Visual event
detection and analysis
Surveillance
camera #1
Acoustic
sensor
Soft message
generation
1 2 3 4
5 6 7 8 9 10
16
15
20
19
14
13
18
12
17
11
Subject 1 got off fast from the driver side
Message-1
Message-2
Message-3
Subject 1 opened the trunk of the car and
unloaded a large object
Subject 1 left the scene (GPS) at 1345 hr
HVI protocol: (Subject-predicate-object-time-space)
2
1.5
1
–0.5
–1
–1.5
–2
0
0.5
0 0.5 1 1.5 2 2.5
FIGURE 1.5 Fused human–vehicle activity estimation scheme. (From Shirkhodaie, A. et al., “Acoustic
and Imagery Semantic Labeling and Fusion of Human-Vehicle Interactions,” in SPIE Defense and Security
Conference, Orlando, FL, 2011.)
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-40-320.jpg)
![16 Multisensor Data Fusion
The goal of this chapter is to provide readers with a comprehensive review of contemporary
data fusion methodologies, as well as an overview of the most recent developments and emerging
trends in this field. The existing data fusion methodologies are examined according to a data-centric
taxonomy, that is, the specific data-related issue they aim to tackle. Inspired by recent advances in
mobile and ubiquitous sensing, cloud storage and computing, and prevalence of social networks,
the new and emerging directions in data fusion research, such as social data fusion, cloud-enabled
and big data fusion, and fusion of streaming data, are briefly discussed. The rest of this chapter is
organized as follows. A brief discussion of multisensor data fusion definitions and most common
applications are presented in Section 2.2. Relying on a data-centric taxonomy, a review of the exist-
ing data fusion literature is provided in Section 2.3. Section 2.4 is dedicated to a short discussion on
research on data fusion evaluation methodologies and frameworks. The emerging research trends
within the data fusion community are enumerated and briefly examined in Section 2.5. Lastly, in
Section 2.6 the concluding remarks are presented.
2.2
MULTISENSOR DATA FUSION
2.2.1
What Is Multisensor Data Fusion?
From a data integration perspective, Luo defines multisensor fusion as any stage in the integration pro-
cess in which there is an actual combination (or fusion) of different sources of sensory information into
one representational format [1]. The boundary between sensor fusion and sensor integration is quite
fuzzy and the terms are used interchangeably sometimes. Joshi and Sanderson describe multisensor
fusion as part of the multisensor integration process [2]. This process refers to the synergistic use of
multiple sensors to improve operation of the system as a whole and it also includes sensor planning and
sensor architecture. Multisensor planning deals with the acquisition of sensor data while multisensor
architecture is responsible for the organization of data processing and data flow in the system. Joshi
and Sanderson define multisensor fusion as a process that deals with the combination of data from
multiple sensors into one coherent and consistent internal representation or action [2].
Many other definitions for data fusion exist in the literature. Joint Directors of Laboratories
(JDL) [3] defines data fusion as a “multi-level, multifaceted process handling the automatic detec-
tion, association, correlation, estimation, and combination of data and information from several
sources.” Klein [4] generalizes this definition, stating that data can be provided either by a single
source or by multiple sources. Both definitions are general and can be applied in different fields
including remote sensing. In Ref. [5], Bostrom et al. present a review and discussion of many data
fusion definitions. Based on the identified strengths and weaknesses of previous work, a principled
definition of information fusion is proposed as “Information fusion is the study of efficient methods
for automatically or semi-automatically transforming information from different sources and dif-
ferent points in time into a representation that provides effective support for human or automated
decision making.” Data fusion is a multidisciplinary research area borrowing ideas from many
Brain
Uncertain observations
Sensor 1
Sensor 2
Sensor n
Fusion node
Estimate of feature
Feature
z1
z2
zn
X
FIGURE 2.1 Analogy between the brain operation and the fusion node in a data fusion system.
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-44-320.jpg)
![17
Review of the State of the Art and Overview of Emerging Trends
diverse fields such as signal processing, information theory, statistical estimation and inference, and
artificial intelligence. This is indeed reflected in the variety of techniques presented in Section 2.3.
Various conceptualizations of the fusion process exist in the literature. The most common and
popular conceptualization of fusion systems is the JDL model [3]. The JDL classification is based on
the input data and produced outputs, and originated in the military domain. The original JDL model
considers the fusion process in four increasing levels of abstraction: object, situation, impact, and
process refinement. Despite its popularity, the JDL model has many shortcomings, such as being
too restrictive and especially tuned to military applications. The JDL formalization is focused on
data (input/output) rather than processing. An alternative is Dasarathy’s framework [6], which views
the fusion system, from a software engineering perspective, as a data flow characterized by input/
output as well as functionalities (processes). Another general conceptualization of fusion is the
work of Goodman et al. [7], which is based on the notion of random sets. The distinctive aspects of
this framework are its ability to combine decision uncertainties with decisions themselves, as well
as presenting a fully generic scheme of uncertainty representation. One of the most abstract fusion
frameworks was proposed by Kokar et al. [8]. This formalization is based on category theory and
is claimed to be sufficiently general to capture all kinds of fusion, including data fusion, feature
fusion, decision fusion, and fusion of relational information. It can be considered as the first step
toward development of a formal theory of fusion. The major novelty of this work is the ability to
express all aspects of multisource information processing, that is, both data and processing. Further,
it allows for consistent combination of the processing elements (algorithms) with measurable and
provable performance. Such formalization of fusion paves the way for the application of formal
methods to standardized and automatic development of fusion systems.
2.2.2 Applications of Multisensor Data Fusion
Multisensor data fusion aims to overcome the limitations of individual sensors and produce
accurate, robust, and reliable estimates of the world state based on multisensory information [9].
Multisensor data fusion has attracted many researchers from academia and industry because of
its foreseen benefits in many applications. These benefits include, but are not limited to, enhanced
confidence and reliability of measurements, extended spatial and/or temporal coverage, and reduced
data imperfection aspects. Mitchell listed four main advantages of multisensor data fusion [10]: a
greater granularity in the representation of information; greater certainty in data and results; elimi-
nation of noise and errors, producing a greater accuracy; and allowing a more complete view on
the environment. These foreseen benefits of multisensor data fusion result in its wide applicability
in a variety of military and civilian applications. As part of a comprehensive survey on multisen-
sor integration and fusion in intelligent systems, Luo and Kay described a number of military and
industrial applications in this area [11].
Data fusion is an established military technology and available for numerous military applica-
tions. These military applications include, but are not limited to, surveillance [12], anomaly detec-
tion [13] and behavior monitoring [14], target tracking [15,16], target engageability improvement
[17], fire control [18], and landmine detection [19]. For example, modern military Command
Control (C2) systems are making increasing use of data fusion and resource management technol-
ogy and tools [20]. By reducing uncertainty in the existing pieces of information and providing a
means to infer about the missing pieces, data fusion supports the decision makers in compiling and
analyzing the tactical/operational picture, and ultimately improving their situation awareness [21].
Examples of nonmilitary applications of data fusion include air traffic control [22], healthcare
[23,24], speaker detection and tracking [25], mobile robot navigation [26], mobile robot localiza-
tion [27], intelligent transportation systems [28], remote sensing [29,30], environment monitoring
[31,32], and situational awareness [33]. For example, a Bayesian approach with pre- and postfilter-
ing to handle data uncertainty and inconsistency in mobile robot local positioning is described in
Ref. [27]. Mobile robot positioning provides an answer for the question: Where is the robot? The
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-45-320.jpg)
![18 Multisensor Data Fusion
robot positioning solutions can be roughly categorized into relative position measurements (dead
reckoning) and absolute position measurements. In the former, the robot position is estimated by
applying to a previously determined position the course and distance traveled since. In the latter,
the absolute position of the robot is computed by measuring the direction of incidence of three or
more actively transmitting beacons, using artificial or natural landmarks, or using model matching
to estimate the absolute location of the robot. There will always be an error in the readings provided
by these techniques, and therefore the notion of multisensor data fusion is commonly used to tackle
various imperfection aspects of data and yield a more accurate estimate for the robot position [27].
2.3
A DATA-CENTRIC TAXONOMY FOR MULTISENSOR
DATA FUSION ALGORITHMS
Regardless of how different components (modules) of the data fusion system are organized, which
is specified by the given fusion architecture, the underlying fusion algorithms must ultimately pro-
cess (fuse) the input data. Real-world data fusion applications have to deal with several data-related
challenges. As a result, we explore data fusion algorithms according to a data-centric taxonomy
[34]. Figure 2.2 illustrates an overview of data-related challenges that are typically tackled by data
fusion algorithms. The input data to the fusion system may be imperfect, correlated, inconsistent,
and/or in disparate forms/modalities. Each of these four main categories of challenging problems
can be further subcategorized into more specific problems, as shown in Figure 2.2 and discussed in
the following.
Various classifications of imperfect data have been proposed in the literature [35–37]. Our classi-
fication of imperfect data is inspired by the pioneering work of Smets’ [36] as well as an elaboration
by Dubois and Prade [38]. Three aspects of data imperfection are considered in our classification:
uncertainty, imprecision, and granularity.
Data are uncertain when the associated degree of confidence about what is stated by the data is
less than 1. On the other hand, imprecise data are those data that refer to several, rather than only
one, object(s). Finally, data granularity refers to the ability to distinguish among objects, which are
Data-related fusion aspects
Imperfection Correlation
Conflict Outlier
Granularity
Imprecision
Uncertainty
Vagueness Ambiguity Incompleteness
Disorder
Inconsistensy Disparateness
FIGURE 2.2 A taxonomy of data fusion methodologies: Different data fusion algorithms can be roughly
categorized based on one of the four challenging problems of input data that are mainly tackled: data imper-
fection, data correlation, data inconsistency, and disparateness of data form.
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-46-320.jpg)
![19
Review of the State of the Art and Overview of Emerging Trends
described by data, being dependent on the provided set of attributes. Mathematically speaking,
assume the given data d (for each described object of interest) to be structured as the following:
object O attribute A statement S
representing that the data d is stating S regarding the relationship of some attribute(s) A to some
object O in the world. Further assume C(S) to represent the degree of confidence we assign to the
given statement S. Then, data are regarded to be uncertain if C(S) 1 while being precise, that is,
a singleton. Similarly, data are deemed as imprecise if the implied attribute A or degree of confi-
dence C are more than 1, for example, an interval or set. Please note, the statement part of the data
is almost always precise.
The imprecise A or C may be well defined or ill defined and/or miss some information. Thus,
imprecision can manifest itself as ambiguity, vagueness, or incompleteness of data. The ambiguous
data refers to those data where the A or C is exact and well defined yet imprecise. For instance, in the
sentence “Target position is between 2 and 5” the assigned attribute is the well-defined imprecise
interval [2 5]. The vague data is characterized by having ill-defined attributes, that is, the attribute is
more than 1 and not a well-defined set or interval. For instance, in the sentence “The tower is large”
the assigned attribute “large” is not well defined as it can be interpreted subjectively, that is, have
different meaning from one observer to the other. The imprecise data that has some information
missing is called incomplete data. For instance, in the sentence “It is possible to see the chair,” only
the upper limit on the degree of confidence C is given, that is, C τ for some τ [39].
Consider an information system [40] in which a number of (rather than one) objects O = {o1,…,
ok} are described using a set of attributes A = {V1, V2, …, Vn} with respective domains D1, D2, …,
Dn. Let F = D1 × D2 × … × Dn to represent the set of all possible descriptions given the attributes in
A, also called the frame. It is possible for several objects to share the same description in terms of
these attributes. Let [o]F to be the set of objects that are equivalently described (thus indistinguish-
able) within the frame F, also called the equivalence class. Now, let T ⊆ O represent the target set
of objects. In general, it is not possible to exactly describe T using F, because T may include and
exclude objects that are indistinguishable within the frame F. However, one can approximate T by
the lower and upper limit sets that can be described exactly within F in terms of the induced equiva-
lence classes. Indeed, the rough set theory provides a systematic approach to this end. In summary,
data granularity refers to the fact that the choice of data frame F (granule) has a significant impact
on the resultant data imprecision. In other words, different attribute subset selections B ⊆ A will lead
to different frames, and thus different sets of indiscernible (imprecise) objects.
Correlated (dependent) data are also a challenge for data fusion systems and must be treated
appropriately. We consider inconsistency in input data to stem from (highly) conflicting, spurious,
or out of sequence data. Finally, fusion data may be provided in different forms, that is, in one or
several modalities, as well as generated by physical sensors (hard data) or human operators (soft
data).
We believe such categorization of fusion algorithms is beneficial as it enables explicit exploration
of popular fusion techniques according to the specific data-related fusion challenge(s) they target.
Further, our taxonomy is intended to facilitate ease of development by supplying fusion algorithm
designers with an outlook of the appropriate and established techniques to tackle the data-related chal-
lenges their given application may involve. Finally, such exposition would be more intuitive and there-
fore helpful to nonexperts in data fusion by providing them with an easy-to-grasp view of the field.
2.3.1
Fusion of Imperfect Data
The inherent imperfection of data is the most fundamental challenging problem of data fusion
systems, and thus the bulk of research work has been focused on tackling this issue. A number of
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-47-320.jpg)
![20 Multisensor Data Fusion
mathematical theories are available to represent data imperfection [41], such as probability the-
ory [42], fuzzy set theory [43,44], possibility theory [45], rough set theory [46], and Dempster–
Shafer evidence theory (DSET) [47]. Most of these approaches are capable of representing specific
aspect(s) of imperfect data. For example, a probabilistic distribution expresses data uncertainty,
fuzzy set theory can represent vagueness of data, and evidential belief theory can represent uncer-
tain as well as ambiguous data. Historically, the probability theory was used for a long time to deal
with almost all kinds of imperfect information, because it was the only existing theory. Alternative
techniques such as fuzzy set theory and evidential reasoning have been proposed to deal with per-
ceived limitations in probabilistic methods, such as complexity, inconsistency, precision of models,
and uncertainty about uncertainty [42]. There are also hybridizations of these approaches that aim
for a more comprehensive treatment of data imperfection. Examples of such hybrid frameworks are
fuzzy rough set theory (FRST) [48] and fuzzy Dempster–Shafer theory (fuzzy DSET) [49]. Lastly,
there is the fairly new field of fusion using random sets, which could be used to develop a unified
framework for treatment of data imperfections [50]. Figure 2.3 provides an overview of the afore-
mentioned mathematical theories of dealing with data imperfections. On the x-axis, various aspects
of data imperfection, introduced in Figure 2.3, are depicted. The box around each of the mathemati-
cal theories designates the range of imperfection aspects targeted mainly by that theory. Interested
readers are referred to Refs. [39] and [34] for a comprehensive review of the classical theories of
representing data imperfections, describing each of them along with their interrelations.
2.3.2
Fusion of Correlated Data
Many data fusion algorithms, including the popular Kalman filter (KF) approach, require either
independence or prior knowledge of the cross covariance of data to produce consistent results.
Unfortunately, in many applications fusion data are correlated with potentially unknown cross
covariance. This can occur as a result of common noise acting on the observed phenomena [51] in
centralized fusion settings, or the rumor propagation issue, also known as the data incest or double
counting problem [52], in which measurements are inadvertently used several times in distributed
Imperfection
Fuzzy
set
Possibility
Rough
set
Probability
U
n
c
e
r
t
a
i
n
t
y
A
m
b
i
g
u
i
t
y
V
a
g
u
e
n
e
s
s
I
n
c
o
m
p
l
e
t
e
n
e
s
s
G
r
a
n
u
l
a
r
i
t
y
DSET
Theory
Random set
Fuzzy DSET
FIGURE 2.3 Overview of theoretical frameworks of imperfect data treatment (note: the fuzzy rough set
theory is omitted from the diagram to avoid confusion).
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-48-320.jpg)
![21
Review of the State of the Art and Overview of Emerging Trends
fusion settings [53]. If not addressed properly, data correlation can lead to biased estimation, for
example, artificially high confidence value, or even divergence of fusion algorithm [54]. Most of the
proposed solutions to correlated data fusion attempt to solve it by either eliminating the cause of
correlation [55,56] or tackling the impact of correlation in fusion process [57–59].
2.3.3
Fusion of Inconsistent Data
2.3.3.1 Spurious Data
Data provided by sensors to the fusion system may be spurious as a result of unexpected situations
such as permanent failures, short duration spike faults, or slowly developing failure [60]. If fused
with correct data, such spurious data can lead to dangerously inaccurate estimates. For instance,
KF would easily break down if exposed to outliers. The majority of work on treating spurious data
has been focused on identification/prediction and subsequent elimination of outliers from the fusion
process. Indeed, the literature work on sensor validation is partially aiming at the same target [61–63].
The problem with most of these techniques is the requirement for prior information, often in the
form of specific failure model(s). As a result, they would perform poorly in a general case in which
prior information is not available or unmodeled failures occur [64]. In Refs. [60,65] a general frame-
work for detection of spurious data has been proposed that relies on stochastic adaptive modeling
of sensors and is thus not specific to any prior sensor failure model. Extensive experimental simula-
tions have shown the promising performance of this technique in dealing with spurious data [64].
2.3.3.2 Out-of-Sequence Data
The input data to the fusion system are usually organized as discrete pieces each labeled with a
timestamp designating its time of origin. Several factors such as variable propagation times for dif-
ferent data sources as well as having heterogeneous sensors operating at multiple rates can lead to
data arriving out of sequence at the fusion system. Such out-of-sequence measurements (OOSM)
can appear as inconsistent data to the fusion algorithm. The main issue is how to use these, usually
old, data to update the current estimate while taking care of the correlated process noise between
the current time and the time of the delayed measurement [66].
Most of the early work on OOSM assumed only single-lag data. For example, an approximate
suboptimal solution to OOSM called Algorithm B [67], as well as its famous optimal counterpart
Algorithm A [68], both assume single-lag data. Some researchers have proposed algorithms to enable
handling of OOSM with arbitrary lags [69–71]. Among these methods the work in Ref. [71] is par-
ticularly interesting as it provides a unifying framework for treating OOSM with Algorithm A as a
special case. Nonetheless, it was shown in Ref. [72] that this approach, along with many other multilag
OOSM methods, is usually very expensive in terms of computational complexity and storage. The
same authors proposed an extension to the Algorithm A and Algorithm B called Algorithm Al1 and
Algorithm Bl1, respectively. They further showed that these new algorithms have requirements similar
to their single-lag counterparts and are therefore recommended for practical applications; Algorithm
Bl1 especially is preferred because it is almost optimal and very efficient. Research work also inves-
tigates the OOSM problem in the case of having both single-lag and multiple-lag data, termed the
mixed-lag OOSM problem. The proposed algorithm is claimed to handle all three types of OOSM
data and is shown to be suboptimal in the linear MMSE sense under one approximation [73].
2.3.3.3 Conflicting Data
Fusion of conflicting data, when, for instance, several experts have very different ideas about
the same phenomenon, has long been identified as a challenging task in the data fusion com-
munity. In particular, this issue has been heavily studied for fusion within the Dempster–Shafer
evidence theory framework. As shown in a famous counterexample by Zadeh [74], naive applica-
tion of Dempster’s rule of combination to fusion of highly conflicting data results in unintuitive
results. Since then, Dempster’s rule of combination has been subject to much criticism for rather
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-49-320.jpg)
![22 Multisensor Data Fusion
counterintuitive behavior [75]. Most of the solutions proposed alternatives to Dempster’s rule of
combinations [76–79]. On the other hand, some authors have defended this rule, arguing that the
counterintuitive results are due to improper application of this rule [50,80,81]. For example, in
Ref. [50] Mahler shows that the supposed unintuitive result of Dempster’s combination rule can be
resolved using a simple corrective strategy, i.e. to assign arbitrary small but nonzero belief masses
to hypotheses deemed extremely unlikely.
Fusion of conflicting data within the Bayesian probabilistic framework has also been explored by
some authors. For example, the Covariance Union (CU) algorithm is developed to complement the
Contextual Information (CI) method, and enable data fusion where input data is not just correlated
but may also be conflicting [82]. Furthermore, a new Bayesian framework for fusion of uncertain,
imprecise, as well as conflicting data was proposed in Ref. [83].
2.3.4
Fusion of Disparate Data
The input data to a fusion system may be generated by a wide variety of sensors, humans, or even
archived sensory data. Fusion of such disparate data to build a coherent and accurate global view
or the observed phenomena is a very difficult task. Nonetheless, in some fusion applications such as
human–computer interaction (HCI), such diversity of sensors is necessary to enable natural interac-
tion with humans. Our focus of discussion is on fusion of human generated data (soft data) as well
as fusion of soft and hard data, as research in this direction has attracted attention in recent years.
This is motivated by the inherent limitations of electronic (hard) sensors and recent availability
of communication infrastructure that allow humans to act as soft sensors [84]. Further, although
a tremendous amount of research has been done on data fusion using conventional sensors, very
limited work has studied fusion of data produced by human and nonhuman sensors. An example of
preliminary research in this area includes the work on generating a dataset for hard/soft data fusion
intended to serve as a foundation and a verification/validation resource for future research [85,86].
Also in Ref. [84], Hall et al. provide a brief review on ongoing work on dynamic fusion of soft/
hard data, identifying its motivation and advantages, challenges, and requirements. A Dempster–
Shafer theoretic framework for soft/hard data fusion is presented that relies on a novel conditional
approach to updating as well as a new model to convert propositional logic statements from text
into forms usable by Dempster–Shafer theory [87]. Another recent example of data fusion systems
capable of leveraging soft data is presented in Ref. [88], where Seifzadeh et al. describe a solution to
the problem of agile target tracking using fuzzy inference applied to soft data reports that character-
ize the target agility level.
2.4
EVALUATION OF DATA FUSION SYSTEMS
Performance evaluation aims at studying the behavior of a data fusion system operated by various
algorithms and comparing their pros and cons based on a set of measures or metrics. The outcome
is typically a mapping of different algorithms into different real values or partial orders for ranking
[89]. Generally speaking, the obtained performance of a data fusion system is deemed to be depen-
dent on two components: the quality of input data and the efficiency of fusion algorithm. As a result,
the literature work on (low-level) fusion evaluation can be categorized into the following groups:
• Evaluating the quality of input data to the fusion system. The target here is to develop
approaches that enable quality assessment of the data, which are fed to the fusion system,
and calculation of the degree of confidence in data in terms of attributes such as reliability
and credibility [90]. The most notable work in this group is perhaps the standardization
agreements (STANAG) 2022 of the North Atlantic Treaty Organization (NATO). STANAG
adopts an alphanumeric system of rating, which combines a measurement of the reliability
of the source of information with a measurement of the credibility of that information, both
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-50-320.jpg)
![23
Review of the State of the Art and Overview of Emerging Trends
evaluated using the existing knowledge. STANAG recommendations are expressed using
natural language statements, which makes them quite imprecise and ambiguous. Some
researchers attempted to analyze these recommendations and provide a formal mathematical
system of information evaluation in compliance with the NATO recommendations [90,91].
The proposed formalism relies on the observation that three notions underline an informa-
tion evaluation system: the number of independent sources supporting a piece of information,
their reliability, and that the information may conflict with some available/prior information.
Accordingly, a model of evaluation is defined and its fusion method, which accounts for the
three aforementioned notions, is formulated. The same authors have extended their work to
enable dealing with the notion of degree of conflict, in contrast to merely conflicting or non-
conflicting information [92]. Nonetheless, the current formalism is still not complete as there
are some foreseen notions of the STANAG recommendations, such as total ignorance about
the reliability of the information source, that are not being considered. Another important
aspect related to input information quality, which is largely ignored, is the rate at which it is
provided to the fusion system. The information rate is a function of many factors, including
the revisit rate of the sensors, the rate at which data sets are communicated, and also the qual-
ity of the communication link [93]. The effect of information rate is particularly important in
decentralized fusion settings where imperfect communication is common.
• Assessing the performance of the fusion system. The performance of fusion systems itself
is computed and compared using a specific set of measures referred to as measures of
performance (MOPs). The literature work on MOP is rather extensive and includes a wide
variety of measures. The choice of the specific MOP(s) of interest depends on the charac-
teristics of the fusion system. For instance, there is more to evaluate in a multiple-sensor
system than there is in a single-sensor system. Further, in the case of multitarget problems,
the data/track association part of the system also needs to be evaluated along with the
estimation part. The commonly used MOPs may be broadly categorized into the metrics
computed for each target and metrics computed over an ensemble of targets. Some of the
MOPs belonging to the former category are track accuracy, track covariance consistency,
track jitter, track estimate bias, track purity, and track continuity. Examples of measures
in the latter category are average number of missed targets, average number of extra tar-
gets, average track initiation time, completeness history, and cross-platform commonality
history [94,95]. There are also other less popular measures related to the discrimination
and/or classification capability of the fusion system that can be useful to collect in some
applications. Aside from the conventional approaches for performance measurement, there
is some notable work on development of MOPs for multitarget fusion systems within the
finite set theory framework [96,97]. The key observation is that a multitarget system is
fundamentally different from a single-target system. In the former case, the system state
is indeed a finite set of vectors rather than a single vector. This is due to the appearance/
disappearance of targets, which leads to the number of states varying with time. In addi-
tion, it is more natural to mathematically represent the collection of states as a finite set, as
the order in which the states are listed has no physical significance [98]. This approach is
especially useful in fusion applications in which the number of targets is not known and has
to be inferred along with their positions. Finally, it is worth pointing out some of the fusion
evaluation tools and testbeds that have recently become available. The Fusion Performance
Analysis (FPA) tool from Boeing is software that enables computation of technical per-
formance measures (TPMs) for virtually any fusion system. It is developed in Java (thus
is platform-independent) and implements numerous TPMs in three main categories: state
estimation, track quality, and discrimination [99]. Another interesting development is the
multisensor–multitarget tracking testbed [100], which has been lately introduced and is the
first step toward the realization of a state-of-the-art testbed for evaluation of large-scale
distributed fusion systems.
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-51-320.jpg)
![24 Multisensor Data Fusion
To the best of our knowledge, there is no standard and well-established evaluation framework to
assess the performance of data fusion algorithms. Most of the work is being done in simulation and
based on sometimes idealized assumption(s), which make it difficult to predict how the algorithm
would perform in real-life applications. A review of literature on data fusion performance evalu-
ation is presented in Ref. [101], where the challenging aspects of data fusion performance evalua-
tion, in practice, are discussed. Having analyzed more than 50 of the related literature work, it has
been shown that only very little (i.e., about 6%) of the surveyed research work treats the fusion
evaluation problem from a practical perspective. Indeed, it is demonstrated that most of the existing
work is focused on performing evaluation in simulation or unrealistic test environments, which is
substantially different from practical cases. Regarding the preceding discussions, there appears to
be a serious need for further research on development and standardizing measures of performance
applicable to the practical evaluation of data fusion systems.
2.5
NEW DIRECTIONS IN MULTISENSOR DATA FUSION
2.5.1 Social Data Fusion
The advent of social network services such as Facebook and Twitter has enabled users to share
their social data—pictures, videos, news, ideas—on the Web. The social data are highly rich in
content and thus provide an unprecedented opportunity for both researchers in social sciences and
practitioners in industry to study human behavior, identify customer preferences, and much more.
Within the context of data fusion, the conventional sensory data and the recently availability social
data can be deemed as mutually compensatory in numerous data fusion and processing applications
[102]. For instance, social network services can be leveraged in a participatory sensing manner, that
is, human as a social sensor [103], to collect data in areas where physical sensors are not available.
Similarly, data provided by conventional sensors can be used to construct context regarding the
available social data, thus enabling them to be analyzed more effectively. On the other hand, social
data are typically provided as streams of massive unstructured data. Accordingly, exploiting social
data in fusion applications involves tackling challenges in stream data processing, scalable and
distributed data storage and processing, and ability to model and interpret unstructured data. The
aforementioned advantages along with the inevitable theoretical and practical challenges has made
social data fusion a highly attractive and promising area of research within the fusion community,
as reflected in the plethora of recent publications [103–106].
2.5.2 Opportunistic Data Fusion
Regarding the limitations of traditional data fusion systems, which are designed mostly to use ded-
icated sensor and information resources, and the availability of new ubiquitous computing and
communication technologies, the opportunistic data fusion paradigm considers the possibility of
treating sensors as shared resources and performing fusion in an opportunistic manner [107]. New
challenging problems associated with such fusion systems are identified and novel approaches to
tackle them are explored. Some of the distinctions of the opportunistic information fusion model
(OIFM) compared to the conventional approach are the need for on-the-fly discovery of sensors, ad
hoc computational load, and dynamic (not predefined) fusion rules. The key enabling component
required to realize an OIFM is a new approach toward middleware development called opportunis-
tic middleware model (OMM). This is because the existing middleware platforms do not scale to the
device diversity, size, and runtime dynamics required by OIFM applications [107]. Unfortunately,
current specifications for the OMM do not address many issues related to its implementation and
thus future research is still needed to make OIFM viable. Nonetheless, some preliminary research
work is reported in the literature. For instance, in Ref. [108] an opportunistic fusion of data across
time, space, and feature level is performed in a visual sensor network to achieve human gesture
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-52-320.jpg)
![25
Review of the State of the Art and Overview of Emerging Trends
analysis. In Ref. [109], the authors study the problem of optimal camera placement in a visual sensor
network designed to serve multiple applications (each to be operated in an opportunistic manner).
The problem is formulated as a multiobjective optimization problem and solved efficiently using a
multiobjective genetic algorithm.
2.5.3 Adaptive Fusion and Learning
Early work on adaptive data fusion dates back to the early 1990s [110]. Nonetheless, this problem
has rarely been explored in the fusion literature until recently. Some of the existing work is focused
on incorporation of adaptivity into the KF algorithm. In Ref. [111] an adaptive fusion system capable
of intelligent allocation of limited resources is described that enables efficient tracking of moving
targets in three dimensions. An adaptive variant of KF called FL-AKF that relies on fuzzy infer-
ence based on covariance matching to adaptively estimate the covariance matrix of measurement
noise is proposed in Ref. [112]. In a similar approach, in Ref. [113] Tafti and Sadati present a novel
adaptive Kalman filter (NAKF) that achieves adaptation using a mathematical function termed
degree of matching (DoM), which is based on covariance matching. An adaptive UKF algorithm
with multiple fading factors-based gain correction is proposed and applied to the picosatellite atti-
tude estimation problem [114]. Another trend of work investigates explicit integration of machine
learning algorithms into the fusion process to accomplish adaptation. For example, machine learn-
ing methods are deployed in Ref. [115] to achieve online adaptation to users’ multimodal temporal
thresholds within a human–computer interaction application framework. Some other works study
application of reinforcement learning to adaptive fusion systems to perform dynamic data reliability
estimation [116,117]. Another research work also proposed using kernel-based learning methods to
achieve adaptive decision fusion rules [118].
2.5.4
Data Reliability and Trust
The majority of data fusion literature work is based on an optimistic assumption about the reli-
ability of underlying models producing the beliefs associated with imperfect data. For instance,
sensory data are commonly considered as equally reliable and play a symmetrical role in the fusion
process [119]. Nonetheless, different models usually have different reliabilities and are valid only
for a specific range. A recent trend in data fusion has addressed this issue mostly by attempting to
account for reliability of beliefs. This has been accomplished through introduction of the notion of
a second level of uncertainty, that is, uncertainty about uncertainty, represented as reliability coeffi-
cients. The main challenges are first to estimate these coefficients and then to incorporate them into
the fusion process. A number of approaches to estimate reliability coefficients have been proposed
that rely on domain knowledge and contextual information [120], learning through training [121],
possibility theory [122], and expert judgments [123]. Further, the problem of reliability incorpora-
tion has been studied within several fusion frameworks such as Dempster–Shafer theory [124],
fuzzy and possibility theory [125], transferable belief model [126], and probability theory [127].
Another research work also investigates the impact of belief reliability on high-level data fusion
[128]. The issue of reliability in data fusion is still not well established, and several open questions
such as interrelationship between reliabilities, reliability of heterogeneous data, and a comprehen-
sive architecture to manage data fusion algorithm and reliability of data sources remain part of
future research [119,124]. Taking research to the next level, more recent work attempts to address
the overarching issues of information quality and higher level quality [129].
2.5.5
Data Fusion in the Cloud and Big Data Fusion
Recent advances in cloud computing technologies have led to the availability of interesting new
capabilities for data fusion systems. Some of the most notable benefits of cloud-based computing
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-53-320.jpg)
![26 Multisensor Data Fusion
include scalable and flexible data storage and processing, while maintaining a high level of reli-
ability and security. Enabled by the power of the cloud, modern data fusion algorithms can now be
performed over vast amounts of entities across multiple databases [130]. Efficient implementation
of such big data fusion systems, however, requires attending to data management, system design,
and real-time execution. For instance, the Google fusion table can be deemed as a preliminary
cloud-enabled data management and fusion service that allows for uploading, sharing, filtering,
and visualization. In particular, it supports the fusion of data from multiple sources through join-
ing across tables sourced by different users [131]. The notion of cloud robotics, initially proposed
by James Kuffner at Google [132], is another example of the potential of cloud-enabled data fusion
and processing to revolutionize modern robotics. A cloud-enabled team of robots is capable of off-
loading expensive computational and/or storage robotic tasks, hence allowing users to focus on the
application at hand rather than worrying about the underlying IT infrastructure. A recent exemplary
work in cloud robotics is described in Ref. [133], where a holistic robotic system is able to leverage
cloud services to enhance the performance of video tracking. The experimental results demonstrate
the feasibility of offloading computation to the cloud, which is especially beneficial when there are
a large number of robot networks demanding image processing tasks.
2.5.6
Fusion of Data Streams
The so-called tidal wave of big data is typically characterized by its three-V properties: volume,
velocity, and variety. In particular, advances in mobile technologies have led to the proliferation of
numerous online data-intensive applications where data streams are being collected continuously
in large volume and high speed. When it comes to time-sensitive big data fusion applications that
involve processing such extremely large data feeds produced at high speeds by multiple sources,
the conventional static database technologies are not sufficient. Examples of such modern real-time
applications are traffic monitoring and management, stock price prediction, and flight schedule
checking. An alternative solution is to fuse data streams on-the-fly as soon as they are available.
Research work on to the stream data fusion problem is gradually gaining popularity. Schueller
and Behrend argue [134] in favor of the Reactive Programming (RE) paradigm and the Language
Integrated Query (LINQ) language as a promising solution to the problem of storing and fus-
ing real-time streams of data. Dynamic trust assessment over data-in-motion is another important
challenge, which has been addressed in Ref. [135]. Their proposal is perhaps the first attempt to
develop a dynamic trust assessment framework applicable to data streams with subjective logic as
the underlying computational toolset. In a related study, Zhao et al. present a probabilistic model to
transform the problem of truth discovery over data streams into a probabilistic inference problem
[136]. The proposed approach is claimed to possess advantages such as requiring only a single
pass over data, limited memory usage, and short response time, which are backed by preliminary
experimental results.
2.5.7 Low-Level versus High-Level Data Fusion
The discussion on high-level data fusion may appear to be outside the scope of this chapter. However,
as argued in Ref. [137], as soft human-generated data, in the form of complex natural language state-
ments, play an ever-increasing role in modern fusion systems, the clear distinction between low-
level and high-level fusion processes might no longer be applicable. The key observation is that the
interpretation and analysis of soft data necessitate development of complicated models not restricted
to the immediate time frame, similar to those used by the high-level data fusion processes. In partic-
ular, knowledge of the context within which a piece of soft data is uttered is crucial in our ability to
exploit soft data [138]. A similar line of thought is presented in Ref. [139], where Dragos examines
the main challenges involved in dealing with various forms of uncertainties potentially expressed
by soft data and the need for high-level ontological analysis to assess them properly.
© 2016 by Taylor Francis Group, LLC](https://image.slidesharecdn.com/2637782-250523060649-bd26fa7b/85/Multisensor-Data-Fusion-From-Algorithms-And-Architectural-Design-To-Applications-Fourati-54-320.jpg)
Multisensor Data Fusion From Algorithms And Architectural Design To Applications Fourati
- 1. Multisensor Data Fusion From Algorithms And Architectural Design To Applications Fourati download https://ebookbell.com/product/multisensor-data-fusion-from- algorithms-and-architectural-design-to-applications- fourati-5275564 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Handbook Of Multisensor Data Fusion Theory And Practice Second Edition Electrical Engineering Applied Signal Processing Series 2nd Edition Martin E Liggins https://ebookbell.com/product/handbook-of-multisensor-data-fusion- theory-and-practice-second-edition-electrical-engineering-applied- signal-processing-series-2nd-edition-martin-e-liggins-2418368 Uncertainty Theories And Multisensor Data Fusion 1st Edition Alain Appriou https://ebookbell.com/product/uncertainty-theories-and-multisensor- data-fusion-1st-edition-alain-appriou-4725634 Handbook Of Multisensor Data Fusion 1st Edition David L Hall https://ebookbell.com/product/handbook-of-multisensor-data-fusion-1st- edition-david-l-hall-922862 Multisensor Image Fusion And Data Mining For Environmental Remote Sensing First Edition Bai https://ebookbell.com/product/multisensor-image-fusion-and-data- mining-for-environmental-remote-sensing-first-edition-bai-6992734
- 3. Multisensor Fusion And Integration In The Wake Of Big Data Deep Learning And Cyber Physical System 1st Ed Sukhan Lee https://ebookbell.com/product/multisensor-fusion-and-integration-in- the-wake-of-big-data-deep-learning-and-cyber-physical-system-1st-ed- sukhan-lee-7150938 Multisensor Systems And Data Fusion In Remote Sensing Piotr Kaniewski https://ebookbell.com/product/multisensor-systems-and-data-fusion-in- remote-sensing-piotr-kaniewski-50655984 Multisensor Filtering Fusion With Censored Data Under A Constrained Network Environment Hang Geng https://ebookbell.com/product/multisensor-filtering-fusion-with- censored-data-under-a-constrained-network-environment-hang- geng-58455028 Multisensor Filtering Fusion With Censored Data Under A Constrained Network Environment Hang Geng Zidong Wang Yuhua Cheng https://ebookbell.com/product/multisensor-filtering-fusion-with- censored-data-under-a-constrained-network-environment-hang-geng- zidong-wang-yuhua-cheng-58466530 Advances And Challenges In Multisensor Data And Information Processing Volume 8 Nato Security Through Science Series Information And Communication Security D Sinformation And Communication Security Eric Lefebvre https://ebookbell.com/product/advances-and-challenges-in-multisensor- data-and-information-processing-volume-8-nato-security-through- science-series-information-and-communication-security-d-sinformation- and-communication-security-eric-lefebvre-2135690
- 5. From Algorithms and Architectural Design to Applications © 2016 by Taylor & Francis Group, LLC
- 6. Devices, Circuits, and Systems Series Editor Krzysztof Iniewski CMOS Emerging Technologies Research Inc., Vancouver, British Columbia, Canada PUBLISHED TITLES: Atomic Nanoscale Technology in the Nuclear Industry Taeho Woo Biological and Medical Sensor Technologies Krzysztof Iniewski Building Sensor Networks: From Design to Applications Ioanis Nikolaidis and Krzysztof Iniewski Circuits at the Nanoscale: Communications, Imaging, and Sensing Krzysztof Iniewski CMOS: Front-End Electronics for Radiation Sensors Angelo Rivetti Design of 3D Integrated Circuits and Systems Rohit Sharma Electrical Solitons: Theory, Design, and Applications David Ricketts and Donhee Ham Electronics for Radiation Detection Krzysztof Iniewski Electrostatic Discharge Protection of Semiconductor Devices and Integrated Circuits Juin J. Liou Embedded and Networking Systems: Design, Software, and Implementation Gul N. Khan and Krzysztof Iniewski Energy Harvesting with Functional Materials and Microsystems Madhu Bhaskaran, Sharath Sriram, and Krzysztof Iniewski Gallium Nitride (GaN): Physics, Devices, and Technology Farid Medjdoub Graphene, Carbon Nanotubes, and Nanostuctures: Techniques and Applications James E. Morris and Krzysztof Iniewski High-Speed Devices and Circuits with THz Applications Jung Han Choi © 2016 by Taylor & Francis Group, LLC
- 7. High-Speed Photonics Interconnects Lukas Chrostowski and Krzysztof Iniewski High Frequency Communication and Sensing: Traveling-Wave Techniques Ahmet Tekin and Ahmed Emira Integrated Microsystems: Electronics, Photonics, and Biotechnology Krzysztof Iniewski Integrated Power Devices and TCAD Simulation Yue Fu, Zhanming Li, Wai Tung Ng, and Johnny K.O. Sin Internet Networks: Wired, Wireless, and Optical Technologies Krzysztof Iniewski Labs on Chip: Principles, Design, and Technology Eugenio Iannone Laser-Based Optical Detection of Explosives Paul M. Pellegrino, Ellen L. Holthoff, and Mikella E. Farrell Low Power Emerging Wireless Technologies Reza Mahmoudi and Krzysztof Iniewski Medical Imaging: Technology and Applications Troy Farncombe and Krzysztof Iniewski Metallic Spintronic Devices Xiaobin Wang MEMS: Fundamental Technology and Applications Vikas Choudhary and Krzysztof Iniewski Micro- and Nanoelectronics: Emerging Device Challenges and Solutions Tomasz Brozek Microfluidics and Nanotechnology: Biosensing to the Single Molecule Limit Eric Lagally MIMO Power Line Communications: Narrow and Broadband Standards, EMC, and Advanced Processing Lars Torsten Berger, Andreas Schwager, Pascal Pagani, and Daniel Schneider Mixed-Signal Circuits Thomas Noulis Mobile Point-of-Care Monitors and Diagnostic Device Design Walter Karlen Multisensor Data Fusion: From Algorithm and Architecture Design to Applications Hassen Fourati Nano-Semiconductors: Devices and Technology Krzysztof Iniewski PUBLISHED TITLES: © 2016 by Taylor & Francis Group, LLC
- 8. Nanoelectronic Device Applications Handbook James E. Morris and Krzysztof Iniewski Nanopatterning and Nanoscale Devices for Biological Applications Šeila Selimovic´ Nanoplasmonics: Advanced Device Applications James W. M. Chon and Krzysztof Iniewski Nanoscale Semiconductor Memories: Technology and Applications Santosh K. Kurinec and Krzysztof Iniewski Novel Advances in Microsystems Technologies and Their Applications Laurent A. Francis and Krzysztof Iniewski Optical, Acoustic, Magnetic, and Mechanical Sensor Technologies Krzysztof Iniewski Optical Fiber Sensors: Advanced Techniques and Applications Ginu Rajan Optical Imaging Devices: New Technologies and Applications Ajit Khosla and Dongsoo Kim Organic Solar Cells: Materials, Devices, Interfaces, and Modeling Qiquan Qiao Radiation Detectors for Medical Imaging Jan S. Iwanczyk Radiation Effects in Semiconductors Krzysztof Iniewski Reconfigurable Logic: Architecture, Tools, and Applications Pierre-Emmanuel Gaillardon Semiconductor Radiation Detection Systems Krzysztof Iniewski Smart Grids: Clouds, Communications, Open Source, and Automation David Bakken Smart Sensors for Industrial Applications Krzysztof Iniewski Soft Errors: From Particles to Circuits Jean-Luc Autran and Daniela Munteanu Solid-State Radiation Detectors: Technology and Applications Salah Awadalla Technologies for Smart Sensors and Sensor Fusion Kevin Yallup and Krzysztof Iniewski Telecommunication Networks Eugenio Iannone PUBLISHED TITLES: © 2016 by Taylor & Francis Group, LLC
- 9. Testing for Small-Delay Defects in Nanoscale CMOS Integrated Circuits Sandeep K. Goel and Krishnendu Chakrabarty VLSI: Circuits for Emerging Applications Tomasz Wojcicki Wireless Technologies: Circuits, Systems, and Devices Krzysztof Iniewski Wireless Transceiver Circuits: System Perspectives and Design Aspects Woogeun Rhee FORTHCOMING TITLES: Advances in Imaging and Sensing Shuo Tang, Dileepan Joseph, and Krzysztof Iniewski Analog Electronics for Radiation Detection Renato Turchetta Cell and Material Interface: Advances in Tissue Engineering, Biosensor, Implant, and Imaging Technologies Nihal Engin Vrana Circuits and Systems for Security and Privacy Farhana Sheikh and Leonel Sousa CMOS Time-Mode Circuits and Systems: Fundamentals and Applications Fei Yuan Ionizing Radiation Effects in Electronics: From Memories to Imagers Marta Bagatin and Simone Gerardin Magnetic Sensors: Technologies and Applications Kirill Poletkin MRI: Physics, Image Reconstruction, and Analysis Angshul Majumdar and Rabab Ward Multisensor Attitude Estimation: Fundamental Concepts and Applications Hassen Fourati and Djamel Eddine Chouaib Belkhiat Nanoelectronics: Devices, Circuits, and Systems Nikos Konofaos Nanomaterials: A Guide to Fabrication and Applications Sivashankar Krishnamoorthy and Gordon Harling Physical Design for 3D Integrated Circuits Aida Todri-Sanial and Chuan Seng Tan Power Management Integrated Circuits and Technologies Mona M. Hella and Patrick Mercier Radio Frequency Integrated Circuit Design Sebastian Magierowski PUBLISHED TITLES: © 2016 by Taylor & Francis Group, LLC
- 10. FORTHCOMING TITLES: Silicon on Insulator System Design Bastien Giraud Semiconductor Devices in Harsh Conditions Kirsten Weide-Zaage and Malgorzata Chrzanowska-Jeske Smart eHealth and eCare Technologies Handbook Sari Merilampi, Lars T. Berger, and Andrew Sirkka Structural Health Monitoring of Composite Structures Using Fiber Optic Methods Ginu Rajan and Gangadhara Prusty Terahertz Sensing and Imaging: Technology and Devices Daryoosh Saeedkia and Wojciech Knap Tunable RF Components and Circuits: Applications in Mobile Handsets Jeffrey L. Hilbert Wireless Medical Systems and Algorithms: Design and Applications Pietro Salvo and Miguel Hernandez-Silveira © 2016 by Taylor & Francis Group, LLC
- 11. CRC Press is an imprint of the Taylor & Francis Group, an informa business Boca Raton London NewYork EDITED BY Hassen Fourati GIPSA-LAB DEPARTMENT OF CONTROL SYSTEMS UNIVERSITY GRENOBLE ALPES GRENOBLE, FRANCE Krzysztof Iniewski MANAGING EDITOR CMOS EMERGING TECHNOLOGIES RESEARCH INC. VANCOUVER, BRITISH COLUMBIA, CANADA From Algorithms and Architectural Design to Applications © 2016 by Taylor & Francis Group, LLC
- 12. MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does not warrant the accuracy of the text or exercises in this book. This book’s use or discussion of MATLAB® software or related products does not constitute endorsement or sponsorship by The MathWorks of a particular pedagogical approach or particular use of the MATLAB® software. CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2016 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Version Date: 20150309 International Standard Book Number-13: 978-1-4822-6375-6 (eBook - PDF) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid- ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or uti- lized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopy- ing, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http:// www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com © 2016 by Taylor & Francis Group, LLC
- 13. Dedicated to my wife, Emna, whose Zen-like patience continues to amaze; to my parents, without whom I would not be where I am today. Hassen Fourati © 2016 by Taylor & Francis Group, LLC
- 14. © 2016 by Taylor & Francis Group, LLC
- 15. xi Contents Preface..............................................................................................................................................xv Editors.............................................................................................................................................xvii Contributors.....................................................................................................................................xix Section I Novel Advances in Multisensor Data Fusion Algorithm Design Chapter 1 Challenges in Information Fusion Technology Capabilities for Modern Intelligence and Security Problems...............................................................................3 James Llinas Chapter 2 Multisensor Data Fusion: A Data-Centric Review of the State of the Art and Overview of Emerging Trends.............................................................................15 Bahador Khaleghi, Alaa Khamis, and Fakhri Karray Chapter 3 Information Fusion: Theory at Work..........................................................................35 Jean-François Grandin Chapter 4 JDL Model (III) Updates for an Information Management Enterprise......................55 Erik Blasch Chapter 5 Elements of Random Set Information Fusion.............................................................75 Ronald Mahler Chapter 6 Optimal Fusion for Dynamic Systems with Process Noise........................................89 Chee-Yee Chong and Shozo Mori Chapter 7 A Fuzzy Multicriteria Approach for Data Fusion.....................................................109 André D. Mora, António J. Falcão, Luís Miranda, Rita A. Ribeiro, and José M. Fonseca Chapter 8 Distributed Detection and Data Fusion with Heterogeneous Sensors......................127 Satish G. Iyengar, Hao He, Arun Subramanian, Ruixin Niu, Pramod K. Varshney, and Thyagaraju Damarla Chapter 9 Fusion Systems Evaluation: An Information Quality Perspective............................ 147 Ion-George Todoran, Laurent Lecornu, Ali Khenchaf, and Jean-Marc Le Caillec © 2016 by Taylor Francis Group, LLC
- 16. xii Contents Chapter 10 Sensor Failure Robust Fusion.................................................................................... 157 Matt Higger, Murat Akcakaya, Umut Orhan, and Deniz Erdogmus Chapter 11 Treatment of Dependent Information in Multisensor Kalman Filtering and Data Fusion......................................................................................................... 169 Benjamin Noack, Joris Sijs, Marc Reinhardt, and Uwe D. Hanebeck Chapter 12 Cubature Information Filters: Theory and Applications to Multisensor Fusion....... 193 Ienkaran Arasaratnam and Kumar Pakki Bharani Chandra Chapter 13 Estimation Fusion for Linear Equality Constrained Systems...................................207 Zhansheng Duan and X. Rong Li Chapter 14 Nonlinear Information Fusion Algorithm of an Asynchronous Multisensor Based on the Cubature Kalman Filter.......................................................................223 Wei Gao, Ya Zhang, and Qian Sun Chapter 15 The Analytic Implementation of the Multisensor Probability Hypothesis Density Filter.............................................................................................................235 Fangming Huang, Kun Wang, Jian Xu, and Zhiliang Huang Chapter 16 Information Fusion Estimation for Multisensor Multirate Systems with Multiplicative Noises.........................................................................................253 Shuli Sun, Jing Ma, and Fangfang Peng Chapter 17 Optimal Distributed Kalman Filtering Fusion with Singular Covariances of Filtering Errors and Measurement Noises............................................................267 Enbin Song Chapter 18 Accumulated State Densities and Their Applications in Object Tracking...............295 Wolfgang Koch Chapter 19 Belief Function Based Multisensor Multitarget Classification Solution................... 331 Samir Hachour, François Delmotte, and David Mercier Chapter 20 Decision Fusion in Cognitive Wireless Sensor Networks.........................................349 Andrea Abrardo, Marco Martalò, and Gianluigi Ferrari Chapter 21 Dynamics of Consensus Formation among Agent Opinions....................................363 Thanuka Wickramarathne, Kamal Premaratne, and Manohar Murthi © 2016 by Taylor Francis Group, LLC
- 17. xiii Contents Chapter 22 Decentralized Bayesian Fusion in Networks with Non-Gaussian Uncertainties......383 Nisar R. Ahmed, Simon J. Julier, Jonathan R. Schoenberg, and Mark E. Campbell Chapter 23 Attack-Resilient Sensor Fusion for CPS....................................................................409 Radoslav Ivanov, Miroslav Pajic, and Insup Lee Section II Multisensor Data Fusion Showcases Advancements Chapter 24 Multisensor Data Fusion for Water Quality Evaluation Using Dempster–Shafer Evidence Theory.......................................................................................................425 Zhou Jian Chapter 25 A Granular Sensor-Fusion Method for Regenerative Life Support Systems............ 435 Gregorio E. Drayer and Ayanna M. Howard Chapter 26 Evaluating Image Fusion Performance: From Metrics to Cognitive Assessment..... 453 Zheng Liu and Erik Blasch Chapter 27 A Review of Feature and Data Fusion with Medical Images................................... 491 Alex Pappachen James and Belur V. Dasarathy Chapter 28 Multisensor Data Fusion: Architecture Design and Application in Physical Activity Assessment..................................................................................................509 Shaopeng Liu and Robert X. Gao Chapter 29 Data Fusion for Attitude Estimation of a Projectile: From Theory to In-Flight Demonstration........................................................................................................... 519 Sébastien Changey and Emmanuel Pecheur Chapter 30 Data Fusion for Telemonitoring: Application to Health and Autonomy................... 535 Céline Franco, Nicolas Vuillerme, Bruno Diot, Jacques Demongeot, and Anthony Fleury Chapter 31 Sensor Data Fusion for Automotive Systems............................................................549 Max Mauro Dias Santos Chapter 32 Data Fusion in Intelligent Traffic and Transportation Engineering: Recent Advances and Challenges.............................................................................563 Nour-Eddin El Faouzi and Lawrence A. Klein © 2016 by Taylor Francis Group, LLC
- 18. xiv Contents Chapter 33 Application of Multisensor Data Fusion for Traffic Congestion Analysis................595 Shrikant Fulari, Lelitha Vanajakshi, Shankar C. Subramanian, and T. Ajitha Chapter 34 Consensus-Based Decentralized Extended Kalman Filter for State Estimation of Large-Scale Freeway Networks............................................................................ 611 Liguo Zhang Index...............................................................................................................................................623 © 2016 by Taylor Francis Group, LLC
- 19. xv Preface The technology of multisensor data fusion seeks to combine information coming from multiple and different sources and sensors, resulting in an enhanced overall system performance with respect to separate sensors and sources. Multisensor data fusion has gained in importance over the last decades and found applications in an impressive variety of areas within diverse disciplines: naviga- tion, sensor networks, intelligent transportation systems, security, medical diagnosis, biometrics, environmental monitoring, remote sensing, measurements, robotics, and so forth. Different con- cepts, techniques, and architectures have been developed to optimize the overall system output in applications for which sensor fusion might be useful and enables development of concrete solutions. The idea for this book arose as a response to the immense interest and strong activities in the field of multisensor data fusion during the last few years, both in theoretical and practical aspects. This book is targeted toward researchers, academics, engineers, and graduate students working in the field of sensor fusion, estimation and observation, filtering, and signal processing. This book captures the latest data fusion concepts and techniques drawn from a broad array of disciplines. With contributions from the world’s leading fusion researchers and academicians, this book has 34 chapters, divided roughly into two sections, and covers the fundamental theory and recent theoretical advances, as well as showcasing applications of multisensor data fusion. Each chapter is complete in itself and can be read in isolation or in conjunction with other chapters of the book. Chapters 1 through 23 in Section I are devoted to the state of the art and novel advances in multisensor data fusion algorithm design. New materials and achievements in optimal fusion and multisensor filters are provided. In Section II, Chapters 24 through 34 mostly showcase multisensor data fusion advancements in fields such as medical applications, navigation, traffic analysis, and so on. We are grateful to all the contributors for sharing their valuable knowledge and we expect this book to offer a good balance between academic and industrial research throughout the different chapters. We sincerely hope that this book will be a source of inspiration for new concepts and applications and stimulate further the development of data fusion architecture. We would also like to acknowledge CRC Press and its staff for technical and editorial assistance that improved the quality of this book and resulted in its publication. Finally, we hope readers will enjoy this book and that it will prove to be a useful addition to the increasingly important and expanding field of data fusion. Hassen Fourati Univ. Grenoble Alpes, Gipsa-Lab, F-38000 Grenoble, France CNRS, Gipsa-Lab, F-38000 Grenoble, France Inria, Grenoble, France MATLAB® is a registered trademark of The MathWorks, Inc. For product information, please contact: The MathWorks, Inc. 3 Apple Hill Drive Natick, MA 01760-2098 USA Tel: 508 647 7000 Fax: 508-647-7001 E-mail: info@mathworks.com Web: www.mathworks.com © 2016 by Taylor Francis Group, LLC
- 20. © 2016 by Taylor Francis Group, LLC
- 21. xvii Editors Hassen Fourati, PhD, is currently an associate professor in the Electrical Engineering and Computer Science Department at the University Grenoble Alpes, Grenoble, France and a mem- ber of the Networked Controlled Systems Team (NeCS), affiliated with the Automatic Control Department of the Laboratoire Grenoble Images Parole Signal Automatique (GIPSA-LAB) and the Institut National de Recherche en Informatique et en Automatique (INRIA). He received the B.Eng. in Electrical Engineering from the National Engineering School of Sfax, Tunisia, a MS in Automated Systems and Control from the University of Claude Bernard, Lyon, France, and a PhD in Automatic Control from the University of Strasbourg, France, in 2006, 2007, and 2010, respectively. His research interests include nonlinear filtering, estimation, and multi sensor fusion with applications in navigation, inertial and magnetic sensors, robotics, and traffic management. He has published several research journal articles, papers in international conferences, and book chapters. He can be reached at hassen.fourati@gipsa-lab.fr. Krzysztof (Kris) Iniewski is managing RD at Redlen Technologies Inc., a start-up company in Vancouver, Canada. Redlen’s revolutionary production process for advanced semiconductor materi- als enables a new generation of more accurate, all-digital, radiation-based imaging solutions. Kris is also a president of CMOS Emerging Technologies Research Inc. (www.cmosetr.com), an organi- zation of high-tech events covering communications, microsystems, optoelectronics, and sensors. In his carrier, Dr. Iniewski held numerous faculty and management positions at University of Toronto, University of Alberta, SFU, and PMC-Sierra Inc. He has published over 100 research papers in international journals and conferences. He holds 18 international patents granted in the United States, Canada, France, Germany, and Japan. He is a frequently invited speaker and has consulted for multiple organizations internationally. He has written and edited several books for CRC Press, Cambridge University Press, IEEE Press, Wiley, McGraw-Hill, Artech House, and Springer. His personal goal is to contribute to healthy living and sustainability through innovative engineering solutions. In his leisurely time, Kris can be found hiking, sailing, skiing, or biking in beautiful British Columbia. He can be reached at kris.iniewski@gmail.com. © 2016 by Taylor Francis Group, LLC
- 22. © 2016 by Taylor Francis Group, LLC
- 23. xix Andrea Abrardo University of Siena Siena, Italy abrardo@dii.unisi.it Nisar R. Ahmed University of Colorado Boulder Boulder, Colorado, USA Nisar.Ahmed@Colorado.EDU T. Ajitha Indian Institute of Technology Madras Chennai, Tamilnadu, India tajitha98@gmail.com Murat Akcakaya University of Pittsburgh Pittsburgh, Pennsylvania, USA akcakaya@pitt.edu Ienkaran Arasaratnam Apple Inc. Cupertino, California, USA haran@ieee.org Erik Blasch Air Force Research Laboratory Rome, New York, USA erik.blasch@us.af.mil Jean-Marc Le Caillec Telecom Bretagne Brest Cedex 3, France jm.lecaillec@telecom-bretagne.eu Mark E. Campbell Cornell University Ithaca, New York, USA mc288@cornell.edu Kumar Pakki Bharani Chandra University of Exeter Exeter, UK b.c.k.pakki@exeter.ac.uk Sébastien Changey GNC Department ISL–French-German Research Institute of Saint-Louis Saint-Louis Cedex, France sebastien.changey@isl.eu Chee-Yee Chong Independent Consultant Los Altos, California, USA cychong@ieee.org Thyagaraju Damarla US Army Research Laboratory Adelphi, Maryland, USA thyagaraju.damarla.civ@mail.mil Belur V. Dasarathy Independent Consultant Huntsville, Alabama, USA fusion-consultant@ieee.org François Delmotte Université Lille Nord de France Béthune, France francois.delmotte@univ-artois.fr Jacques Demongeot Laboratoire AGIM Université Grenoble-Alpes La Tronche, Grenoble, France and Institut Universitaire de France Paris, France and Mines Douai IA Douai, France jacques.demongeot@yahoo.fr Contributors © 2016 by Taylor Francis Group, LLC
- 24. xx Contributors Bruno Diot Laboratoire AGIM Université Grenoble-Alpes Grenoble, France and IDS Montceau-les-Mines, France bruno.diot@ids-assistance.com Gregorio E. Drayer Georgia Institute of Technology Atlanta, Georgia, USA drayer@gatech.edu Zhansheng Duan Center for Information Engineering Science Research Xi’an Jiaotong University Xi’an, China zduan@uno.edu Deniz Erdogmus Northeastern University Boston, Massachusetts, USA erdogmus@ece.neu.edu António J. Falcão Uninova-CA3 Monte da Caparica, Portugal ajf@uninova.pt Nour-Eddin El Faouzi Transport and Traffic Engineering Laboratory Bron, France and ENTPE LICIT Vaulx-en-Velin, France and University of Lyon Lyon, France nour-eddin.elfaouzi@ifsttar.fr Gianluigi Ferrari University of Parma Parma, Italy gianluigi.ferrari@unipr.it Anthony Fleury Mines Douai IA Douai, France anthony.fleury@mines-douai.fr José M. Fonseca Uninova-CA3 Monte da Caparica, Portugal jmf@uninova.pt Céline Franco Laboratoire AGIM Université Grenoble-Alpes La Tronche, Grenoble, France and Institut Universitaire de France Paris, France and Mines Douai IA Douai, France celine.franco@imag.fr Shrikant Fulari Indian Institute of Technology Madras Chennai, India shrikant.f@gmail.com Robert X. Gao Department of Mechanical and Aerospace Engineering Case Western Reserve University Cleveland, Ohio, USA robert.gao@case.edu Wei Gao College of Automation Harbin Engineering University Harbin, China gaow@hrbeu.edu.cn Jean-François Grandin THALES Systèmes Aéroportés Elancourt, France jean-francois.grandin@fr.thalesgroup.com © 2016 by Taylor Francis Group, LLC
- 25. xxi Contributors Samir Hachour Université Lille Nord de France Béthune, France samirhachour@yahoo.fr Uwe D. Hanebeck Karlsruhe Institute of Technology (KIT) Karlsruhe, Germany uwe.hanebeck@ieee.org Hao He Department of EECS Syracuse University Syracuse, New York, USA hhe02@syr.edu Matt Higger Northeastern University Boston, Massachusetts, USA higger@ece.neu.edu Ayanna M. Howard Georgia Institute of Technology Atlanta, Georgia, USA ayanna.howard@ece.gatech.edu Fangming Huang Nanjing Research Institute of Electronics Engineering Nanjing, China HFM3000@sina.com Zhiliang Huang Nanjing Research Institute of Electronics Engineering Nanjing, China zhiliangh28@163.com Radoslav Ivanov University of Pennsylvania Philadelphia, Pennsylvania, USA rivanov@seas.upenn.edu Satish G. Iyengar General Electric Global Research Corporation Niskayuna, New York, USA iyengar@ge.com Alex Pappachen James Department of Electrical and Electronics Engineering Nazarbayev University Astana, Kazakhstan apj@ieee.org Zhou Jian College of Computer Nanjing University of Posts and Telecommunications Nanjing, China zhoujian@njupt.edu.cn Simon J. Julier Department of Computer Science University College of London London, UK s.julier@cs.ucl.ac.uk Fakhri Karray Department of Electrical and Computer Engineering University of Waterloo Waterloo, Ontario, Canada karray@uwaterloo.ca Bahador Khaleghi IMS Inc. Waterloo, Ontario, Canada bkhalegh@uwaterloo.ca Alaa Khamis Vestec Inc. and Suez University Waterloo, Ontario, Canada akhamis@pami.uwaterloo.ca Ali Khenchaf ENSTA Bretagne Brest Cedex 9, France ali.khenchaf@ensta-bretagne.fr Lawrence A. Klein Klein Associates Santa Ana, California, USA larry@laklein.com © 2016 by Taylor Francis Group, LLC
- 26. xxii Contributors Wolfgang Koch Fraunhofer/University of Bonn Wachtberg, Germany wolfgang.koch@fkie.fraunhofer.de Laurent Lecornu Telecom Bretagne Brest Cedex 3, France Laurent.lecornu@telecom-bretagne.eu Insup Lee University of Pennsylvania Philadelphia, Pennsylvania, USA lee@cis.upenn.edu X. Rong Li Department of Electrical Engineering University of New Orleans New Orleans, Louisiana, USA xli@uno.edu Shaopeng Liu Distributed Intelligent Systems Lab GE Global Research Niskayuna, New York, USA victorlsp@gmail.com Zheng Liu Toyota Technological Institute Nagoya, Japan zheng.liu@ieee.org James Llinas Center for Multisource Information Fusion University at Buffalo Buffalo, New York, USA llinas@buffalo.edu Jing Ma Department of Automation Heilongjiaang University Harbin, China jingma427@163.com Ronald Mahler Intelligent Robotics Laboratory Lockheed Martin Advanced Technology Laboratories Eagan, Minnesota, USA MahlerRonald@comcast.net Marco Martalò University of Parma Parma, Italy and E-Campus University Novedrate (CO), Italy marco.martalo@unipr.it David Mercier Université Lille Nord de France Béthune, France david.mercier@univ-artois.fr Luís Miranda Uninova-CA3 Monte da Caparica, Portugal lmm@ca3-uninova.org André D. Mora Uninova-CA3 Monte da Caparica, Portugal atm@uninova.pt Shozo Mori Systems Technology Research Sunnyvale, California, USA shozo.mori@stresearch.com Manohar Murthi University of Miami Coral Gables, Florida, USA mmurthi@miami.edu Benjamin Noack Karlsruhe Institute of Technology (KIT) Karlsruhe, Germany benjamin.noack@ieee.org Ruixin Niu Department of Electrical and Computer Engineering Virginia Commonwealth University Richmond, Virginia, USA rniu@vcu.edu Umut Orhan Honeywell Aerospace Redmond, Washington, USA uorhan@cu.edu.tr © 2016 by Taylor Francis Group, LLC
- 27. xxiii Contributors Miroslav Pajic University of Pennsylvania Philadelphia, Pennsylvania, USA pajic@seas.upenn.edu Emmanuel Pecheur GNC Department ISL–French-German Research Institute of Saint-Louis Saint-Louis, France emmanuel.pecheur@isl.eu Fangfang Peng Department of Automation Heilongjiaang University Harbin, China pengfangfang2013@163.com Kamal Premaratne University of Miami Coral Gables, Florida, USA kamal@miami.edu Marc Reinhardt Karlsruhe Institute of Technology (KIT) Karlsruhe, Germany marc.reinhardt@ieee.org Rita A. Ribeiro Uninova-CA3 Monte da Caparica, Portugal rar@uninova.pt Max Mauro Dias Santos Department of Electronics Federal University of Technology–Paraná (UTFPR) Ponta Grossa, Brazil maxsantos@utfpr.edu.br Jonathan R. Schoenberg Arzentech, Inc. Fishers, Indiana, USA jon@arzentech.com Joris Sijs TNO Technical Sciences The Hague, The Netherlands joris.sijs@tno.nl Enbin Song College of Mathematics Sichuan University Chengdu, China e.b.song@163.com Arun Subramanian Department of EECS Syracuse University Syracuse, New York, USA arsubram@syr.edu Shankar C. Subramanian Indian Institute of Technology Madras Chennai, India shankarram@iitm.ac.in Qian Sun College of Automation Harbin Engineering University Harbin, China sunsl@hlju.edu.cn Shuli Sun Department of Automation Heilongjiaang University Harbin, China sunsl@hlju.edu.cn Ion-George Todoran Telecom Bretagne Brest Cedex 3, France iongeorge.todoran@telecom-bretagne.eu Lelitha Vanajakshi Indian Institute of Technology Madras Chennai, India lelitha@iitm.ac.in Pramod K. Varshney Department of EECS Syracuse University Syracuse, New York, USA varshney@syr.edu © 2016 by Taylor Francis Group, LLC
- 28. xxiv Contributors Nicolas Vuillerme Laboratoire AGIM Université Grenoble-Alpes Grenoble, France and Institut Universitaire de France Paris, France nicolas.vuillerme@agim.eu Kun Wang Nanjing Research Institute of Electronics Engineering Nanjing, China kun.wang1981@gmail.com Thanuka Wickramarathne University of Notre Dame Notre Dame, Indiana, USA twickram@nd.edu Jian Xu Nanjing Research Institute of Electronics Engineering Nanjing, China xujian2001-1@163.com Liguo Zhang School of Electronic Information and Control Engineering Beijing University of Technology Beijing, China zhangliguo@bjut.edu.cn Ya Zhang College of Automation Harbin Engineering University Harbin, China yzhang@hrbeu.edu.cn © 2016 by Taylor Francis Group, LLC
- 29. Section I Novel Advances in Multisensor Data Fusion Algorithm Design © 2016 by Taylor Francis Group, LLC
- 30. © 2016 by Taylor Francis Group, LLC
- 31. 3 1 Challenges in Information Fusion Technology Capabilities for Modern Intelligence and Security Problems James Llinas 1.1 HETEROGENEITY OF SUPPORTING INFORMATION Experiences in dealing with intelligence and security problems in Iraq and Afghanistan and other places in the world have required the (ongoing) formulation of new paradigms of intelligence analy sis and dynamic decision making. Broadly, these problems fall into the categories of counter terrorism and counterinsurgency (COIN) as well as stability operations. Depending on the phases of COIN or other operations, the nature of decision making ranges from conventional military-like to sociopolitical. Because of this wide spectrum of action, the nature of information support required for analysis has an equally wide range. As automated information fusion (IF) processes provide some of the support to such decision making, requirements for IF process design must address these varying requirements, resulting in considerable challenges in IF process design. 1.1.1 Observational Data Further, these experiences have also shown that some of the key observational and intelligence data in such operations come not only from traditional sensor systems, but also from dismounted soldiers or other human observers reporting on their patrol activities. These data are naturally com municated in language in the form of various military and intelligence reports and messages. Such “soft” data finds its way into IF processes as both structured and unstructured digitized text, and CONTENTS 1.1 Heterogeneity of Supporting Information.................................................................................3 1.1.1 Observational Data........................................................................................................3 1.1.2 Open Source and Social Media Data.............................................................................4 1.1.3 Contextual Data.............................................................................................................4 1.1.4 Ontological Data............................................................................................................5 1.1.5 Learned Information......................................................................................................5 1.2 Common Referencing and Data Association.............................................................................6 1.3 Semantics...................................................................................................................................7 1.4 Graphical Representations and Methods...................................................................................8 1.5 Overall System Architectures and Analysis Frameworks.........................................................9 1.6 Capability Shortfalls and Research Needs..............................................................................12 1.7 Conclusion...............................................................................................................................13 Acknowledgment..............................................................................................................................13 References.........................................................................................................................................13 © 2016 by Taylor Francis Group, LLC
- 32. 4 Multisensor Data Fusion this input modality creates new challenges to IF process designs, contrasted with more traditional IF applications involving the use of highly calibrated, numerically precise observational data from sensors. Combined with the data from the usual repertoire of “hard” or sensor data from various radio frequency (RF) sensors, video and other imaging systems, as well as signals intelligence (SIGINT) and satellite imagery, the observational data stream is a composite of data of highly dif ferent quality, sampling rates, content, and structure. 1.1.2 Open Source and Social Media Data Soft or hard data can also find their way into modern IF processes in the form of monitored open source and social media feeds such as newswire feeds, Twitter, and blog sources judged to be pos sibly helpful. Getting such data into an IF system will require automated Web crawlers and related capabilities, as well as subsequent natural language processing capabilities. 1.1.3 Contextual Data Modern problems also afford (and demand) the use of additional data and information beyond just observational data. A major category of such data and information is Contextual Information (CI). CI is that information that can be said to “surround” a situation of interest in the world (many defini tions and characterizations exist but we do not address such issues here). It is information that aids in understanding the (estimated) situation and also in reacting to the situation, if a reaction is required. CI can be relatively or fully static or can be dynamic, possibly changing along the same timeline as the situation (e.g., weather). It is also likely that it may not be possible to know the full characteriza tion and specification of CI at system/algorithm design time, except in very closed worlds. Thus, we envision an “a priori” framework of exploitation of CI that attempts to account for the effects on situational estimation of that CI that is known at design time. Even if such effects are known at design time, there is a question of the ease or difficulty involved in integrating CI effects into a fusion system design or into any algorithm designs. This issue is influenced in part by the nature of the CI and the manner of its native representation, for example, as numeric or symbolic, and the nature of the corresponding algorithm; for example, cases can arise that involve integrating symbolic CI into a numeric algorithm. Strategies for a priori exploitation of CI may thus require the invention of new hybrid methods that incorporate whatever information an algorithm normally employs in estimation (usually observational data) with an adjunct CI exploitation process. Note too that CI may, like obser vational data, have errors and inconsistencies itself, and accommodation of such errors is a consider ation for hybrid algorithm design. Similarly, we envision the need for an “a posteriori” CI exploitation process, owing to at least two factors: (1) that all relevant CI may not be able to be known at system/ algorithm design time and may have to be searched for and discovered at runtime, as a function of the current situation estimate, and (2) that such CI may not be of a type that was integrated into the system/algorithm designs at design time and so may not be able to be easily integrated into the situation estimation process. In this case we then envision that at least part of the job of a posteriori CI exploitation would involve checking the consistency of a current situational hypothesis with the newly discovered (and situationally relevant) CI. There are yet other system engineering issues. The first is the question of accessibility; CI must be accessible to use it, but accessibility may not be a straightforward matter in all cases. One question is whether the most current CI is available; another may be that some CI is controlled or secure and may have limited availability. The other question is one of representational form. CI data can be expected to be of a type that has been created by “native” users; for example, weather data, important in many fusion applications as CI, are generated by meteorologists, for meteorologists (not for fusion system designers). Thus, even if these data are available, there is likely to be a need for a “middleware” layer that incorporates some logic and algorithms both to sample these data and shape them into a form suitable for use in fusion processes. In even simpler cases, this middleware may be required to © 2016 by Taylor Francis Group, LLC
- 33. 5 IF Technology Capabilities for Modern Security Problems reformat the data from some native form to a usable one. In spite of some a priori mapping of how CI influences or constrains the way in which situational inferences or estimates can be developed, which may serve certain environments, the defense and security type applications, with their various dynamic and uncertain types of CI, demand a more adaptive approach. Given a nominated situational hypothesis Hf from a fusion process or “engine,” the first question is: What CI type information is relevant to this hypothesis? Relevant CI is only that information that influences our interpretation or understanding of Hf. Presuming a “relevancy filter” can be crafted, a search function would explore the available CI and make this CI available to an “a posteriori” reasoning engine. That reasoning engine would then use (1) a CI-guided subset of Domain Knowledge and (2) the retrieved CI to reason over Hf to first determine consistency of Hf with the relevant CI. If it is inconsistent, then some type of adjudication logic will need to be applied to reconcile this inconsistency between (1) the fusion process that produced Hf and (2) the a posteriori reasoning process that judges it as inconsistent. If, however, Hf is judged as consistent with the additional CI, an expanded interpretation of Hf could be developed, providing a deeper situational understanding. This overall process, which can be consid ered a “Process Refinement” operation, would be a so-called “Level 4” process in the context of the Joint Directors of Laboratories (JDL) Data Fusion Process Model [1], that is, as an adaptive operation for fusion process enhancement. The overall ideas discussed here are elaborated in Ref. [2]. 1.1.4 Ontological Data IF processes and algorithms historically have been developed in a framework that has assumed the a priori availability of a reliable body of procedural and dynamic knowledge about the problem domain, that is, knowledge that supports a more direct approach to temporal reasoning about the unfolding patterns of interest in the problem domain. In COIN and other complex problems, such a priori and reliable knowledge is most often not available—the Tactics, Techniques, and Procedures of modern-day adversaries are highly adaptive and extremely hard to model with confidence. The US DARPA COMPOEX Program [3] attempted to develop such models but achieved only partial success, experiencing gaps in the overall modeling space of such desired behavioral models. We label these types of problems as “weak knowledge” problems, implying that only fragmentary a priori behavioral model type knowledge is available to aid in IF-based reasoning, inferencing, and estimation. Ontological information, however, that does not attempt to form such comprehensive behavioral and temporal models overtly but does include temporal primitives along with structural/syntactic relations among entities can be specified a priori with reasonably good confidence, and thus pro vides a declarative knowledge base to support IF reasoning and estimation. Note that such knowl edge is also represented in language and is available as digital text, in the same way as data from messages, documents, Twitter, and so forth. The use of ontological information in IF systems can be varied; ontological information can augment observed data and can aid in asserting possible rela tionships, directing search and also sensor management (to acquire expected information based on ontological relations), and yet in other ways. Importantly, specified ontologies can also serve as pro viding consistent and grounded semantic terminology for any given system. In our current research, we employ ontologies primarily for augmenting observational data with asserted ontological data whose relevance is algorithmically determined using “spreading activation” and then integrated to enrich the evidential basis for reasoning [4]. The broader implications of ontologies for intelligence analysis are described in Ref. [5], which come from the University of Buffalo’s National Center for Ontological Research (see http://ncorwiki.buffalo.edu/index.php/Main_Page). 1.1.5 Learned Information Finally, there is the class of information that could be learned (online) from all of the aforemen tioned sources if the IF process is designed with a Data Mining/Inductive or Abductive Learning © 2016 by Taylor Francis Group, LLC
- 34. 6 Multisensor Data Fusion functional component. Very little research and prototyping of such dual-process type IF systems has been done although the conceptualization of such IF schemes and architectures was put forward some time ago (e.g., Ref. [6]), as shown in Figure 1.1. The runtime integration of learned informa tion raises a number of both algorithmic issues as well as architectural issues. For example, if mean ingful patterns of behavior can be learned and can be measured/judged as persistent or enduring, such patterns could be incorporated in a dynamically modifiable knowledge base to be reused (as a Level 4 Process refinement function). Such learning processes will also not be perfect and have some uncertainty that also needs to be factored into the traditional Common Referencing and Data Association functions of the target fusion process. 1.2 COMMON REFERENCING AND DATA ASSOCIATION Common Referencing (CR) is that traditional IF system function that is sometimes called “Alignment” and is the function that normalizes these input sources for any given fusion applica tion or design. CR addresses such issues as coordinate system normalization, temporal alignment, and uncertainty alignment across the input streams, among others. With the highly disparate input streams described earlier, the design of required CR techniques is a nontrivial challenge. There are at least two major CR issues that these heterogeneous data represent temporal align ment and uncertainty alignment. Consider a textual input message whose free text, in just a few lines, could have past–present–future tense expressions, for example, “3 days ago I saw…”, “past precedents lead me to believe that tomorrow I should see…” and so forth. Other sources can also have varied temporal structures regarding their input. Such data lead to the issue of what the IF community has called “OOSM: out-of-sequence measurements” for hard/sensor data but the issue carries over to all sources as well and requires complex temporal alignment techniques for CR; it also raises the issue of retrospective fusion processing operations to correct for delayed inputs (if warranted; this is a design choice). Temporal alignment methods we have used for soft data are described in Ref. [7]. Data mining Data fusion Discovery modeling Data mining search operations Object base Knowledge Information understood and explained Data transform Data cleansing Data warehouse Operational process data storage Sensor 1 Sensor 2 Sensor 3 Data Observations and measurements Information Data, organized and placed in context Model Common user visualization Visualization, management Situation Level 3 Impact Level 2 Situation refinement Object base Level 1 Object refinement Level 0 Signal data refinement Situations Impacts Objects Level 4 Resource refinement Visualization, validation FIGURE 1.1 Notional fusion process architecture combining data mining and data fusion. (From ISCAS ‘98—Proceedings of the 1998 IEEE International Symposium on Circuits and Systems, Ann Arbor, 1998.) © 2016 by Taylor Francis Group, LLC
- 35. 7 IF Technology Capabilities for Modern Security Problems The uncertainty alignment requirement evolves as a result of the high likelihood that any uncer tainty in the widely various sources will be represented in disparate forms. Consider the basic differences between the uncertainty in sensor (hard) data and textual (soft) data; sensor data uncer tainty is sensibly always expressed in probabilistic form whereas, owing to the problem of imprecise adjectives and adverbs in language, linguistic uncertainty is often expressed in possibilistic (fuzzy) terms. It can be expected that uncontrolled open source or social media data may use yet other uncertainty formalisms to express or tag inputs. Transformation and normalization of disparate forms of uncertainty is a specialized topic in the uncertainty/statistical literature (e.g., Ref. [8]), and is among the high-priority issues in the IF community [9]. It should be noted that such trans formations largely can be developed only by invoking some statistical type qualities that are pre served across the transform, such as some form of total uncertainty; that is, the transform does not create an “equivalent” value of a probability in say a possibilistic space; seminal papers on the probability– possibility transformation issue are in Refs. [10–12]. In our research, we have addressed the probabilistic– possibilistic transformation issue in an approach that satisfies the consistency and preference preservation principles [13], while resulting in the most specific distribution for a speci fied portion of a probabilistic representation, resulting in the use of a truncated triangular transfor mation in our case [14]. Regarding the Data Association (DA) function, which some consider the heart of a fusion process, these varied data raise the level of DA complexity in significant ways. The soft data category, which inherently reports about Entities and (judged) Relationships, and is inherently in semantic format (language/words), raises the important issue of how to measure semantic similarity of such elements as reported in these various input streams. Such scores are needed in the Hypothesis Evaluation step of the DA process (see Ref. [15] on these DA subfunctions). But there are further DA complications that arise due to the soft data: linguistic phrases have verbs that reflect inter-Entity (noun) relationships; also of note is that the Natural Language Processing (NLP) community has employed graphical methods for the representation of linguis tic structures. As a result, the DA process now involves interassociation of both Entities (nouns) and Relations (verbs), that is, of graphical structures. This requirement extends to the hard data as well because that data need to be cast in a semantic framework to enable the overall DA process for the combined hard and soft data. Developing DA methods for graphical structures represents an entirely new challenge for the DA function. In such approaches for these applica tions, a scoring approach also needs to be developed to assess Relational similarity as well as Entity similarity, and a composite association scheme for these graphical substructures needs to be evolved. Historical approaches to DA have often employed solution methods drawn from assignment problems in operations research. When association is required between many non graphical data sources, this can be handled by the multidimensional assignment problem [16,17]. The main difference between the multidimensional assignment problem and graph association is how topological information from the graphs is used. Our research center has attacked this problem and has developed research prototype algorithms, as described in Ref. [18], where the graph association problem is formulated as a binary linear program and a heuristic for solving the multiple graph association is developed using a Lagrangian relaxation approach to address issues with a between-graph transitivity requirement. 1.3 SEMANTICS The introduction of linguistic information, as well as the transformation of sensor + algorithm estimation process outputs into a semantic frame, also adds to the complexity of IF process design and development. Semantic complexity is also added by the very nature of modern intelligence and security problems wherein the situations of interest relate to both military operations and also sociopolitical behaviors and entities. Clear meanings of such notions of interest as “patterns of life,” © 2016 by Taylor Francis Group, LLC
- 36. 8 Multisensor Data Fusion “rhythm of the city,” and “radicalization” as patterns or situations of interest—to be estimated by IF systems—have proven difficult to specify in clear semantic terms. Although the use of ontolo gies helps in this regard, standardization issues remain when considering networked and distributed systems, which are typical in the modern era. For example, in distributed intelligence or military systems there is typically no single point of architectural authority that can mandate a single onto logical framework for the network. For large-scale real systems there is also the problem of large legacy systems that were never designed with ontological formalisms in mind; this creates a “retro fit” problem of adjusting the semantic framework of that system to some new ontological standard, a costly and complex operation. It must also be noted that the way in which all textual/linguistic information gets into an IF system is through processing in some type of NLP or text extraction system. Such systems serve as a front-end filter for the admission of fundamental entity and relationship data, the raw soft data of the system, and so any imperfections in such extractions bound the capture of evidential informa tion for the subsequent reasoning and estimation processes. Whereas errors in hard sensor data are typically known with reasonable accuracy because of sensor calibrations, the errors in text extrac tion and NLP systems are either weakly known or unknown, sometimes as a result of proprietary constraints. Other strategies to deal with the complexities of semantics involve the use of controlled languages, to bound the grammatical structures and also the extent of the vocabulary that has to be dealt with. A good example for military/intelligence applications is the Battle Management Language (BML) [19] that has been under development since about 2003 for both Command and Control simulation studies but also for IF applications (e.g., Refs. [20,21]). There is a corresponding need to understand better the nature of semantic (and syntactic) com plexity in language, and also to develop measures and metrics that aid in developing better NLP processes and controlled languages. There is a reasonably rich literature on these topics (e.g., Ref. [22]) that should be exploited in regard to the integrated design of IF systems that today have to deal with a wide range of semantic difficulties. 1.4 GRAPHICAL REPRESENTATIONS AND METHODS There are a number of reasons that, for COIN and asymmetric warfare-type problems, graphs are becoming a dominant representational form for the information in and the processes involved in IF systems. In the information domain, many of the components discussed in Section 1.1 are textual/ linguistic and to capture this information in digital form, graphs are the representational form of choice. The problem domain is also described in the ontologies that are also typically couched in graphical forms. Note that ontologies describe inter-Entity relations of various types. Note too that the inferences and estimates of interest in these problems are of the higher level type in the sense of the JDL Model of Information Fusion, that is, estimates of situations and threat states. These higher level states—the conditions of interest for intelligence and security applications—are also best described as graphs, as situations can in the most abstract sense be considered as a graph of entities and relations. As a result, it is not unexpected to see that the core functions of IF, such as DA as previously described, are employing graphical methods in these fusion function operations. The US Army’s primary intelligence support system, the Distributed Common Ground Station-Army (DCGS-A), employs a global graph approach to capture all of the evidentiary information that supports IF and other intelligence analysis operations; see Ref. [23] and Figure 1.2, which shows the top-level struc ture of this graphical concept. Developing a comprehensive understanding of these problems thus involves a logical synthesis of the many situational substructures or subgraphs in these problem domains. The subgraphs are somewhat thematic and can be thought of as revolving about the Political, Military, Economic, © 2016 by Taylor Francis Group, LLC
- 37. 9 IF Technology Capabilities for Modern Security Problems Social, Infrastructure, and Information (PMESII) notion of the heterogeneity of the classes of information of interest in such problems. Thus, it is not surprising to see social network analysis tools—which are by the way graph-theoretic and graph-centric—employed in support of intelli gence analysis, here with the focus on the social and infrastructure patterns and subgraphs of the problem space. In our own work for such problems, we considered that it would be broadly helpful in analysis to enable a subgraph-querying capability as a generalized analysis tool. In such an approach, the analyst forms a query in text that can be transformed to a graph (we call these template graphs in that they are subgraph structures of interest) that is then searched for in the associated-evidence graph that is formed by the DA process. This search operation is in effect an stochastic inexact graph-matching problem, as the nodes and arcs of the evidential (or perhaps the template graph) have uncertainty values associated with them, and also because what is sought is the best match to the query, not an exact match, as there may be no exact match in such unpredictable problem situa tions. Other complexities arise in trying to realize such capability, such as executing such operations incrementally for streaming data, and also doing them in a computationally efficient way because the graphs can get quite large. As a consequence of several PhD efforts, we have realized today a rather mature graph-matching capability for intelligence analysis that is implemented in a cloud- based process; see Refs. [24–26], among other of our works. 1.5 OVERALL SYSTEM ARCHITECTURES AND ANALYSIS FRAMEWORKS It can perhaps be appreciated from the preceding discussion that the major challenge for intel ligence analysis in these modern problems is the synthesis of a total situational picture. In the face of highly heterogeneous data of varying uncertainty and of a problem domain that has many sub structures and relations and entities of interest, and has a varying temporal operational tempo, these problems— even with state of the art automated support/analysis systems such as IF systems— create a cognitive challenge even for the best analysts. Global graph “object” Relationship or verb Action Person Organization Account Task Event Society Place Region Feature Material Reference Equipment Consumable (Docs, media, etc.) • 12 primary types of objects • An object is a “thing” whose existence is not predicated on the existence of some other “thing” Facility FIGURE 1.2 US Army’s “global graph” concept for DCGS-A. (From Walsh, D. Relooking the JDL Model for fusion on a global graph. In National Symposium on Sensor and Data Fusion, Las Vegas, July 2010.) © 2016 by Taylor Francis Group, LLC
- 38. 10 Multisensor Data Fusion Automated methods for aiding such synthesis are in research and are just now being experi mented with (e.g., Complex Event Processing [CEP], Probabilistic Argumentation, Graph-Based Relational Learning, and other methods). At the moment, intelligence support systems comprise suites of disparate tools and hopefully some agile visualization schemes that aid analysts in the hypothesis-synthesis challenge. Our research prototype system, supported by the US Army Research Office, has addressed a number of the issues discussed here (as commented on within the chapter) and is just now entering the phase where the user end of the system is being designed and developed. The current Tool Suite comprises Dynamic Social Network Analysis Tools (uses a random graph approach), the Graph Matching Tools mentioned previously, a Link Analysis Tool (finds a wide variety of inter-Entity connections), Named Entity Recognizer (part of the NLP system), Entity and Activity Recognizers (uses automated semantic labeling methods from imagery and video), and an early prototype Abductive Reasoner. Together, these form the Community of Interest Service Layer or analyst layer in our architecture, which is basically a service-oriented architecture. That layer is shown in Figure 1.3, and comprises three main services: Evidence and Entity-estimate Foraging Service (this includes the CR and DA functions previously described), a Sensemaking Service where the above tools reside, and an Analytic Support Service that includes Visualization support, Pedigree Service, and other processes. Note that the Hypothesis Composition Service is still in conceptualization; at present we are exploring a CEP approach. CEP addresses the challenge of combining multiple heterogeneous data streams into a hierarchical structure that can represent higher order events and semantic meaning through the application of rules and filters at multiple levels of information (e.g., Ref. [27]). The Core Enterprise Services that do all of the front-end data processing and conditioning are shown in Figure 1.4; this figure does not show much of the detail but the flavor of these operations can be appreciated. Multiple hard data streams (in our case these are LIDAR, EO/IR, and Visible imagery, video sources, and acoustic devices) are processed individually to the point where seman tic information is developed from various estimation algorithms. Multiple soft message streams as arising from multiple soldier reports enter the NLP-based soft processing stream and the primary Hypothesis composition services Data association services Graph matching tool Common ref services Dynamic social net tool Group activity tool Link analysis tool Intelligence analysts Metadata and pedigree services Workflow services Alert services Visualization services Enterprise service bus Analytic support services Sensemaking services Evidence and entity-estimate foraging services FIGURE 1.3 Analyst layer in our service-oriented architecture for counterinsurgency analysis support. © 2016 by Taylor Francis Group, LLC
- 39. 11 IF Technology Capabilities for Modern Security Problems entities and relations are extracted. All of this semantic information flows to the Enterprise Bus, where it is accessible to the Community of Interest (Analysts) Service Layer for inferencing and estimation operations. Space permits us to describe only one of our tools and we choose to show our Activity Recognizer Tool system; this is developed by our colleagues at Tennessee State University and is described in Ref. [28]. This tool focuses on human–vehicle interactions and activities, a type of activity that is critically important for the problem of improvised explosive devices (IEDs). It is assumed that video and acoustic sensing of the human–vehicle settings is feasible. An approach that is based on human–vehicle activity ontology is used, wherein discrete activities (e.g., door opening as detected by acoustics, human entering vehicle as detected by video) are detected by each sensing modality. Spatiotemporal and semantic association of these discrete activities, along with the activity-class ontology, allows fusion-based inferencing of aggregated activity classes of interest. Further, as previously described, these inferences are framed into “pseudo- messages” that are sent to the Enterprise Bus for access by the hard + soft DA service, to allow combination with soft message data on the same activities. These operations are depicted in Figure 1.5. Single-sensor-based evidence Single-sensor-based evidence Single-sensor-based evidence Single-sensor-based evidence Single-sensor-based evidence Single-sensor-based evidence State estimation D C Entity-level evidence formation Entity-level evidence formation Entity-level evidence formation Hb Ha State estimation State estimation Common referencing Common referencing Data association Common referencing Data association Data association Direct human inference Direct human inference Sensor-specific preprocessing Sensor-specific preprocessing Sensor-specific preprocessing Enterprise service bus A B FIGURE 1.4 Core enterprise services in our service-oriented architecture for counterinsurgency analysis support. © 2016 by Taylor Francis Group, LLC
- 40. 12 Multisensor Data Fusion 1.6 CAPABILITY SHORTFALLS AND RESEARCH NEEDS Intelligence and security problems of the type discussed here are very likely to continue for the foreseeable future, although the prospects for conventional nation-state conflict still remain as well, and will drive yet other technology requirements, and it should be noted that there are some over laps in such requirements. Although the IF community is reacting to the COIN and asymmetric/ irregular warfare and stability operations needs of the type described here, there are very few well- tested capabilities that have been transitioned to operational systems. The IF community also has to make judgments about research investments that are peculiar to IF process needs and those that are supportive of IF processes; a good example is in NLP and text extraction—this is a core capability supportive of IF but that is actively being matured by the NLP community. For IF processing in particular, there are needs to improve CR methods for temporal and uncertainty alignment, and to improve capabilities for retrospective and reliable temporal estimation when meaningful amounts of input are continually shifting in time. Improvements in DA techniques are similarly required, both for new ideas in measures for semantic similarity and scoring in support of DA, but equally in graphical or other methods that in addition exhibit computational efficiency, as DA is typically the computational bottleneck in IF systems. Regarding state estimation, we see the major chal lenge being in the creation of automated methods to support synthesis of the disparate hypotheses emanating from data and theme-specific inferencing/estimation tools, to aid intelligence analysts in forming more comprehensive assessments of the “story” of interest suggested by the evidence. New studies and implementations of information foraging theory [29] are needed as one means to achieve more informative and efficient examinations of both associated evidence and inferences. Deeper examinations are also needed of the various Sensemaking paradigms [30,31] and ways that IF technologies can support them. Event alignment and fusion Acoustic event detection and analysis Visual event detection and analysis Surveillance camera #1 Acoustic sensor Soft message generation 1 2 3 4 5 6 7 8 9 10 16 15 20 19 14 13 18 12 17 11 Subject 1 got off fast from the driver side Message-1 Message-2 Message-3 Subject 1 opened the trunk of the car and unloaded a large object Subject 1 left the scene (GPS) at 1345 hr HVI protocol: (Subject-predicate-object-time-space) 2 1.5 1 –0.5 –1 –1.5 –2 0 0.5 0 0.5 1 1.5 2 2.5 FIGURE 1.5 Fused human–vehicle activity estimation scheme. (From Shirkhodaie, A. et al., “Acoustic and Imagery Semantic Labeling and Fusion of Human-Vehicle Interactions,” in SPIE Defense and Security Conference, Orlando, FL, 2011.) © 2016 by Taylor Francis Group, LLC
- 41. 13 IF Technology Capabilities for Modern Security Problems 1.7 CONCLUSION Evolving international sociopolitical events and dynamics, coupled with rapid growth of a wide variety of informational technologies, has given rise to a marked increase in the complexity of intel ligence analysis and efforts to design and develop technological capabilities to aid such analyses. As IF technologies have been a major contributor for intelligence analysis, these complexities have carried over to create significant challenges in IF system design. This chapter has reviewed what are considered to be many of these challenges and, by referencing a major academic research pro gram at our research center on such challenges, provided some examples of candidate methods to deal with these complexities. Much of the research on modern IF system design is still in the basic research domain, and far from being proven, much remains to be done and explored. ACKNOWLEDGMENT This research activity has been supported in part by a Multidisciplinary University Research Initiative (MURI) grant (No. W911NF-09-1-0392) for Unified Research on Network-based Hard/ Soft Information Fusion, issued by the US Army Research Office (ARO) under the program man agement of Dr. John Lavery. REFERENCES 1. D. L. Hall and J. Llinas, Introduction to multisensor data fusion. Proceedings of IEEE, 85(1):6–23, 1997. 2. J. Gómez-Romero et al., Strategies and techniques for use and exploitation of contextual information in high-level fusion architectures. 13th International Conference on Information Fusion (Fusion 2010), Edinburgh, UK, July 2010. 3. A. Kott and P. S. Corpac, COMPOEX technology to assist leaders in planning and executing cam paigns in complex operational environments. 12th International Command and Control Research and Technology Symposium, Newport, RI, 2007. 4. M. Kandefer and S. C. Shapiro, Evaluating spreading activation for soft information fusion. 14th International Conference on Information Fusion, Chicago, July 5–8, 2011. 5. B. Smith et al., Ontology for the intelligence analyst. Crosstalk: The Journal of Defense Software Engineering, 25(6):18–25, 2012. 6. E. Waltz, Information understanding: Integrating data fusion and data mining processes. ISCAS ‘98— Proceedings of the 1998 IEEE International Symposium on Circuits and Systems, Ann Arbor, MI, 1998. 7. D. McMaster, Temporal alignment in soft information processing. 14th International Conference on Information Fusion, Chicago, July 2011. 8. G. J. Klir, A principle of uncertainty and information invariance. International Journal of General Systems, 17:249–275, 1990. 9. E. Blasch et al., High level information fusion developments, issues, and grand challenges. Fusion10 Panel Discussion, 13th International Conference on Information Fusion (Fusion 10), 2010. 10. M. Oussalah, On the probability/possibility transformations: A comparative analysis. International Journal of General Systems, 29(5):671–718, 2000. 11. J. F. Geer and G. J. Klir, A mathematical analysis of information-preserving transformations between probabilistic and possibilistic formulations of uncertainty. International Journal of General Systems, 20:143–176, 1992. 12. G. Klir and B. Parviz, Probability-possibility transformations: A comparison. International Journal of General Systems, 21(1):291–310, 1992. 13. D. Dubois and H. Prade, Unfair coins and necessity measures: A possibilistic interpretation of histo grams. Fuzzy Sets and Systems, 10:15–20, 1983. 14. G. Gross, R. Nagi and K. Sambhoos, A fuzzy graph matching approach in intelligence analysis and maintenance of continuous situational awareness. Journal of Information Fusion, 18:43–61, 2014. 15. D. L. Hall and J. Llinas, Handbook of Multisensor Data Fusion. CRC Press, Boca Raton, FL, 2001. © 2016 by Taylor Francis Group, LLC
- 42. 14 Multisensor Data Fusion 16. A. Poore, S. Lu and B. Suchomel, Data association using multiple-frame assignments. In Handbook of Multisensor Data Fusion, 2nd ed., M. Liggins, D. Hall and J. Llinas (eds.). CRC Press, Boca Raton, FL, 2009, pp. 299–318. 17. A. Poore and N. Rijavec, A Lagrangian relaxation algorithm for multidimensional assignment problems arising from multitarget tracking. SIAM Journal on Optimization, 3:544–563, 1993. 18. G. Tauer, R. Nagi and M. Sudit, The graph association problem: Mathematical models and a lagrangian heuristic. Naval Research Logistics (NRL), 60(3):251–268, 2013. 19. J. M. Pullen et al., Joint Battle Management Language (JBML)—US Contribution to the C-BML PDG and NATO MSG-048 TA. IEEE European Simulation Interoperability Workshop, Genoa, Italy, June 2007. 20. U. Schade, J. Biermann, M. Frey and K. Kruger, From Battle Management Language (BML) to auto matic information fusion. Proceedings of Information Fusion Geographic Information Systems (GIS), 2007, pp. 84–95. 21. H. Lee and B. P. Zeigler, SES-based ontological process for high level information fusion. The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, 7(4), Art. No. 129, 2010. 22. S. Pollard and A. W. Biermann, A measure of semantic complexity for natural language systems. Proceedings of NLP Complexity Workshop: Syntactic and Semantic Complexity in Natural Language Processing Systems, Stroudsburg, PA, 2000. 23. D. Walsh, Relooking the JDL model for fusion on a global graph. National Symposium on Sensor and Data Fusion, Las Vegas, NV, July 2010. 24. A. Stotz, R. Nagi and M. Sudit, Incremental graph matching for situation awareness. 12th International Conference on Information Fusion, Seattle, WA, July 6–9, 2009. 25. K. Sambhoos, R. Nagi, M. Sudit and A. Stotz, Enhancements to high level data fusion using graph matching and state space search. Information Fusion, 11(4):351–364, 2010. 26. G. Gross, Continuous preservation of situational awareness through incremental/stochastic graphical methods. 14th International Conference on Information Fusion, Chicago, July 5–8, 2011. 27. A. Buchmann and B. Koldehofe, Complex event processing. IT-Information Technology, 51:241–242, 2009. 28. A. Shirkhodaie, A. Rababaah and V. Elangovan, Acoustic and imagery semantic labeling and fusion of human-vehicle interactions. SPIE Defense and Security Conference, Orlando, FL, 2011. 29. P. Pirolli, Information Foraging Theory: Adaptive Interaction with Information. Oxford University Press, New York, 2007. 30. P. Pirolli and S. K. Card, The sensemaking process and leverage points for analyst technology. Proceedings of the 2005 International Conference on Intelligence Analysis, McLean, VA, 2005. 31. G. Klein, B. Moon and R. F. Hoffman, Making sense of sensemaking I: Alternative perspectives. IEEE Intelligent Systems, 21(4):70–73, 2006. © 2016 by Taylor Francis Group, LLC
- 43. 15 2 Multisensor Data Fusion A Data-Centric Review of the State of the Art and Overview of Emerging Trends Bahador Khaleghi, Alaa Khamis, and Fakhri Karray 2.1 INTRODUCTION All living organisms have the ability to gain information about their environment, as well as to interpret this information to take appropriate decisions. Building a complete picture of the envi- ronment could be achieved using a single sensing element or by the fusion of the data gathered from multiple sensing elements. The operation of the human brain is probably the best analogy to a multisensor data fusion system, where the brain acts as the fusion node and makes sense of input provided by our five sense organs, as illustrated in Figure 2.1. CONTENTS 2.1 Introduction.............................................................................................................................15 2.2 Multisensor Data Fusion.......................................................................................................... 16 2.2.1 What Is Multisensor Data Fusion?............................................................................... 16 2.2.2 Applications of Multisensor Data Fusion.................................................................... 17 2.3 A Data-Centric Taxonomy for Multisensor Data Fusion Algorithms..................................... 18 2.3.1 Fusion of Imperfect Data.............................................................................................19 2.3.2 Fusion of Correlated Data............................................................................................20 2.3.3 Fusion of Inconsistent Data......................................................................................... 21 2.3.3.1 Spurious Data................................................................................................ 21 2.3.3.2 Out-of-Sequence Data................................................................................... 21 2.3.3.3 Conflicting Data............................................................................................21 2.3.4 Fusion of Disparate Data.............................................................................................22 2.4 Evaluation of Data Fusion Systems.........................................................................................22 2.5 New Directions in Multisensor Data Fusion...........................................................................24 2.5.1 Social Data Fusion.......................................................................................................24 2.5.2 Opportunistic Data Fusion...........................................................................................24 2.5.3 Adaptive Fusion and Learning....................................................................................25 2.5.4 Data Reliability and Trust............................................................................................25 2.5.5 Data Fusion in the Cloud and Big Data Fusion...........................................................25 2.5.6 Fusion of Data Streams................................................................................................26 2.5.7 Low-Level versus High-Level Data Fusion.................................................................26 2.5.8 Evolution of the JDL Model........................................................................................27 2.6 Conclusion...............................................................................................................................27 References.........................................................................................................................................27 © 2016 by Taylor Francis Group, LLC
- 44. 16 Multisensor Data Fusion The goal of this chapter is to provide readers with a comprehensive review of contemporary data fusion methodologies, as well as an overview of the most recent developments and emerging trends in this field. The existing data fusion methodologies are examined according to a data-centric taxonomy, that is, the specific data-related issue they aim to tackle. Inspired by recent advances in mobile and ubiquitous sensing, cloud storage and computing, and prevalence of social networks, the new and emerging directions in data fusion research, such as social data fusion, cloud-enabled and big data fusion, and fusion of streaming data, are briefly discussed. The rest of this chapter is organized as follows. A brief discussion of multisensor data fusion definitions and most common applications are presented in Section 2.2. Relying on a data-centric taxonomy, a review of the exist- ing data fusion literature is provided in Section 2.3. Section 2.4 is dedicated to a short discussion on research on data fusion evaluation methodologies and frameworks. The emerging research trends within the data fusion community are enumerated and briefly examined in Section 2.5. Lastly, in Section 2.6 the concluding remarks are presented. 2.2 MULTISENSOR DATA FUSION 2.2.1 What Is Multisensor Data Fusion? From a data integration perspective, Luo defines multisensor fusion as any stage in the integration pro- cess in which there is an actual combination (or fusion) of different sources of sensory information into one representational format [1]. The boundary between sensor fusion and sensor integration is quite fuzzy and the terms are used interchangeably sometimes. Joshi and Sanderson describe multisensor fusion as part of the multisensor integration process [2]. This process refers to the synergistic use of multiple sensors to improve operation of the system as a whole and it also includes sensor planning and sensor architecture. Multisensor planning deals with the acquisition of sensor data while multisensor architecture is responsible for the organization of data processing and data flow in the system. Joshi and Sanderson define multisensor fusion as a process that deals with the combination of data from multiple sensors into one coherent and consistent internal representation or action [2]. Many other definitions for data fusion exist in the literature. Joint Directors of Laboratories (JDL) [3] defines data fusion as a “multi-level, multifaceted process handling the automatic detec- tion, association, correlation, estimation, and combination of data and information from several sources.” Klein [4] generalizes this definition, stating that data can be provided either by a single source or by multiple sources. Both definitions are general and can be applied in different fields including remote sensing. In Ref. [5], Bostrom et al. present a review and discussion of many data fusion definitions. Based on the identified strengths and weaknesses of previous work, a principled definition of information fusion is proposed as “Information fusion is the study of efficient methods for automatically or semi-automatically transforming information from different sources and dif- ferent points in time into a representation that provides effective support for human or automated decision making.” Data fusion is a multidisciplinary research area borrowing ideas from many Brain Uncertain observations Sensor 1 Sensor 2 Sensor n Fusion node Estimate of feature Feature z1 z2 zn X FIGURE 2.1 Analogy between the brain operation and the fusion node in a data fusion system. © 2016 by Taylor Francis Group, LLC
- 45. 17 Review of the State of the Art and Overview of Emerging Trends diverse fields such as signal processing, information theory, statistical estimation and inference, and artificial intelligence. This is indeed reflected in the variety of techniques presented in Section 2.3. Various conceptualizations of the fusion process exist in the literature. The most common and popular conceptualization of fusion systems is the JDL model [3]. The JDL classification is based on the input data and produced outputs, and originated in the military domain. The original JDL model considers the fusion process in four increasing levels of abstraction: object, situation, impact, and process refinement. Despite its popularity, the JDL model has many shortcomings, such as being too restrictive and especially tuned to military applications. The JDL formalization is focused on data (input/output) rather than processing. An alternative is Dasarathy’s framework [6], which views the fusion system, from a software engineering perspective, as a data flow characterized by input/ output as well as functionalities (processes). Another general conceptualization of fusion is the work of Goodman et al. [7], which is based on the notion of random sets. The distinctive aspects of this framework are its ability to combine decision uncertainties with decisions themselves, as well as presenting a fully generic scheme of uncertainty representation. One of the most abstract fusion frameworks was proposed by Kokar et al. [8]. This formalization is based on category theory and is claimed to be sufficiently general to capture all kinds of fusion, including data fusion, feature fusion, decision fusion, and fusion of relational information. It can be considered as the first step toward development of a formal theory of fusion. The major novelty of this work is the ability to express all aspects of multisource information processing, that is, both data and processing. Further, it allows for consistent combination of the processing elements (algorithms) with measurable and provable performance. Such formalization of fusion paves the way for the application of formal methods to standardized and automatic development of fusion systems. 2.2.2 Applications of Multisensor Data Fusion Multisensor data fusion aims to overcome the limitations of individual sensors and produce accurate, robust, and reliable estimates of the world state based on multisensory information [9]. Multisensor data fusion has attracted many researchers from academia and industry because of its foreseen benefits in many applications. These benefits include, but are not limited to, enhanced confidence and reliability of measurements, extended spatial and/or temporal coverage, and reduced data imperfection aspects. Mitchell listed four main advantages of multisensor data fusion [10]: a greater granularity in the representation of information; greater certainty in data and results; elimi- nation of noise and errors, producing a greater accuracy; and allowing a more complete view on the environment. These foreseen benefits of multisensor data fusion result in its wide applicability in a variety of military and civilian applications. As part of a comprehensive survey on multisen- sor integration and fusion in intelligent systems, Luo and Kay described a number of military and industrial applications in this area [11]. Data fusion is an established military technology and available for numerous military applica- tions. These military applications include, but are not limited to, surveillance [12], anomaly detec- tion [13] and behavior monitoring [14], target tracking [15,16], target engageability improvement [17], fire control [18], and landmine detection [19]. For example, modern military Command Control (C2) systems are making increasing use of data fusion and resource management technol- ogy and tools [20]. By reducing uncertainty in the existing pieces of information and providing a means to infer about the missing pieces, data fusion supports the decision makers in compiling and analyzing the tactical/operational picture, and ultimately improving their situation awareness [21]. Examples of nonmilitary applications of data fusion include air traffic control [22], healthcare [23,24], speaker detection and tracking [25], mobile robot navigation [26], mobile robot localiza- tion [27], intelligent transportation systems [28], remote sensing [29,30], environment monitoring [31,32], and situational awareness [33]. For example, a Bayesian approach with pre- and postfilter- ing to handle data uncertainty and inconsistency in mobile robot local positioning is described in Ref. [27]. Mobile robot positioning provides an answer for the question: Where is the robot? The © 2016 by Taylor Francis Group, LLC
- 46. 18 Multisensor Data Fusion robot positioning solutions can be roughly categorized into relative position measurements (dead reckoning) and absolute position measurements. In the former, the robot position is estimated by applying to a previously determined position the course and distance traveled since. In the latter, the absolute position of the robot is computed by measuring the direction of incidence of three or more actively transmitting beacons, using artificial or natural landmarks, or using model matching to estimate the absolute location of the robot. There will always be an error in the readings provided by these techniques, and therefore the notion of multisensor data fusion is commonly used to tackle various imperfection aspects of data and yield a more accurate estimate for the robot position [27]. 2.3 A DATA-CENTRIC TAXONOMY FOR MULTISENSOR DATA FUSION ALGORITHMS Regardless of how different components (modules) of the data fusion system are organized, which is specified by the given fusion architecture, the underlying fusion algorithms must ultimately pro- cess (fuse) the input data. Real-world data fusion applications have to deal with several data-related challenges. As a result, we explore data fusion algorithms according to a data-centric taxonomy [34]. Figure 2.2 illustrates an overview of data-related challenges that are typically tackled by data fusion algorithms. The input data to the fusion system may be imperfect, correlated, inconsistent, and/or in disparate forms/modalities. Each of these four main categories of challenging problems can be further subcategorized into more specific problems, as shown in Figure 2.2 and discussed in the following. Various classifications of imperfect data have been proposed in the literature [35–37]. Our classi- fication of imperfect data is inspired by the pioneering work of Smets’ [36] as well as an elaboration by Dubois and Prade [38]. Three aspects of data imperfection are considered in our classification: uncertainty, imprecision, and granularity. Data are uncertain when the associated degree of confidence about what is stated by the data is less than 1. On the other hand, imprecise data are those data that refer to several, rather than only one, object(s). Finally, data granularity refers to the ability to distinguish among objects, which are Data-related fusion aspects Imperfection Correlation Conflict Outlier Granularity Imprecision Uncertainty Vagueness Ambiguity Incompleteness Disorder Inconsistensy Disparateness FIGURE 2.2 A taxonomy of data fusion methodologies: Different data fusion algorithms can be roughly categorized based on one of the four challenging problems of input data that are mainly tackled: data imper- fection, data correlation, data inconsistency, and disparateness of data form. © 2016 by Taylor Francis Group, LLC
- 47. 19 Review of the State of the Art and Overview of Emerging Trends described by data, being dependent on the provided set of attributes. Mathematically speaking, assume the given data d (for each described object of interest) to be structured as the following: object O attribute A statement S representing that the data d is stating S regarding the relationship of some attribute(s) A to some object O in the world. Further assume C(S) to represent the degree of confidence we assign to the given statement S. Then, data are regarded to be uncertain if C(S) 1 while being precise, that is, a singleton. Similarly, data are deemed as imprecise if the implied attribute A or degree of confi- dence C are more than 1, for example, an interval or set. Please note, the statement part of the data is almost always precise. The imprecise A or C may be well defined or ill defined and/or miss some information. Thus, imprecision can manifest itself as ambiguity, vagueness, or incompleteness of data. The ambiguous data refers to those data where the A or C is exact and well defined yet imprecise. For instance, in the sentence “Target position is between 2 and 5” the assigned attribute is the well-defined imprecise interval [2 5]. The vague data is characterized by having ill-defined attributes, that is, the attribute is more than 1 and not a well-defined set or interval. For instance, in the sentence “The tower is large” the assigned attribute “large” is not well defined as it can be interpreted subjectively, that is, have different meaning from one observer to the other. The imprecise data that has some information missing is called incomplete data. For instance, in the sentence “It is possible to see the chair,” only the upper limit on the degree of confidence C is given, that is, C τ for some τ [39]. Consider an information system [40] in which a number of (rather than one) objects O = {o1,…, ok} are described using a set of attributes A = {V1, V2, …, Vn} with respective domains D1, D2, …, Dn. Let F = D1 × D2 × … × Dn to represent the set of all possible descriptions given the attributes in A, also called the frame. It is possible for several objects to share the same description in terms of these attributes. Let [o]F to be the set of objects that are equivalently described (thus indistinguish- able) within the frame F, also called the equivalence class. Now, let T ⊆ O represent the target set of objects. In general, it is not possible to exactly describe T using F, because T may include and exclude objects that are indistinguishable within the frame F. However, one can approximate T by the lower and upper limit sets that can be described exactly within F in terms of the induced equiva- lence classes. Indeed, the rough set theory provides a systematic approach to this end. In summary, data granularity refers to the fact that the choice of data frame F (granule) has a significant impact on the resultant data imprecision. In other words, different attribute subset selections B ⊆ A will lead to different frames, and thus different sets of indiscernible (imprecise) objects. Correlated (dependent) data are also a challenge for data fusion systems and must be treated appropriately. We consider inconsistency in input data to stem from (highly) conflicting, spurious, or out of sequence data. Finally, fusion data may be provided in different forms, that is, in one or several modalities, as well as generated by physical sensors (hard data) or human operators (soft data). We believe such categorization of fusion algorithms is beneficial as it enables explicit exploration of popular fusion techniques according to the specific data-related fusion challenge(s) they target. Further, our taxonomy is intended to facilitate ease of development by supplying fusion algorithm designers with an outlook of the appropriate and established techniques to tackle the data-related chal- lenges their given application may involve. Finally, such exposition would be more intuitive and there- fore helpful to nonexperts in data fusion by providing them with an easy-to-grasp view of the field. 2.3.1 Fusion of Imperfect Data The inherent imperfection of data is the most fundamental challenging problem of data fusion systems, and thus the bulk of research work has been focused on tackling this issue. A number of © 2016 by Taylor Francis Group, LLC
- 48. 20 Multisensor Data Fusion mathematical theories are available to represent data imperfection [41], such as probability the- ory [42], fuzzy set theory [43,44], possibility theory [45], rough set theory [46], and Dempster– Shafer evidence theory (DSET) [47]. Most of these approaches are capable of representing specific aspect(s) of imperfect data. For example, a probabilistic distribution expresses data uncertainty, fuzzy set theory can represent vagueness of data, and evidential belief theory can represent uncer- tain as well as ambiguous data. Historically, the probability theory was used for a long time to deal with almost all kinds of imperfect information, because it was the only existing theory. Alternative techniques such as fuzzy set theory and evidential reasoning have been proposed to deal with per- ceived limitations in probabilistic methods, such as complexity, inconsistency, precision of models, and uncertainty about uncertainty [42]. There are also hybridizations of these approaches that aim for a more comprehensive treatment of data imperfection. Examples of such hybrid frameworks are fuzzy rough set theory (FRST) [48] and fuzzy Dempster–Shafer theory (fuzzy DSET) [49]. Lastly, there is the fairly new field of fusion using random sets, which could be used to develop a unified framework for treatment of data imperfections [50]. Figure 2.3 provides an overview of the afore- mentioned mathematical theories of dealing with data imperfections. On the x-axis, various aspects of data imperfection, introduced in Figure 2.3, are depicted. The box around each of the mathemati- cal theories designates the range of imperfection aspects targeted mainly by that theory. Interested readers are referred to Refs. [39] and [34] for a comprehensive review of the classical theories of representing data imperfections, describing each of them along with their interrelations. 2.3.2 Fusion of Correlated Data Many data fusion algorithms, including the popular Kalman filter (KF) approach, require either independence or prior knowledge of the cross covariance of data to produce consistent results. Unfortunately, in many applications fusion data are correlated with potentially unknown cross covariance. This can occur as a result of common noise acting on the observed phenomena [51] in centralized fusion settings, or the rumor propagation issue, also known as the data incest or double counting problem [52], in which measurements are inadvertently used several times in distributed Imperfection Fuzzy set Possibility Rough set Probability U n c e r t a i n t y A m b i g u i t y V a g u e n e s s I n c o m p l e t e n e s s G r a n u l a r i t y DSET Theory Random set Fuzzy DSET FIGURE 2.3 Overview of theoretical frameworks of imperfect data treatment (note: the fuzzy rough set theory is omitted from the diagram to avoid confusion). © 2016 by Taylor Francis Group, LLC
- 49. 21 Review of the State of the Art and Overview of Emerging Trends fusion settings [53]. If not addressed properly, data correlation can lead to biased estimation, for example, artificially high confidence value, or even divergence of fusion algorithm [54]. Most of the proposed solutions to correlated data fusion attempt to solve it by either eliminating the cause of correlation [55,56] or tackling the impact of correlation in fusion process [57–59]. 2.3.3 Fusion of Inconsistent Data 2.3.3.1 Spurious Data Data provided by sensors to the fusion system may be spurious as a result of unexpected situations such as permanent failures, short duration spike faults, or slowly developing failure [60]. If fused with correct data, such spurious data can lead to dangerously inaccurate estimates. For instance, KF would easily break down if exposed to outliers. The majority of work on treating spurious data has been focused on identification/prediction and subsequent elimination of outliers from the fusion process. Indeed, the literature work on sensor validation is partially aiming at the same target [61–63]. The problem with most of these techniques is the requirement for prior information, often in the form of specific failure model(s). As a result, they would perform poorly in a general case in which prior information is not available or unmodeled failures occur [64]. In Refs. [60,65] a general frame- work for detection of spurious data has been proposed that relies on stochastic adaptive modeling of sensors and is thus not specific to any prior sensor failure model. Extensive experimental simula- tions have shown the promising performance of this technique in dealing with spurious data [64]. 2.3.3.2 Out-of-Sequence Data The input data to the fusion system are usually organized as discrete pieces each labeled with a timestamp designating its time of origin. Several factors such as variable propagation times for dif- ferent data sources as well as having heterogeneous sensors operating at multiple rates can lead to data arriving out of sequence at the fusion system. Such out-of-sequence measurements (OOSM) can appear as inconsistent data to the fusion algorithm. The main issue is how to use these, usually old, data to update the current estimate while taking care of the correlated process noise between the current time and the time of the delayed measurement [66]. Most of the early work on OOSM assumed only single-lag data. For example, an approximate suboptimal solution to OOSM called Algorithm B [67], as well as its famous optimal counterpart Algorithm A [68], both assume single-lag data. Some researchers have proposed algorithms to enable handling of OOSM with arbitrary lags [69–71]. Among these methods the work in Ref. [71] is par- ticularly interesting as it provides a unifying framework for treating OOSM with Algorithm A as a special case. Nonetheless, it was shown in Ref. [72] that this approach, along with many other multilag OOSM methods, is usually very expensive in terms of computational complexity and storage. The same authors proposed an extension to the Algorithm A and Algorithm B called Algorithm Al1 and Algorithm Bl1, respectively. They further showed that these new algorithms have requirements similar to their single-lag counterparts and are therefore recommended for practical applications; Algorithm Bl1 especially is preferred because it is almost optimal and very efficient. Research work also inves- tigates the OOSM problem in the case of having both single-lag and multiple-lag data, termed the mixed-lag OOSM problem. The proposed algorithm is claimed to handle all three types of OOSM data and is shown to be suboptimal in the linear MMSE sense under one approximation [73]. 2.3.3.3 Conflicting Data Fusion of conflicting data, when, for instance, several experts have very different ideas about the same phenomenon, has long been identified as a challenging task in the data fusion com- munity. In particular, this issue has been heavily studied for fusion within the Dempster–Shafer evidence theory framework. As shown in a famous counterexample by Zadeh [74], naive applica- tion of Dempster’s rule of combination to fusion of highly conflicting data results in unintuitive results. Since then, Dempster’s rule of combination has been subject to much criticism for rather © 2016 by Taylor Francis Group, LLC
- 50. 22 Multisensor Data Fusion counterintuitive behavior [75]. Most of the solutions proposed alternatives to Dempster’s rule of combinations [76–79]. On the other hand, some authors have defended this rule, arguing that the counterintuitive results are due to improper application of this rule [50,80,81]. For example, in Ref. [50] Mahler shows that the supposed unintuitive result of Dempster’s combination rule can be resolved using a simple corrective strategy, i.e. to assign arbitrary small but nonzero belief masses to hypotheses deemed extremely unlikely. Fusion of conflicting data within the Bayesian probabilistic framework has also been explored by some authors. For example, the Covariance Union (CU) algorithm is developed to complement the Contextual Information (CI) method, and enable data fusion where input data is not just correlated but may also be conflicting [82]. Furthermore, a new Bayesian framework for fusion of uncertain, imprecise, as well as conflicting data was proposed in Ref. [83]. 2.3.4 Fusion of Disparate Data The input data to a fusion system may be generated by a wide variety of sensors, humans, or even archived sensory data. Fusion of such disparate data to build a coherent and accurate global view or the observed phenomena is a very difficult task. Nonetheless, in some fusion applications such as human–computer interaction (HCI), such diversity of sensors is necessary to enable natural interac- tion with humans. Our focus of discussion is on fusion of human generated data (soft data) as well as fusion of soft and hard data, as research in this direction has attracted attention in recent years. This is motivated by the inherent limitations of electronic (hard) sensors and recent availability of communication infrastructure that allow humans to act as soft sensors [84]. Further, although a tremendous amount of research has been done on data fusion using conventional sensors, very limited work has studied fusion of data produced by human and nonhuman sensors. An example of preliminary research in this area includes the work on generating a dataset for hard/soft data fusion intended to serve as a foundation and a verification/validation resource for future research [85,86]. Also in Ref. [84], Hall et al. provide a brief review on ongoing work on dynamic fusion of soft/ hard data, identifying its motivation and advantages, challenges, and requirements. A Dempster– Shafer theoretic framework for soft/hard data fusion is presented that relies on a novel conditional approach to updating as well as a new model to convert propositional logic statements from text into forms usable by Dempster–Shafer theory [87]. Another recent example of data fusion systems capable of leveraging soft data is presented in Ref. [88], where Seifzadeh et al. describe a solution to the problem of agile target tracking using fuzzy inference applied to soft data reports that character- ize the target agility level. 2.4 EVALUATION OF DATA FUSION SYSTEMS Performance evaluation aims at studying the behavior of a data fusion system operated by various algorithms and comparing their pros and cons based on a set of measures or metrics. The outcome is typically a mapping of different algorithms into different real values or partial orders for ranking [89]. Generally speaking, the obtained performance of a data fusion system is deemed to be depen- dent on two components: the quality of input data and the efficiency of fusion algorithm. As a result, the literature work on (low-level) fusion evaluation can be categorized into the following groups: • Evaluating the quality of input data to the fusion system. The target here is to develop approaches that enable quality assessment of the data, which are fed to the fusion system, and calculation of the degree of confidence in data in terms of attributes such as reliability and credibility [90]. The most notable work in this group is perhaps the standardization agreements (STANAG) 2022 of the North Atlantic Treaty Organization (NATO). STANAG adopts an alphanumeric system of rating, which combines a measurement of the reliability of the source of information with a measurement of the credibility of that information, both © 2016 by Taylor Francis Group, LLC
- 51. 23 Review of the State of the Art and Overview of Emerging Trends evaluated using the existing knowledge. STANAG recommendations are expressed using natural language statements, which makes them quite imprecise and ambiguous. Some researchers attempted to analyze these recommendations and provide a formal mathematical system of information evaluation in compliance with the NATO recommendations [90,91]. The proposed formalism relies on the observation that three notions underline an informa- tion evaluation system: the number of independent sources supporting a piece of information, their reliability, and that the information may conflict with some available/prior information. Accordingly, a model of evaluation is defined and its fusion method, which accounts for the three aforementioned notions, is formulated. The same authors have extended their work to enable dealing with the notion of degree of conflict, in contrast to merely conflicting or non- conflicting information [92]. Nonetheless, the current formalism is still not complete as there are some foreseen notions of the STANAG recommendations, such as total ignorance about the reliability of the information source, that are not being considered. Another important aspect related to input information quality, which is largely ignored, is the rate at which it is provided to the fusion system. The information rate is a function of many factors, including the revisit rate of the sensors, the rate at which data sets are communicated, and also the qual- ity of the communication link [93]. The effect of information rate is particularly important in decentralized fusion settings where imperfect communication is common. • Assessing the performance of the fusion system. The performance of fusion systems itself is computed and compared using a specific set of measures referred to as measures of performance (MOPs). The literature work on MOP is rather extensive and includes a wide variety of measures. The choice of the specific MOP(s) of interest depends on the charac- teristics of the fusion system. For instance, there is more to evaluate in a multiple-sensor system than there is in a single-sensor system. Further, in the case of multitarget problems, the data/track association part of the system also needs to be evaluated along with the estimation part. The commonly used MOPs may be broadly categorized into the metrics computed for each target and metrics computed over an ensemble of targets. Some of the MOPs belonging to the former category are track accuracy, track covariance consistency, track jitter, track estimate bias, track purity, and track continuity. Examples of measures in the latter category are average number of missed targets, average number of extra tar- gets, average track initiation time, completeness history, and cross-platform commonality history [94,95]. There are also other less popular measures related to the discrimination and/or classification capability of the fusion system that can be useful to collect in some applications. Aside from the conventional approaches for performance measurement, there is some notable work on development of MOPs for multitarget fusion systems within the finite set theory framework [96,97]. The key observation is that a multitarget system is fundamentally different from a single-target system. In the former case, the system state is indeed a finite set of vectors rather than a single vector. This is due to the appearance/ disappearance of targets, which leads to the number of states varying with time. In addi- tion, it is more natural to mathematically represent the collection of states as a finite set, as the order in which the states are listed has no physical significance [98]. This approach is especially useful in fusion applications in which the number of targets is not known and has to be inferred along with their positions. Finally, it is worth pointing out some of the fusion evaluation tools and testbeds that have recently become available. The Fusion Performance Analysis (FPA) tool from Boeing is software that enables computation of technical per- formance measures (TPMs) for virtually any fusion system. It is developed in Java (thus is platform-independent) and implements numerous TPMs in three main categories: state estimation, track quality, and discrimination [99]. Another interesting development is the multisensor–multitarget tracking testbed [100], which has been lately introduced and is the first step toward the realization of a state-of-the-art testbed for evaluation of large-scale distributed fusion systems. © 2016 by Taylor Francis Group, LLC
- 52. 24 Multisensor Data Fusion To the best of our knowledge, there is no standard and well-established evaluation framework to assess the performance of data fusion algorithms. Most of the work is being done in simulation and based on sometimes idealized assumption(s), which make it difficult to predict how the algorithm would perform in real-life applications. A review of literature on data fusion performance evalu- ation is presented in Ref. [101], where the challenging aspects of data fusion performance evalua- tion, in practice, are discussed. Having analyzed more than 50 of the related literature work, it has been shown that only very little (i.e., about 6%) of the surveyed research work treats the fusion evaluation problem from a practical perspective. Indeed, it is demonstrated that most of the existing work is focused on performing evaluation in simulation or unrealistic test environments, which is substantially different from practical cases. Regarding the preceding discussions, there appears to be a serious need for further research on development and standardizing measures of performance applicable to the practical evaluation of data fusion systems. 2.5 NEW DIRECTIONS IN MULTISENSOR DATA FUSION 2.5.1 Social Data Fusion The advent of social network services such as Facebook and Twitter has enabled users to share their social data—pictures, videos, news, ideas—on the Web. The social data are highly rich in content and thus provide an unprecedented opportunity for both researchers in social sciences and practitioners in industry to study human behavior, identify customer preferences, and much more. Within the context of data fusion, the conventional sensory data and the recently availability social data can be deemed as mutually compensatory in numerous data fusion and processing applications [102]. For instance, social network services can be leveraged in a participatory sensing manner, that is, human as a social sensor [103], to collect data in areas where physical sensors are not available. Similarly, data provided by conventional sensors can be used to construct context regarding the available social data, thus enabling them to be analyzed more effectively. On the other hand, social data are typically provided as streams of massive unstructured data. Accordingly, exploiting social data in fusion applications involves tackling challenges in stream data processing, scalable and distributed data storage and processing, and ability to model and interpret unstructured data. The aforementioned advantages along with the inevitable theoretical and practical challenges has made social data fusion a highly attractive and promising area of research within the fusion community, as reflected in the plethora of recent publications [103–106]. 2.5.2 Opportunistic Data Fusion Regarding the limitations of traditional data fusion systems, which are designed mostly to use ded- icated sensor and information resources, and the availability of new ubiquitous computing and communication technologies, the opportunistic data fusion paradigm considers the possibility of treating sensors as shared resources and performing fusion in an opportunistic manner [107]. New challenging problems associated with such fusion systems are identified and novel approaches to tackle them are explored. Some of the distinctions of the opportunistic information fusion model (OIFM) compared to the conventional approach are the need for on-the-fly discovery of sensors, ad hoc computational load, and dynamic (not predefined) fusion rules. The key enabling component required to realize an OIFM is a new approach toward middleware development called opportunis- tic middleware model (OMM). This is because the existing middleware platforms do not scale to the device diversity, size, and runtime dynamics required by OIFM applications [107]. Unfortunately, current specifications for the OMM do not address many issues related to its implementation and thus future research is still needed to make OIFM viable. Nonetheless, some preliminary research work is reported in the literature. For instance, in Ref. [108] an opportunistic fusion of data across time, space, and feature level is performed in a visual sensor network to achieve human gesture © 2016 by Taylor Francis Group, LLC
- 53. 25 Review of the State of the Art and Overview of Emerging Trends analysis. In Ref. [109], the authors study the problem of optimal camera placement in a visual sensor network designed to serve multiple applications (each to be operated in an opportunistic manner). The problem is formulated as a multiobjective optimization problem and solved efficiently using a multiobjective genetic algorithm. 2.5.3 Adaptive Fusion and Learning Early work on adaptive data fusion dates back to the early 1990s [110]. Nonetheless, this problem has rarely been explored in the fusion literature until recently. Some of the existing work is focused on incorporation of adaptivity into the KF algorithm. In Ref. [111] an adaptive fusion system capable of intelligent allocation of limited resources is described that enables efficient tracking of moving targets in three dimensions. An adaptive variant of KF called FL-AKF that relies on fuzzy infer- ence based on covariance matching to adaptively estimate the covariance matrix of measurement noise is proposed in Ref. [112]. In a similar approach, in Ref. [113] Tafti and Sadati present a novel adaptive Kalman filter (NAKF) that achieves adaptation using a mathematical function termed degree of matching (DoM), which is based on covariance matching. An adaptive UKF algorithm with multiple fading factors-based gain correction is proposed and applied to the picosatellite atti- tude estimation problem [114]. Another trend of work investigates explicit integration of machine learning algorithms into the fusion process to accomplish adaptation. For example, machine learn- ing methods are deployed in Ref. [115] to achieve online adaptation to users’ multimodal temporal thresholds within a human–computer interaction application framework. Some other works study application of reinforcement learning to adaptive fusion systems to perform dynamic data reliability estimation [116,117]. Another research work also proposed using kernel-based learning methods to achieve adaptive decision fusion rules [118]. 2.5.4 Data Reliability and Trust The majority of data fusion literature work is based on an optimistic assumption about the reli- ability of underlying models producing the beliefs associated with imperfect data. For instance, sensory data are commonly considered as equally reliable and play a symmetrical role in the fusion process [119]. Nonetheless, different models usually have different reliabilities and are valid only for a specific range. A recent trend in data fusion has addressed this issue mostly by attempting to account for reliability of beliefs. This has been accomplished through introduction of the notion of a second level of uncertainty, that is, uncertainty about uncertainty, represented as reliability coeffi- cients. The main challenges are first to estimate these coefficients and then to incorporate them into the fusion process. A number of approaches to estimate reliability coefficients have been proposed that rely on domain knowledge and contextual information [120], learning through training [121], possibility theory [122], and expert judgments [123]. Further, the problem of reliability incorpora- tion has been studied within several fusion frameworks such as Dempster–Shafer theory [124], fuzzy and possibility theory [125], transferable belief model [126], and probability theory [127]. Another research work also investigates the impact of belief reliability on high-level data fusion [128]. The issue of reliability in data fusion is still not well established, and several open questions such as interrelationship between reliabilities, reliability of heterogeneous data, and a comprehen- sive architecture to manage data fusion algorithm and reliability of data sources remain part of future research [119,124]. Taking research to the next level, more recent work attempts to address the overarching issues of information quality and higher level quality [129]. 2.5.5 Data Fusion in the Cloud and Big Data Fusion Recent advances in cloud computing technologies have led to the availability of interesting new capabilities for data fusion systems. Some of the most notable benefits of cloud-based computing © 2016 by Taylor Francis Group, LLC
- 54. 26 Multisensor Data Fusion include scalable and flexible data storage and processing, while maintaining a high level of reli- ability and security. Enabled by the power of the cloud, modern data fusion algorithms can now be performed over vast amounts of entities across multiple databases [130]. Efficient implementation of such big data fusion systems, however, requires attending to data management, system design, and real-time execution. For instance, the Google fusion table can be deemed as a preliminary cloud-enabled data management and fusion service that allows for uploading, sharing, filtering, and visualization. In particular, it supports the fusion of data from multiple sources through join- ing across tables sourced by different users [131]. The notion of cloud robotics, initially proposed by James Kuffner at Google [132], is another example of the potential of cloud-enabled data fusion and processing to revolutionize modern robotics. A cloud-enabled team of robots is capable of off- loading expensive computational and/or storage robotic tasks, hence allowing users to focus on the application at hand rather than worrying about the underlying IT infrastructure. A recent exemplary work in cloud robotics is described in Ref. [133], where a holistic robotic system is able to leverage cloud services to enhance the performance of video tracking. The experimental results demonstrate the feasibility of offloading computation to the cloud, which is especially beneficial when there are a large number of robot networks demanding image processing tasks. 2.5.6 Fusion of Data Streams The so-called tidal wave of big data is typically characterized by its three-V properties: volume, velocity, and variety. In particular, advances in mobile technologies have led to the proliferation of numerous online data-intensive applications where data streams are being collected continuously in large volume and high speed. When it comes to time-sensitive big data fusion applications that involve processing such extremely large data feeds produced at high speeds by multiple sources, the conventional static database technologies are not sufficient. Examples of such modern real-time applications are traffic monitoring and management, stock price prediction, and flight schedule checking. An alternative solution is to fuse data streams on-the-fly as soon as they are available. Research work on to the stream data fusion problem is gradually gaining popularity. Schueller and Behrend argue [134] in favor of the Reactive Programming (RE) paradigm and the Language Integrated Query (LINQ) language as a promising solution to the problem of storing and fus- ing real-time streams of data. Dynamic trust assessment over data-in-motion is another important challenge, which has been addressed in Ref. [135]. Their proposal is perhaps the first attempt to develop a dynamic trust assessment framework applicable to data streams with subjective logic as the underlying computational toolset. In a related study, Zhao et al. present a probabilistic model to transform the problem of truth discovery over data streams into a probabilistic inference problem [136]. The proposed approach is claimed to possess advantages such as requiring only a single pass over data, limited memory usage, and short response time, which are backed by preliminary experimental results. 2.5.7 Low-Level versus High-Level Data Fusion The discussion on high-level data fusion may appear to be outside the scope of this chapter. However, as argued in Ref. [137], as soft human-generated data, in the form of complex natural language state- ments, play an ever-increasing role in modern fusion systems, the clear distinction between low- level and high-level fusion processes might no longer be applicable. The key observation is that the interpretation and analysis of soft data necessitate development of complicated models not restricted to the immediate time frame, similar to those used by the high-level data fusion processes. In partic- ular, knowledge of the context within which a piece of soft data is uttered is crucial in our ability to exploit soft data [138]. A similar line of thought is presented in Ref. [139], where Dragos examines the main challenges involved in dealing with various forms of uncertainties potentially expressed by soft data and the need for high-level ontological analysis to assess them properly. © 2016 by Taylor Francis Group, LLC
- 55. Random documents with unrelated content Scribd suggests to you:
- 56. THE FULL PROJECT GUTENBERG LICENSE
- 57. PLEASE READ THIS BEFORE YOU DISTRIBUTE OR USE THIS WORK To protect the Project Gutenberg™ mission of promoting the free distribution of electronic works, by using or distributing this work (or any other work associated in any way with the phrase “Project Gutenberg”), you agree to comply with all the terms of the Full Project Gutenberg™ License available with this file or online at www.gutenberg.org/license. Section 1. General Terms of Use and Redistributing Project Gutenberg™ electronic works 1.A. By reading or using any part of this Project Gutenberg™ electronic work, you indicate that you have read, understand, agree to and accept all the terms of this license and intellectual property (trademark/copyright) agreement. If you do not agree to abide by all the terms of this agreement, you must cease using and return or destroy all copies of Project Gutenberg™ electronic works in your possession. If you paid a fee for obtaining a copy of or access to a Project Gutenberg™ electronic work and you do not agree to be bound by the terms of this agreement, you may obtain a refund from the person or entity to whom you paid the fee as set forth in paragraph 1.E.8. 1.B. “Project Gutenberg” is a registered trademark. It may only be used on or associated in any way with an electronic work by people who agree to be bound by the terms of this agreement. There are a few things that you can do with most Project Gutenberg™ electronic works even without complying with the full terms of this agreement. See paragraph 1.C below. There are a lot of things you can do with Project Gutenberg™ electronic works if you follow the terms of this agreement and help preserve free future access to Project Gutenberg™ electronic works. See paragraph 1.E below.
- 58. 1.C. The Project Gutenberg Literary Archive Foundation (“the Foundation” or PGLAF), owns a compilation copyright in the collection of Project Gutenberg™ electronic works. Nearly all the individual works in the collection are in the public domain in the United States. If an individual work is unprotected by copyright law in the United States and you are located in the United States, we do not claim a right to prevent you from copying, distributing, performing, displaying or creating derivative works based on the work as long as all references to Project Gutenberg are removed. Of course, we hope that you will support the Project Gutenberg™ mission of promoting free access to electronic works by freely sharing Project Gutenberg™ works in compliance with the terms of this agreement for keeping the Project Gutenberg™ name associated with the work. You can easily comply with the terms of this agreement by keeping this work in the same format with its attached full Project Gutenberg™ License when you share it without charge with others. 1.D. The copyright laws of the place where you are located also govern what you can do with this work. Copyright laws in most countries are in a constant state of change. If you are outside the United States, check the laws of your country in addition to the terms of this agreement before downloading, copying, displaying, performing, distributing or creating derivative works based on this work or any other Project Gutenberg™ work. The Foundation makes no representations concerning the copyright status of any work in any country other than the United States. 1.E. Unless you have removed all references to Project Gutenberg: 1.E.1. The following sentence, with active links to, or other immediate access to, the full Project Gutenberg™ License must appear prominently whenever any copy of a Project Gutenberg™ work (any work on which the phrase “Project Gutenberg” appears, or with which the phrase “Project Gutenberg” is associated) is accessed, displayed, performed, viewed, copied or distributed:
- 59. This eBook is for the use of anyone anywhere in the United States and most other parts of the world at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org. If you are not located in the United States, you will have to check the laws of the country where you are located before using this eBook. 1.E.2. If an individual Project Gutenberg™ electronic work is derived from texts not protected by U.S. copyright law (does not contain a notice indicating that it is posted with permission of the copyright holder), the work can be copied and distributed to anyone in the United States without paying any fees or charges. If you are redistributing or providing access to a work with the phrase “Project Gutenberg” associated with or appearing on the work, you must comply either with the requirements of paragraphs 1.E.1 through 1.E.7 or obtain permission for the use of the work and the Project Gutenberg™ trademark as set forth in paragraphs 1.E.8 or 1.E.9. 1.E.3. If an individual Project Gutenberg™ electronic work is posted with the permission of the copyright holder, your use and distribution must comply with both paragraphs 1.E.1 through 1.E.7 and any additional terms imposed by the copyright holder. Additional terms will be linked to the Project Gutenberg™ License for all works posted with the permission of the copyright holder found at the beginning of this work. 1.E.4. Do not unlink or detach or remove the full Project Gutenberg™ License terms from this work, or any files containing a part of this work or any other work associated with Project Gutenberg™. 1.E.5. Do not copy, display, perform, distribute or redistribute this electronic work, or any part of this electronic work, without prominently displaying the sentence set forth in paragraph 1.E.1
- 60. with active links or immediate access to the full terms of the Project Gutenberg™ License. 1.E.6. You may convert to and distribute this work in any binary, compressed, marked up, nonproprietary or proprietary form, including any word processing or hypertext form. However, if you provide access to or distribute copies of a Project Gutenberg™ work in a format other than “Plain Vanilla ASCII” or other format used in the official version posted on the official Project Gutenberg™ website (www.gutenberg.org), you must, at no additional cost, fee or expense to the user, provide a copy, a means of exporting a copy, or a means of obtaining a copy upon request, of the work in its original “Plain Vanilla ASCII” or other form. Any alternate format must include the full Project Gutenberg™ License as specified in paragraph 1.E.1. 1.E.7. Do not charge a fee for access to, viewing, displaying, performing, copying or distributing any Project Gutenberg™ works unless you comply with paragraph 1.E.8 or 1.E.9. 1.E.8. You may charge a reasonable fee for copies of or providing access to or distributing Project Gutenberg™ electronic works provided that: • You pay a royalty fee of 20% of the gross profits you derive from the use of Project Gutenberg™ works calculated using the method you already use to calculate your applicable taxes. The fee is owed to the owner of the Project Gutenberg™ trademark, but he has agreed to donate royalties under this paragraph to the Project Gutenberg Literary Archive Foundation. Royalty payments must be paid within 60 days following each date on which you prepare (or are legally required to prepare) your periodic tax returns. Royalty payments should be clearly marked as such and sent to the Project Gutenberg Literary Archive Foundation at the address specified in Section 4, “Information
- 61. about donations to the Project Gutenberg Literary Archive Foundation.” • You provide a full refund of any money paid by a user who notifies you in writing (or by e-mail) within 30 days of receipt that s/he does not agree to the terms of the full Project Gutenberg™ License. You must require such a user to return or destroy all copies of the works possessed in a physical medium and discontinue all use of and all access to other copies of Project Gutenberg™ works. • You provide, in accordance with paragraph 1.F.3, a full refund of any money paid for a work or a replacement copy, if a defect in the electronic work is discovered and reported to you within 90 days of receipt of the work. • You comply with all other terms of this agreement for free distribution of Project Gutenberg™ works. 1.E.9. If you wish to charge a fee or distribute a Project Gutenberg™ electronic work or group of works on different terms than are set forth in this agreement, you must obtain permission in writing from the Project Gutenberg Literary Archive Foundation, the manager of the Project Gutenberg™ trademark. Contact the Foundation as set forth in Section 3 below. 1.F. 1.F.1. Project Gutenberg volunteers and employees expend considerable effort to identify, do copyright research on, transcribe and proofread works not protected by U.S. copyright law in creating the Project Gutenberg™ collection. Despite these efforts, Project Gutenberg™ electronic works, and the medium on which they may be stored, may contain “Defects,” such as, but not limited to, incomplete, inaccurate or corrupt data, transcription errors, a copyright or other intellectual property infringement, a defective or
- 62. damaged disk or other medium, a computer virus, or computer codes that damage or cannot be read by your equipment. 1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except for the “Right of Replacement or Refund” described in paragraph 1.F.3, the Project Gutenberg Literary Archive Foundation, the owner of the Project Gutenberg™ trademark, and any other party distributing a Project Gutenberg™ electronic work under this agreement, disclaim all liability to you for damages, costs and expenses, including legal fees. YOU AGREE THAT YOU HAVE NO REMEDIES FOR NEGLIGENCE, STRICT LIABILITY, BREACH OF WARRANTY OR BREACH OF CONTRACT EXCEPT THOSE PROVIDED IN PARAGRAPH 1.F.3. YOU AGREE THAT THE FOUNDATION, THE TRADEMARK OWNER, AND ANY DISTRIBUTOR UNDER THIS AGREEMENT WILL NOT BE LIABLE TO YOU FOR ACTUAL, DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE OR INCIDENTAL DAMAGES EVEN IF YOU GIVE NOTICE OF THE POSSIBILITY OF SUCH DAMAGE. 1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you discover a defect in this electronic work within 90 days of receiving it, you can receive a refund of the money (if any) you paid for it by sending a written explanation to the person you received the work from. If you received the work on a physical medium, you must return the medium with your written explanation. The person or entity that provided you with the defective work may elect to provide a replacement copy in lieu of a refund. If you received the work electronically, the person or entity providing it to you may choose to give you a second opportunity to receive the work electronically in lieu of a refund. If the second copy is also defective, you may demand a refund in writing without further opportunities to fix the problem. 1.F.4. Except for the limited right of replacement or refund set forth in paragraph 1.F.3, this work is provided to you ‘AS-IS’, WITH NO OTHER WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED,
- 63. INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PURPOSE. 1.F.5. Some states do not allow disclaimers of certain implied warranties or the exclusion or limitation of certain types of damages. If any disclaimer or limitation set forth in this agreement violates the law of the state applicable to this agreement, the agreement shall be interpreted to make the maximum disclaimer or limitation permitted by the applicable state law. The invalidity or unenforceability of any provision of this agreement shall not void the remaining provisions. 1.F.6. INDEMNITY - You agree to indemnify and hold the Foundation, the trademark owner, any agent or employee of the Foundation, anyone providing copies of Project Gutenberg™ electronic works in accordance with this agreement, and any volunteers associated with the production, promotion and distribution of Project Gutenberg™ electronic works, harmless from all liability, costs and expenses, including legal fees, that arise directly or indirectly from any of the following which you do or cause to occur: (a) distribution of this or any Project Gutenberg™ work, (b) alteration, modification, or additions or deletions to any Project Gutenberg™ work, and (c) any Defect you cause. Section 2. Information about the Mission of Project Gutenberg™ Project Gutenberg™ is synonymous with the free distribution of electronic works in formats readable by the widest variety of computers including obsolete, old, middle-aged and new computers. It exists because of the efforts of hundreds of volunteers and donations from people in all walks of life. Volunteers and financial support to provide volunteers with the assistance they need are critical to reaching Project Gutenberg™’s goals and ensuring that the Project Gutenberg™ collection will
- 64. remain freely available for generations to come. In 2001, the Project Gutenberg Literary Archive Foundation was created to provide a secure and permanent future for Project Gutenberg™ and future generations. To learn more about the Project Gutenberg Literary Archive Foundation and how your efforts and donations can help, see Sections 3 and 4 and the Foundation information page at www.gutenberg.org. Section 3. Information about the Project Gutenberg Literary Archive Foundation The Project Gutenberg Literary Archive Foundation is a non-profit 501(c)(3) educational corporation organized under the laws of the state of Mississippi and granted tax exempt status by the Internal Revenue Service. The Foundation’s EIN or federal tax identification number is 64-6221541. Contributions to the Project Gutenberg Literary Archive Foundation are tax deductible to the full extent permitted by U.S. federal laws and your state’s laws. The Foundation’s business office is located at 809 North 1500 West, Salt Lake City, UT 84116, (801) 596-1887. Email contact links and up to date contact information can be found at the Foundation’s website and official page at www.gutenberg.org/contact Section 4. Information about Donations to the Project Gutenberg Literary Archive Foundation Project Gutenberg™ depends upon and cannot survive without widespread public support and donations to carry out its mission of increasing the number of public domain and licensed works that can be freely distributed in machine-readable form accessible by the widest array of equipment including outdated equipment. Many
- 65. small donations ($1 to $5,000) are particularly important to maintaining tax exempt status with the IRS. The Foundation is committed to complying with the laws regulating charities and charitable donations in all 50 states of the United States. Compliance requirements are not uniform and it takes a considerable effort, much paperwork and many fees to meet and keep up with these requirements. We do not solicit donations in locations where we have not received written confirmation of compliance. To SEND DONATIONS or determine the status of compliance for any particular state visit www.gutenberg.org/donate. While we cannot and do not solicit contributions from states where we have not met the solicitation requirements, we know of no prohibition against accepting unsolicited donations from donors in such states who approach us with offers to donate. International donations are gratefully accepted, but we cannot make any statements concerning tax treatment of donations received from outside the United States. U.S. laws alone swamp our small staff. Please check the Project Gutenberg web pages for current donation methods and addresses. Donations are accepted in a number of other ways including checks, online payments and credit card donations. To donate, please visit: www.gutenberg.org/donate. Section 5. General Information About Project Gutenberg™ electronic works Professor Michael S. Hart was the originator of the Project Gutenberg™ concept of a library of electronic works that could be freely shared with anyone. For forty years, he produced and distributed Project Gutenberg™ eBooks with only a loose network of volunteer support.
- 66. Project Gutenberg™ eBooks are often created from several printed editions, all of which are confirmed as not protected by copyright in the U.S. unless a copyright notice is included. Thus, we do not necessarily keep eBooks in compliance with any particular paper edition. Most people start at our website which has the main PG search facility: www.gutenberg.org. This website includes information about Project Gutenberg™, including how to make donations to the Project Gutenberg Literary Archive Foundation, how to help produce our new eBooks, and how to subscribe to our email newsletter to hear about new eBooks.
- 67. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com