IoT4BPMers

IoT for BPMers
Challenges, case studies and successful
applications
Francesco Leotta
Andrea Marrella
Massimo Mecella
<surname>@diag.uniroma1.it

CITE THIS SLIDE AS:
LEOTTA F., MARRELLA A., MECELLA M. (2019) IOT FOR BPMERS.
CHALLENGES, CASE STUDIES AND SUCCESSFUL APPLICATIONS. IN:
HILDEBRANDT T., VAN DONGEN B., RÖGLINGER M., MENDLING J. (EDS)
BUSINESS PROCESS MANAGEMENT. BPM 2019. LECTURE NOTES IN
COMPUTER SCIENCE, VOL 11675. SPRINGER, CHAM
COPYRIGHT AND NO WARRANTY NOTICE
THESE SLIDES ARE DISTRIBUTED UNDER THE CREATIVE COMMONS LICENSE. ANY VIEWS OR
OPINIONS PRESENTED ARE SOLELY THOSE OF THE AUTHOR AND DO NOT NECESSARILY REPRESENT
THOSE OF ANY ORGANIZATIONS/COMPANIES (INCLUDING SUBSIDIARIES) MENTIONED IN THE
SLIDES.
THE SLIDES ARE DISTRIBUTED IN THE HOPE THAT THEY WILL BE USEFUL, BUT WITHOUT ANY
WARRANTY. THEY ARE PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND
PERFORMANCE OF THE SLIDES IS WITH YOU.
Tutorial @ BPM 2019 – 3 September 2019

Who are we ?
Basically the core of the Information Systems Engineering and BPM group of
Sapienza
• Francesco Leotta
– Fixed-term assistant professor
– Main research interests in Ambient Intelligence, HCI and SOA
• Andrea Marrella
– Assistant professor
– Visiting scholar in Canada (York University in 2012 and 2013) and Estonia
(Tartu University in 2016)
– Main research interests in Process Intelligence, HCI, HRI
• Massimo Mecella
– Associate professor (qualification to full professorship)
– Visiting scholar in US (Telcordia Technologies in 1999, Purdue University in
2005 and 2006)
– Main research interests in SOA, BPM, HCI, data management, smart
applications (AI applied to new scenarios), software engineering
– Wide experience in EU and National projects
– Chair of various conferences in the last years (lately CAiSE 2019 in Roma)

Outline of the talk
• A short introduction to IoT
• Surveying research on business processes
meeting IoT
• Data abstraction and management in BPM
• Adaptation of cyber-physical processes
• Habit mining via process mining in smart
environments

A SHORT INTRODUCTION TO IOT
Tutorial @ BPM 2019 – 3 September 2019 4

Internet-of-Things (IoT)
A system vision of interrelated compu1ng devices,
mechanical and digital machines, objects, animals
or people that are provided with:
• unique idenMﬁers (UIDs)
• the ability to transfer data over a network
without necessarily requiring human-to-human
or human-to-computer interac1on

Smart object
• An object that enhances the interaction with not only people but also with
other smart objects
– smart connected product
– smart connected thing
– smart device
• product, asset, other thing embedded with processors, sensors, software
and connectivity that allow data to be exchanged between the product
and its environment, manufacturer, operator/user, and other products
and systems
– Connectivity enables some capabilities of the product to exist outside
the physical device, in what is known as the product cloud
– The data collected from this product can be then analyzed to inform
decision-making, enable operational efficiencies and continuously
improve the performance of the product

A bit of history (1)
@ 1982 : a modified Coke vending machine at Carnegie Mellon
University was the first Internet-connected appliance, able to report
its inventory and whether newly loaded drinks were cold or not
§ hOps://www.cs.cmu.edu/~coke/history_long.txt
@ 1991 : seminal paper Weiser, Mark: «The Computer for the Twenty-
First Century». ScienAfic American. 265 (3): 94–104
§ doi:10.1038/scien1ficamerican0991-94
@ 1999 : at the World Economic Forum in Davos, Bill Joy (BSD Unix, vi,
Sun Microsystems) envisioned device-to-device communicaAon as a
part of his "Six Webs" framework
@ 1999 : Kevin Ashton of Procter & Gamble, later MIT's Auto-ID
Center, conied the term «Internet of things», though he preferred
the phrase «Internet for things»
§ hOps://www.rfidjournal.com/ar1cles/view?4986

A bit of history (2)
@2005 : Arduino (a single-board microcontroller to be
used in interactive projects) is invented at the
Interaction Design Institute Ivrea (IDII), Italy
• Pervasive / Ubiquitous computing conferences
provided scientific/technical advancements to the field
– Cf. Ubicomp series (since 1999), merged since 2012 with
Pervasive Computing, see http://www.ubicomp.org/sc/
• https://dblp1.uni-trier.de/db/conf/huc/
– Cf. PerCom series (since 2003)
• https://dblp1.uni-trier.de/db/conf/percom/

Enabling technologies for IoT
• Addressability
– RFID
– EPC – Electronic Product Code
– IPv6
– URI
• Wireless/wired communicaMons
– Bluetooth mesh networking
– Light-Fidelity (Li-Fi)
– Near-ﬁeld communica1on (NFC)
– RFID
– Wi-Fi (IEEE 802.11)
– ZigBee (IEEE 802.15.4)
– Z-Wave
– LTE-Advanced
– Low-power wide-area networking (LPWAN) : LoRaWan, Sigfox, NB-IoT, Weightless, RPMA
– 5G
– Very small aperture terminal (VSAT)
– Ethernet
– Power-line communica1on (PLC)

Smart objects as building blocks (1)
• IoT built from
smart objects
raises several important research questions in terms of:
– system architecture, design and development
– human involvement
• E.g.,
– What is the right balance for the distribution of functionality
between smart objects and the supporting infrastructure?
– How do we model and represent smart objects’ intelligence?
– What are appropriate programming models?
– How can people make sense of and interact with smart physical
objects?
G. Kortuem, F. Kawsar, V. Sundramoorthy, D. Fitton. Smart Objects as
Building Blocks for the Internet of Things. IEEE Internet Computing
14, 1 (January 2010), 44-51

• Activity-aware objects
• Policy-aware objects
• Process-aware objects
Awareness - the smart object’s ability to understand (that is, sense,
interpret, and react to) events and human activities occurring in the
physical world
Representation – the smart object’s application and programming
model — in particular, programming abstractions
Interaction – the object’s ability to converse with the user in terms
of input, output, control, and feedback

Research challenges (speciﬁc for IoTers)
• Scalability (massive number of devices)
• Reliable coverage
• Move cloud services to edge of the network
(fog computing)
• Handle data generated by + 50 billion devices
– Reduce data to be stored (processing and storage)
• Power consumption problem
(energy harvesting; software optimization)
• SDN (software-defined networking)/NFV (network
functions virtualization) -based IoT
In brief, see:
https://www.cisco.com/c/en/us/solutions/software-
defined-networking/sdn-vs-nfv.html

Trends to be monitored
• IoT drives demand for data broker and data
analyAcs
– Data must be managed, integrated and analyzed
• IoT drives demand for cloud compuAng
• Interoperability issues
• Security and privacy concerns
I.F. Akyildiz: INTERNET OF THINGS.
Trends, Directions, Opportunities,
Challenges. IFA 2017, Berlin

BUSINESS PROCESSES MEET IOT
Interes1ng research eﬀorts

BP-Meets-IoT Manifesto
https://arxiv.org/abs/1709.03628

Sensing (physical
objects, systems,
humans)
Actuation (physical
objects, systems,
humans)
Event Processing + Learning
Discover
Pre-deﬁned
model
Enact Response
Predict and
Adapt
raw event data
higher level knowledge
Interaction between BPM and IoT
IoT
BPM
17Tutorial @ BPM 2019 – 3 September 2019

Research challenges

Example C2 - Automatic Documentation
of Treatment Processes
Daily Morning Care Routine in a Nursing Home
Physical Objects
Equipped with NFC Tags
Physical objects (care utilities) used:
- Change incontinence pants: Pants
- Go to toilet with resident: Toilet paper
- Wash resident: Washing cloth
- Conduct intimite care: Washing cloth
- Brush Teeth: Toothbrush
- Comb hair: Hair brush
- Apply ointments: Ointment package
Florian Stertz, Juergen Mangler, Stefanie Rinderle-Ma:
NFC-Based Task Enactment for AutomaAc DocumentaAon of Treatment Processes. BPMDS/EMMSAD@CAiSE
2017: 34-48

Example C2 - Autonomous Checkout
Autonomous
checkout solu1ons
are emerging
Machine vision
and AI-based
approach (Amazon
Go; Standard
Cognition)
Bar Code Scanning
(Walmart Scan and Go)

Challenge for C3 and C8 - Mapping of Events to
AcMviMes and AcMviMes to Process Instances
Solution to mapping challenge is highly domain-specific
Autonomous checkout
- Customer identified through mobile app
- Check-in into shop starts process instance
- Check-out stops process instance
Automatic Treatment
Documentation
- Patient identified through NFC
reader
- Process instance starts when
first activity is executed
- Process instance stops when all
activities have been conducted
- 1 NFC sensor per phsyical object

…
Arik Senderovich, Andreas Rogge-Solti, Avigdor Gal, Jan Mendling, Avishai Mandelbaum: The ROAD from Sensor Data to Process
Instances via Interaction Mining. CAiSE 2016: 257-273
Francesco LeoOa, Massimo Mecella, Daniele Sora: Visual process maps: a visualiza1on tool for discovering habits in smart homes.
Journal of Ambient Intelligence and Humanized Compu1ng hOps://doi.org/10.1007/s12652-019-01211-7

BPM modeling languages and execuMon
environments considering IoT
Schönig, S., Ackermann, L. and Jablonski, S. Internet of Things Meets BPM: A Conceptual Integra1on
Framework. DOI: 10.5220/0006824803070314. In Proceedings of 8th nternaAonal Conference on SimulaAon
and Modeling Methodologies, Technologies and ApplicaAons (SIMULTECH 2018)
Model
ImplementExecute
Monitor &
Optimize

Model
ImplementExecute
Monitor &
Optimize
Extensions to the modeling language in order to explicitly
consider IoT, e.g.,
- Internet-of-Things-Aware Process Modeling Method (IAPMM),
by Petrasch and Hentschke
- BPMN4CPS, by Graja et al.
- the approach of Sonja Meyer et al. (cf. CAiSE 2013/2015)
- the work of Giovanni Meroni (CAiSE 2019 PhD thesis award)
using artifact-centric modeling for smart objects

Model
ImplementExecute
Monitor &
Optimize
Extensions to some execu1on engines:
- WS-BPEL-based, by George (2008), Domingos et al. (2014),
Mateo et al. (2012)
- BPMN-based, by Beyer, Weske et al. (cf. BPM Demo Track
2016), Schönig, Jablonski et al. (cf. BPM 2018 Workshops)

Considering processes/IoT during the
design of applicaMons (1)
• Besides modeling, it is important to consider
how IoT-based applications/scenarios can
benefit from BPM
– «The single app-controlled Phillips Hue lamp will
not profit from BPM concepts, while a scenario
that schedules maintenance appointments for a
fleet of cars might»

Considering processes/IoT during the
design of applicatios (2)
Classiﬁca1ons of scenarios based on
• ParMcipants (sensor, actuator, display, controller,
complex device, web service, human beings)
• Control (central, on device, distributed)
• InteracMon (things-to-things, things-to-controller)
• Data
• AutomaMon
• Ownership
Sankalita Mandal, Marcin Hewelt, Maarten Oestreich, and Mathias
Weske (2018). A Classiﬁca1on Framework for IoT Scenarios. Proc. BPM
2018 Workshops

IoT and … blockchain (of course J)
• IoT devices generate data that need to be shared for
demonstra1ng some proper1es with the guarantees
of no modiﬁca1ons
– In a distributed and mul1-organiza1on context, in which
there is no trusted third party available
• Examples:
– Modum.io
(from the Weber’s talk)
– Siemens
SIMATIC IoT 2040 Gateway

WRAPPING UP
Data abstraction and management
is the challenge (!!)

Wrapping-up : the notion of Cyber-
Physical Processes (CPPs)
• Smart objects, having physical impact on the
world, allows to digitally improve the
automa1on of processes in so-called Cyber-
Physical Systems (CPSs)
CPSs vs. IoT ? See an interes1ng discussion on
https://www.researchgate.net/post/What_is_the_diff
erence_between_Internet_of_Things_IoT_and_Cyber_Ph
ysical_Systems_CPS
My preferred answers are that both terms are interchangeable (cf. ISO/IEC
JTC1 2015) and diﬀerent scien1ﬁc/technical communi1es prefer one term
over the other (e.g., CPS more preferred in US and among engineering
disciplines - EE, ME, AE - and CS, whereas IoT is more preferred in EU and
China and among industry and network people

Wrapping-up : the notion of Cyber-
Physical Processes (CPPs)
• Smart objects, having physical impact on the
world, allows to digitally improve the automa1on
of processes in so-called Cyber-Physical Systems
(CPSs)
• This is the IoT scenario in which we are interested
as BPMers, the one of Cyber-Physical Processes
(CPPs), i.e., processes in which (some) tasks are
autonomously enacted by smart objects and
have a physical eﬀect

Focusing on data abstracMon (1)
• One of the cri1cal points, which we will
address in the following, is how to properly
abstract data
• Why ?
– IoT data are very ﬁne-grained, not at the proper
granularity level typical of BPM approaches
• The Digital Twin abstrac1on is the proper abstrac1on
layer in which such granularity-mismatch is solved, but
it should be devised how to solve the mismatch semi-
automa1cally and not necessarily ad-hoc

Focusing on data abstraction (2)
– IoT data can be con1nuous values over huge
domains
• Not good if you like to adopt automa1c reasoning /
veriﬁca1on techniques
– IoT data require learning to be exploitable in
processes. But such learning should be
explainable, as BPM is inherently to make
processes «white box» and therefore all should be
clear and explainable
• ???? Research here is much needed

SMARTPM
Adapta&on of cyber-physical processes

• It is the ability of a process to cope with excep@ons and deviate at run-
@me from the execu&on path prescribed by the process.
• Exis&ng BPM environments provide support for the handling of :
– an@cipated excep@ons, captured in the process model at design-&me.
– unan@cipated excep@ons, managed manually at run-&me.
35
Process Adaptation

Adaptation of (traditional) business processes
• Tradi&onal business processes (e.g., administra&ve and
financial processes) are usually easily predictable.
– They reflect rou@ne work with low flexibility requirements.
– AJer being modeled, they can be repeatedly instan@ated and
executed in a controlled way.
– Excep&on handlers can be properly modeled at design-@me.
– Data flows do not play a relevant role in process adapta&on.

Cyber-Physical Systems
• Cyber-physical systems are characterized by the presence of heterogeneous devices
with diﬀerent architectures, compu&ng and communica&on capabili&es.
– Example: In a modern manufacturing plant, each element of the factory is a5ached to
sensors and actuators. Data sharing is enforced throughout the plant, to allow produc8on
lines to run with real-8me ﬂexibility ensuring trouble-free manufacturing.

Cyber-Physical Process (CPP)
• In a ceramic plant, a dedicated cyber-physical process (CPP) coordinates the working
of the robot arms and the machinery in the various steps of the produc&on line.
ROTOMOULDING DRYING GLAZING FIRING
Starting from a digital CAD model,
an initial raw model of a ceramic
product is generated.
When a step completes, a quality check is performed by
ac&va&ng a digital 3D scanner that analyzes the surface of the
ceramic elements to iden&fy the presence of ruptures or defects.
A raw model has an higher volume than the final product,
since it will lose part of its volume during the next steps of
the process (e.g., due to humidity, temperature, etc.)
A conveyor belt is used to move ceramic elements from a step
to another. Each step is performed by a diﬀerent sta&c robo&c
arm or machinery located in a ﬁxed posi&on of the plant.

Example of 3D scanner

Poten@al excep@ons
• Some exceptions may be caused by the deformation of ceramic
materials during the drying/glazing/firing steps.
– An incorrect thermal expansion of the elements’ body may cause their rupture
If a deforma&on aJer the glazing step is evaluated
as cri&c and not anymore “adjustable”, it is useless
to proceed to the next step of the process.
Recovery procedure: A moving robot pick up and
deposit the broken element in a warehouse and a
technician clean the conveyor belt from debris.

Poten@al excep@ons
• A CPP can also be jeopardized by the occurrence of exogenous events, which
can asynchronously change the contextual properties of the environment.
– An anomalous value of an environmental parameter (temperature, humidity, pressure, etc.)
may affect the quality of the transformations of the ceramic material.
If during the ﬁring step the temperature of an element
reaches a dangerous value, it must be stopped before it
causes defects to the ceramic materials under ﬁring.
Recovery procedure: A technician configures the oven
system to modify its temperature to a reasonable value.

Adapta@on of CPPs
Key fact: recovery procedures depend on the actual context (e.g., the
posi&ons of actors and robots, robot’s baYery levels, the range of the sensors,
whether a loca&on has become dangerous to get it, etc.)
1) the number of an@cipated excep@ons to be iden&ﬁed at the outset (and ways to
overcome them) is oJen too large;
2) many unan@cipated excep@ons may arise during process execu&on, and their
resolu&on should be performed on a case-by-case basis, by exploi&ng informa&on
gathered at run-&me.
Challenge: Build real-&me monitoring and automated adapta@on features
during process execu&on, in order to:
1) synthesize on-the-ﬂy recovery procedures that solve all excep@ons (an&cipated and
not an&cipated) into the original process;
2) achieve the overall objec&ves of the original process s&ll preserving its structure by
minimizing any human interven@on.

SmartPM Approach
43
Modeling approach towards a declara@ve specifica@on of process tasks.
n Each task is described with the needed precondi&ons for execu&ng it and the expected effects
produced aJer the task execu&on. Data
Process Adapta@on: the ability to reduce the gap from the expected reality ψ(s) – the
(idealized) model of reality used to reason – and the physical reality φ(s).
The aim is to find a recovery procedure that turns φ(s) (the faulty physical reality) into
ψ(s) (the desired expected reality).
φ(s)
φ(s+1)
ψ(s+1)
Intui@on: for each execu1on step
if φ(s+1) is different from ψ(s+1)
then adapt
Physical reality at
situation s records the
actual values of task
outcomes.
Each task has a set of effects that
turn φ(s) into φ(s+1).
Expected reality records the
desired effects of each task.

Deﬁning data in SmartPM
• In SmartPM, contextual information is represented through a
domain theory consisting of discrete objects and variables which
may change as effects of task outcomes and exogenous events.
• For example:
– Location: <loc_glazing, loc_firing, ...>
– Object: <obj1, obj2, obj3, …>
– Status(obj: Object) = [ok, high_temp, low_pressure, …]
• Physical reality can be seen as the set of all variable values in a
specific state of the execution.
• Expected reality records the desired effects of each task, as defined
at design-time.

Example
φ(s) = …AND status(obj1) = ok AND status(obj2) = ok …AND …
ψ(s) = …AND status(obj1) = ok AND status(obj2) = ok …AND …
φ(s+1) = …AND status(obj1) = high_temp AND status(obj2) = high_temp …AND …
ψ(s+1) = …AND status(obj1) = ok AND status(obj2) = ok …AND …
The exogenous event asynchrounously changes φ(s)
PROCESS ADAPTATION REQUIRED!

Physical-to-digital interface
(where data abstrac@on happens)
• Sensors data are oJen con@nuous values over huge domains.
• To exploit automa&c reasoning/verifica&on techniques, such
data must be abstracted as discrete variables grounded into
finite domains.
Physical-to-Digital interface
• SmartPM provides some web tools that allow us to associate
some of the data objects defined in the domain theory with
the con&nuous data values collected from the environment.

The architecture of SmartPM
Sensors and actuators that affect the state of the
physical environment. A physical-to-digital interface
transforms raw data collected by the sensors into
machine-readable events, and converts high-level
commands sent by the upper layers into raw
instructions readable by the actuators.
Set of services offered by the real-world en&&es
(robots, humans, etc.) to perform specific tasks. High-
level commands can be composed into complex ones.
Process execu&on, monitoring and adapta&on of
running instances in case of (un)an&cipated excep&ons
or exogenous events.
Defini&on of process specifica&ons in terms of control
flow, tasks precondi&on and effects, and formaliza&on
of the data reflec&ng the contextual knowledge of the
cyber-physical environment under observa&on.

Design Tool
Modeling canvas to deﬁne the control ﬂow of
the process and an editor to create and
modify the data, the resource perspec&ve
and all the contextual informa@on of the
scenario in which the process will be
executed.

Action based languages for SmartPM
Intui@on: Resor&ng to ac@on-based languages in AI
• Situa@on Calculus to model:
– the contextual sekng in which the process is meant to run
– the support framework for managing the task life cycle
• Customiza&on of an IndiGolog Interpreter to:
– monitor the online execu&on of running processes
– detect poten&al mismatches at run-&me
– invoke a state-of-the-art planner to synthesize a recovery procedure
• Automated Planning to generate a recovery plan that turns φ(s) into ψ(s)
– Planning domain:
• process tasks represented as planning ac&ons in PDDL
• predicates to capture the contextual data describing the process domain
– Planning problem: instan&a&on of the contextual data in a star&ng state (the faulty
physical reality φ(s)) and in a goal state (the desired expected reality ψ(s))

Task handler of SmartPM
The Task Handler is realized for
Android devices. It supports the
visualiza@on of assigned tasks and
enables star@ng task execu@on and
no@fying of task comple@on by
selec&ng appropriate outcomes.

The SmartPM Location Tool
51
SmartPM provides a Loca@on web tool (as a
Google Maps plugin) that allows a process
designer to mark areas of interest from a real map
(by selec&ng la&dude/longitude values) and
associate them to the discrete loca@ons (e.g.,
loc00, loc01, etc.) deﬁned during the design stage
of a process

The SmartPM Arduino Tool
53
Arduino has a large variety of sensors available to
measure different environmental values, for example
different gas levels in the air, water quality, radiation
level, etc. Arduino can be connected with Android via
Bluetooth for transferring the data.

Main references on SmartPM
• A. Marrella, M. Mecella, S. Sardina. Suppor@ng Adap@veness of
Cyber-Physical Processes through Ac@on-based Formalisms. AI
Communica8ons, Volume 31, Issue 1, IOS Press, 2018
• A. Marrella, M. Mecella, S. Sardina. Intelligent Process Adapta@on
in the SmartPM System. ACM Transac8ons on Intelligent Systems
and Technology (TIST), Vol. 8(2), 2017
• A. Marrella, M. Mecella. Adap@ve Process Management in Cyber-
Physical Domains. Book Chapter, Advances in Intelligent Process-
Aware Informa8on Systems, Intelligent Systems Reference Library,
Volume 123, Springer, 2017

VISUAL PROCESS MAPS
Process mining from sensor logs for analyzing
human habits in smart environments

The Ambient Intelligence Loop
Context
Extraction
Acting
Sensing
Decision
Making
Knowledge
Smart
Cyber-
Physical
Environment
Runtime
Learning
Direct
Human
Control
Cyber-Physical
Ambient Intelligence

Knowledge: Models for What? (1)
57
• Context: the state of the environment
including the human inhabitants with their
ac8ons/ac8vi8es/habits
• AcTon: atomic interac8on of the human with
the environment or a part of it (e.g., a device)
– Some techniques in literature focuses only on
ac8ons
– Other techniques skip ac8ons while recognizing
ac8vi8es
• Human Preferences: a speciﬁc set of rules
over contextual variables. The goal here is
user sa8sfac8on.
– Controllable and Uncontrollable variables

Knowledge: Models for What? (2)
58
• Activity: a sequence of actions (one in the
extreme case) or sensor
measurements/events with a ﬁnal goal
– Activities can be collaborative
• Habit: a set of interleaving of activities that
happen in speciﬁc contextual conditions
– E.g., what a user does each morning between
08:00 and 10:00am
– E.g., what a user does between very specific
actions (e.g., leaving the bed and leaving the
house)

Classification of Modeling Methods (1)
• SpeciﬁcaTon-based methods
– Knowledge expressed in terms of some kind of
logic language
– Pros J: Human readable à easy to validate
– Cons L: Hand made by experts à feasible only
with a limited number of sensors

Classification of Modeling Methods (2)
• Learning-based methods
– Techniques from both machine learning and
data mining
• Supervised, Unsupervised, Semi-Supervised
methods
– Pros J: No need for hand-made models
• Supervised methods s8ll require a lot of labeled data
– Cons L: Usually not human readable
• E.g., sta8s8cal formalisms as HMM

An Idea: BPM?
• Business Process Management - BPM can be
helpful at modeling human habits and ac8vi8es
– Due to the different applicaTon contexts, challenges
must be addressed
• Few approaches using workflows already
proposed but they do not leverage the strong and
recent research in process mining
• Great benefits from the point of view of visual
analysis
• Grounded in logics and very descrip8ve
• PotenTally a trade-off between specificaTon-
based and learning-based approaches!!!
64
Az8ria, A.; Izaguirre, A.; Basagoi8, R.; Augusto, J.C.; Cook, D. Automa8c modeling of frequent user
behaviours in intelligent environments. In Proc. Intelligent Environments 2010.Tutorial @ BPM 2019 – 3 September 2019

Dealing with Granularity
• Clear gap between the granularity of sensor logs
and the traces used for process mining [Baier2013]
• No one-to-one correspondence between sensor
measurements and performed actions (tasks)
– A single user action may trigger many sensor
measurements
– A single sensor measurement may be related to
several actions
• Required approach:
1. Aggregate sensor measurements to recognize actions
2. Apply process mining
• The kind of available sensors strongly influences the
granularity and confidence of recognized actions
Baier, T., Mendling, J.: Bridging abstrac8on layers in process mining by automated
matching of events and ac8vi8es. In BPM 2013.

Log SegmentaTon (1/2)
66
• A common prerequisite of process mining techniques is to
have an event log explicitly segmented into cases (process
instances)
– Case “start” and case “end” events
– For each event, which case it belongs to
– Relatively easy to instrument a process in an industrial or
business environments
• This assumption is usually not met by sensor logs, as
labeling is generally an expensive task to be performed by
humans
– Especially difficult to associate actions (derived from sensor
measurements) to activities and habits in the interleaved case and
in presence of multiple users

Log Segmentation (2/2)
67
• How do we define habits and activities?
– Manually defined?
– Automatically learned and adapted?
– Active learning?
• What about multiple users?
– Usually sensor logs do not contain any information about
which user(s) caused a certain sensor to trigger or to provide
a specific measurement
• The employment of body-area sensors and tags is usually perceived
as invasive by the user and do not solve all the issues
– Mining habits in a multi-user scenario is significantly harder
• e.g., even though multiple users can be identified by the spatial
distance between sensors triggering close in time, when trajectories
intersect tracking techniques or reasoning must be employed to keep
following users

Which Formalism?
68
• Question: Does a human habit resemble a “spaghetti” process?
– Approaches to deal with unstructured processes do exist as both imperative and
declarative modeling formalisms
– Human processes in smart spaces are very similar to “artful” processes (e.g.,
treating patients in hospitals)

Visual Process Maps
• Our proposed pipeline for learning and visual analysis
• Based on fuzzy mining
• Human expert is involved in visually analyzing the log
and the extracted models
• Final goal is analysis of human habits

The Dataset (1/3)
• Aruba Dataset from CASAS project
hfp://casas.wsu.edu/datasets/
– Dataset covering life of a
woman for two years
– Labeled dataset
• Meal_Prepara8on (1606)
• Relax (2910)
• Ea8ng (257)
• Work (171)
• Sleeping (401)
• Wash_Dishes (65)
• Bed_to_Toilet (157)
• Enter_Home (431)
• Leave_Home (431)
(F) (G)
(A) (B)
(C)
(D) (E)

The Dataset (2/3)
• PIR sensors MXXX
• Door closure sensors
DXXX
• Temperature sensors
TXXX
• Sensor log row format
<date time sensor value [label]>
– The label denotes whether an activity starts or ends
– Interleaved activities

The Dataset (3/3)
• The input to our tool includes:
– The data file containing the dataset
– A map of the house
– The aruba_sensor_map.csv containing rows in the
format <Sensor X Y floor Room Object Note> where:
• Sensor is the name of a sensor inside the dataset
• X Y floor and Room represent the loca8on with respect to
the aruba.jpg file (X and Y are pixels)
• Object is the name of the physical object in
correspondence of the sensor

SegmenTng the Dataset
• We split the dataset into process cases the
original dataset
• Two folders are generated in the
segmentation folder:
– Output date: here we have a file for each day (the
habit here is the daily routine)
– Output task: here we have a file for each task
(here we consider the activities separately)
– A different file for each repetition of the
habit/activity (process case)

Playing the Log (1)
74
Simula8on
Speed
Current
Event
Event
Range
Play/Pause
Current
Event
Descrip8on
Sensor
Shown
Export
Image

Playing the Log (2)
75
While playing, the path of
the user is shown:
• Path color denotes the
age (from green – more
recent – to blue – older)
of the measurement
• A number is associated
to each edge (the bigger
the newer)
• The size of the dot
reﬂects the 8me spent
under the sensor

CompuTng Subtrajectories
76
The «Calculate
subtrajectories» bufon
allows to turn sensor
measurements into
ac8ons…let’s see how

Bridging the Gap between Sensor
Logs and Event Logs
• TRACLUS [Lee2007]: Trajectory clustering algorithm
– Two phases:
• Trajectory partitioning
• Density-based line-segment clustering
• We can now classify each trajectory as a specific
movement action: STAY, AREA, MOVEMENT

Bridging the Gap between Sensor Logs
and Event Logs
Given a trajectory ! returned by TRACLUS
"# $ reflects how many sensors are involved in
the trajectory
%& ! =
()*+,- ./ 01231(43 2,(2.-2
3.356 ()*+,- ./ 2,(2.-2
"7 $ reflects how trajectory time is distributed
among sensors (Gini coefficient)
"8 $ reflects how much time is spent under a
single sensor
%9 ! =
31*, 2:,(3 )(0,- 3ℎ, *.23 /-,<),(3 2,(2.-
3.356 31*, ./ 3-5=,43.->

Bridging the Gap between Sensor Logs
and Event Logs

Computing Subtrajectories
80
The list of user actions. By
clicking on each of them,
player controls are adjusted
accordingly to play it.
Once ac8ons are computed,
we can apply process
mining by clicking on Run
ProM/Save Disco File.

Discovering Human Habits
• Once the sensor log is turned into a
(movement) action log, we can apply fuzzy
mining:
– Process discovery by using
ProM implementation of
fuzzy mining
– Nodes representing actions
• In our case STAY or AREA actions
• MOVEMENT actions ignored

Discovering Human Habits
• We initially segment traces splitting on:
– Entire days, i.e., we extract fuzzy models of the «daily habit»
– Portions of the logs manually indicated by user,
i.e., we extract fuzzy models of «activities»

Main references on VPM
• Leotta F. Mecella M., Sora D., Spinelli G. "Pipelining user
trajectory analysis and visual process maps for habit
mining." 2017 IEEE Ubiquitous Intelligence & Computing,
2017.
• Leotta F., Mecella M., Sora D. "Visual analysis of sensor logs
in smart spaces: Activities vs. situations." 2018 IEEE Fourth
International Conference on Big Data Computing Service
and Applications (BigDataService). IEEE, 2018.
• Leotta F., Mecella M., Sora D. “Visual Process Maps: A
Visualization Tool for Discovering Habits in Smart Homes.”
Journal of Ambient Intelligence and Humanized Computing,
2019.

CONCLUDING REMARKS

Why IoT ?
h"ps://azure.microso1.com/mediahandler/ﬁles/resourceﬁles/
iot-signals/IoT-Signals-Microso1-072019.pdf

A lot of road ahead
Cf. Daniele Mazzei’s invited talk @ 3rd InternaKonal Workshop on BP-
Meets-IoT

Automation vs. Autonomy
Cf. Joseph Sifakis (ACM Turing Award): Autonomous Systems – An
Architectural CharacterizaKon. Keynote at ICWS 2019, on
arxiv.org

BPM can help

IoT4BPMers

More Related Content

What's hot (20)

Similar to IoT4BPMers (20)

Recently uploaded (20)

IoT4BPMers