Towards a Benchmark for BPMN Engines

Vincenzo Ferme, Cesare Pautasso | University of Lugano (USI)
Towards a
Benchmark for
BPMN Engines

Towards a Benchmark for BPMN Engines
2
University of Stuttgart (IAAS)
Germany
University of Lugano (USI)
Switzerland
Prof. Cesare Pautasso
Vincenzo Ferme
Ana Ivanchikj
Marco Argenti
Gabriele Cerfoglio
Simone D’Avico
Abdil Cakal,Andreia Faria Carvalho
Nicolò Linder, Sonny Monti
Prof. Frank Leymann
Dr. Dieter H. Roller
Marigianna Skouradaki
Abdul Wahab
Nagarjuna Siddam
Tayyaba Azad
BaluVenu Thayil
Credits

3
The Impact of TPC Benchmark on DBMS Performance
Source: http://www.tpc.org/tpcc/results/tpcc_results.asp

4
What is a Workﬂow?
Register
Reserv.
Send
Doc.
Conﬁrm
Request

4
Register
Reserv.
Send
Doc.
Conﬁrm
Request
Bob
1

4
Register
Reserv.
Send
Doc.
Conﬁrm
Request
Bob
1
Reservation
Service
2
Bob
2

4
Register
Reserv.
Send
Doc.
Conﬁrm
Request
Bob
1
Reservation
Service
2
Bob
2
Alice
3

4
Register
Reserv.
Send
Doc.
Conﬁrm
Request
Bob
1
Reservation
Service
2
Bob
2
Alice
3
4
Conﬁrmation
Handler

Workflow Management System
4
Register
Reserv.
Send
Doc.
Confirm
Request
Bob
1
Reservation
Service
2
Bob
2
Alice
3
4
Confirmation
Handler

Applications
What is a Workﬂow Management System (WfMS)?
5
WfMS Users
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C

6Application Server
Workﬂow Management System’s Main Components
WES
Job
Executor
Core
Engine
Application Server

6Application Server
WES
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
Application Server

6Application Server
WES
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
Task Dispatcher Users
Application
Application Server

6Application Server
WES
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
Application
Application Server
Event Handler
Service Invoker
Web
Service
Application

6Application Server
WES
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
Instance
Database
DBMS
Persistent Manager
Transaction Manager
Application
Application Server
Event Handler
Service Invoker
Web
Service
Application

System’s Architecture
• Distributed workflow support
• Migrating workflow objects support
• Transactional workflow support
Deployment Infrastructure
• Standalone
• Cluster deployment
• Cloud deployment
• Mobile deployment
Supported Languages
• BPMN, BPEL, Petri-Nets, …
7
Functionality
• Dynamic handling of Workflows
• Integration capabilities
Workflow Management Systems’ Diversity
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011

• Standalone
Supported Languages
8
Functionality
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011

• Standalone
Supported Languages
9
Functionality
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011

• Standalone
Supported Languages
10
Functionality
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011

11
https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines
Jan 2011
BPMN 2.0
Jan 2014
BPMN 2.0.2
ISO/IEC 19510
Aug 2009
BETA
BPMN 2.0
NumberofBPMN2.0WfMSs
0
5
10
15
20
25
Year of the First Version Supporting BPMN 2.0
2009 2010 2011 2012 2013 2014 2015
1
BPMN 2.0: A Widely Adopted Standard (?)

12
BPMN 2.0

12
B
A
Control Flow
BPMN 2.0

12
Data Flow
A B
C
B
A
Control Flow
BPMN 2.0

12
Data Flow
A B
C
B
A
Control Flow
Events
BPMN 2.0

12
Data Flow
A B
C
B
A
Control Flow
Events Activities Task Types Execution Behaviour
A B
C D
E F
G
H
I
BPMN 2.0

13
Design and implement the ﬁrst benchmark to
assess and compare the performance of WfMSs
that are compliant with Business Process Model
and Notation 2.0 standard.
”
“
BenchFlow Project
the goal
http://benchflow.inf.usi.ch

14
end-users, vendors, developers
Why do we Need a Benchmark?

14
1. How to choose the best WfMS in accordance  
with the company’s technical requirements?
2. How to choose the best WfMS  
in accordance with the company’s  
business process models (workﬂows)?
B
C
D

14
1. How to choose the best WfMS in accordance  
with the company’s technical requirements?
2. How to choose the best WfMS  
in accordance with the company’s  
business process models (workﬂows)?
B
C
D
4. How to identify WfMS’s bottlenecks?
3. How to evaluate performance improvements 
during WfMS’s development?

15
Workload
Mix
Load Functions
Test Data
Workload
Model
WfMS
Conﬁgurations
Test
Types
Workload
Metrics
KPIs
Derive
Performance
Data
Benchmark Elements for BenchFlow

16
Test Types Load Test, Stress Test
Benchmark Element Limitations
State of the Art Limitations (BPEL WfMS)
[SEKE ’14]
Röck et al.

16
Test Types Load Test, Stress TestWorkloadModel
Workload Mix
Load Functions
Test Data
Unclear Deﬁnition
Simplistic
Not Realistic
[SEKE ’14]
Röck et al.

16
Workload Mix
Load Functions
Test Data
Unclear Deﬁnition
Simplistic
Not Realistic
Systems
Number of WfMS
WfMS Conﬁgurations
2–5
Not Extensive
[SEKE ’14]
Röck et al.

16
Workload Mix
Load Functions
Test Data
Unclear Deﬁnition
Simplistic
Not Realistic
Systems
Number of WfMS
WfMS Conﬁgurations
2–5
Not Extensive
Analyses
Metrics
KPIs
Too General
Too General
[SEKE ’14]
Röck et al.

• Relevant
• Representative
• Portable
• Scalable
• Simple
Benchmarking Requirements
• Repeatable
• Vendor-neutral
• Accessible
• Efﬁcient
• Affordable
• K. Huppler, The art of building a good benchmark, 2009
• J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993
• S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A
challenge to software engineering, 2003
References:

18
Main Challenges in Benchmarking BPMN 2.0 WfMSs

18
WORKLOAD MODEL

18
Workload Mix
20%
80%
C
A
B
D
WORKLOAD MODEL

18
Workload Mix
20%
80%
C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S

18
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S

18
20%
80%
C
A
B
D
BENCHMARK EXECUTION
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S

18
20%
80%
C
A
B
D
WfMS-speciﬁc APIs
and BPMN 2.0 Customisations
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S

18
20%
80%
C
A
B
D
WfMS-speciﬁc APIs
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S
Asynchronous Execution
of Workﬂows
WfMS
?
start

18
20%
80%
C
A
B
D
WfMS-speciﬁc APIs
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
C
A
B
U S
of Workﬂows
WfMS
?
start

18
Performance
Metrics and KPIs
20%
80%
C
A
B
D
WfMS-speciﬁc APIs
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
C
A
B
U S
of Workﬂows
WfMS
?
start

19
Deﬁne the Workload Mix
based on real-world BPMN 2.0 process models
gram
Register
Reserv.
Send
Doc.
Confirm
Request

19
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Obtain Real-World BPMN 2.0 Process Models
MAIN CHALLENGE:

19
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Obtain Real-World BPMN 2.0 Process Models
MAIN CHALLENGE:
gram
Register
Reserv.
Send
Doc.
Confirm
Request
B
A
C

> 100 METRICS:
Size, Structure, External
Interaction, Data Handling,
Complexity
20
characterise real-world BPMN 2.0 process models collections
http://benchflow.inf.usi.ch/bpmeter
BPMeter:
Web Service and application
for static analysis of BPMN
2.0 collections

21
NUMBER OF WfMSs SUPPORTING THE FEATURE
NUMBEROFREAL-WORLDMODELS
USINGTHEFEATURE
M
O
CK
DATA
heterogeneous usage and support of BPMN 2.0

21
NUMBER OF WfMSs SUPPORTING THE FEATURE
NUMBEROFREAL-WORLDMODELS
USINGTHEFEATURE
0
200
4
F400
5
600
2
700
8
800
12
950
10
M
O
CK
DATA
heterogeneous usage and support of BPMN 2.0

22
Different Types of Benchmarks
BPMN 2.0 SUPPORTED ELEMENTS
BPMN2.0APPLICATIONDOMAIN
CUSTOMCORE ADVANCED FULL
core, advanced, full, custom

22
YOUR 
COMPANY

22
YOUR 
COMPANY
DOMAIN
SPECIFIC

22
YOUR 
COMPANY
DOMAIN
SPECIFIC
GENERAL

23
Workload Mix Characterisation Overview
Models
Collection
Characterisation
Characterisation
Criteria
Benchmark
Workﬂows
Characterise
Selection
Criteria

23
Reoccurring Structures
A
H
Models
Collection
Characterisation
Characterisation
Criteria
Benchmark
Workﬂows
Characterise
Selection
Criteria

23
Clusters of ModelsReoccurring Structures
A
H
Models
Collection
Characterisation
Characterisation
Criteria
Benchmark
Workﬂows
Characterise
Selection
Criteria

24
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
discover reoccurring structural patterns

24
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
[SOSE ’15]
a3
a2
a1
a6
a5
a4
Graph
Matching
REOCCURRING STRUCTURES
Skouradaki et al.

24
User Deﬁned
Selection Criteria
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
[SOSE ’15]
a3
a2
a1
a6
a5
a4
Graph
Matching
Skouradaki et al.

24
User Deﬁned
Selection Criteria
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
WORKLOAD MIX
a1
a2
a1
a3 a5
50%
50%
Composition
Criteria
[SOSE ’15]
a3
a2
a1
a6
a5
a4
Graph
Matching
Skouradaki et al.

25
discover clusters of models
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES

25
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
> 100 METRICS: Size, Structure, External Interaction, Data Handling, Complexity
CLUSTERS OF MODELS
F
E
G
B
C
D
A
H
PCA 
+
Clustering

25
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
> 100 METRICS: Size, Structure, External Interaction, Data Handling, Complexity
CLUSTERS OF MODELS
F
E
G
B
C
D
A
H
PCA 
+
Clustering
WORKLOAD MIX
50%
50%
Clustroid
Models
B
C
D
A

Deﬁne the Test Data
26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
different types of test data

26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Test Data:

26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Test Data:
40%
60%
Condition Data

26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Test Data:
…
40%
60%
Condition Data

27
am
Register
Reserv.
Send
Doc.
Confirm
Request
Deﬁne the Load Functions
different types of load functions

27
am
Register
Reserv.
Send
Doc.
Confirm
Request
P(branch)
different types of load functions

28
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
process logs, experts’ opinion

28
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
40%
60%
P(branch)

28
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
Completion Time
Completion Time
40%
60%
P(branch)

28
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
Response Time
Completion Time
Completion Time
40%
60%
P(branch)

28
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
Response Time
Requests Rate
Completion Time
Completion Time
40%
60%
P(branch)

28
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
Response Time
Requests Rate
Completion Time
Completion Time
40%
60%
P(branch)
Experts

29
Test Types and the Workload Model
Performance Test Types
• Load testing
• Stress testing
• Soak testing
• Spike testing
• Scalability testing
• Capacity testing
• Conﬁguration testing
• Isolation testing
• …
Workload Model

29
• Load testing
• Stress testing
• Soak testing
• Spike testing
• …
Workload Model
0
40
80
10
20
30
40
50
LoadFunction
0
200
400
10
20
30
40
50

29
• Load testing
• Stress testing
• Soak testing
• Spike testing
• …
Workload Model
0
40
80
10
20
30
40
50
LoadFunction
0
200
400
10
20
30
40
50
Unsaved diagram
Register
Reserv.
Send
Doc.
Confirm
Request
Unsaved diagram
Register
Reserv.
Send
Doc.
Confirm
Request
Unsaved diagram
Register
Reserv.
Send
Doc.
Conf
Requ
Unsaved diagram
Register
Reserv.
Send
Doc.
Conf
Requ
Unsaved diagram
Register
Reserv.
Send
Doc.
Unsaved diagram
Register
Reserv.
Send
Doc.
WorkloadMix

29
• Load testing
• Stress testing
• Soak testing
• Spike testing
• …
Workload Model
0
40
80
10
20
30
40
50
LoadFunction
0
200
400
10
20
30
40
50
TestData
Unsaved diagram
Register
Reserv.
Send
Doc.
Confirm
Request
Unsaved diagram
Register
Reserv.
Send
Doc.
Confirm
Request
Unsaved diagram
Register
Reserv.
Send
Doc.
Conf
Requ
Unsaved diagram
Register
Reserv.
Send
Doc.
Conf
Requ
Unsaved diagram
Register
Reserv.
Send
Doc.
Unsaved diagram
Register
Reserv.
Send
Doc.
WorkloadMix

30
Performance
Metrics and KPIs
WfMS-speciﬁc APIs
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION ANALYSES
20%
80%
C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S
of Workﬂows
WfMS
?
start

31
Instance
Database
DBMSFaban Drivers
ContainersServers
DATA 
TRANSFORMERS
ANALYSERS
Performance
Metrics
Performance
KPIs
harness
WfMS
TestExecutionAnalyses
Faban
+
Web
Service
Enabling the Benchmark Execution and Analyses
BenchFlow framework
Minio

Performance Benchmark
BenchFlow Framework
32
System Under Test (SUT)
core requirements & functionalities

BenchFlow Framework
32
• Automate the SUT deployment
• Simplify the SUT’s deployment 
conﬁguration

• Manage the benchmark lifecycle
• Simulate the entities interacting  
with the WfMS
• Accommodate and automate  
different kinds of performance tests
• Ensure reliable execution
• Ensure repeatability
BenchFlow Framework
32
• Automate the SUT deployment
• Simplify the SUT’s deployment 
conﬁguration

33
BenchFlow Framework
system under test
Docker Engine

33
BenchFlow Framework
system under test
Docker Engine
Containers

33
BenchFlow Framework
system under test
Docker Engine
Docker Machine
provides
Containers

33
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Machine
provides
Containers

33
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Machine
provides manages
Containers
Servers

33
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Compose
Docker Machine
provides manages
Containers
Servers

33
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Compose
Docker Machine
provides manages
Containers
Servers
SUT’s Deployment Conf.
deploys

Manages the Benchmark Lifecycle
BenchFlow Framework
performance benchmark
34
Faban

BenchFlow Framework
34
Faban
Simulates interacting entities

BenchFlow Framework
34
Faban
Ensures repeatability
• the behaviour of a load driver
is described using code
• load drivers can be released

BenchFlow Framework
34
Faban
paired with Docker:
SUT’s initial state is
frozen in the container

BenchFlow Framework
34
Faban
Ensures reliable execution
• checks the environment condition
• validates the benchmark execution
• ensures as few interferences as
possible on the SUT during
measurements
paired with Docker:

BenchFlow Framework
34
Faban
Ensures reliable execution
• checks the environment condition
• validates the benchmark execution
• ensures as few interferences as
possible on the SUT during
measurements
paired with Docker:
Automates performance tests

35
software adapters
WfMS-speciﬁc APIs
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
Adapters

WfMS-speciﬁc APIs
36
WfMS Users
D
A
B
C
Instance
Database
Application Server
DBMS
Web
Service
Load Driver
custom APIs
• Load Driver APIs
• Web Service APIs
• User APIs
• Event APIs
APIs:

37
WfMS-speciﬁc APIs
requirements from the WfMS
ADVANCED FULLCORE

37
WfMS-speciﬁc APIs
ADVANCED FULLCORE
WfMS
Load Driver APIs
Deploy Workflow
Start Workflow

37
WfMS-speciﬁc APIs
ADVANCED FULLCORE
WfMS
Load Driver APIs
Deploy Workflow
Start Workflow
WfMS
Claim Task Complete Task
User and Web Service APIs
+
Create User Create Group
Pending User Tasks
Invoke WS

37
WfMS-speciﬁc APIs
ADVANCED FULLCORE
WfMS
Load Driver APIs
Deploy Workflow
Start Workflow
+
Event APIs
Issue Event
WfMS
Pending Event Tasks
WfMS
Claim Task Complete Task
User and Web Service APIs
+
Create User Create Group
Pending User Tasks
Invoke WS

38
BPMN 2.0 Customisations
models deployment version
WfMS
Deploy Workflow

38
WfMS
Deploy Workflow
BPMN 2.0 Standard
XML Serialisation
WfMS A
BPMN 2.0
Serialisation
WfMS B
BPMN 2.0
Serialisation

38
WfMS
Deploy Workflow
BPMN 2.0 Standard
XML Serialisation
WfMS A
BPMN 2.0
Serialisation
WfMS B
BPMN 2.0
Serialisation
XML
Processor

38
WfMS
Deploy Workflow
BPMN 2.0 Standard
XML Serialisation
WfMS A
BPMN 2.0
Serialisation
WfMS B
BPMN 2.0
Serialisation
XML
Processor
WfMS A WfMS B
|| ||

38
WfMS
Deploy Workflow
BPMN 2.0 Standard
XML Serialisation
WfMS A
BPMN 2.0
Serialisation
WfMS B
BPMN 2.0
Serialisation
XML
Processor
Customised
Test Data
Formatting
+WfMS A WfMS B
|| ||

39
Asynchronous Execution of Workﬂows
WfMS
Start Workflow
overview

Users
D
A
B
C
Start
Application Server
Web
Service
Load Driver
DBMS
WfMS
Instance
Database
39
WfMS
Start Workflow
overview

Users
D
A
B
C
Start
Application Server
Web
Service
Load Driver
DBMS
WfMS
Instance
Database
39
WfMS
Start Workflow
End
overview

40
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
monitors

40
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
MONITOR
monitors

41
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
• CPU usage
• Database state
Examples of Monitors:Monitors’ Characteristics:
• Microservices
• Lightweight (written in Go)
• As less invasive on the SUT as possible
• Expose REST APIs towards the Drivers
monitors

41
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
CPU
Monitor
DB
Monitor
• CPU usage
• Database state
Examples of Monitors:Monitors’ Characteristics:
• Microservices
• As less invasive on the SUT as possible
• Expose REST APIs towards the Drivers
monitors

42
Performance
Metrics and KPIs
20%
80%
C
A
B
D
WfMS-speciﬁc APIs
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
C
A
B
U S
of Workﬂows
WfMS
?
start

Performance Metrics and KPIs
43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
levels

43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
levels

43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
levels

43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
• Feature Level
levels

43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
• Feature Level
• Interactions
levels

44
examples and relevant users

44
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL

44
Throughput
Response Time
Capacity
Latency
Flexibility to Spike
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL

44
Throughput
Response Time
Capacity
Latency
CPU Usage
RAM Usage
Duration of Process
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL

44
Delay of Timer Event
Space Used by a Data Input
Duration of Message Event
Throughput
Response Time
Capacity
Latency
CPU Usage
RAM Usage
Duration of Process
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL

44
Delay of Timer Event
Space Used by a Data Input
Duration of Message Event
Throughput
Response Time
Capacity
Latency
CPU Usage
RAM Usage
Duration of Process
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
END-USERS
VENDORS
DEVELOPERS

45
minimal data requirements
Accessibility of the Data
DBMS
WfMS

45
• Workﬂow & Construct:
• Start Time
• End Time
• [Duration]
Availability of Timing DataAccessibility of the Data
DBMS
WfMS

45
• Workﬂow & Construct:
• Start Time
• End Time
• [Duration]
Availability of Timing DataAccessibility of the Data
DBMS
WfMS
Availability of Execution State
State of the workﬂow execution. E.g., running, completed, error

46
collect data
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service

46
collect data
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
Minio
Instance
Database
Analyses
COLLECTORS

47
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
• Container’s Stats (e.g., CPU usage)
• Database dump
• Applications Logs
Examples of Collectors:Collectors’ Characteristics:
• Microservices
• Two types: online and ofﬂine
• Buffer data locally
• Expose REST APIs towards Drivers
collect data

47
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
Stats
Collector
DB
Collector
• Container’s Stats (e.g., CPU usage)
• Database dump
• Applications Logs
Examples of Collectors:Collectors’ Characteristics:
• Microservices
• Two types: online and ofﬂine
• Buffer data locally
• Expose REST APIs towards Drivers
collect data

48
amount of data
REALISTIC
DATA
Number of
Repetitions
1
10
5
3

48
amount of data
REALISTIC
DATA
Number of
Repetitions
1
10
5
3
Number of
WfMSs
1

49
transform data
DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
Instance
Database
DATA 
TRANSFORMERS
Analyses
Data Mappers

49
transform data
DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
Instance
Database
DATA 
TRANSFORMERS
Analyses
• custom DB format
• unstructured logs
• …
Data Mappers

49
transform data
DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
Instance
Database
DATA 
TRANSFORMERS
Analyses
• custom DB format
• unstructured logs
• …
ETL Process
Data Mappers

50
transform data
DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
DATA 
TRANSFORMERS
Analyses
fast and general
engine for large-scale
data processing
Data Mappers
Instance
Database

DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
51
Analyses
DATA 
TRANSFORMERS
ANALYSERS
Performance
Metrics
Performance
KPIs
Analyses
analyse data
Data Mappers
Instance
Database

52
Highlights

52
Highlights
Why a Benchmark for WfMSs

52
Highlights
Why a Benchmark for WfMSs Benchmark’s Main Challenges

52
Highlights
Why a Benchmark for WfMSs Benchmark’s Main Challenges
BenchFlow Framework

53
Future Work
BenchFlow framework
1. Finalise the release of the BenchFlow framework as
an open-source project on GitHub and DockerHub;
benchﬂowhttps://github.com/benchﬂow

53
Future Work
BenchFlow framework
1. Finalise the release of the BenchFlow framework as
an open-source project on GitHub and DockerHub;
benchﬂowhttps://github.com/benchﬂow
2. Simplify and automate the execution of common
performance tests: load test, spike test, scalability
test, …

54
Future Work
experiments
1. Perform the ﬁrst real-world experiments

54
Future Work
experiments
Execute Different Test Types
• Stress testing
• Spike testing

54
Future Work
experiments
• Stress testing
• Spike testing
Use Real-World Workﬂows

54
Future Work
experiments
• Stress testing
• Spike testing
Production-like Conﬁgurations

54
Future Work
experiments
• Stress testing
• Spike testing
Realistic:
• Load Functions
• Test Data
Production-like Conﬁgurations

55
Future Work
experiments
2. Increase the number of supported WfMSs
APIs:
CORE: Load Driver APIs
+
ADVANCED: User and Web  
Service APIs
+ FULL: Event APIs

55
Future Work
experiments
APIs:
+
Service APIs
+ FULL: Event APIs
WfMS N
BPMN 2.0
Serialisation
customised test
data formatting+WfMS N
||
Deployable BPMN 2.0 Models:

55
Future Work
experiments
APIs:
+
Service APIs
+ FULL: Event APIs
WfMS N
BPMN 2.0
Serialisation
customised test
data formatting+WfMS N
||
Deployable BPMN 2.0 Models:
Data:
• Accessibility of the data
• Availability of timing data
• Availability of execution
state

56
•We want to characterise the Workload Mix using Real-World process models
• Share your executable BPMN 2.0 process models, even anonymised
Process Models
•We want to characterise the Load Functions using Real-World behaviours
• Share your execution logs, even anonymised
Execution Logs
Call for Collaboration
WfMSs, process models, process logs
•We want to add more and more WfMSs to the benchmark
• Contact us for collaboration, and BenchFlow framework support
WfMSs

Join Us @ ICWE 2016 in Lugano!
http://icwe2016.inf.usi.ch

Vincenzo Ferme, Cesare Pautasso | University of Lugano (USI)
vincenzo.ferme@usi.ch
@VincenzoFerme
http://benchflow.inf.usi.ch
c.pautasso@ieee.org
@pautasso
benchﬂowbenchﬂow

Backup Slides

[SEKE ’14]
C. Röck, S. Harrer, and G. Wirtz. Performance benchmarking of BPEL engines: A
comparison framework, status quo evaluation and challenges. In Proc. of the 26th
International Conference on Software Engineering and Knowledge Engineering, SEKE
2014, pages 31–34, 2014.
60
[SOSE ’15]
M. Skouradaki, K. Goerlach, M. Hahn, and F. Leymann. Application of Sub-Graph
Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models.
In Proc. of the 9th International IEEE Symposium on Service-Oriented System
Engineering, SOSE 2015, 2015.
Cited Work

61
Published Work
[BTW ’15]
C. Pautasso, V. Ferme, D. Roller, F. Leymann, and M. Skouradaki. Towards workflow
benchmarking: Open research challenges. In Proc. of the 16th conference on
Database Systems for Business,Technology, and Web, BTW 2015, pages 331–350, 2015.
[SSP ’14]
M. Skouradaki, D. H. Roller, F. Leymann, V. Ferme, and C. Pautasso. Technical open
challenges on benchmarking workflow management systems. In Proc. of the 2014
Symposium on Software Performance, SSP 2014, pages 105–112, 2014.
[ICPE ’15]
M. Skouradaki, D. H. Roller, L. Frank, V. Ferme, and C. Pautasso. On the Road to
Benchmarking BPMN 2.0 Workflow Engines. In Proc. of the 6th ACM/SPEC
International Conference on Performance Engineering, ICPE ’15, pages 301–304, 2015.

62
[CLOSER ’15]
M. Skouradaki,V. Ferme, F. Leymann, C. Pautasso, and D. H. Roller. “BPELanon”: Protect
business processes on the cloud. In Proc. of the 5th International Conference on
Cloud Computing and Service Science, CLOSER 2015. SciTePress, 2015.
Published Work
[SOSE ’15]
M. Skouradaki, K. Goerlach, M. Hahn, and F. Leymann. Application of Sub-Graph
Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models.
In Proc. of the 9th International IEEE Symposium on Service-Oriented System
Engineering, SOSE 2015, 2015.
[BPM ’15]
V. Ferme, A. Ivanchikj, C. Pautasso. A Framework for Benchmarking BPMN 2.0
Workﬂow Management Systems. In Proc. of the 13th International Conference on
Business Process Management, BPM ’15, pages 251-259, 2015.

63
[BPMD ’15]
A. Ivanchikj,V. Ferme, C. Pautasso. BPMeter: Web Service and Application for Static
Analysis of BPMN 2.0 Collections. In Proc. of the 13th International Conference on
Business Process Management [Demo], BPM ’15, pages 30-34, 2015.
Published Work

Towards a Benchmark for BPMN Engines

More Related Content

What's hot (20)

Similar to Towards a Benchmark for BPMN Engines (20)

More from Vincenzo Ferme (10)

Recently uploaded (20)

Towards a Benchmark for BPMN Engines