Micro Benchmarking WfMS with Workflow Patterns
2 likes655 views
The document discusses a study on micro-benchmarking BPMN 2.0 workflow management systems (WFMS) to understand the performance impacts of core language constructs and identify bottlenecks through simple workload patterns. It outlines the methodology, experimental setup, and presents results comparing mean durations and CPU usage across different workflow patterns in three open-source WFMS: Activiti, jBPM, and Camunda. The findings provide insights into the effectiveness of BPMN 2.0 implementations and open pathways for future research enhancements.
![2323
Research
© Marigianna Skouradaki
Workload Models
Empty
Script 1
Empty
Script 2
Sequence Pattern [SEQ]
parallelSplitPattern
Empty
Script 1
Empty
Script 2
Parallel Split & Synchronization [PAR]
tern
Empty
Script 1
Wait 5
Sec
Explicit Termination Pattern [EXT]ryCyclesPattern
generate
1 or 2
Empty
Script 1
Empty
Script 2
i++
i < 10
i >= 10
case_1
case_2
Arbitrary Cycle Pattern [CYC]
exclusiveChoicePattern
generate
number
Empty
Script 1
Empty
Script 2case_2
case_1
Simple Merge & Exclusive Choice [EXC]](https://image.slidesharecdn.com/caise16presentation-160615120734/85/Micro-Benchmarking-WfMS-with-Workflow-Patterns-9-320.jpg)
![2727
Research
Vincenzo Ferme
Experiments Methodology
n Experiment 1:
Max(1500) concurrent Instance Producers starting
instances for each workflow pattern [SEQ, EXC, EXT,
PAR, CYC]
n Experiment 2:
1500 concurrent Instance Producers starting an equal
number instances of the workflow patterns (20% [MIX]
of [SEQ, EXC, EXT, PAR, CYC])
à Three rounds per experiment
à Each experiment lasts for 10 minutes
à Discarded the data of the warm-up state (first 1min)](https://image.slidesharecdn.com/caise16presentation-160615120734/85/Micro-Benchmarking-WfMS-with-Workflow-Patterns-13-320.jpg)
![2828
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
Empty
Script 1
Empty
Script 2
[SEQ]](https://image.slidesharecdn.com/caise16presentation-160615120734/85/Micro-Benchmarking-WfMS-with-Workflow-Patterns-14-320.jpg)
![2929
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
[EXC]
Pattern
generate
number
Empty
Script 1
Empty
Script 2case_2
case_1](https://image.slidesharecdn.com/caise16presentation-160615120734/85/Micro-Benchmarking-WfMS-with-Workflow-Patterns-15-320.jpg)
![3030
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
Empty
Script 1
Wait 5
Sec
[EXT]](https://image.slidesharecdn.com/caise16presentation-160615120734/85/Micro-Benchmarking-WfMS-with-Workflow-Patterns-16-320.jpg)
![3131
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
Empty
Script 1
Empty
Script 2
[PAR]](https://image.slidesharecdn.com/caise16presentation-160615120734/85/Micro-Benchmarking-WfMS-with-Workflow-Patterns-17-320.jpg)
![3232
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
CYC
Instance Producers:
Camunda: 600
WfMS A: 600
WfMS B: 600
ryCyclesPattern
generate
1 or 2
Empty
Script 1
Empty
Script 2
i++
i < 10
i >= 10
case_1
case_2
[CYC]](https://image.slidesharecdn.com/caise16presentation-160615120734/85/Micro-Benchmarking-WfMS-with-Workflow-Patterns-18-320.jpg)
![3434
Research
© Marigianna Skouradaki
Results: Mean CPU (%) and RAM (MB) usage
munda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
48.53
(b)MeanCPU(%)
rkflow pattern and (b) Mean CPU (%) Usage
SEQ EXC EXT PAR CYC MIX
0
2,000
4,000
12,074.2
12,215.9
12,025.54
12,201.16
12,340.65
12,310.88
2,936.6
2,976.37
2,747.34
2,935.81
2,851.9
2,786.54
807.81
824.96
794.92
828.37
933.31
991.86
MeanRAM(MB)
WfMS A WfMS B Camunda
Fig. 3. (a) Mean RAM (MB) Usage per workflow
per workflow pattern in [MIX]
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR
0
20
40
60
80
100
43.21
57.42
60.2
66.1
5.83
5.73
0.24
5.64
36.75
44.62
33.34
41.76
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CP
per workflow pattern
mentary material. The behaviour of all the WfMSs is discussed th
CPU(%) usage
Memory (MB) Usage](https://image.slidesharecdn.com/caise16presentation-160615120734/85/Micro-Benchmarking-WfMS-with-Workflow-Patterns-20-320.jpg)
![4646
Research
© Marigianna Skouradaki
Findings
Even though the defined workload is simple
SEQ EXC EXT PAR CY
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
6.39
9.3
2,622
10.06
38.65
0.74
0.85
0.4
0.7
(a)MeanDuration(ms)
Activiti jBPM
Fig. 2. (a) Mean Duration (ms)
per workflow pattern
shows that the duration times
those of [SEQ].
More particularly, we have
and 0.85 ms for Camunda. Co
a slight increase with respect
CPU and 12, 215 MB RAM for
jBPM takes approximately the
workflow instances. For this, it
of RAM, and Camunda 43.21%
765, 274 workflow instances in
4.2.3 Explicit Terminatio
[EXT] executes concurrently an
3. Differences in
the duration
of the
workflow instances
2. Discovered
Bottleneck on WfMS B
1. Differences in
resource utilization
show space for
improvement
4. EXT pattern is
not implemented
consistently on WfMS B
5. Sequential patterns
are better candidates
for discovering the
maximum throughputsequencePattern
Empty
Script 1
Empty
Script 2
6. Complex and parallel
workflows
are better candidates
for stressing the WfMS on
Resource Utilization
parallelSplitPattern
Empty
Script 1
Empty
Script 2
attern
Empty
Script 1
Wait 5
Sec
7. The CYC pattern
caused
time out errors
in Camunda
arbitaryCyclesPattern
generate
1 or 2
Empty
Script 1
Empty
Script 2
i++
i < 10
i >= 10
case_1
case_2
8. The execution
of the MIX
had the expected
behavior](https://image.slidesharecdn.com/caise16presentation-160615120734/85/Micro-Benchmarking-WfMS-with-Workflow-Patterns-22-320.jpg)
Micro Benchmarking WfMS with Workflow Patterns
- 1. Research Marigianna Skouradaki, Frank Leymann Institute of Architecture of Application Systems University of Stuttgart, Germany Vincenzo Ferme, Cesare Pautasso Faculty of Informatics, University of Lugano, Switzerland André van Hoorn Institute of Software Technology, University of Stuttgart, Germany Micro – Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns
- 2. 22 Research © Marigianna Skouradaki Motivation *the appearance and company selection are random
- 3. 77 Research © Marigianna Skouradaki Motivation Workload Mix Results 2. What is the impact of the core BPMN 2.0 language constructs on the performance of the Workflow Management Systems? 3. Can simple workload mix reveal performance bottlenecks? 1. How to define meaningful workload models candidates? Benchmark
- 4. 88 Research © Marigianna Skouradaki Agenda n Background n Workload Models n BenchFlow Environment n Experiments Setup n Experiments Methodology n Results n Discussion n Conclusions and Future Work
- 5. 99 Research © Marigianna Skouradaki What is a Workflow Management System (WfMS)? 4.Navigator 3.Web Services 6.Application Server Workflow Engine 5.Instance Database 2.Users & IT tools … 1.Process Models
- 6. 1010 Research © Marigianna Skouradaki What is a Workflow Management System (WfMS)? 4.Navigator 3.Web Services 6.Application Server Workflow Engine 5.Instance Database 2.Users & IT tools … 1.Process Models
- 7. 1313 Research © Marigianna Skouradaki Micro-Benchmark n Simple workload and targets the evaluation of atomic operations1 1Waller, J., Hasselbring, W.: A benchmark engineering methodology to measure the overhead of application-level monitoring. In: KPDAYS. CEUR Workshop Proceedings, vol. 1083, pp. 59–68. CEUR-WS.org (2013) 2Oracle, “Java Microbenchmark Harness”, (2013). Online resource: http://openjdk.java.net/projects/code-tools/jmh/ 3Van Der Aalst, W.M.P., Hofstede, T., et al.: Workflow patterns. Distrib. Parallel Databases 14(1), 5–51 (Jul 2003) HelloWorld() micro-benchmark measures “nothing”. It is a good showcase for the overheads in infrastructure2 à For Workflow Management Systems The basic control flow workflow patterns can be considered as atomic operations. Contribution Introduce & Execute the first Micro-Benchmark on BPMN 2.0 WfMS using Workflow Patterns
- 8. 1818 Research © Marigianna Skouradaki Workflow Patterns as Workload Models (Constraints) 1. Maximize the simplicity of the process models 2. Omit interactions with external systems (i.e., stress the Process Navigator) 3. Most script tasks are empty, unless the implementation of the workflow pattern requires otherwise 4. Define equal probability for passing the control flow to the outgoing branches 5. Combine: - Simple merge + exclusive choice - Parallel split + synchronization
- 9. 2323 Research © Marigianna Skouradaki Workload Models Empty Script 1 Empty Script 2 Sequence Pattern [SEQ] parallelSplitPattern Empty Script 1 Empty Script 2 Parallel Split & Synchronization [PAR] tern Empty Script 1 Wait 5 Sec Explicit Termination Pattern [EXT]ryCyclesPattern generate 1 or 2 Empty Script 1 Empty Script 2 i++ i < 10 i >= 10 case_1 case_2 Arbitrary Cycle Pattern [CYC] exclusiveChoicePattern generate number Empty Script 1 Empty Script 2case_2 case_1 Simple Merge & Exclusive Choice [EXC]
- 10. 2424 Research © Marigianna Skouradaki Three Open-Source WfMS Under Test 1. Widely used in the industry 2. Tested against BPMN 2.0 conformance1 3. Vendor provided configurations are used 1.Geiger, M., Harrer, S., Lenhard, J., Casar, M., Vorndran, A., Wirtz, G.: BPMN conformance in open source engines. In: Proc. of the 9th IEEE International Symposium on Service-Oriented System Engineering (SOSE 2015). SOSE 2015, IEEE, San Francisco Bay, CA, USA (March 30 – April 3 2015)
- 11. 2525 Research Vincenzo Ferme The BenchFlow Environment5 5Ferme, Vincenzo; Ivanchikj, Ana and Pautasso Cesare: "A Framework for Benchmarking BPMN 2.0 Workflow Management Systems", In: Proceedings of BPM’15, Innsbruck, Austria, Springer, August, 2015.
- 13. 2727 Research Vincenzo Ferme Experiments Methodology n Experiment 1: Max(1500) concurrent Instance Producers starting instances for each workflow pattern [SEQ, EXC, EXT, PAR, CYC] n Experiment 2: 1500 concurrent Instance Producers starting an equal number instances of the workflow patterns (20% [MIX] of [SEQ, EXC, EXT, PAR, CYC]) à Three rounds per experiment à Each experiment lasts for 10 minutes à Discarded the data of the warm-up state (first 1min)
- 14. 2828 Research © Marigianna Skouradaki Results: Mean Duration (ms) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 2.91 8.16 6.39 9.3 2,622 10.06 38.65 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) Activiti jBPM Camunda SEQ EXC E 0 20 40 60 80 100 43.21 57.42 5.83 5.73 36.75 44.62 (b)MeanCPU(%) Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 6.23 8.16 6.39 9.3 2,622 10.06 39.36 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) WfMS A WfMS B Camunda SEQ EXC EXT PAR CYC MIX 0 20 40 60 80 100 43.21 57.42 60.2 66.1 70.09 77.69 5.83 5.73 0.24 5.64 4.67 0.66 36.75 44.62 33.34 41.76 41.67 (b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U per workflow pattern Mean duration (ms) Empty Script 1 Empty Script 2 [SEQ]
- 15. 2929 Research © Marigianna Skouradaki Results: Mean Duration (ms) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 2.91 8.16 6.39 9.3 2,622 10.06 38.65 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) Activiti jBPM Camunda SEQ EXC E 0 20 40 60 80 100 43.21 57.42 5.83 5.73 36.75 44.62 (b)MeanCPU(%) Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 6.23 8.16 6.39 9.3 2,622 10.06 39.36 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) WfMS A WfMS B Camunda SEQ EXC EXT PAR CYC MIX 0 20 40 60 80 100 43.21 57.42 60.2 66.1 70.09 77.69 5.83 5.73 0.24 5.64 4.67 0.66 36.75 44.62 33.34 41.76 41.67 (b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U per workflow pattern Mean duration (ms) [EXC] Pattern generate number Empty Script 1 Empty Script 2case_2 case_1
- 16. 3030 Research © Marigianna Skouradaki Results: Mean Duration (ms) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 2.91 8.16 6.39 9.3 2,622 10.06 38.65 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) Activiti jBPM Camunda SEQ EXC E 0 20 40 60 80 100 43.21 57.42 5.83 5.73 36.75 44.62 (b)MeanCPU(%) Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 6.23 8.16 6.39 9.3 2,622 10.06 39.36 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) WfMS A WfMS B Camunda SEQ EXC EXT PAR CYC MIX 0 20 40 60 80 100 43.21 57.42 60.2 66.1 70.09 77.69 5.83 5.73 0.24 5.64 4.67 0.66 36.75 44.62 33.34 41.76 41.67 (b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U per workflow pattern Mean duration (ms) Empty Script 1 Wait 5 Sec [EXT]
- 17. 3131 Research © Marigianna Skouradaki Results: Mean Duration (ms) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 2.91 8.16 6.39 9.3 2,622 10.06 38.65 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) Activiti jBPM Camunda SEQ EXC E 0 20 40 60 80 100 43.21 57.42 5.83 5.73 36.75 44.62 (b)MeanCPU(%) Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 6.23 8.16 6.39 9.3 2,622 10.06 39.36 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) WfMS A WfMS B Camunda SEQ EXC EXT PAR CYC MIX 0 20 40 60 80 100 43.21 57.42 60.2 66.1 70.09 77.69 5.83 5.73 0.24 5.64 4.67 0.66 36.75 44.62 33.34 41.76 41.67 (b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U per workflow pattern Mean duration (ms) Empty Script 1 Empty Script 2 [PAR]
- 18. 3232 Research © Marigianna Skouradaki Results: Mean Duration (ms) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 2.91 8.16 6.39 9.3 2,622 10.06 38.65 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) Activiti jBPM Camunda SEQ EXC E 0 20 40 60 80 100 43.21 57.42 5.83 5.73 36.75 44.62 (b)MeanCPU(%) Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 6.23 8.16 6.39 9.3 2,622 10.06 39.36 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) WfMS A WfMS B Camunda SEQ EXC EXT PAR CYC MIX 0 20 40 60 80 100 43.21 57.42 60.2 66.1 70.09 77.69 5.83 5.73 0.24 5.64 4.67 0.66 36.75 44.62 33.34 41.76 41.67 (b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U per workflow pattern Mean duration (ms) CYC Instance Producers: Camunda: 600 WfMS A: 600 WfMS B: 600 ryCyclesPattern generate 1 or 2 Empty Script 1 Empty Script 2 i++ i < 10 i >= 10 case_1 case_2 [CYC]
- 19. 3333 Research © Marigianna Skouradaki Results: Mean Duration (ms) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 2.91 8.16 6.39 9.3 2,622 10.06 38.65 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) Activiti jBPM Camunda SEQ EXC E 0 20 40 60 80 100 43.21 57.42 5.83 5.73 36.75 44.62 (b)MeanCPU(%) Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 6.23 8.16 6.39 9.3 2,622 10.06 39.36 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) WfMS A WfMS B Camunda SEQ EXC EXT PAR CYC MIX 0 20 40 60 80 100 43.21 57.42 60.2 66.1 70.09 77.69 5.83 5.73 0.24 5.64 4.67 0.66 36.75 44.62 33.34 41.76 41.67 (b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U per workflow pattern Mean duration (ms)
- 20. 3434 Research © Marigianna Skouradaki Results: Mean CPU (%) and RAM (MB) usage munda SEQ EXC EXT PAR CYC MIX 0 20 40 60 80 100 43.21 57.42 60.2 66.1 70.09 77.69 5.83 5.73 0.24 5.64 4.67 0.66 36.75 44.62 33.34 41.76 41.67 48.53 (b)MeanCPU(%) rkflow pattern and (b) Mean CPU (%) Usage SEQ EXC EXT PAR CYC MIX 0 2,000 4,000 12,074.2 12,215.9 12,025.54 12,201.16 12,340.65 12,310.88 2,936.6 2,976.37 2,747.34 2,935.81 2,851.9 2,786.54 807.81 824.96 794.92 828.37 933.31 991.86 MeanRAM(MB) WfMS A WfMS B Camunda Fig. 3. (a) Mean RAM (MB) Usage per workflow per workflow pattern in [MIX] SEQ EXC EXT PAR CYC MIX 0 5 10 15 20 0.39 0.48 14.1 13.29 6.23 8.16 6.39 9.3 2,622 10.06 39.36 540.02 0.74 0.85 0.4 0.7 3.06 1.22 (a)MeanDuration(ms) WfMS A WfMS B Camunda SEQ EXC EXT PAR 0 20 40 60 80 100 43.21 57.42 60.2 66.1 5.83 5.73 0.24 5.64 36.75 44.62 33.34 41.76 (b)MeanCPU(%) Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CP per workflow pattern mentary material. The behaviour of all the WfMSs is discussed th CPU(%) usage Memory (MB) Usage
- 21. 3737 Research © Marigianna Skouradaki Results Throughput (wi/s) SEQ EXC EXT PAR CYC MIX WfMS A 1,447.66 1,436.03 1,426.35 1,429.65 644.02 1,402.33 WfMS B 63.31 49.04 0.38 48.89 13.46 1.78 Camunda 1,456.79 1,417.17 1,455.68 1,455.68 327.99 1,061.27 CYC Instance Producers: WfMS A: 800 WfMS B: 1500 Camunda: 600 WfMS B uses a synchronous instantiation API. Therefore, load drivers must wait for the workflow instances to end Before instantiating the next one
- 22. 4646 Research © Marigianna Skouradaki Findings Even though the defined workload is simple SEQ EXC EXT PAR CY 0 5 10 15 20 0.39 0.48 14.1 13.29 2.91 6.39 9.3 2,622 10.06 38.65 0.74 0.85 0.4 0.7 (a)MeanDuration(ms) Activiti jBPM Fig. 2. (a) Mean Duration (ms) per workflow pattern shows that the duration times those of [SEQ]. More particularly, we have and 0.85 ms for Camunda. Co a slight increase with respect CPU and 12, 215 MB RAM for jBPM takes approximately the workflow instances. For this, it of RAM, and Camunda 43.21% 765, 274 workflow instances in 4.2.3 Explicit Terminatio [EXT] executes concurrently an 3. Differences in the duration of the workflow instances 2. Discovered Bottleneck on WfMS B 1. Differences in resource utilization show space for improvement 4. EXT pattern is not implemented consistently on WfMS B 5. Sequential patterns are better candidates for discovering the maximum throughputsequencePattern Empty Script 1 Empty Script 2 6. Complex and parallel workflows are better candidates for stressing the WfMS on Resource Utilization parallelSplitPattern Empty Script 1 Empty Script 2 attern Empty Script 1 Wait 5 Sec 7. The CYC pattern caused time out errors in Camunda arbitaryCyclesPattern generate 1 or 2 Empty Script 1 Empty Script 2 i++ i < 10 i >= 10 case_1 case_2 8. The execution of the MIX had the expected behavior
- 23. 4747 Research © Marigianna Skouradaki Future Work n Finalize and release the BenchFlow framework in an open-source model on Github and DockerHub: n Generate synthetic process models w.r.t. the present results n Execute a macro benchmark with the synthetic, complex, representative workflow mix n Execute different tests with realistic load functions and test data n Production-like configurations benchflowhttps://github.com/benchflow goo.gl/Olw32m skourama@iaas.uni-stuttgart.de