SlideShare a Scribd company logo

Workflow Engine Performance Benchmarking with BenchFlow

Vincenzo Ferme, profile picture
Vincenzo Ferme

The document discusses benchmarking the performance of workflow management systems (WfMSs) that are compliant with the Business Process Model and Notation (BPMN) 2.0 standard. It introduces the BenchFlow project, which aims to design and implement the first benchmark to assess and compare WfMS performance. It outlines the benchmarking process, which involves a workload model, WfMS configurations, test types, and metrics to measure performance. The methodology requires WfMSs to provide containerized distributions and APIs to deploy processes, start instances, and collect timing data, in order to automate testing and reproduce results.

Technology
Related topics:
Performance Evaluation
1 of 119
Downloaded 22 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
Vincenzo Ferme Faculty of Informatics USI Lugano, Switzerland WORKFLOW ENGINE PERFORMANCE BENCHMARKING WITH BENCHFLOW
Vincenzo Ferme 2 The BenchFlow Project Design and implement the first benchmark to assess and compare the performance of WfMSs that are compliant with Business Process Model and Notation 2.0 standard. ” “
Vincenzo Ferme 3 What is a Workflow Management System? WfMS Users Applications Application Server Instance Database DBMS Web Service D A B C
Vincenzo Ferme 4 Many Vendors of BPMN 2.0 WfMSs https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines Jan 2011 BPMN 2.0 Jan 2014 BPMN 2.0.2 ISO/IEC 19510 Aug 2009 BETA BPMN 2.0 2016 0 23 Grand Total 23 NumberofBPMN2.0WfMSs 0 5 10 15 20 25 Year of the First Version Supporting BPMN 2.0 2009 2010 2011 2012 2013 2014 2015 2016
Vincenzo Ferme 5 end-users, vendors, developers Why do We Need a Benchmark?
Vincenzo Ferme 5 end-users, vendors, developers Why do We Need a Benchmark? 1. How to choose the best WfMS according to the company’s technical requirements? 2. How to choose the best WfMS 
 according to the company’s 
 business process models (workflows)? B C D
Vincenzo Ferme 5 end-users, vendors, developers Why do We Need a Benchmark? 1. How to choose the best WfMS according to the company’s technical requirements? 2. How to choose the best WfMS 
 according to the company’s 
 business process models (workflows)? B C D 4. How to identify WfMS’s bottlenecks? 3. How to evaluate performance improvements
 during WfMS’s development?
Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Workload Metrics KPIs Measure Input/Process/Output Model
Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D
Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data
Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Containers
Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers
Vincenzo Ferme Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Figure 4: Benchmarking Methodology Choreography pending on the WfMS’s architecture. The DBMS Container can refer to an existing publicly available Container distributions. The containerized WfMS should be publicly available (e.g., at the Docker Hub registry8), or the Benchflow team should be granted access to a private registry used by the vendor. The same applies to the Containers’ definition file, i.e., the Dockerfile (Turnbull, 2014, ch. 4). While private reg- istries are a solution that can work with vendors of closed-source systems, they impact the reproducibil- ity of the results. For each WfMS version to be in- cluded in the benchmark, there must be a default con- figuration, i.e., the configuration in which the WfMS Containers start without modifying any configuration parameters, except the ones required in order to cor- rectly start the WfMS, e.g., the database connection. However, if vendors want to benchmark the perfor- mance of their WfMS with different configurations, for example, the configuration provided to users as a “getting started” configuration, or production-grade configurations for real-life usage, they can also pro- vide different configurations. To do that, the Con- tainers must allow to issue the WfMS configurations through additional environment variables9, and/or by access the configuration files. Exposing the volumes allows to access the files defined inside the Contain- ers, on the host operating system. Precisely, the WES Container has to, at least, enable the configuration of: 1) the used DB, i.e., DB driver, url, username and password for connection; 2) the WES itself; and 3) the logging level of the WES, and the application stack layers on which it may depend on. Alternatively, in- stead of providing configuration details, the vendors can provide different Containers for different config- urations. However, even in that case enabling the DB configuration is required. In order to access relevant data, all WfMS Con- tainers have to specify the volumes to access the WfMS log files, and to access all the data useful to setup the system. For instance, if the WfMS defines examples as part of the deployment, we may want to remove those examples by overriding the volume containing them. Moreover, the WES Container (or the DBMS Container) has to create the WfMS’s DB and populate it with data required to run the WfMS. The vendors need to provide authentication configu- ration details of the WfMS components (e.g., the user 7 Methodology Framework + Container Based Methodology and Framework BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness MONITOR Adapters COLLECTORS
Vincenzo Ferme Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Figure 4: Benchmarking Methodology Choreography pending on the WfMS’s architecture. The DBMS Container can refer to an existing publicly available Container distributions. The containerized WfMS should be publicly available (e.g., at the Docker Hub registry8), or the Benchflow team should be granted access to a private registry used by the vendor. The same applies to the Containers’ definition file, i.e., the Dockerfile (Turnbull, 2014, ch. 4). While private reg- istries are a solution that can work with vendors of closed-source systems, they impact the reproducibil- ity of the results. For each WfMS version to be in- cluded in the benchmark, there must be a default con- figuration, i.e., the configuration in which the WfMS Containers start without modifying any configuration parameters, except the ones required in order to cor- rectly start the WfMS, e.g., the database connection. However, if vendors want to benchmark the perfor- mance of their WfMS with different configurations, for example, the configuration provided to users as a “getting started” configuration, or production-grade configurations for real-life usage, they can also pro- vide different configurations. To do that, the Con- tainers must allow to issue the WfMS configurations through additional environment variables9, and/or by access the configuration files. Exposing the volumes allows to access the files defined inside the Contain- ers, on the host operating system. Precisely, the WES Container has to, at least, enable the configuration of: 1) the used DB, i.e., DB driver, url, username and password for connection; 2) the WES itself; and 3) the logging level of the WES, and the application stack layers on which it may depend on. Alternatively, in- stead of providing configuration details, the vendors can provide different Containers for different config- urations. However, even in that case enabling the DB configuration is required. In order to access relevant data, all WfMS Con- tainers have to specify the volumes to access the WfMS log files, and to access all the data useful to setup the system. For instance, if the WfMS defines examples as part of the deployment, we may want to remove those examples by overriding the volume containing them. Moreover, the WES Container (or the DBMS Container) has to create the WfMS’s DB and populate it with data required to run the WfMS. The vendors need to provide authentication configu- ration details of the WfMS components (e.g., the user 7 Methodology Framework + Container Based Methodology and Framework BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness MONITOR Adapters COLLECTORS Provides Great Advantages for Automation and Reproducibility of Results, while Ensuring Negligible Performance Impact
Vincenzo Ferme 8 [CLOSER ’16] Container-centric Methodology for Benchmarking Workflow Management Systems. Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Figure 4: Benchmarking Methodology Choreography pending on the WfMS’s architecture. The DBMS Container can refer to an existing publicly available Container distributions. The containerized WfMS should be publicly available (e.g., at the Docker Hub registry8), or the Benchflow team should be granted access to a private registry used by the vendor. The access the configuration files. Exposing the volumes allows to access the files defined inside the Contain- ers, on the host operating system. Precisely, the WES Container has to, at least, enable the configuration of: 1) the used DB, i.e., DB driver, url, username and password for connection; 2) the WES itself; and 3) the
Vincenzo Ferme 9 BenchFlow Benchmarking Methodology requirements from the WfMS Availability of APIs • Deploy Process • Start Process Instance • Users API • Web Service APIs • Events APIs
Vincenzo Ferme 9 BenchFlow Benchmarking Methodology requirements from the WfMS Availability of APIs • Deploy Process • Start Process Instance • Users API • Web Service APIs • Events APIs • Workflow & Construct: • Start Time • End Time • [Duration] Availability of Timing Data
Vincenzo Ferme 9 BenchFlow Benchmarking Methodology requirements from the WfMS Availability of APIs • Deploy Process • Start Process Instance • Users API • Web Service APIs • Events APIs • Workflow & Construct: • Start Time • End Time • [Duration] Availability of Timing Data Availability of Execution State State of the workflow execution. E.g., running, completed, error
Vincenzo Ferme 10 BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness benchflow [BPM ’15] [ICPE ’16] MONITOR Adapters COLLECTORS
Vincenzo Ferme 11 BenchFlow Framework WfMSs’ specific characteristics •Manages the Automatic WfMS deployment; •Provides a “plugin” mechanism to add new WfMSs; •Performs automatic Process Models deployment; •Collects Client and Server-side data and metrics; •Automatically computes performance metrics and
 statistics.
Vincenzo Ferme 12 Applications of the Methodology and Framework [CAiSE ’16] Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns. WfMS MySQL 3 WfMSsWorkload Empty Script 1 Empty Script 2 … 0 750 1500 0:00 4:00 10:00 Metrics • Engine Level • Process Level • Environment
Vincenzo Ferme Micro-Benchmarking with Workflow Patterns 13 research questions Selected three WfMSs: popular open-source WfMSs actually used in industry 1. What is the impact of individual or a mix of workflow patterns on the performance of each one of the benchmarked BPMN 2.0 WfMSs? 2. Are there performance bottlenecks for the selected WfMSs?
Vincenzo Ferme 14 Empty Script 1 Empty Script 2 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix Sequence Pattern [SEQ] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
Vincenzo Ferme 14 Empty Script 1 Empty Script 2 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix Sequence Pattern [SEQ]exclusiveChoicePattern generate number Empty Script 1 Empty Script 2case_2 case_1 Exclusive Choice and Simple Merge [EXC] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
Vincenzo Ferme 15 lesPattern generate 1 or 2 Empty Script 1 Empty Script 2 i++ i < 10 i >= 10 case_1 case_2 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix Arbitrary Cycle [CYC] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
Vincenzo Ferme 16 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix Empty Script 1 Empty Script 2 Parallel Split and Synchronisation [PAR] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
Vincenzo Ferme 16 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix citTerminationPattern Empty Script 1 Wait 5 Sec Explicit Termination Pattern [EXT] Empty Script 1 Empty Script 2 Parallel Split and Synchronisation [PAR] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
Vincenzo Ferme 17 Micro-Benchmarking with Workflow Patterns workload mix design decisions 1. Maximise the simplicity of the model expressing the workflow pattern 2. Omit the interactions with external systems by implementing all tasks as script tasks
Vincenzo Ferme 18 Load Functions Micro-Benchmarking with Workflow Patterns load function Test Type Load test Instance Producers 0 300 600 900 1200 1500 Time (min:sec) 0:000:301:002:003:004:005:006:007:008:0010:00 Example for u=1500
Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C
Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26
Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79
Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final
Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final Max Java Heap: 32GB
Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final Max Java Heap: 32GB Max DB Connections Number: 100
Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 WfMS’s Containers: official ones, if available on DockerHub O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final Max Java Heap: 32GB Max DB Connections Number: 100
Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 WfMS’s Containers: official ones, if available on DockerHub WfMS’s Configuration: as suggested by vendor’s documentation O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final Max Java Heap: 32GB Max DB Connections Number: 100
Vincenzo Ferme 20 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS deployment TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WfMS CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS
Vincenzo Ferme 20 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS deployment TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WfMS CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS 10 Gbit/s 10 Gbit/s
Vincenzo Ferme 20 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS deployment TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WfMS CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS 10 Gbit/s 10 Gbit/s O.S.: Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS
Vincenzo Ferme 20 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS deployment TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WfMS CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS 10 Gbit/s 10 Gbit/s O.S.: Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS 1.8.2 1.8.2 1.8.2
Vincenzo Ferme 21 Micro-Benchmarking with Workflow Patterns performed experiments Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR])
Vincenzo Ferme 21 Micro-Benchmarking with Workflow Patterns performed experiments Three runs for each execution of the experiments Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR])
Vincenzo Ferme 21 Micro-Benchmarking with Workflow Patterns performed experiments Three runs for each execution of the experiments The data collected in the first minute of the execution has not been included in the analysis Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR])
Vincenzo Ferme 21 Micro-Benchmarking with Workflow Patterns performed experiments Three runs for each execution of the experiments The data collected in the first minute of the execution has not been included in the analysis Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR])
Vincenzo Ferme 22 Number of Process Instances Computed Metrics and Statistics engine level metrics Throughput Execution Time Database Size WfMS Users Load Driver Instance Database Application Server DBMS Web Service D A B C …
Vincenzo Ferme 22 Number of Process Instances Computed Metrics and Statistics engine level metrics Throughput Execution Time Database Size WfMS Users Load Driver Instance Database Application Server DBMS Web Service D A B C …
Vincenzo Ferme 23 individual patterns: throughput (bp/sec) Micro-Benchmarking with Workflow Patterns SEQ EXC EXT PAR CYC WfMS A WfMS B WfMS C 1456.79 1417.17 1455.68 1433.12 327.99 1447.66 1436.03 1426.35 1429.65 644.02 63.31 49.04 0.38 48.89 13.46 u=600 u=800 u=1500
Vincenzo Ferme 24 Micro-Benchmarking with Workflow Patterns individual patterns: WfMS C Scalability for SEQ pattern WfMS C uses a synchronous instantiation API, load drivers must wait for workflows to end before starting new ones
Vincenzo Ferme 24 Micro-Benchmarking with Workflow Patterns individual patterns: WfMS C Scalability for SEQ pattern WfMS C uses a synchronous instantiation API, load drivers must wait for workflows to end before starting new ones Response Time 0 14 28 500 1000 1500 2000 u Instance Producers Throughput 0 40 80 500 1000 1500 2000 u Instance Producers
Vincenzo Ferme 24 Micro-Benchmarking with Workflow Patterns individual patterns: WfMS C Scalability for SEQ pattern WfMS C uses a synchronous instantiation API, load drivers must wait for workflows to end before starting new ones Response Time 0 14 28 500 1000 1500 2000 u Instance Producers Throughput 0 40 80 500 1000 1500 2000 u Instance Producers We have found a bottleneck! (~60 bp/sec)
Vincenzo Ferme 25 Computed Metrics and Statistics process level metrics Throughput WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS … Instance Duration PerProcess Definition
Vincenzo Ferme 25 Computed Metrics and Statistics process level metrics Throughput WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS … Instance Duration PerProcess Definition
Vincenzo Ferme 26 Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time)
Vincenzo Ferme 26 Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time) Empty Script 1 Empty Script 2 [SEQ]
Vincenzo Ferme 26 Empty Script 1 Wait 5 Sec [EXT] Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time)
Vincenzo Ferme 26 Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time) Empty Script 1 Empty Script 2 [PAR]
Vincenzo Ferme 26 Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time) CYC Instance Producers: u=600 [CYC] tern generate 1 or 2 Empty Script 1 Empty Script 2 i++ i < 10 i >= 10 case_1 case_2
Vincenzo Ferme 27 Computed Metrics and Statistics environment metrics WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS CPU/RAM Usage per Second IO Operations/Network Usage per Second …
Vincenzo Ferme 27 Computed Metrics and Statistics environment metrics WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS CPU/RAM Usage per Second IO Operations/Network Usage per Second …
Vincenzo Ferme 28 Micro-Benchmarking with Workflow Patterns individual patterns: mean (RAM) and mean (CPU) usage RAM CPU
Vincenzo Ferme 29 Micro-Benchmarking with Workflow Patterns performed experiments Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR]) Three runs for each execution of the experiments The data collected in the first minute of the execution has not been included in the analysis
Vincenzo Ferme 30 Micro-Benchmarking with Workflow Patterns mix of patterns: duration, RAM, CPU
Vincenzo Ferme 31 Micro-Benchmarking with Workflow Patterns summary Even though the Executed Workflows are Simple
Vincenzo Ferme 31 Micro-Benchmarking with Workflow Patterns summary There are relevant differences in workflows’ duration and throughput among the WfMSs Even though the Executed Workflows are Simple
Vincenzo Ferme 31 Micro-Benchmarking with Workflow Patterns summary There are relevant differences in workflows’ duration and throughput among the WfMSs There are relevant differences in resource usage among WfMSs CPURAM Even though the Executed Workflows are Simple
Vincenzo Ferme 31 Micro-Benchmarking with Workflow Patterns summary There are relevant differences in workflows’ duration and throughput among the WfMSs One WfMS has bottlenecks There are relevant differences in resource usage among WfMSs CPURAM Even though the Executed Workflows are Simple
Highlights
Highlights Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers Benchmarking Process
Highlights Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers Benchmarking Process Vincenzo Ferme 8 [CLOSER ’16] Container-centric Methodology for Benchmarking Workflow Management Systems. Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Benchmarking Methodology
Highlights Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers Benchmarking Process Vincenzo Ferme 8 [CLOSER ’16] Container-centric Methodology for Benchmarking Workflow Management Systems. Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Benchmarking Methodology Vincenzo Ferme 10 BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness benchflow [BPM ’15] [ICPE ’16] MONITOR Adapters COLLECTORS BenchFlow Framework
Highlights Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers Benchmarking Process Vincenzo Ferme 8 [CLOSER ’16] Container-centric Methodology for Benchmarking Workflow Management Systems. Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Benchmarking Methodology Vincenzo Ferme 10 BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness benchflow [BPM ’15] [ICPE ’16] MONITOR Adapters COLLECTORS BenchFlow Framework Vincenzo Ferme 26 Empty Script 1 Wait 5 Sec [EXT] Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time) Empty Script 1 Empty Script 2 [SEQ] Empty Script 1 Empty Script 2 [PAR] CYC Instance Producers: u=600 [CYC] generate 1 or 2 Empty Script 1 Empty Script 2 i++ i < 10 i >= 10 case_1 case_2 Micro-Benchmark with WP
Vincenzo Ferme 33 •We want to characterise the Workload Mix using Real-World process models • Share your executable BPMN 2.0 process models, even anonymised Process Models •We want to characterise the Load Functions using Real-World behaviours • Share your execution logs, even anonymised Execution Logs •We want to add more and more WfMSs to the benchmark • Contact us for collaboration, and BenchFlow framework support WfMSs Call for Collaboration WfMSs, process models, process logs
benchflow benchflow vincenzo.ferme@usi.ch http://benchflow.inf.usi.ch WORKFLOW ENGINE PERFORMANCE BENCHMARKING WITH BENCHFLOW Vincenzo Ferme Faculty of Informatics USI Lugano, Switzerland
BACKUP SLIDES Vincenzo Ferme Faculty of Informatics USI Lugano, Switzerland
Vincenzo Ferme 36 Published Work [BTW ’15] C. Pautasso, V. Ferme, D. Roller, F. Leymann, and M. Skouradaki. Towards workflow benchmarking: Open research challenges. In Proc. of the 16th conference on Database Systems for Business,Technology, and Web, BTW 2015, pages 331–350, 2015. [SSP ’14] M. Skouradaki, D. H. Roller, F. Leymann, V. Ferme, and C. Pautasso. Technical open challenges on benchmarking workflow management systems. In Proc. of the 2014 Symposium on Software Performance, SSP 2014, pages 105–112, 2014. [ICPE ’15] M. Skouradaki, D. H. Roller, L. Frank, V. Ferme, and C. Pautasso. On the Road to Benchmarking BPMN 2.0 Workflow Engines. In Proc. of the 6th ACM/SPEC International Conference on Performance Engineering, ICPE ’15, pages 301–304, 2015.
Vincenzo Ferme 37 Published Work [CLOSER ’15] M. Skouradaki,V. Ferme, F. Leymann, C. Pautasso, and D. H. Roller. “BPELanon”: Protect business processes on the cloud. In Proc. of the 5th International Conference on Cloud Computing and Service Science, CLOSER 2015. SciTePress, 2015. [SOSE ’15] M. Skouradaki, K. Goerlach, M. Hahn, and F. Leymann. Application of Sub-Graph Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models. In Proc. of the 9th International IEEE Symposium on Service-Oriented System Engineering, SOSE 2015, 2015. [BPM ’15] V. Ferme, A. Ivanchikj, C. Pautasso. A Framework for Benchmarking BPMN 2.0 Workflow Management Systems. In Proc. of the 13th International Conference on Business Process Management, BPM ’15, pages 251-259, 2015.
Vincenzo Ferme 38 Published Work [BPMD ’15] A. Ivanchikj,V. Ferme, C. Pautasso. BPMeter: Web Service and Application for Static Analysis of BPMN 2.0 Collections. In Proc. of the 13th International Conference on Business Process Management [Demo], BPM ’15, pages 30-34, 2015. [ICPE ’16] V. Ferme, and C. Pautasso. Integrating Faban with Docker for Performance Benchmarking. In Proc. of the 7th ACM/SPEC International Conference on Performance Engineering, ICPE ’16, 2016. [CLOSER ’16] V. Ferme, A. Ivanchikj, C. Pautasso., M. Skouradaki, F. Leymann. A Container-centric Methodology for Benchmarking Workflow Management Systems. In Proc. of the 6th International Conference on Cloud Computing and Service Science, CLOSER 2016. SciTePress, 2016.
Vincenzo Ferme [CAiSE ’16] M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns . In Proc. of the 28th International Conference on Advanced Information Systems Engineering, CAiSE ’16, 2016. 39 Published Work [ICWE ’16] C. Jürgen,V. Ferme, H.C. Gall. Using Docker Containers to Improve Reproducibility in Software and Web Engineering Research. In Proc. of the 16th International Conference on Web Engineering, 2016. [ICWS ’16] M. Skouradaki,V.Andrikopoulos, F. Leymann. Representative BPMN 2.0 Process Model Generation from Recurring Structures. In Proc. of the 23rd IEEE International Conference on Web Services, ICWS '16, 2016.
Vincenzo Ferme 40 Published Work [SummerSOC ’16] M. Skouradaki, T. Azad, U. Breitenbücher, O. Kopp, F. Leymann. A Decision Support System for the Performance Benchmarking of Workflow Management Systems. In Proc. of the 10th Symposium and Summer School On Service-Oriented Computing, SummerSOC '16, 2016. [OTM ’16] M. Skouradaki, V. Andrikopoulos, O. Kopp, F. Leymann. RoSE: Reoccurring Structures Detection in BPMN 2.0 Process Models Collections. In Proc. of On the Move to Meaningful Internet Systems Conference, OTM '16, 2016. (to appear) [BPM Forum ’16] V. Ferme, A. Ivanchikj, C. Pautasso. Estimating the Cost for Executing Business Processes in the Cloud. In Proc. of the 14th International Conference on Business Process Management, BPM Forum ’16, 2016. (to appear)
Vincenzo Ferme 41 Docker Performance [IBM ’14] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An updated performance comparison of virtual machines and Linux containers. IBM Research Report, 2014. Although containers themselves have almost no overhead, Docker is not without performance gotchas. Docker volumes have noticeably better performance than files stored in AUFS. Docker’s NAT also introduces overhead for workloads with high packet rates. These features represent a tradeoff between ease of management and performance and should be considered on a case-by-case basis. ” “Our results show that containers result in equal or better performance than VMs in almost all cases. “ ” BenchFlow Configures Docker for Performance by Default
Vincenzo Ferme 42 Benchmarking Requirements • Relevant • Representative • Portable • Scalable • Simple • Repeatable • Vendor-neutral • Accessible • Efficient • Affordable • K. Huppler, The art of building a good benchmark, 2009 • J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993 • S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A challenge to software engineering, 2003
Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE WfMS Initialisation APIs Deploy Process Start Process Instance Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE WfMS Initialisation APIs Deploy Process Start Process Instance + WfMS Claim Task Complete Task User APIs Create User Create Group Pending User Tasks Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE WfMS Initialisation APIs Deploy Process Start Process Instance + WfMS Claim Task Complete Task User APIs Create User Create Group Pending User Tasks Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work Invoke WS Web Service APIs
Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE WfMS Initialisation APIs Deploy Process Start Process Instance Event APIs Issue Event WfMS Pending Event Tasks + WfMS Claim Task Complete Task User APIs Create User Create Group Pending User Tasks Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work Invoke WS Web Service APIs
Vincenzo Ferme 44 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS Table 1: Summary of Core and Non-core APIs to be imple- mented by the WfMS Functionality Min Response Data CoreAPIs Initialisation APIs Deploy a process Deployed process ID Start a process instance Process instance ID Non-coreAPIs User APIs Create a user User ID Create a group of users User group ID Access pending tasks Pending tasks IDs Claim a task* Complete a task Event APIs Access pending events Pending events IDs Issue events Web service APIs Map tasks to Web service endpoints *Optional depending on the WfMS implementation ing the a u (U ate by acc a g and mo ing (Pi has col Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Vincenzo Ferme 45 BenchFlow Framework system under test Docker Engine
Vincenzo Ferme 45 BenchFlow Framework system under test Docker Engine Containers
Vincenzo Ferme 45 BenchFlow Framework system under test Docker Engine Docker Machine provides Containers
Vincenzo Ferme 45 BenchFlow Framework system under test Docker Swarm Docker Engine Docker Machine provides Containers
Vincenzo Ferme 45 BenchFlow Framework system under test Docker Swarm Docker Engine Docker Machine provides manages Containers Servers
Vincenzo Ferme 45 BenchFlow Framework system under test Docker Swarm Docker Engine Docker Compose Docker Machine provides manages Containers Servers
Vincenzo Ferme 45 BenchFlow Framework system under test Docker Swarm Docker Engine Docker Compose Docker Machine provides manages Containers Servers SUT’s Deployment Conf. deploys
Vincenzo Ferme 46 Server-side Data and Metrics Collection WfMS Start Workflow asynchronous execution of workflows
Vincenzo Ferme Users D A B C Start Application Server Web Service Load Driver DBMS WfMS Instance Database 46 Server-side Data and Metrics Collection WfMS Start Workflow asynchronous execution of workflows
Vincenzo Ferme Users D A B C Start Application Server Web Service Load Driver DBMS WfMS Instance Database 46 Server-side Data and Metrics Collection WfMS Start Workflow End asynchronous execution of workflows
Vincenzo Ferme 47 Server-side Data and Metrics Collection monitors DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service
Vincenzo Ferme 47 Server-side Data and Metrics Collection monitors DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service MONITOR
Vincenzo Ferme 48 Server-side Data and Metrics Collection monitors DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service • CPU usage • Database state Examples of Monitors:Monitors’ Characteristics: • RESTful services • Lightweight (written in Go) • As less invasive on the SUT as possible
Vincenzo Ferme 48 Server-side Data and Metrics Collection monitors DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service CPU Monitor DB Monitor • CPU usage • Database state Examples of Monitors:Monitors’ Characteristics: • RESTful services • Lightweight (written in Go) • As less invasive on the SUT as possible
Vincenzo Ferme 49 Server-side Data and Metrics Collection collect data DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service
Vincenzo Ferme 49 Server-side Data and Metrics Collection collect data DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service Minio Instance Database Analyses COLLECTORS
Vincenzo Ferme 50 Server-side Data and Metrics Collection collect data DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service • Container’s Stats (e.g., CPU usage) • Database dump • Applications Logs Examples of Collectors:Collectors’ Characteristics: • RESTful services • Lightweight (written in Go) • Two types: online and offline • Buffer data locally
Vincenzo Ferme 50 Server-side Data and Metrics Collection collect data DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service Stats Collector DB Collector • Container’s Stats (e.g., CPU usage) • Database dump • Applications Logs Examples of Collectors:Collectors’ Characteristics: • RESTful services • Lightweight (written in Go) • Two types: online and offline • Buffer data locally
Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector Synchronisation with the Analyses schedule analyses
Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector COLLECT Synchronisation with the Analyses schedule analyses
Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector COLLECT Publish Synchronisation with the Analyses schedule analyses
Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector COLLECT Publish Subscribe Synchronisation with the Analyses schedule analyses
Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector COLLECT Publish Subscribe Read Synchronisation with the Analyses schedule analyses
Vincenzo Ferme 52 Performance Metrics and KPIs amount of data REALISTIC DATA Number of Repetitions 1 10 5 3
Vincenzo Ferme 52 Performance Metrics and KPIs amount of data REALISTIC DATA Number of Repetitions 1 10 5 3 Number of WfMSs 1
Vincenzo Ferme 53 Micro-Benchmarking with Workflow Patterns mix of patterns: throughput (bp/sec) MIX WfMS A WfMS B WfMS C 1061.27 1402.33 1.78
Vincenzo Ferme 54 Computed Metrics and Statistics feature level metrics WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS Throughput … Instance Duration PerConstruct (type,name)
Vincenzo Ferme 54 Computed Metrics and Statistics feature level metrics WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS Throughput … Instance Duration PerConstruct (type,name)
Vincenzo Ferme 55 Computed Metrics and Statistics interactions metrics Response Time Latency … WfMS Users Load Driver Instance Database Application Server DBMS Web Service D A B C
Vincenzo Ferme 55 Computed Metrics and Statistics interactions metrics Response Time Latency … WfMS Users Load Driver Instance Database Application Server DBMS Web Service D A B C
Vincenzo Ferme 56 Computed Metrics and Statistics descriptive and homogeneity statistics • Descriptive Statistics (e.g., Mean, Confidence Interval, 
 Standard Deviation, Percentiles, Quartiles, …) • Coefficient ofVariation across trials • Levene Test on trials

More Related Content

PDF
Towards a Benchmark for BPMN Engines
Vincenzo Ferme
 
DOCX
SAP SQL Anywhere High Availability Tutorial for Business Objects
Sitesh Patel
 
PPT
Meet the expert
Karthik Subramanian
 
PPT
Karthik Subramanian online portfolio
Karthik Subramanian
 
PPT
Subversion workshop
TrafeX
 
PDF
Version Control with SVN
PHPBelgium
 
PDF
Dell 3-2-1 Reference Configurations: High available and scalable performance ...
Principled Technologies
 
ODP
Nguyễn Vũ Hưng: Subversion best practices
Vu Hung Nguyen
 
Towards a Benchmark for BPMN Engines
Vincenzo Ferme
 
SAP SQL Anywhere High Availability Tutorial for Business Objects
Sitesh Patel
 
Meet the expert
Karthik Subramanian
 
Karthik Subramanian online portfolio
Karthik Subramanian
 
Subversion workshop
TrafeX
 
Version Control with SVN
PHPBelgium
 
Dell 3-2-1 Reference Configurations: High available and scalable performance ...
Principled Technologies
 
Nguyễn Vũ Hưng: Subversion best practices
Vu Hung Nguyen
 

What's hot (20)

DOCX
Cognos 10 upgrade migrate fixpack by bhawani nandan prasad
Bhawani N Prasad
 
PPTX
02.28.13 WANDisco SVN Training: Getting Info Out of SVN
WANdisco Plc
 
PPTX
Getting Started With Subversion
Jordan Hatch
 
PPTX
Symfony 2 under control
Max Romanovsky
 
PPT
Subversion in 2010 and Beyond
guest1243d91
 
PDF
OpenPOWER Application Optimization
Ganesan Narayanasamy
 
PDF
How to use CVS applied to SOLab
Pablo Arriazu
 
KEY
SVN Best Practices
abackstrom
 
PDF
Subversion Best Practices
Matt Wood
 
PPT
SVN Usage & Best Practices
Ashraf Fouad
 
ODP
Svn Basic Tutorial
Marco Pivetta
 
PPTX
How to set up an ASP.NET 5 Continuous Delivery Pipeline using IBM Bluemix Dev...
Richard Johansson
 
PDF
Svn tutorial
kalyansiri
 
PDF
Building A Simple Web Service With CXF
Carl Lu
 
PPT
Less13 performance
Amit Bhalla
 
PPT
SVN Tool Information : Best Practices
Maidul Islam
 
PDF
Database Change Management | Change Manager 5.1 Beta
Michael Findling
 
ODP
Subversion User Guide
Muthuselvam RS
 
PDF
Doors quality analyzer deployment guide
ReuseCompany
 
PDF
Oracle Release 12 E-Business Suite Patching Best Practices
Scott Jenner
 
Cognos 10 upgrade migrate fixpack by bhawani nandan prasad
Bhawani N Prasad
 
02.28.13 WANDisco SVN Training: Getting Info Out of SVN
WANdisco Plc
 
Getting Started With Subversion
Jordan Hatch
 
Symfony 2 under control
Max Romanovsky
 
Subversion in 2010 and Beyond
guest1243d91
 
OpenPOWER Application Optimization
Ganesan Narayanasamy
 
How to use CVS applied to SOLab
Pablo Arriazu
 
SVN Best Practices
abackstrom
 
Subversion Best Practices
Matt Wood
 
SVN Usage & Best Practices
Ashraf Fouad
 
Svn Basic Tutorial
Marco Pivetta
 
How to set up an ASP.NET 5 Continuous Delivery Pipeline using IBM Bluemix Dev...
Richard Johansson
 
Svn tutorial
kalyansiri
 
Building A Simple Web Service With CXF
Carl Lu
 
Less13 performance
Amit Bhalla
 
SVN Tool Information : Best Practices
Maidul Islam
 
Database Change Management | Change Manager 5.1 Beta
Michael Findling
 
Subversion User Guide
Muthuselvam RS
 
Doors quality analyzer deployment guide
ReuseCompany
 
Oracle Release 12 E-Business Suite Patching Best Practices
Scott Jenner
 
Ad

Similar to Workflow Engine Performance Benchmarking with BenchFlow (20)

PDF
A Container-Centric Methodology for Benchmarking Workflow Management Systems
Vincenzo Ferme
 
PDF
BenchFlow, a Framework for Benchmarking BPMN 2.0 Workflow Management Systems
Vincenzo Ferme
 
PDF
On the Road to Benchmarking BPMN 2.0 Workflow Engines
Vincenzo Ferme
 
PDF
CamundaCon 2018: Big Workflow in Action: Zeebe Live Hacking (Camunda)
camunda services GmbH
 
PDF
Micro Benchmarking WfMS with Workflow Patterns
Marigianna Skouradaki
 
PPTX
Flink vs. Spark
Slim Baltagi
 
PPTX
Slim Baltagi – Flink vs. Spark
Flink Forward
 
PPTX
Improving Business Performance Through Big Data Benchmarking, Todor Ivanov, B...
DataBench
 
PPTX
Edge 2016 SCL-2484: a software defined scalable and flexible container manage...
Yong Feng
 
PDF
Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...
HostedbyConfluent
 
PDF
Benchmarking Hadoop and Big Data
Nicolas Poggi
 
PDF
Flink Forward Berlin 2018: Raj Subramani - "A streaming Quantitative Analytic...
Flink Forward
 
PDF
Drive Smarter Decisions with Big Data Using Complex Event Processing
Perficient, Inc.
 
PPTX
Big Data Technical Benchmarking, Arne Berre, BDVe Webinar series, 09/10/2018
DataBench
 
PPTX
Log Data Analysis Platform by Valentin Kropov
SoftServe
 
PPTX
Log Data Analysis Platform
Valentin Kropov
 
PDF
Confluent Partner Tech Talk with QLIK
confluent
 
PDF
ABench: Big Data Architecture Stack Benchmark
t_ivanov
 
PPTX
Creating a Benchmarking Infrastructure That Just Works
Tim Callaghan
 
PDF
Estimating the Cost for Executing Business Processes in the Cloud
Vincenzo Ferme
 
A Container-Centric Methodology for Benchmarking Workflow Management Systems
Vincenzo Ferme
 
BenchFlow, a Framework for Benchmarking BPMN 2.0 Workflow Management Systems
Vincenzo Ferme
 
On the Road to Benchmarking BPMN 2.0 Workflow Engines
Vincenzo Ferme
 
CamundaCon 2018: Big Workflow in Action: Zeebe Live Hacking (Camunda)
camunda services GmbH
 
Micro Benchmarking WfMS with Workflow Patterns
Marigianna Skouradaki
 
Flink vs. Spark
Slim Baltagi
 
Slim Baltagi – Flink vs. Spark
Flink Forward
 
Improving Business Performance Through Big Data Benchmarking, Todor Ivanov, B...
DataBench
 
Edge 2016 SCL-2484: a software defined scalable and flexible container manage...
Yong Feng
 
Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Conflue...
HostedbyConfluent
 
Benchmarking Hadoop and Big Data
Nicolas Poggi
 
Flink Forward Berlin 2018: Raj Subramani - "A streaming Quantitative Analytic...
Flink Forward
 
Drive Smarter Decisions with Big Data Using Complex Event Processing
Perficient, Inc.
 
Big Data Technical Benchmarking, Arne Berre, BDVe Webinar series, 09/10/2018
DataBench
 
Log Data Analysis Platform by Valentin Kropov
SoftServe
 
Log Data Analysis Platform
Valentin Kropov
 
Confluent Partner Tech Talk with QLIK
confluent
 
ABench: Big Data Architecture Stack Benchmark
t_ivanov
 
Creating a Benchmarking Infrastructure That Just Works
Tim Callaghan
 
Estimating the Cost for Executing Business Processes in the Cloud
Vincenzo Ferme
 
Ad

More from Vincenzo Ferme (7)

PDF
Declarative Performance Testing Automation - Automating Performance Testing f...
Vincenzo Ferme
 
PDF
Continuous Performance Testing for Microservices
Vincenzo Ferme
 
PDF
A Declarative Approach for Performance Tests Execution in Continuous Software...
Vincenzo Ferme
 
PDF
BenchFlow: A Platform for End-to-end Automation of Performance Testing and An...
Vincenzo Ferme
 
PDF
Towards Holistic Continuous Software Performance Assessment
Vincenzo Ferme
 
PDF
Using Docker Containers to Improve Reproducibility in Software and Web Engine...
Vincenzo Ferme
 
PPT
Open Data
Vincenzo Ferme
 
Declarative Performance Testing Automation - Automating Performance Testing f...
Vincenzo Ferme
 
Continuous Performance Testing for Microservices
Vincenzo Ferme
 
A Declarative Approach for Performance Tests Execution in Continuous Software...
Vincenzo Ferme
 
BenchFlow: A Platform for End-to-end Automation of Performance Testing and An...
Vincenzo Ferme
 
Towards Holistic Continuous Software Performance Assessment
Vincenzo Ferme
 
Using Docker Containers to Improve Reproducibility in Software and Web Engine...
Vincenzo Ferme
 
Open Data
Vincenzo Ferme
 

Recently uploaded (20)

PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PPTX
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Encapsulation theory and applications.pdf
gurumoop
 
Teaching material agriculture food technology
LiaRayya
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 

Workflow Engine Performance Benchmarking with BenchFlow

  • 2. Vincenzo Ferme 2 The BenchFlow Project Design and implement the first benchmark to assess and compare the performance of WfMSs that are compliant with Business Process Model and Notation 2.0 standard. ” “
  • 3. Vincenzo Ferme 3 What is a Workflow Management System? WfMS Users Applications Application Server Instance Database DBMS Web Service D A B C
  • 4. Vincenzo Ferme 4 Many Vendors of BPMN 2.0 WfMSs https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines Jan 2011 BPMN 2.0 Jan 2014 BPMN 2.0.2 ISO/IEC 19510 Aug 2009 BETA BPMN 2.0 2016 0 23 Grand Total 23 NumberofBPMN2.0WfMSs 0 5 10 15 20 25 Year of the First Version Supporting BPMN 2.0 2009 2010 2011 2012 2013 2014 2015 2016
  • 5. Vincenzo Ferme 5 end-users, vendors, developers Why do We Need a Benchmark?
  • 6. Vincenzo Ferme 5 end-users, vendors, developers Why do We Need a Benchmark? 1. How to choose the best WfMS according to the company’s technical requirements? 2. How to choose the best WfMS 
 according to the company’s 
 business process models (workflows)? B C D
  • 7. Vincenzo Ferme 5 end-users, vendors, developers Why do We Need a Benchmark? 1. How to choose the best WfMS according to the company’s technical requirements? 2. How to choose the best WfMS 
 according to the company’s 
 business process models (workflows)? B C D 4. How to identify WfMS’s bottlenecks? 3. How to evaluate performance improvements
 during WfMS’s development?
  • 8. Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Workload Metrics KPIs Measure Input/Process/Output Model
  • 9. Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D
  • 10. Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data
  • 11. Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Containers
  • 12. Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers
  • 13. Vincenzo Ferme Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Figure 4: Benchmarking Methodology Choreography pending on the WfMS’s architecture. The DBMS Container can refer to an existing publicly available Container distributions. The containerized WfMS should be publicly available (e.g., at the Docker Hub registry8), or the Benchflow team should be granted access to a private registry used by the vendor. The same applies to the Containers’ definition file, i.e., the Dockerfile (Turnbull, 2014, ch. 4). While private reg- istries are a solution that can work with vendors of closed-source systems, they impact the reproducibil- ity of the results. For each WfMS version to be in- cluded in the benchmark, there must be a default con- figuration, i.e., the configuration in which the WfMS Containers start without modifying any configuration parameters, except the ones required in order to cor- rectly start the WfMS, e.g., the database connection. However, if vendors want to benchmark the perfor- mance of their WfMS with different configurations, for example, the configuration provided to users as a “getting started” configuration, or production-grade configurations for real-life usage, they can also pro- vide different configurations. To do that, the Con- tainers must allow to issue the WfMS configurations through additional environment variables9, and/or by access the configuration files. Exposing the volumes allows to access the files defined inside the Contain- ers, on the host operating system. Precisely, the WES Container has to, at least, enable the configuration of: 1) the used DB, i.e., DB driver, url, username and password for connection; 2) the WES itself; and 3) the logging level of the WES, and the application stack layers on which it may depend on. Alternatively, in- stead of providing configuration details, the vendors can provide different Containers for different config- urations. However, even in that case enabling the DB configuration is required. In order to access relevant data, all WfMS Con- tainers have to specify the volumes to access the WfMS log files, and to access all the data useful to setup the system. For instance, if the WfMS defines examples as part of the deployment, we may want to remove those examples by overriding the volume containing them. Moreover, the WES Container (or the DBMS Container) has to create the WfMS’s DB and populate it with data required to run the WfMS. The vendors need to provide authentication configu- ration details of the WfMS components (e.g., the user 7 Methodology Framework + Container Based Methodology and Framework BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness MONITOR Adapters COLLECTORS
  • 14. Vincenzo Ferme Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Figure 4: Benchmarking Methodology Choreography pending on the WfMS’s architecture. The DBMS Container can refer to an existing publicly available Container distributions. The containerized WfMS should be publicly available (e.g., at the Docker Hub registry8), or the Benchflow team should be granted access to a private registry used by the vendor. The same applies to the Containers’ definition file, i.e., the Dockerfile (Turnbull, 2014, ch. 4). While private reg- istries are a solution that can work with vendors of closed-source systems, they impact the reproducibil- ity of the results. For each WfMS version to be in- cluded in the benchmark, there must be a default con- figuration, i.e., the configuration in which the WfMS Containers start without modifying any configuration parameters, except the ones required in order to cor- rectly start the WfMS, e.g., the database connection. However, if vendors want to benchmark the perfor- mance of their WfMS with different configurations, for example, the configuration provided to users as a “getting started” configuration, or production-grade configurations for real-life usage, they can also pro- vide different configurations. To do that, the Con- tainers must allow to issue the WfMS configurations through additional environment variables9, and/or by access the configuration files. Exposing the volumes allows to access the files defined inside the Contain- ers, on the host operating system. Precisely, the WES Container has to, at least, enable the configuration of: 1) the used DB, i.e., DB driver, url, username and password for connection; 2) the WES itself; and 3) the logging level of the WES, and the application stack layers on which it may depend on. Alternatively, in- stead of providing configuration details, the vendors can provide different Containers for different config- urations. However, even in that case enabling the DB configuration is required. In order to access relevant data, all WfMS Con- tainers have to specify the volumes to access the WfMS log files, and to access all the data useful to setup the system. For instance, if the WfMS defines examples as part of the deployment, we may want to remove those examples by overriding the volume containing them. Moreover, the WES Container (or the DBMS Container) has to create the WfMS’s DB and populate it with data required to run the WfMS. The vendors need to provide authentication configu- ration details of the WfMS components (e.g., the user 7 Methodology Framework + Container Based Methodology and Framework BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness MONITOR Adapters COLLECTORS Provides Great Advantages for Automation and Reproducibility of Results, while Ensuring Negligible Performance Impact
  • 15. Vincenzo Ferme 8 [CLOSER ’16] Container-centric Methodology for Benchmarking Workflow Management Systems. Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Figure 4: Benchmarking Methodology Choreography pending on the WfMS’s architecture. The DBMS Container can refer to an existing publicly available Container distributions. The containerized WfMS should be publicly available (e.g., at the Docker Hub registry8), or the Benchflow team should be granted access to a private registry used by the vendor. The access the configuration files. Exposing the volumes allows to access the files defined inside the Contain- ers, on the host operating system. Precisely, the WES Container has to, at least, enable the configuration of: 1) the used DB, i.e., DB driver, url, username and password for connection; 2) the WES itself; and 3) the
  • 16. Vincenzo Ferme 9 BenchFlow Benchmarking Methodology requirements from the WfMS Availability of APIs • Deploy Process • Start Process Instance • Users API • Web Service APIs • Events APIs
  • 17. Vincenzo Ferme 9 BenchFlow Benchmarking Methodology requirements from the WfMS Availability of APIs • Deploy Process • Start Process Instance • Users API • Web Service APIs • Events APIs • Workflow & Construct: • Start Time • End Time • [Duration] Availability of Timing Data
  • 18. Vincenzo Ferme 9 BenchFlow Benchmarking Methodology requirements from the WfMS Availability of APIs • Deploy Process • Start Process Instance • Users API • Web Service APIs • Events APIs • Workflow & Construct: • Start Time • End Time • [Duration] Availability of Timing Data Availability of Execution State State of the workflow execution. E.g., running, completed, error
  • 19. Vincenzo Ferme 10 BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness benchflow [BPM ’15] [ICPE ’16] MONITOR Adapters COLLECTORS
  • 20. Vincenzo Ferme 11 BenchFlow Framework WfMSs’ specific characteristics •Manages the Automatic WfMS deployment; •Provides a “plugin” mechanism to add new WfMSs; •Performs automatic Process Models deployment; •Collects Client and Server-side data and metrics; •Automatically computes performance metrics and
 statistics.
  • 21. Vincenzo Ferme 12 Applications of the Methodology and Framework [CAiSE ’16] Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns. WfMS MySQL 3 WfMSsWorkload Empty Script 1 Empty Script 2 … 0 750 1500 0:00 4:00 10:00 Metrics • Engine Level • Process Level • Environment
  • 22. Vincenzo Ferme Micro-Benchmarking with Workflow Patterns 13 research questions Selected three WfMSs: popular open-source WfMSs actually used in industry 1. What is the impact of individual or a mix of workflow patterns on the performance of each one of the benchmarked BPMN 2.0 WfMSs? 2. Are there performance bottlenecks for the selected WfMSs?
  • 23. Vincenzo Ferme 14 Empty Script 1 Empty Script 2 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix Sequence Pattern [SEQ] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
  • 24. Vincenzo Ferme 14 Empty Script 1 Empty Script 2 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix Sequence Pattern [SEQ]exclusiveChoicePattern generate number Empty Script 1 Empty Script 2case_2 case_1 Exclusive Choice and Simple Merge [EXC] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
  • 25. Vincenzo Ferme 15 lesPattern generate 1 or 2 Empty Script 1 Empty Script 2 i++ i < 10 i >= 10 case_1 case_2 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix Arbitrary Cycle [CYC] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
  • 26. Vincenzo Ferme 16 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix Empty Script 1 Empty Script 2 Parallel Split and Synchronisation [PAR] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
  • 27. Vincenzo Ferme 16 Workload Mix Micro-Benchmarking with Workflow Patterns workload mix citTerminationPattern Empty Script 1 Wait 5 Sec Explicit Termination Pattern [EXT] Empty Script 1 Empty Script 2 Parallel Split and Synchronisation [PAR] • N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006 Pattern Reference:
  • 28. Vincenzo Ferme 17 Micro-Benchmarking with Workflow Patterns workload mix design decisions 1. Maximise the simplicity of the model expressing the workflow pattern 2. Omit the interactions with external systems by implementing all tasks as script tasks
  • 29. Vincenzo Ferme 18 Load Functions Micro-Benchmarking with Workflow Patterns load function Test Type Load test Instance Producers 0 300 600 900 1200 1500 Time (min:sec) 0:000:301:002:003:004:005:006:007:008:0010:00 Example for u=1500
  • 30. Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C
  • 31. Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26
  • 32. Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79
  • 33. Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final
  • 34. Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final Max Java Heap: 32GB
  • 35. Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final Max Java Heap: 32GB Max DB Connections Number: 100
  • 36. Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 WfMS’s Containers: official ones, if available on DockerHub O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final Max Java Heap: 32GB Max DB Connections Number: 100
  • 37. Vincenzo Ferme 19 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS environments WfMS A WfMS B WfMS C MySQL MySQL MySQL MySQL: Community Server 5.6.26 WfMS’s Containers: official ones, if available on DockerHub WfMS’s Configuration: as suggested by vendor’s documentation O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01 J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79 App. Server: Ap.Tomcat 7.0.62 Ap.Tomcat 7.0.62 Wildfly 8.1.0. Final Max Java Heap: 32GB Max DB Connections Number: 100
  • 38. Vincenzo Ferme 20 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS deployment TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WfMS CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS
  • 39. Vincenzo Ferme 20 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS deployment TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WfMS CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS 10 Gbit/s 10 Gbit/s
  • 40. Vincenzo Ferme 20 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS deployment TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WfMS CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS 10 Gbit/s 10 Gbit/s O.S.: Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS
  • 41. Vincenzo Ferme 20 WfMS Configurations Micro-Benchmarking with Workflow Patterns configurations: WfMS deployment TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WfMS CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS 10 Gbit/s 10 Gbit/s O.S.: Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS 1.8.2 1.8.2 1.8.2
  • 42. Vincenzo Ferme 21 Micro-Benchmarking with Workflow Patterns performed experiments Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR])
  • 43. Vincenzo Ferme 21 Micro-Benchmarking with Workflow Patterns performed experiments Three runs for each execution of the experiments Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR])
  • 44. Vincenzo Ferme 21 Micro-Benchmarking with Workflow Patterns performed experiments Three runs for each execution of the experiments The data collected in the first minute of the execution has not been included in the analysis Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR])
  • 45. Vincenzo Ferme 21 Micro-Benchmarking with Workflow Patterns performed experiments Three runs for each execution of the experiments The data collected in the first minute of the execution has not been included in the analysis Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR])
  • 46. Vincenzo Ferme 22 Number of Process Instances Computed Metrics and Statistics engine level metrics Throughput Execution Time Database Size WfMS Users Load Driver Instance Database Application Server DBMS Web Service D A B C …
  • 47. Vincenzo Ferme 22 Number of Process Instances Computed Metrics and Statistics engine level metrics Throughput Execution Time Database Size WfMS Users Load Driver Instance Database Application Server DBMS Web Service D A B C …
  • 48. Vincenzo Ferme 23 individual patterns: throughput (bp/sec) Micro-Benchmarking with Workflow Patterns SEQ EXC EXT PAR CYC WfMS A WfMS B WfMS C 1456.79 1417.17 1455.68 1433.12 327.99 1447.66 1436.03 1426.35 1429.65 644.02 63.31 49.04 0.38 48.89 13.46 u=600 u=800 u=1500
  • 49. Vincenzo Ferme 24 Micro-Benchmarking with Workflow Patterns individual patterns: WfMS C Scalability for SEQ pattern WfMS C uses a synchronous instantiation API, load drivers must wait for workflows to end before starting new ones
  • 50. Vincenzo Ferme 24 Micro-Benchmarking with Workflow Patterns individual patterns: WfMS C Scalability for SEQ pattern WfMS C uses a synchronous instantiation API, load drivers must wait for workflows to end before starting new ones Response Time 0 14 28 500 1000 1500 2000 u Instance Producers Throughput 0 40 80 500 1000 1500 2000 u Instance Producers
  • 51. Vincenzo Ferme 24 Micro-Benchmarking with Workflow Patterns individual patterns: WfMS C Scalability for SEQ pattern WfMS C uses a synchronous instantiation API, load drivers must wait for workflows to end before starting new ones Response Time 0 14 28 500 1000 1500 2000 u Instance Producers Throughput 0 40 80 500 1000 1500 2000 u Instance Producers We have found a bottleneck! (~60 bp/sec)
  • 52. Vincenzo Ferme 25 Computed Metrics and Statistics process level metrics Throughput WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS … Instance Duration PerProcess Definition
  • 53. Vincenzo Ferme 25 Computed Metrics and Statistics process level metrics Throughput WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS … Instance Duration PerProcess Definition
  • 54. Vincenzo Ferme 26 Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time)
  • 55. Vincenzo Ferme 26 Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time) Empty Script 1 Empty Script 2 [SEQ]
  • 56. Vincenzo Ferme 26 Empty Script 1 Wait 5 Sec [EXT] Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time)
  • 57. Vincenzo Ferme 26 Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time) Empty Script 1 Empty Script 2 [PAR]
  • 58. Vincenzo Ferme 26 Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time) CYC Instance Producers: u=600 [CYC] tern generate 1 or 2 Empty Script 1 Empty Script 2 i++ i < 10 i >= 10 case_1 case_2
  • 59. Vincenzo Ferme 27 Computed Metrics and Statistics environment metrics WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS CPU/RAM Usage per Second IO Operations/Network Usage per Second …
  • 60. Vincenzo Ferme 27 Computed Metrics and Statistics environment metrics WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS CPU/RAM Usage per Second IO Operations/Network Usage per Second …
  • 61. Vincenzo Ferme 28 Micro-Benchmarking with Workflow Patterns individual patterns: mean (RAM) and mean (CPU) usage RAM CPU
  • 62. Vincenzo Ferme 29 Micro-Benchmarking with Workflow Patterns performed experiments Individual patterns 1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR] Mix of patterns 2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR]) Three runs for each execution of the experiments The data collected in the first minute of the execution has not been included in the analysis
  • 63. Vincenzo Ferme 30 Micro-Benchmarking with Workflow Patterns mix of patterns: duration, RAM, CPU
  • 64. Vincenzo Ferme 31 Micro-Benchmarking with Workflow Patterns summary Even though the Executed Workflows are Simple
  • 65. Vincenzo Ferme 31 Micro-Benchmarking with Workflow Patterns summary There are relevant differences in workflows’ duration and throughput among the WfMSs Even though the Executed Workflows are Simple
  • 66. Vincenzo Ferme 31 Micro-Benchmarking with Workflow Patterns summary There are relevant differences in workflows’ duration and throughput among the WfMSs There are relevant differences in resource usage among WfMSs CPURAM Even though the Executed Workflows are Simple
  • 67. Vincenzo Ferme 31 Micro-Benchmarking with Workflow Patterns summary There are relevant differences in workflows’ duration and throughput among the WfMSs One WfMS has bottlenecks There are relevant differences in resource usage among WfMSs CPURAM Even though the Executed Workflows are Simple
  • 69. Highlights Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers Benchmarking Process
  • 70. Highlights Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers Benchmarking Process Vincenzo Ferme 8 [CLOSER ’16] Container-centric Methodology for Benchmarking Workflow Management Systems. Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Benchmarking Methodology
  • 71. Highlights Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers Benchmarking Process Vincenzo Ferme 8 [CLOSER ’16] Container-centric Methodology for Benchmarking Workflow Management Systems. Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Benchmarking Methodology Vincenzo Ferme 10 BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness benchflow [BPM ’15] [ICPE ’16] MONITOR Adapters COLLECTORS BenchFlow Framework
  • 72. Highlights Vincenzo Ferme 6 The BenchFlow Benchmarking Process Workload Model WfMS Configurations Test Types Workload Metrics KPIs Measure Input/Process/Output Model Workload Mix 80% C A B 20% D Load Functions Test Data Performance Data Process Instance Duration Throughput Containers Benchmarking Process Vincenzo Ferme 8 [CLOSER ’16] Container-centric Methodology for Benchmarking Workflow Management Systems. Vendor BenchFlow Provide Benchmarking Methodology Vendor BenchFlow Agree on adding Vendor's WfMS to the Benchmark Benchmarking Methodology Agreement Proposal Signed Agreement Vendor BenchFlow Provide Containerized Distribution of WfMS Containerized WfMS Request Containerized WfMS Vendor BenchFlow Verify the Benchmark Results Results Verification Outcome Results Verification Request Vendor BenchFlow Provide Draft Benchmark Results Draft Benchmark ResultsVerified Benchmark Results Not Valid Are the Results Valid? Community BenchFlow Publish Benchmark Results Valid Benchmarking Methodology Vincenzo Ferme 10 BenchFlow Framework architecture Instance Database DBMSFaban Drivers ContainersServers DATA
 TRANSFORMERS ANALYSERS Performance Metrics Performance KPIs WfMS TestExecutionAnalyses Faban + Web Service Minio harness benchflow [BPM ’15] [ICPE ’16] MONITOR Adapters COLLECTORS BenchFlow Framework Vincenzo Ferme 26 Empty Script 1 Wait 5 Sec [EXT] Micro-Benchmarking with Workflow Patterns individual patterns: mean (instance duration time) Empty Script 1 Empty Script 2 [SEQ] Empty Script 1 Empty Script 2 [PAR] CYC Instance Producers: u=600 [CYC] generate 1 or 2 Empty Script 1 Empty Script 2 i++ i < 10 i >= 10 case_1 case_2 Micro-Benchmark with WP
  • 73. Vincenzo Ferme 33 •We want to characterise the Workload Mix using Real-World process models • Share your executable BPMN 2.0 process models, even anonymised Process Models •We want to characterise the Load Functions using Real-World behaviours • Share your execution logs, even anonymised Execution Logs •We want to add more and more WfMSs to the benchmark • Contact us for collaboration, and BenchFlow framework support WfMSs Call for Collaboration WfMSs, process models, process logs
  • 76. Vincenzo Ferme 36 Published Work [BTW ’15] C. Pautasso, V. Ferme, D. Roller, F. Leymann, and M. Skouradaki. Towards workflow benchmarking: Open research challenges. In Proc. of the 16th conference on Database Systems for Business,Technology, and Web, BTW 2015, pages 331–350, 2015. [SSP ’14] M. Skouradaki, D. H. Roller, F. Leymann, V. Ferme, and C. Pautasso. Technical open challenges on benchmarking workflow management systems. In Proc. of the 2014 Symposium on Software Performance, SSP 2014, pages 105–112, 2014. [ICPE ’15] M. Skouradaki, D. H. Roller, L. Frank, V. Ferme, and C. Pautasso. On the Road to Benchmarking BPMN 2.0 Workflow Engines. In Proc. of the 6th ACM/SPEC International Conference on Performance Engineering, ICPE ’15, pages 301–304, 2015.
  • 77. Vincenzo Ferme 37 Published Work [CLOSER ’15] M. Skouradaki,V. Ferme, F. Leymann, C. Pautasso, and D. H. Roller. “BPELanon”: Protect business processes on the cloud. In Proc. of the 5th International Conference on Cloud Computing and Service Science, CLOSER 2015. SciTePress, 2015. [SOSE ’15] M. Skouradaki, K. Goerlach, M. Hahn, and F. Leymann. Application of Sub-Graph Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models. In Proc. of the 9th International IEEE Symposium on Service-Oriented System Engineering, SOSE 2015, 2015. [BPM ’15] V. Ferme, A. Ivanchikj, C. Pautasso. A Framework for Benchmarking BPMN 2.0 Workflow Management Systems. In Proc. of the 13th International Conference on Business Process Management, BPM ’15, pages 251-259, 2015.
  • 78. Vincenzo Ferme 38 Published Work [BPMD ’15] A. Ivanchikj,V. Ferme, C. Pautasso. BPMeter: Web Service and Application for Static Analysis of BPMN 2.0 Collections. In Proc. of the 13th International Conference on Business Process Management [Demo], BPM ’15, pages 30-34, 2015. [ICPE ’16] V. Ferme, and C. Pautasso. Integrating Faban with Docker for Performance Benchmarking. In Proc. of the 7th ACM/SPEC International Conference on Performance Engineering, ICPE ’16, 2016. [CLOSER ’16] V. Ferme, A. Ivanchikj, C. Pautasso., M. Skouradaki, F. Leymann. A Container-centric Methodology for Benchmarking Workflow Management Systems. In Proc. of the 6th International Conference on Cloud Computing and Service Science, CLOSER 2016. SciTePress, 2016.
  • 79. Vincenzo Ferme [CAiSE ’16] M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns . In Proc. of the 28th International Conference on Advanced Information Systems Engineering, CAiSE ’16, 2016. 39 Published Work [ICWE ’16] C. Jürgen,V. Ferme, H.C. Gall. Using Docker Containers to Improve Reproducibility in Software and Web Engineering Research. In Proc. of the 16th International Conference on Web Engineering, 2016. [ICWS ’16] M. Skouradaki,V.Andrikopoulos, F. Leymann. Representative BPMN 2.0 Process Model Generation from Recurring Structures. In Proc. of the 23rd IEEE International Conference on Web Services, ICWS '16, 2016.
  • 80. Vincenzo Ferme 40 Published Work [SummerSOC ’16] M. Skouradaki, T. Azad, U. Breitenbücher, O. Kopp, F. Leymann. A Decision Support System for the Performance Benchmarking of Workflow Management Systems. In Proc. of the 10th Symposium and Summer School On Service-Oriented Computing, SummerSOC '16, 2016. [OTM ’16] M. Skouradaki, V. Andrikopoulos, O. Kopp, F. Leymann. RoSE: Reoccurring Structures Detection in BPMN 2.0 Process Models Collections. In Proc. of On the Move to Meaningful Internet Systems Conference, OTM '16, 2016. (to appear) [BPM Forum ’16] V. Ferme, A. Ivanchikj, C. Pautasso. Estimating the Cost for Executing Business Processes in the Cloud. In Proc. of the 14th International Conference on Business Process Management, BPM Forum ’16, 2016. (to appear)
  • 81. Vincenzo Ferme 41 Docker Performance [IBM ’14] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An updated performance comparison of virtual machines and Linux containers. IBM Research Report, 2014. Although containers themselves have almost no overhead, Docker is not without performance gotchas. Docker volumes have noticeably better performance than files stored in AUFS. Docker’s NAT also introduces overhead for workloads with high packet rates. These features represent a tradeoff between ease of management and performance and should be considered on a case-by-case basis. ” “Our results show that containers result in equal or better performance than VMs in almost all cases. “ ” BenchFlow Configures Docker for Performance by Default
  • 82. Vincenzo Ferme 42 Benchmarking Requirements • Relevant • Representative • Portable • Scalable • Simple • Repeatable • Vendor-neutral • Accessible • Efficient • Affordable • K. Huppler, The art of building a good benchmark, 2009 • J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993 • S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A challenge to software engineering, 2003
  • 83. Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
  • 84. Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE WfMS Initialisation APIs Deploy Process Start Process Instance Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
  • 85. Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE WfMS Initialisation APIs Deploy Process Start Process Instance + WfMS Claim Task Complete Task User APIs Create User Create Group Pending User Tasks Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
  • 86. Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE WfMS Initialisation APIs Deploy Process Start Process Instance + WfMS Claim Task Complete Task User APIs Create User Create Group Pending User Tasks Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work Invoke WS Web Service APIs
  • 87. Vincenzo Ferme 43 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS NON-CORECORE WfMS Initialisation APIs Deploy Process Start Process Instance Event APIs Issue Event WfMS Pending Event Tasks + WfMS Claim Task Complete Task User APIs Create User Create Group Pending User Tasks Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work Invoke WS Web Service APIs
  • 88. Vincenzo Ferme 44 BenchFlow Benchmarking MethodologyWfMS Configurations requirements from the WfMS Table 1: Summary of Core and Non-core APIs to be imple- mented by the WfMS Functionality Min Response Data CoreAPIs Initialisation APIs Deploy a process Deployed process ID Start a process instance Process instance ID Non-coreAPIs User APIs Create a user User ID Create a group of users User group ID Access pending tasks Pending tasks IDs Claim a task* Complete a task Event APIs Access pending events Pending events IDs Issue events Web service APIs Map tasks to Web service endpoints *Optional depending on the WfMS implementation ing the a u (U ate by acc a g and mo ing (Pi has col Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
  • 90. Vincenzo Ferme 45 BenchFlow Framework system under test Docker Engine Containers
  • 91. Vincenzo Ferme 45 BenchFlow Framework system under test Docker Engine Docker Machine provides Containers
  • 92. Vincenzo Ferme 45 BenchFlow Framework system under test Docker Swarm Docker Engine Docker Machine provides Containers
  • 93. Vincenzo Ferme 45 BenchFlow Framework system under test Docker Swarm Docker Engine Docker Machine provides manages Containers Servers
  • 94. Vincenzo Ferme 45 BenchFlow Framework system under test Docker Swarm Docker Engine Docker Compose Docker Machine provides manages Containers Servers
  • 95. Vincenzo Ferme 45 BenchFlow Framework system under test Docker Swarm Docker Engine Docker Compose Docker Machine provides manages Containers Servers SUT’s Deployment Conf. deploys
  • 96. Vincenzo Ferme 46 Server-side Data and Metrics Collection WfMS Start Workflow asynchronous execution of workflows
  • 97. Vincenzo Ferme Users D A B C Start Application Server Web Service Load Driver DBMS WfMS Instance Database 46 Server-side Data and Metrics Collection WfMS Start Workflow asynchronous execution of workflows
  • 98. Vincenzo Ferme Users D A B C Start Application Server Web Service Load Driver DBMS WfMS Instance Database 46 Server-side Data and Metrics Collection WfMS Start Workflow End asynchronous execution of workflows
  • 99. Vincenzo Ferme 47 Server-side Data and Metrics Collection monitors DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service
  • 100. Vincenzo Ferme 47 Server-side Data and Metrics Collection monitors DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service MONITOR
  • 101. Vincenzo Ferme 48 Server-side Data and Metrics Collection monitors DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service • CPU usage • Database state Examples of Monitors:Monitors’ Characteristics: • RESTful services • Lightweight (written in Go) • As less invasive on the SUT as possible
  • 102. Vincenzo Ferme 48 Server-side Data and Metrics Collection monitors DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service CPU Monitor DB Monitor • CPU usage • Database state Examples of Monitors:Monitors’ Characteristics: • RESTful services • Lightweight (written in Go) • As less invasive on the SUT as possible
  • 103. Vincenzo Ferme 49 Server-side Data and Metrics Collection collect data DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service
  • 104. Vincenzo Ferme 49 Server-side Data and Metrics Collection collect data DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service Minio Instance Database Analyses COLLECTORS
  • 105. Vincenzo Ferme 50 Server-side Data and Metrics Collection collect data DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service • Container’s Stats (e.g., CPU usage) • Database dump • Applications Logs Examples of Collectors:Collectors’ Characteristics: • RESTful services • Lightweight (written in Go) • Two types: online and offline • Buffer data locally
  • 106. Vincenzo Ferme 50 Server-side Data and Metrics Collection collect data DBMSFaban Drivers ContainersServers harness WfMS TestExecution Web Service Stats Collector DB Collector • Container’s Stats (e.g., CPU usage) • Database dump • Applications Logs Examples of Collectors:Collectors’ Characteristics: • RESTful services • Lightweight (written in Go) • Two types: online and offline • Buffer data locally
  • 107. Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector Synchronisation with the Analyses schedule analyses
  • 108. Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector COLLECT Synchronisation with the Analyses schedule analyses
  • 109. Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector COLLECT Publish Synchronisation with the Analyses schedule analyses
  • 110. Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector COLLECT Publish Subscribe Synchronisation with the Analyses schedule analyses
  • 111. Vincenzo Ferme 51 TestExecution A high-throughput distributed messaging system Minio Instance Database Analyses DBMSFaban Drivers ContainersServers harness WfMS Web Service Stats Collector DB Collector COLLECT Publish Subscribe Read Synchronisation with the Analyses schedule analyses
  • 112. Vincenzo Ferme 52 Performance Metrics and KPIs amount of data REALISTIC DATA Number of Repetitions 1 10 5 3
  • 113. Vincenzo Ferme 52 Performance Metrics and KPIs amount of data REALISTIC DATA Number of Repetitions 1 10 5 3 Number of WfMSs 1
  • 114. Vincenzo Ferme 53 Micro-Benchmarking with Workflow Patterns mix of patterns: throughput (bp/sec) MIX WfMS A WfMS B WfMS C 1061.27 1402.33 1.78
  • 115. Vincenzo Ferme 54 Computed Metrics and Statistics feature level metrics WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS Throughput … Instance Duration PerConstruct (type,name)
  • 116. Vincenzo Ferme 54 Computed Metrics and Statistics feature level metrics WfMS Users Load Driver Instance Database Application Server Web Service D A B C DBMS Throughput … Instance Duration PerConstruct (type,name)
  • 117. Vincenzo Ferme 55 Computed Metrics and Statistics interactions metrics Response Time Latency … WfMS Users Load Driver Instance Database Application Server DBMS Web Service D A B C
  • 118. Vincenzo Ferme 55 Computed Metrics and Statistics interactions metrics Response Time Latency … WfMS Users Load Driver Instance Database Application Server DBMS Web Service D A B C
  • 119. Vincenzo Ferme 56 Computed Metrics and Statistics descriptive and homogeneity statistics • Descriptive Statistics (e.g., Mean, Confidence Interval, 
 Standard Deviation, Percentiles, Quartiles, …) • Coefficient ofVariation across trials • Levene Test on trials