BPMS2@BPM2018
Download as PPTX, PDF0 likes276 views
This document summarizes a study that investigated whether there is a trade-off between effectiveness and efficiency in process modeling. 146 master's students created business process models. The study measured time spent, number of errors made and solved, and whether students engaged in a revision phase at the end. The results showed no relationship between time/effort and errors made, but a relationship between time/effort and errors solved. Of the students, 81 engaged in a revision phase, with 48% finding and fixing more errors during revision. However, the study found only mild support for the hypotheses due to limitations such as subjective error evaluations and a narrow focus only on syntax errors.
BPMS2@BPM2018
- 1. 1/15 | www.janclaes.info Jeroen Bolle and Jan Claes Investigating the trade-off between the effectiveness and efficiency of process modeling
- 2. 2/15 | www.janclaes.info Motivation Previous research Differentiated process modeling technique Proposed effects on effectiveness and efficiency Dataset and info can be downloaded at http://www.janclaes.info/experiments February 2015 Ghent University 146 master students Business Engineering BPM course
- 3. 3/15 | www.janclaes.info Motivation Jan Claes, Irene Vanderfeesten, Frederik Gailly, Paul Grefen, Geert Poels, The Structured Process Modeling Method (SPMM) - What is the best way for me to construct a process model?, Decision Support Systems, Vol 100, p. 57-76, 2017 EFFECTIVENESS EFFICIENCY Low R2 values High R2 values
- 4. 4/15 | www.janclaes.info Research question Is there a trade-off between effectiveness and efficiency in process modeling? Is there a revision phase at the end of modeling? (revising = investing time and effort = lower efficiency) Does this revision phase result in higher quality? (revising = correcting errors = higher effectiveness)
- 5. 5/15 | www.janclaes.info Measurements Time: number of seconds in timeframe Effort: number of operations in timeframe Syntactic quality Number of syntax errors made in timeframe Number of syntax errors solved in timeframe Dataset of this paper at DOI 10.17632/5b8by4k244.1 For syntax error determination in partial models, see De Bock & Claes (2018)
- 6. 6/15 | www.janclaes.info Results Numberoferrorsper100elementsinfinalmodel Total modeling time Errors made Errors solved Errors remaining
- 7. 7/15 | www.janclaes.info Results Total time Total effort Errors made -0,200 (p=0,820) -0,003 (p=0,970) Errors solved 0,253* (p=0,050) 0,349** (p=0,000) Errors remaining -0,183* (p=0,044) -0,229* (p=0,012)
- 8. 8/15 | www.janclaes.info Revision phase How to determine time ‘t’ ? Modeling phase: most of the creation Revision phase: moving, deleting, or renaming elements, extensive scrolling Revision phase: also creates after deletes or to close gaps etc. Result: 81 of 121 modelers had a revision phase Modeling phase Revision phase t
- 9. 9/15 | www.janclaes.info Revision phase RevisionModeling RevisionModeling
- 10. 10/15 | www.janclaes.info Results Errors made in revision phase minus errors made in modeling phase Errors solved in revision phase minus errors solved in modeling phase Relatively more errors made in modeling phase per100elementsperseconds
- 11. 11/15 | www.janclaes.info Results Errors made in revision phase minus errors solved in revision phase Errors made in revision phase and Errors solved in revision phase 21% Relatively more errors made than solved in revision phase per100elementsperseconds
- 12. 12/15 | www.janclaes.info Conclusions Is there a trade-off between effectiveness and efficiency in process modeling? No relation between time/effort and errors made Relation between time/effort and errors solved Is there a revision phase at the end of modeling? 81 of 121 modelers (66%) had a revision phase Does this revision phase result in higher quality? 48% higher, 41% equal, 21% lower number of errors
- 13. 13/15 | www.janclaes.info Discussion Only mild support for hypotheses 33% no revision phase 14% contra-productive revision phase 28% no productive revision phase 25% productive revision phase
- 14. 14/15 | www.janclaes.info Limitations Difficult comparison We normalized to model size and model duration We did not normalize to number of errors made No discrimination between error types Solving a major error while introducing a minor error… Only syntax errors Most objective to measure, but least relevant? Subjective measures by limited number of people E.g., naïve revision phase approach Limited external validity 121 master students Business Engineering at UGent
- 15. 15/15 | www.janclaes.info Do you have any questions? Do you have feedback? Thanks for you attention! Jan Claes jan.claes@ugent.be www.janclaes.info