Evaluating the efficiency of using a search-based automated model merge technique
0 likes36 views
The document presents a novel search-based automated model merge technique aimed at assisting engineers in merging models more effectively, efficiently, and with lower cognitive effort. An experiment was conducted to compare this technique with existing alternatives, revealing significant advantages in user satisfaction and reduced cognitive workload. Future work includes extending the study to domain modelers and exploring crowdsourcing platforms for further validation.
![Experiment instruments
5 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al.
Instrument Scale
Profile Availability Form
Background Questionnaire
[0 -> 5]
Duration Video recording second
Success Quality of delivered Task solutions [0 -> 1]
Cognitive Effort NASA TLX Scale [0 -> 100]
Satisfaction Satisfaction Questionnaire [(-1) -> 1]
Preference Feedback Questionnaire 0 or 1](https://image.slidesharecdn.com/efficiencyeval-190520133456/85/Evaluating-the-efficiency-of-using-a-search-based-automated-model-merge-technique-5-320.jpg)
![Participants
8 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al.
Total G1 G2
Number 15 6 9
Profile [0-5] 1.65 1.92 1.39
Industry 56% 67% 44%](https://image.slidesharecdn.com/efficiencyeval-190520133456/85/Evaluating-the-efficiency-of-using-a-search-based-automated-model-merge-technique-8-320.jpg)
Evaluating the efficiency of using a search-based automated model merge technique
- 1. Evaluating the efficiency of using a search-based automated model merge technique Ankica Barišić, Csaba Debrecani, Daniel Varro, Vasco Amaral, Miguel Goulão
- 2. DSE Merge 2 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al. ❏ a novel technique for search-based automated model merge ❏ uses guided rule-based design-space exploration (DSE) for merging models ❏ support domain modelers without requiring from these experts a high level of programming expertise
- 3. Experiment Objective Engineers can perform model merge operations ... ❏ H1: more effectively, producing correct results (i.e. merged models are of better quality) ❏ H2: more efficiently (i.e. obtained faster merged models) ❏ H3: more satisfactory (i.e. the modelling activity is perceived as more pleasant) ❏ H4: with less cognitive effort (i.e. lower modelling workload) 3 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al. How usable is the proposed technique for performing the model merge operations when compared to the alternative?
- 4. Experiment context 4 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al. Scoping problem to the alternative supports: • User profile - Software engineers • Technology - Eclipse IDE @ Windows 7 • Social and physical environment - ‘Work-alone’ office environment • Domain - Wind turbine • Workflow - Find the best merge solution between local and remote changes on the given instance model Alternative supports for software engineers during the model merge process: ❏ EMF Compare ❏ Diff Merge
- 5. Experiment instruments 5 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al. Instrument Scale Profile Availability Form Background Questionnaire [0 -> 5] Duration Video recording second Success Quality of delivered Task solutions [0 -> 1] Cognitive Effort NASA TLX Scale [0 -> 100] Satisfaction Satisfaction Questionnaire [(-1) -> 1] Preference Feedback Questionnaire 0 or 1
- 6. Tasks 6 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al. Task Model Size Change Size Solutions Cognitive Effort Time Success T1 Small 4 2 25.83 3:32 1 T2 Small 12 8 28.61 4:59 1 T3 Big 6 2 20.55 3:18 0.88 T4 Big 54 >million 24.02 4:27 0.83 1 0.75 0.5 0.25 0 T1 A possible solution is provided Only one conflict is resolved correctly None of the two conflicts are resolved correctly. Other parts of the model are modified Nothing happened T2 A possible solution is provided Not conflicting changes are not applied Only non-conflicting changes are applied Other parts of the model are modified Nothing happened T3 A possible solution is provided Only one conflict is resolved correctly None of the two conflicts are resolved correctly Other parts of the model are modified Nothing happened T4 A possible solution is provided At least half of changes are applied Only local or only remote changes are applied Other parts of the model are modified Nothing happened
- 7. Experiment execution 7 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al.
- 8. Participants 8 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al. Total G1 G2 Number 15 6 9 Profile [0-5] 1.65 1.92 1.39 Industry 56% 67% 44%
- 9. 9 Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al. DSE Merge Diff Merge Sig. (2-tailed) H1 Effectiveness 0.90 0.82 0.16 H2 Efficiency 1289.36 1355.71 0.73 H3 Satisfaction 0.27 0.04 0.01 H4 Cognitive Effort 53.09 65.31 0.05 Results (Welch T test)
- 10. Conclusions & Future Work 10 DSE Merge has clear advantages regarding the satisfaction (H3) of their users and the cognitive effort (H4) required to use it FUTURE WORK: ● extend the study to domain modellers enabling them to incorporate domain-specific knowledge into the merge process ● replicate experiments using crowdsourcing platforms Threats to validity ○ selection validity threat ○ hypothesis guessing ○ experimenter’s expectations Evaluating the efficiency of using a search-based automated model merge techniqueAnkica Barišić et. al.
- 11. Acknowledgments ● COST Action IC1404 MPM4CPS ● DSML4MA TUBITAK/0008/2014 ● FCT/MEC NOVA LINCS; PEst UID/ CEC/04516/ 2013