MAV-Vis: A Notation for Model Uncertainty
1 like877 views
This document describes a user study that evaluated MAV-Vis, a new graphical notation for modeling uncertainty in models, compared to the existing textual notation MAV-Text. The study found that MAV-Vis was considered more intuitive and easier to read, although it was slightly slower for writing. Specifically, participants had fewer errors reading models with alternative refinements represented using MAV-Vis. However, more errors occurred when writing with MAV-Vis, mainly due to confusion over the use of color to represent points of uncertainty. The results provide preliminary evidence that MAV-Vis improves cognitive effectiveness for reading but may require more practice for writing complex models with uncertainty.
![MAV-Vis: A
Notation for
Model
Uncertainty
M.Famelis,
S.Santosa
Introduction
Design
Uncertainty
MAV-Vis
User Study
Conclusion
Design Uncertainty
“The reality of today’s software systems requires us to consider
uncertainty as a first-class concern in the design, implementation,
and deployment of those systems.” [Garlan, 2010]
Our focus:
Uncertainty about design decisions – the contents of a model.
Our agenda:
Enable MBSE with design uncertainty using Partial models.
2 / 27](https://image.slidesharecdn.com/slides-130519134348-phpapp02/85/MAV-Vis-A-Notation-for-Model-Uncertainty-2-320.jpg)
![MAV-Vis: A
Notation for
Model
Uncertainty
M.Famelis,
S.Santosa
Introduction
Design
Uncertainty
MAV-Vis
User Study
Conclusion
Design Uncertainty
“The reality of today’s software systems requires us to consider
uncertainty as a first-class concern in the design, implementation,
and deployment of those systems.” [Garlan, 2010]
Our focus:
Uncertainty about design decisions – the contents of a model.
Our agenda:
Enable MBSE with design uncertainty using Partial models.
2 / 27](https://image.slidesharecdn.com/slides-130519134348-phpapp02/85/MAV-Vis-A-Notation-for-Model-Uncertainty-3-320.jpg)
![MAV-Vis: A
Notation for
Model
Uncertainty
M.Famelis,
S.Santosa
Introduction
Design
Uncertainty
MAV-Vis
User Study
Conclusion
Usable Notation for Uncertainty
Partial models: effective for automated reasoning.
However:
Existing notation (“MAV-Text”) developed ad-hoc.
Is it effective for human communication?
Need to maximize cognitive effectiveness.
“Physics of Notations” [Moody, 2009]
Principles for designing graphical languages.
What we did:
• Developed a new notation: MAV-Vis
• Evaluated our implementation with a user study:
“Does MAV-Vis improve cognitive effectiveness?”
Speed, ease, accuracy for reading and writing.
3 / 27](https://image.slidesharecdn.com/slides-130519134348-phpapp02/85/MAV-Vis-A-Notation-for-Model-Uncertainty-4-320.jpg)
![MAV-Vis: A
Notation for
Model
Uncertainty
M.Famelis,
S.Santosa
Introduction
Design
Uncertainty
MAV-Vis
User Study
Conclusion
“Extended” Partial Models
Expressing dependencies between points of uncertainty.
e.g. May Model: variant presented in [ICSE’12]
alternative refinements and relations between them.
8 / 27](https://image.slidesharecdn.com/slides-130519134348-phpapp02/85/MAV-Vis-A-Notation-for-Model-Uncertainty-18-320.jpg)
![MAV-Vis: A
Notation for
Model
Uncertainty
M.Famelis,
S.Santosa
Introduction
Design
Uncertainty
MAV-Vis
User Study
Conclusion
Assessing MAV-Text
MAV-Text: Textual annotations + Propositional formula
Assessment (based on [Moody, 2009]):
Pros:
• Graphically economical
• 1:1 symbol-concept correspondence
Cons:
• Hard to visually distinguish symbols
• Does not intuitively suggest meaning
• Must annotate each element individually
• Does not take advantage of graphics
• Dependencies “hidden” in formula
• Must know how to read propositional formulas
10 / 27](https://image.slidesharecdn.com/slides-130519134348-phpapp02/85/MAV-Vis-A-Notation-for-Model-Uncertainty-20-320.jpg)
![MAV-Vis: A
Notation for
Model
Uncertainty
M.Famelis,
S.Santosa
Introduction
Design
Uncertainty
MAV-Vis
User Study
Conclusion
Improvements in MAV-Vis
Based on the principles in [Moody, 2009]:
• 1:1 symbol-concept correspondence
• Different retinal vars for each symbol (shape, texture)
• Notation (more) suggestive of concepts
• Relationships are visualized
• Grouping of annotations (not per-element)
• Visually expressive (using shape, color, texture, size)
• Dual coding with text and color
• No need to know propositional logic (per se)
• Relatively economical, cognitively manageable
13 / 27](https://image.slidesharecdn.com/slides-130519134348-phpapp02/85/MAV-Vis-A-Notation-for-Model-Uncertainty-36-320.jpg)
![MAV-Vis: A
Notation for
Model
Uncertainty
M.Famelis,
S.Santosa
Introduction
Design
Uncertainty
MAV-Vis
User Study
Conclusion
Overview
Motivation Evaluate MAV-Vis as an implementation of the
theory in [Moody, 2009].
Goal User study to confirm or refute the hypothesis:
“MAV-Vis improves cognitive effectiveness for
reading and writing compared to MAV-Text”
Cognitive Effectiveness:
Ease, Speed, Accuracy
Participants 12 unpaid particip., with Bach. in CS or higher
Average experience in MAVO: 2.2/5
(3 participants were experts in MAVO,
already familiar with MAV-Text)
16 / 27](https://image.slidesharecdn.com/slides-130519134348-phpapp02/85/MAV-Vis-A-Notation-for-Model-Uncertainty-39-320.jpg)
![MAV-Vis: A
Notation for
Model
Uncertainty
M.Famelis,
S.Santosa
Introduction
Design
Uncertainty
MAV-Vis
User Study
Conclusion
Setup
Procedure:
• Tutorial
• Freeform excercise
• [ Reading, Writing ]x2
• Questionnaire
Design:
• Within subjects to allow comparison and minimize
selection bias
• 2x2 Latin square to control for:
• Order of syntaxes (MAV-Vis, MAV-Text)
• Modeling scenario
(“Hotel Admin” in UML, “School Personnel” in E-R)
Measurements:
Ease Questionnaire responses
Speed Task completion time
Accuracy Error counts and comprehension scores
17 / 27](https://image.slidesharecdn.com/slides-130519134348-phpapp02/85/MAV-Vis-A-Notation-for-Model-Uncertainty-40-320.jpg)
![MAV-Vis: A
Notation for
Model
Uncertainty
M.Famelis,
S.Santosa
Introduction
Design
Uncertainty
MAV-Vis
User Study
Conclusion
Summary
Partial models:
Formalism for management of design uncertainty.
Existing work:
Automated reasoning vs human communication
Ad-hoc notation (MAV-Text)
What we did:
Developed MAV-Vis, using [Moody, 2009]
Performed user study to evaluate our implementation
Cognitive Effectiveness for reading and writing:
Ease, Speed, Accuracy
Results:
MAV-Vis more efficient but more writing errors.
23 / 27](https://image.slidesharecdn.com/slides-130519134348-phpapp02/85/MAV-Vis-A-Notation-for-Model-Uncertainty-46-320.jpg)
MAV-Vis: A Notation for Model Uncertainty
- 1. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion MAV-Vis: A Notation for Model Uncertainty Michalis Famelis and Stephanie Santosa University of Toronto May 18th, 2013 Models in Software Engineering Workshop at ICSE 1 / 27
- 2. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Design Uncertainty “The reality of today’s software systems requires us to consider uncertainty as a first-class concern in the design, implementation, and deployment of those systems.” [Garlan, 2010] Our focus: Uncertainty about design decisions – the contents of a model. Our agenda: Enable MBSE with design uncertainty using Partial models. 2 / 27
- 3. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Design Uncertainty “The reality of today’s software systems requires us to consider uncertainty as a first-class concern in the design, implementation, and deployment of those systems.” [Garlan, 2010] Our focus: Uncertainty about design decisions – the contents of a model. Our agenda: Enable MBSE with design uncertainty using Partial models. 2 / 27
- 4. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Usable Notation for Uncertainty Partial models: effective for automated reasoning. However: Existing notation (“MAV-Text”) developed ad-hoc. Is it effective for human communication? Need to maximize cognitive effectiveness. “Physics of Notations” [Moody, 2009] Principles for designing graphical languages. What we did: • Developed a new notation: MAV-Vis • Evaluated our implementation with a user study: “Does MAV-Vis improve cognitive effectiveness?” Speed, ease, accuracy for reading and writing. 3 / 27
- 5. 1 Introduction 2 Design Uncertainty 3 Designing MAV-Vis 4 User Study 5 Conclusion
- 6. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion What is Design Uncertainty Example: a simple class diagram. What does the modeler know? 5 / 27
- 7. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion What is Design Uncertainty Example: a simple class diagram. What does the modeler not know? 5 / 27
- 8. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion What is Design Uncertainty Example: a simple class diagram. What does the modeler not know? 5 / 27
- 9. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion What is Design Uncertainty Example: a simple class diagram. What does the modeler not know? 5 / 27
- 10. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion What is Design Uncertainty Example: a simple class diagram. What does the modeler not know? 5 / 27
- 11. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Uncertainty: a Set of Possible Refinements If we remove all uncertainty, we have a concrete refinement. 6 / 27
- 12. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Modeling Uncertainty with Partial Models Explicating uncertainty in a partial model. 7 / 27
- 13. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Modeling Uncertainty with Partial Models Explicating uncertainty in a partial model. • May: Element is optional. 7 / 27
- 14. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Modeling Uncertainty with Partial Models Explicating uncertainty in a partial model. • May: Element is optional. • Abs: Element can be multiplied to many copies. 7 / 27
- 15. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Modeling Uncertainty with Partial Models Explicating uncertainty in a partial model. • May: Element is optional. • Abs: Element can be multiplied to many copies. • Var: Element can be merged with others. 7 / 27
- 16. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Modeling Uncertainty with Partial Models Explicating uncertainty in a partial model. • May: Element is optional. • Abs: Element can be multiplied to many copies. • Var: Element can be merged with others. • OW: Model is incomplete. 7 / 27
- 17. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Modeling Uncertainty with Partial Models Explicating uncertainty in a partial model. • May: Element is optional. • Abs: Element can be multiplied to many copies. • Var: Element can be merged with others. • OW: Model is incomplete. 7 / 27
- 18. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion “Extended” Partial Models Expressing dependencies between points of uncertainty. e.g. May Model: variant presented in [ICSE’12] alternative refinements and relations between them. 8 / 27
- 19. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Existing Notation: MAV-Text 9 / 27
- 20. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Assessing MAV-Text MAV-Text: Textual annotations + Propositional formula Assessment (based on [Moody, 2009]): Pros: • Graphically economical • 1:1 symbol-concept correspondence Cons: • Hard to visually distinguish symbols • Does not intuitively suggest meaning • Must annotate each element individually • Does not take advantage of graphics • Dependencies “hidden” in formula • Must know how to read propositional formulas 10 / 27
- 21. 1 Introduction 2 Design Uncertainty 3 Designing MAV-Vis 4 User Study 5 Conclusion
- 22. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis 12 / 27
- 23. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing Var 12 / 27
- 24. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing Abs 12 / 27
- 25. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing May: a color for each PoU 12 / 27
- 26. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing May: identify alternative 12 / 27
- 27. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing May: grouping elements in alternatives 12 / 27
- 28. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing May: the other alternative 12 / 27
- 29. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing May: numbers for different alternatives 12 / 27
- 30. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing May: alternative with many parts 12 / 27
- 31. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing May: a different PoU 12 / 27
- 32. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis Representing May: expressing PoU dependencies 12 / 27
- 33. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis 12 / 27
- 34. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis 12 / 27
- 35. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Introducing MAV-Vis 12 / 27
- 36. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Improvements in MAV-Vis Based on the principles in [Moody, 2009]: • 1:1 symbol-concept correspondence • Different retinal vars for each symbol (shape, texture) • Notation (more) suggestive of concepts • Relationships are visualized • Grouping of annotations (not per-element) • Visually expressive (using shape, color, texture, size) • Dual coding with text and color • No need to know propositional logic (per se) • Relatively economical, cognitively manageable 13 / 27
- 37. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion MAV-Vis Limitations Portability Annotation language → cannot guarrantee symbols won’t conflict! Implemented for Class Diagrams, E-R Diagrams. Porting to other notations not automatable. BUT: can use with any abstract syntax (MOF) Less powerful than propositional logic (of course) But dependency sub-language can be extended. No OW OW annotates entire model. (Megamodeling?) No tooling Out of scope here: focus on ideal notation. 14 / 27
- 38. 1 Introduction 2 Design Uncertainty 3 Designing MAV-Vis 4 User Study 5 Conclusion
- 39. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Overview Motivation Evaluate MAV-Vis as an implementation of the theory in [Moody, 2009]. Goal User study to confirm or refute the hypothesis: “MAV-Vis improves cognitive effectiveness for reading and writing compared to MAV-Text” Cognitive Effectiveness: Ease, Speed, Accuracy Participants 12 unpaid particip., with Bach. in CS or higher Average experience in MAVO: 2.2/5 (3 participants were experts in MAVO, already familiar with MAV-Text) 16 / 27
- 40. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Setup Procedure: • Tutorial • Freeform excercise • [ Reading, Writing ]x2 • Questionnaire Design: • Within subjects to allow comparison and minimize selection bias • 2x2 Latin square to control for: • Order of syntaxes (MAV-Vis, MAV-Text) • Modeling scenario (“Hotel Admin” in UML, “School Personnel” in E-R) Measurements: Ease Questionnaire responses Speed Task completion time Accuracy Error counts and comprehension scores 17 / 27
- 41. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Results Ease: • MAV-Vis considered more intuitive, easier to remember, efficient to read • MAV-Text and MAV-Vis almost tied for writing efficiency • MAV-Vis gathered more preferences Speed: • MAV-Text took 2:08min longer for reading. • MAV-Vis was 13sec slower for writing. Accuracy: • No difference in reading Abs, Var. • Noticeable difference for reading May (on average 1.4 fewer errors) • On average 0.7 more errors writing with MAV-Vis. (most errors were about coloring PoUs) 18 / 27
- 42. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Result Tables 19 / 27
- 43. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Threats to Validity • Small sample size: no statistics Results to be understood as preliminary evidence • Prior exposure to MAVO • Familiarity with propositional logic • Confusion regarding uncertainty concepts (MAVO) (But both syntaxes affected equally) • Selection bias from imbalanced knowledge of UML/E-R (Reported by 1 subject) 20 / 27
- 44. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Discussion Overall: MAV-Vis more efficient but more writing errors. Most writing errors: PoU colors. (PoU not a formal concept) However: Not necessarily universal solution! Cognitive Fit Learning Style and Expertise Freeform: dashes, piles question marks, ellipses 21 / 27
- 45. 1 Introduction 2 Design Uncertainty 3 Designing MAV-Vis 4 User Study 5 Conclusion
- 46. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Summary Partial models: Formalism for management of design uncertainty. Existing work: Automated reasoning vs human communication Ad-hoc notation (MAV-Text) What we did: Developed MAV-Vis, using [Moody, 2009] Performed user study to evaluate our implementation Cognitive Effectiveness for reading and writing: Ease, Speed, Accuracy Results: MAV-Vis more efficient but more writing errors. 23 / 27
- 47. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Next Steps Larger picture: Notation only one aspect. Effective methodological support for Design Uncertainty. Patterns of uncertainty? Sources? How do users already cope with Design Uncertainty? Focused on notation design, not tooling. Future: Tooling integration and impact on cognitive effectiveness “MAVO-isation” of arbitrary languages Dependency sublanguage: What are reasonable extensions? Relation to patterns. Impact on tooling. 24 / 27
- 48. Questions?
- 49. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Bibliography I Famelis, M., Ben-David, S., Chechik, M., and Salay, R. (2011). “Partial Models: A Position Paper”. In Proceedings of MoDeVVa’11, pages 1–6. Famelis, M., Chechik, M., and Salay, R. (2012a). “Partial Models: Towards Modeling and Reasoning with Uncertainty”. In Proceedings of ICSE’12. Famelis, M., Salay, R., and Chechik, M. (2012b). The semantics of partial model transformations. In MISE at ICSE’12, pages 64 –69. Garlan, D. (2010). Software engineering in an uncertain world. In FoSER ’10, pages 125–128. Moody, D. (2009). The “Physics” of Notations: Toward a Scientific Basis for Constructing Visual Notations in Software Engineering. TSE, 35(6):756–779. Saadatpanah, P., Famelis, M., Gorzny, J., Robinson, N., Chechik, M., and Salay, R. (2012). Comparing the effectiveness of reasoning formalisms for partial models. In MoDeVVa’12. Salay, R., Chechik, M., and Gorzny, J. (2012a). “Towards a Methodology for Verifying Partial Model Refinements”. In Proceedings of VOLT’12. 26 / 27
- 50. MAV-Vis: A Notation for Model Uncertainty M.Famelis, S.Santosa Introduction Design Uncertainty MAV-Vis User Study Conclusion Bibliography II Salay, R., Famelis, M., and Chechik, M. (2012b). “Language Independent Refinement using Partial Modeling”. In Proceedings of FASE’12. 27 / 27