Multi-Paradigm Modeling of HMI Applications
2 likes1,284 views
The document discusses the challenges of using Model-Driven Engineering (MDE) for Human-Machine Interfaces (HMI) and outlines multi-paradigm modeling approaches. It emphasizes research challenges such as the development of domain-specific modeling languages, enabling formal reasoning over HMI models, and combining various modeling paradigms for effective UI design. Additionally, it highlights examples of modeling techniques and the need for formal analysis to detect vulnerabilities and improve system interactions.
![Challenges in using MDE for HMI
HMI modeling
Behavioural modeling
Example : Statecharts model of hand behaviour [DM11]
Tom Mens – UMONS 6 / 23](https://image.slidesharecdn.com/kistates-antwerp-111228073907-phpapp02/85/Multi-Paradigm-Modeling-of-HMI-Applications-6-320.jpg)
![Challenges in using MDE for HMI
HMI modeling
Behavioural modeling
Example : Statecharts mode of UI component behaviour [DM11]
Tom Mens – UMONS 7 / 23](https://image.slidesharecdn.com/kistates-antwerp-111228073907-phpapp02/85/Multi-Paradigm-Modeling-of-HMI-Applications-7-320.jpg)
![Challenges in using MDE for HMI
HMI modeling
Behavioural modeling
Example : Statecharts model of graphical scene behaviour [DM11]
Tom Mens – UMONS 8 / 23](https://image.slidesharecdn.com/kistates-antwerp-111228073907-phpapp02/85/Multi-Paradigm-Modeling-of-HMI-Applications-8-320.jpg)
![Challenges in using MDE for HMI
HMI modeling
Behavioural modeling
Example : Petri net model of double click behaviour
(Palanque - Toulouse) [NPLB09]
Tom Mens – UMONS 9 / 23](https://image.slidesharecdn.com/kistates-antwerp-111228073907-phpapp02/85/Multi-Paradigm-Modeling-of-HMI-Applications-9-320.jpg)
![Challenges in using MDE for HMI
Research Challenges
Formal reasoning - Example
Work by Combéfis and Pecheur to analyse HMI vulnerabilities
[CP09]
Using labeled-transition systems (LTS) semantics to generate and
analyse users’ mental models
Example : LTS of user’s mental model of a lamp, compared to the
real system model.
Tom Mens – UMONS 13 / 23](https://image.slidesharecdn.com/kistates-antwerp-111228073907-phpapp02/85/Multi-Paradigm-Modeling-of-HMI-Applications-13-320.jpg)
![Challenges in using MDE for HMI
Research Challenges
Formal reasoning - Example
Work by Combéfis and Pecheur to analyse HMI vulnerabilities
Example : accident with Therac-25 medical system due to a
incorrect operator-machine interaction
Fig. 7. Statechart model of the Therac-25 medical system [16]. The enter command is represented with and the up command is represented with ↑.
Tom Mens – UMONS 14 / 23](https://image.slidesharecdn.com/kistates-antwerp-111228073907-phpapp02/85/Multi-Paradigm-Modeling-of-HMI-Applications-14-320.jpg)
Multi-Paradigm Modeling of HMI Applications
- 1. Challenges in using MDE for HMI Multi-Paradigm Modeling of HMI Applications Tom Mens Romuald Deshayes Software Engineering Lab MoVES event, Universiteit Antwerpen, 20 December 2011 Tom Mens – UMONS 1 / 23
- 2. Challenges in using MDE for HMI HMI modeling Outline : HMI modeling HMI modeling Tom Mens – UMONS 2 / 23
- 3. Challenges in using MDE for HMI HMI modeling Example of a HMI application 3D visual drawing tool Haptic (gestural) user interface (using hands) based on Kinect Creation and manipulation of 3D objects, using OpenGL Complex UI behaviour represented using statecharts www.youtube.com/ watch?v=lVcqzWTnpMY Tom Mens – UMONS 3 / 23
- 4. Challenges in using MDE for HMI HMI modeling Multi-Paradigm Modeling for HMI Goal Use MDE techniques to advance the state-of-the-art in HMI (HCI) Multi-paradigm modeling. Existing approaches focus on : Task modeling : understand user tasks to improve UIs (cf. Jean Vanderdonckt - UCL) Presentation modeling : focus on visual appearance of the application (e.g., OpenGL) User modeling : focus on the user characteristics Behavioural modeling : model how the application reacts to users actions (e.g. event-based) Physical modeling : physical behaviour of real or virtual objects ... Tom Mens – UMONS 4 / 23
- 5. Challenges in using MDE for HMI HMI modeling Task modeling Example : ConcurTaskTrees task model for a digital home Tom Mens – UMONS 5 / 23
- 6. Challenges in using MDE for HMI HMI modeling Behavioural modeling Example : Statecharts model of hand behaviour [DM11] Tom Mens – UMONS 6 / 23
- 7. Challenges in using MDE for HMI HMI modeling Behavioural modeling Example : Statecharts mode of UI component behaviour [DM11] Tom Mens – UMONS 7 / 23
- 8. Challenges in using MDE for HMI HMI modeling Behavioural modeling Example : Statecharts model of graphical scene behaviour [DM11] Tom Mens – UMONS 8 / 23
- 9. Challenges in using MDE for HMI HMI modeling Behavioural modeling Example : Petri net model of double click behaviour (Palanque - Toulouse) [NPLB09] Tom Mens – UMONS 9 / 23
- 10. Challenges in using MDE for HMI HMI modeling Physical modeling Goal Modeling physical behaviour or interaction of virtual objects E.g., falling, rolling, colliding, bouncing, breaking, . . . Requires continuous models (e.g., differential equations) Challenge Combining discrete and continuous models Possible solution : hybrid automata Discrete automata containing differential equations in the states Tom Mens – UMONS 10 / 23
- 11. Challenges in using MDE for HMI Research Challenges Outline : Research Challenges Research Challenges Tom Mens – UMONS 11 / 23
- 12. Challenges in using MDE for HMI Research Challenges Research Challenges Top-down Come up with domain-specific modeling languages (DSML) that combine multiple views on HMI modeling. Bottom-up Enable formal reasoning over HMI models Detect unintended behavioural problems (nondeterminism, non-termination, ...) Detect HMI vulnerabilities, inconsistencies, incompleteness Facilitate code generation ... Tom Mens – UMONS 12 / 23
- 13. Challenges in using MDE for HMI Research Challenges Formal reasoning - Example Work by Combéfis and Pecheur to analyse HMI vulnerabilities [CP09] Using labeled-transition systems (LTS) semantics to generate and analyse users’ mental models Example : LTS of user’s mental model of a lamp, compared to the real system model. Tom Mens – UMONS 13 / 23
- 14. Challenges in using MDE for HMI Research Challenges Formal reasoning - Example Work by Combéfis and Pecheur to analyse HMI vulnerabilities Example : accident with Therac-25 medical system due to a incorrect operator-machine interaction Fig. 7. Statechart model of the Therac-25 medical system [16]. The enter command is represented with and the up command is represented with ↑. Tom Mens – UMONS 14 / 23
- 15. Challenges in using MDE for HMI Research Challenges Domain-Specific Modeling for HMI DSML How to define a DSML that combines the multiple modeling languages/paradigms used for HMI ? Tom Mens – UMONS 15 / 23
- 16. Challenges in using MDE for HMI Research Challenges Domain-Specific Modeling for HMI Domain-specific model : specific to application domain E.g., robotics, gaming, home automation, virtual reality Platform model : describes platform specificities modality (vocal, tactile, WIMP) input devices (hardware sensors, keyboard, mouse, Kinect) output (screen, VR goggles, voice) ... Transformation model : weaves both models into platform-specific model, using model transformation technology Tom Mens – UMONS 16 / 23
- 17. Challenges in using MDE for HMI Research Challenges Visual formalisms Are used for formal reasoning and analysis over the models Questions : Which formalisms to use for which purpose ? How to combine results from different formalisms ? How to feed back results into DSML ? Remark : model transformations are models too, amenable to formal analysis Tom Mens – UMONS 17 / 23
- 18. Challenges in using MDE for HMI Research Challenges Visual Formalisms Use most appropriate formalism, at most suitable abstraction level, for a given purpose Possible candidates Statecharts : for event-driven, deterministic, reactive systems Petri nets : for analysing concurrency-related issues Labelled Transition Systems Hybrid automata (combining discrete and continuous models) Logic-based or constraint-based formalisms Graph transformation (e.g., for model transformation) Tom Mens – UMONS 18 / 23
- 19. Challenges in using MDE for HMI Research Challenges Visual Formalisms Which semantics to associate to each formalism ? Which tools to use for formal reasoning ? Graph transformation tools (e.g., AGG, Fujaba, GrGen) Model manipulation tools (ATL, EMF, Epsilon, AtoMPM, TCore, . . . ) HyTech tool (for hybrid automata) Model checkers SAT solvers ... Tom Mens – UMONS 19 / 23
- 20. Challenges in using MDE for HMI Research Challenges More challenges How to facilitate evolution ? How to generate code from models ? How to develop generic framework (independent of particular DSM or platform model) ? Where and how to use graph transformation ? Killer app ? Augmented reality, gaming, home automation, robotics ? Tom Mens – UMONS 20 / 23
- 21. Challenges in using MDE for HMI Research Challenges Thank you Thank you for your attention ! Questions, suggestions ? Tom Mens – UMONS 21 / 23
- 22. Challenges in using MDE for HMI Research Challenges Bibliography I Sébastien Combéfis and Charles Pecheur. A bisimulation-based approach to the analysis of human-computer interaction. In EICS, pages 101–110, 2009. Romuald Deshayes and Tom Mens. Statechart modelling of interactive gesture-based applications. In CoDeisMoto Workshop, co-located with Interact 2011, 2011. Tom Mens – UMONS 22 / 23
- 23. Challenges in using MDE for HMI Research Challenges Bibliography II David Navarre, Philippe A. Palanque, Jean-François Ladry, and Eric Barboni. Icos : A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scalability. ACM Trans. Comput.-Hum. Interact., 16(4) :56, 2009. Tom Mens – UMONS 23 / 23