![Why?
➔ Lowering barriers to HCI. Eg: gestures
research.
➔ Accessibility [1]
➔ Killer feature
➔ IoT](https://image.slidesharecdn.com/multimodalinteractions-170104225347/85/Multimodal-Interactions-JS-The-What-The-Why-and-The-How-Diego-Paez-Despegar-10-320.jpg)
![Architecture proposed
by Dumas et al [2]
1
2
3](https://image.slidesharecdn.com/multimodalinteractions-170104225347/85/Multimodal-Interactions-JS-The-What-The-Why-and-The-How-Diego-Paez-Despegar-12-320.jpg)
Multimodal Interactions & JS: The What, The Why and The How - Diego Paez, Despegar
- 2. Multimodal Interactions The what, the why and the how Diego Paez, Despegar
- 3. About Me Diego Paez => DEKΔ G E U T co-founder JS developer with a passion for HCI ♥ Currently working at:
- 4. AGENDA ➔ Definitions ➔ Foundations ➔ An alternative approach ➔ Future work
- 5. What is a multimodal interaction (MMI)? #The what
- 7. Senses 5 major senses: sight, hearing, touch, taste and smell. But there are more; eg: Proprioception. You Real World Interaction
- 8. Highway 401 by Nayuki Senses
- 10. Why? ➔ Lowering barriers to HCI. Eg: gestures research. ➔ Accessibility [1] ➔ Killer feature ➔ IoT
- 12. Architecture proposed by Dumas et al [2] 1 2 3
- 14. Plusultra Gyes
- 15. Proposed Architecture Modalities (HW, distributed) Pub/sub pattern used to separate layers Web Clients (distributed) 1 2 3
- 17. Getting started npm install plusultra Docker-compose up mkdir myMultimodalDeviceDriver cd myMultimodalDeviceDriver npm install gyes
- 18. Creating a MM Device Driver var Inherits = require( 'inherits' ); var ModalityChannel = require( 'gyes' ).ModalityDriver; module.exports = MyModality; function MyModality( opts ){ // constructor code... } Inherits(MyModality, ModalityChannel); MyModality.prototype.fission = function( time ){ … }; MyModality.prototype.newEvent = function( data ){ … };
- 19. Inside the app // lets create a new plusultra client plusultraClient = new gyes( app_key, socketURI, options ); plusultraClient.authenticate( app_key ); // create a new modality (this is like metadata) var myMod = new gyes.Modality( modName, 'input', {} ); // instance a driver for the modality var modDriver = new myMultimodalDeviceDriver( options ); myMod.use( modDriver ); // finally add the modality into the system plusultraClient.addModality( app_key, airMod );
- 20. Future Work aka: Help is welcome!
- 21. ➔ Experimenting with WebRTC to enable P2P interactions. ➔ Define Interpretations as components. W3C MMI. React can help? ➔ Improve API. Follow a reactive/functional approach. Cycle/rxjs? Future Work
- 22. Project repos ➔ Plusultra: https://github.com/dpaez/plusultra ➔ Gyes: https://github.com/dpaez/gyes ➔ Demo app (shapes): https://github.com/dpaez/shapes_app ➔ Gitter: https://gitter.im/gyes_mmi/Lobby
- 23. Thanks! Who is the hipster? Follow me @carax Github @dpaez Get the GEUT experience: geutstudio.com / Some cool code @geut