Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation
Download as PPTX, PDF0 likes551 views
The document discusses a hidden Markov model (HMM)-based algorithm for fusing multimodal inputs in interactive systems, addressing challenges in real-time user-induced adaptation. It evaluates the algorithm through qualitative tests and benchmarks, demonstrating its superior performance and ability to handle ambiguous input scenarios. The HMM-based approach integrates into the Hephaistk framework, offering improved recognition rates over existing methods.
Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation
- 1. 2 December 2005 Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation Bruno Dumas1, Beat Signer1 and Denis Lalanne2 1 WISE Research Lab, Vrije Universiteit Brussel, Belgium 2 DIVA Research Group, University of Fribourg, Switzerland
- 2. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 2 June 26, 2012 Outline • Fusion of multimodal input • Multimodal interaction: from design to implementation • Algorithms for fusion of multimodal input • A Hidden Markov Model-based fusion algorithm • Instantiation of the algorithm • Evaluation of the algorithm • Qualitative test • Quantitative evaluation • Performance assessment • Conclusion
- 3. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 3 June 26, 2012 Fusion of Multimodal Input Challenges in multimodal interaction engineering: continous and probabilistic inputs, real-time…
- 4. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 4 June 26, 2012 Challenging/Ambiguous Fusion Cases • “Put-that-there” Complementarity case • “Play next track” Different meanings based on order and context
- 5. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 5 June 26, 2012 CARE Properties at the Design Level Describe how modalities can be combined Complementarity All modalities are necessary Assignment Absence of choice Redundancy Same expressive power between modalities Equivalence Any modality is sufficient
- 6. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 6 June 26, 2012 HephaisTK Multimodal Framework
- 7. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 7 June 26, 2012 Description of Multimodal Dialogues • Syntactic-level description • For more details: check SMUIML language
- 8. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 8 June 26, 2012 Fusion Algorithms • Frame-based algorithms & Unification- based algorithms • Symbolic approaches • Both are a de facto standard • Symbolic-statistical algorithms
- 9. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 9 June 26, 2012 Symbolic-Statistical Approaches Symbolic data Interpretation of low-level data Statistical processing Machine learning algorithms
- 10. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 10 June 26, 2012 Our HMM-Based Fusion Algorithm Why hidden Markov models? Very good modelling of time-related events Takes advantage of input data coming from probabilistic input modalities On-the-fly adaptation of the model - e.g. based on user feedback
- 11. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 11 June 26, 2012 Put-That-There Example
- 12. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 12 June 26, 2012 Put-That-There Example
- 13. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 13 June 26, 2012 Put-That-There Example
- 14. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 14 June 26, 2012 Put-That-There Example
- 15. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 15 June 26, 2012 Put-That-There Example
- 16. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 16 June 26, 2012 Put-That-There Example
- 17. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 17 June 26, 2012 Instantiation of the Algorithm
- 18. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 18 June 26, 2012 Evaluation of the Algorithm Evaluation on three levels Qualitative “gut check” test Benchmarks on real-world examples Performance assessment
- 19. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 19 June 26, 2012 Evaluation: Qualitative Test • Three expert users were asked to assess the behaviour of the algorithm frames-based and HMM-based fusion 5 minutes per condition + interview • Coherent behaviour between the algorithms • No noticeable difference in responsiveness
- 20. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 20 June 26, 2012 Evaluation: Benchmark
- 21. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 21 June 26, 2012 Evaluation: Benchmark
- 22. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 22 June 26, 2012 Benchmark Results (1) • CARE properties: equivalence and redundancy tests
- 23. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 23 June 26, 2012 Benchmark Results (2) • Sequential and non-sequential complementarity tests
- 24. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 24 June 26, 2012 Benchmark Results (3) • “Play next track” example Meaning frames HMM-based
- 25. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 25 June 26, 2012 Evaluation: Performance • Per fusion algorithm: • 5 runs of 40 input pieces • 5 x 20 expected fusion results • Frame-based: 18.2 ms • Standard deviation: 12.7 ms • HMM-based: 16.6 ms • Standard deviation: 11.6 ms
- 26. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 26 June 26, 2012 Conclusion • A new symbolic-statistical multimodal fusion algorithm based on Hidden Markov Models • Integrated into the HephaisTK framework • Superior recognition rates compared to existing algorithms • Handling of ambiguous cases • Processing of real-time user feedback http://wise.vub.ac.be/bruno-dumas