![VR Environment
Open-source tools
oTilt Brush export tool: https://github.com/MrMMu/tiltbrushfbxexport
oUnity script: https://github.com/Murtada100/MTAP_Data_and_Experiment
{‘fbxname’:FBXNAME,
‘fbxmeta’:
[{‘meshname’:MESHNAME,
‘meshmeta’:
{‘brush_name’:BRUSHNAME,
‘brush_guid’:BRUSH_GUID,
‘v’:V, #list of positions (3-tuples)
‘n’:N, #list of normals (3-tuples, or None if missing)
‘uv0’:UV0, #list of uv0 (2-, 3-, 4-tuples, or None if
missing)
‘uv1’:UV1, #see uv0
‘c’:C, #list of colors, as a uint32. abgr little-endian, rgba
big-endian
‘t’:T, #list of tangents (4-tuples, or None if missing)
‘tri’:TRI #list of triangles (3-tuples of ints)
}
},{},…]
}](https://image.slidesharecdn.com/talk-220706100908-53ca9b2b/85/Grab-your-attention-with-AI-Next-steps-for-VR-12-320.jpg)
![Eye orientation
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in
Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]](https://image.slidesharecdn.com/talk-220706100908-53ca9b2b/85/Grab-your-attention-with-AI-Next-steps-for-VR-18-320.jpg)
![Eye orientation
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in
Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]](https://image.slidesharecdn.com/talk-220706100908-53ca9b2b/85/Grab-your-attention-with-AI-Next-steps-for-VR-19-320.jpg)
![Gender
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in
Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]](https://image.slidesharecdn.com/talk-220706100908-53ca9b2b/85/Grab-your-attention-with-AI-Next-steps-for-VR-20-320.jpg)
![Neural Network
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in Springer Multimedia Tools and
Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]](https://image.slidesharecdn.com/talk-220706100908-53ca9b2b/85/Grab-your-attention-with-AI-Next-steps-for-VR-21-320.jpg)
![Experiment
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in Springer Multimedia Tools and
Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]](https://image.slidesharecdn.com/talk-220706100908-53ca9b2b/85/Grab-your-attention-with-AI-Next-steps-for-VR-22-320.jpg)
Grab your attention with AI - Next steps for VR
- 1. Dr Mu Mu, 6 July 2022 Grab your attention with AI Next steps for VR Murtada Dohan (PhD Comp), Cleyon Johns (Game Design) Yoana Slavova (Business Comp), Petar Stepanić (Comp)
- 2. • Human Computer Interaction (HCI) • multidisciplinary field with a focus on the interaction between humans (the users) and computer systems. • Objective video quality assessment – assess user experience using machine learning • Better networking, better video streaming standard. • Video -> Immersive media -> virtual reality • Machine learning -> deep learning / neural networks Background
- 3. •VR journey ( learning, art, AI navigation, mental health… ) •The role of human attention in VR designs
- 4. VR takes a front seat • Education, training, entertainment, advertisement, creative art, healthcare.
- 5. • Motivation: Does “WOW!” translate to an A? • how VR could impact the learning of science in universities. • Comparative study • Stonehenge VR vs PowerPoint • Multiple choice test (recognising numerical, textual, visual info) and interviews • 50 students (age 18-26, 60% female) • Results, observations, etc. • VR not always better (some contributing factors). • More work on peer interactions and interactive tools in VR. A Comparative Study of the Learning Outcomes and Experience of VR in Education - Yoana Slavova • Slavova, Y. and Mu, M., A Comparative Study of the Learning Outcomes and Experience of VR in Education, in Proceedings of the 25th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2018), Germany, 05/2018 • Panel, Westminster HE Forum – Technologies in higher education, https://www.westminsterforumprojects.co.uk/publication/technology-in- higher-education-19
- 6. A Comparative Study of the Learning Outcomes and Experience of VR in Education - Yoana Slavova
- 7. Attention is the answer.
- 8. attention visual audio haptic olfactory, taste movement Human attention “Everyone knows what attention is. It is the taking possession by the mind, in clear, and vivid form, of one out of what seems several simultaneously possible objects or trains of thought.” - William James (experimental psychology) -
- 9. • We designed a gaze-controlled Unity VR game for this study and implemented additional libraries to bridge raw eye-tracking data with game elements and mechanics. • The experimental data show distinctive patterns of fixation spans which are paired with user interviews to help us explore characteristics of user attention. Understanding User Attention In VR Using Gaze Controlled Games – Murtada Dohan (MSc Dissertation) Dohan, M. and Mu, M., Understanding User Attention In VR Using Gaze Controlled Games. In Proceedings of ACM International Conference on Interactive Experiences for TV and Online Video (ACM TVX ’19), June 5–7, 2019, Salford (Manchester), United Kingdom. https://doi.org/10.1145/3317697.3325118 06/2019
- 10. • This work indicates future avenues for content creation in this emerging field and what this might mean for artists and art institutions to experiment with methods of exhibiting innovative content. Abstract Painting Practice: Expanding in a Virtual World Goodyear, A. and Mu, M., Abstract Painting Practice: Expanding in a Virtual World. In Proceedings of ACM International Conference on Interactive Experiences for TV and Online Video (ACM TVX ’19), June 5–7, 2019, Salford (Manchester), United Kingdom. 06/2019
- 11. VR Environment Open-source tools oTilt Brush export tool: https://github.com/MrMMu/tiltbrushfbxexport oUnity script: https://github.com/Murtada100/MTAP_Data_and_Experiment
- 12. VR Environment Open-source tools oTilt Brush export tool: https://github.com/MrMMu/tiltbrushfbxexport oUnity script: https://github.com/Murtada100/MTAP_Data_and_Experiment {‘fbxname’:FBXNAME, ‘fbxmeta’: [{‘meshname’:MESHNAME, ‘meshmeta’: {‘brush_name’:BRUSHNAME, ‘brush_guid’:BRUSH_GUID, ‘v’:V, #list of positions (3-tuples) ‘n’:N, #list of normals (3-tuples, or None if missing) ‘uv0’:UV0, #list of uv0 (2-, 3-, 4-tuples, or None if missing) ‘uv1’:UV1, #see uv0 ‘c’:C, #list of colors, as a uint32. abgr little-endian, rgba big-endian ‘t’:T, #list of tangents (4-tuples, or None if missing) ‘tri’:TRI #list of triangles (3-tuples of ints) } },{},…] }
- 13. Experiment ● The experiment involved participants freely navigating through an VR artwork ● It consists of thousands of different brushstrokes ● HTC VIVE PRO EYE ● Data collected: Walk data, and eye gaze data, hand tracking, verbal response, environment data. Dataset: https://doi.org/10.6084/m9.figshare.15090309.v1
- 14. Experiment
- 15. Experiment
- 16. Walk
- 17. Eye orientation Eye Tracking Coordinate System (ETCS). The ETCS is a cartesian, right handed system and the coordinate axes are oriented
- 18. Eye orientation Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]
- 19. Eye orientation Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]
- 20. Gender Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]
- 21. Neural Network Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]
- 22. Experiment Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]
- 23. Experiment Participant 5170 commented that he loved the experiences of “going into the paint and looking up”, which really gave him “a sense of scale”. He preferred walking to explore the artwork more than the idea of “flying” using VR controllers because “tethering to the ground” gave him “a point of reference”. Participant 4475 felt that she was “in a virtual jungle” and “loved the lights shining through the paint”. She also suggested “more pink trees” and “tropical music” in the background. Participant 1157 “felt a bit lost in the painting...”. The feedback reflects her walking patterns shown in Figure 8(b) where she spent little time adventuring inside the artwork. Participant 4646 “really liked a glowing triangular area in the middle of nowhere” because it reminded her of a song. She also suggested “swimming in the paint” as a better navigation method compared to walking and flying.
- 25. AI-assisted navigation manual Community activities Director’s cut AI model automated Navigation recommend recommend recommend recommend recommend
- 27. Recurrent Neural Network https://colah.github.io/posts/2015-08-Understanding-LSTMs/
- 28. AI Navigation
- 29. AI Navigation • Implemented based on LSTM model. • Integrated to Unity • Collects walking data in real-time to predict the best destination in VR. • Navigator was embedded using the most common navigation techniques: Avatars and Arrows. • We have used 3 avatars, 2 arrows for showing top(3) and top (2) predictions. • 15 participants joined the experiment. Dataset: https://github.com/Murtada100/EyeVRDataset
- 31. AI Navigation • ML recommender can recognise mobility pattern. • The model is embedded in the VR environment. • We have employed two different techniques to illustrate these recommendations to the participants via directed arrows and Avatars. • Implicit recommendation (Live-avatars) are less distracted than explicit recommendation (Directed-arrows).
- 33. Autonomous VR psychiatry for the early detection and treatment of mental illness
- 34. Fix social anxiety disorder? - Petar Johan Stepanić (BSc dissertation)
- 35. Conclusions Attention is the answer. Machine learning is the way. Mu Mu, Technology, FAST mu.mu@northampton.ac.uk Murtada Dohan (PhD Comp), Cleyon Johns (Game) Yoana Slavova (Business Comp), Petar Stepanić (Comp)
Editor's Notes
- #28: https://colah.github.io/posts/2015-08-Understanding-LSTMs/