IVE 2024 Short Course - Lecture18- Hacking Emotions in VR Collaboration.
0 likes121 views
This document discusses the exploration of sharing emotional cues in virtual reality collaboration to enhance emotional consensus and engagement among participants. It highlights gaps in existing research on emotional sharing and proposes methods to incorporate facial cues in collaborative tasks, emphasizing their positional relevance for emotional understanding. The document concludes with suggestions for future research, including adaptive environments and multimodal empathy sharing to support emotional interactions in mixed reality settings.
IVE 2024 Short Course - Lecture18- Hacking Emotions in VR Collaboration.
- 1. Hacking Emotions in VR Collaboration Sharing Cues that Facilitate Emotional Consensus Theo Teo Empathic Computing Lab The Psychology XR 19 July 2024
- 2. Existing Works on XR Social Computing
- 3. XR Social Computing since 2012 • Visual cue have been widely used for sharing thoughts and emotions • Audiotory and Tactile feedback are minor but increasing since 2020 • 2X publications in last 2 years # of publications on XR Social Computing Three major modalities used in XR Social Computing
- 4. So.. Is this area well-studied? Not really…
- 7. Human Interaction | Task Collaboration
- 8. Human Interaction | Task Collaboration | Psychology XR
- 9. Existing Gaps • Limited studies on sharing emotions in a collaboration scenario • Emotional consensus • Conflict resolution • Engagement and Communication • Empathy • Cognitive Empathy • Emotional Empathy • Why it happens • How it happens
- 10. Journey towards Hacking Emotions • Virtual Reality collaboration • Sharing and understanding emotions between collaborators • Addresses existing gaps • Technique that is transferable (design implementation)
- 11. Identifying the Research Topic • Sharing emotions were built on the foundation of ‘putting the cue closer to your body or perspective’ • Is there a way to bring this to collaboration scenario where there is always a task to focus (together)?
- 12. Prologue: Bodily Emotional Cues • Sharing emotions • Facial expressions • Facial emojis • Emotional cues / effects • Attaching emotional cues • Own Hand • Grabbing Object
- 13. A Quick Glance Hand Object Gaze
- 14. Is there a way for in-depth perception?
- 15. 3D Face on Grabbing Object 3D Face on Hand
- 16. 2D Face vs 3D Face - A Pilot Test • 2D Face • Less obstructing • More natural (akin to screen display) • Harder to notice • 3D Face • Able to see from wider angle • More suitable for interaction tasks • Emotions were understood better
- 17. Impact of Facial Cue Positions on Understanding Emotions - A User Study • What is the role of position in VR collaboration that concerns emotional understanding? • Three measurements (attachments) • Baseline (No Facial Cue) • Own’s Hand • Own’s Grabbing Object
- 18. The Furniture Negotiation Task • Dyad study • Furniture Arrangement based on task requirements • Instructions were unique to each side (Conflict, Non-conflict)
- 22. RTECM PSY How satisfied are you? How satisfied you think your partner are? 1 - NOT VERY SATISFIED 5 - VERY SATISFIED 2 … 3 … 4 … 1 - NOT VERY SATISFIED 5 - VERY SATISFIED 2 … 3 … 4 …
- 23. CMT MVE PSY This is .. GREAT!!! I preferred A because B and C.. I’m so sorry for breathing the air!!
- 24. CMT MVE PSY This is .. GREAT!!! I preferred A because B and C.. I’m so sorry for breathing the air!! Blink Focus
- 25. CMT MVE PSY This is .. GREAT!!! I preferred A because B and C.. I’m so sorry for breathing the air!! Blink Heart Rate Galvanic Skin Response
- 26. Results • Correlations were found on emotional consensus when the facial cue attaches to hand (MH) • Self-reported emotional consensus is better when there is facial cue A: I feel… B: I think my partner feel… C: My partner feel… D: My parter think I feel…
- 27. Results - Continued Unnatural feeling… A bit scary to me to see other's face on my hand.. Facial cue is not that obvious hand face was upsetting to look at It makes me scare, so I don't pay attention to the avatar I didn't like a face staring me on the object looks a bit... mystical (like Harry Porter) face was a bit creepy Cue on Hand was the easiest way to see my partners emotions having two separate representations of my partner Facial cue on Hand is helpful Seeing the face on the hand was cool. Poke my partner I could easily see my partner’s face at all times when moving objects having facial cue pop up on object reminded me to check in with how my partner was feeling.
- 28. *
- 29. Key takeaways • Attaching facial cues closer to view of interest can influence awareness and expression of emotions between collaborators • There might be also some effects of hand attached facial cue on gaze behaviour too.. • Except… • The 3D face is unnatural, creepy and scary • Task can be built more ‘engaging’
- 30. Improving the Facial Cue from our Findings • Less creepy while maintaining similar or better interpretation of emotions • Task to only include mutual goal but complex enough to require discussion • More variations on attachable positions for the facial cue
- 31. SelfHand
- 32. SelfObject
- 33. PartnerHand
- 34. PartnerObject
- 35. Revising the Study of Facial Cue Positions on Understanding Emotions • What is the role of position in VR collaboration that concerns emotional understanding? • Five measurements (attachments) • Baseline (No Facial Cue) • Own’s Hand • Own’s Grabbing Object • Task Partner’s Hand • Task Partner’s Grabbing Object
- 36. Urban Planning Discussion Task • Dyad study • Discuss to arrange task object to the correct task grid • Instructions were split into half (one for each side) • A red button that randomly appears every 20-25s to increase the work load • Emotion Sync Indicator
- 38. Demo Submitted to SA24 - XR Theo Teo, Allison Jing, Xuan Tien, Gun A. Lee
- 40. So.. next step?
- 42. Adaptive Environments: Create environments that change in real-time based on the group's emotional state. For instance, calming visuals and sounds when stress levels are high. Emotion Recognition: Implement AI systems that analyze facial expressions, body language, and voice tones to assess and respond to participants' emotional states in real-time. Group Biofeedback: Study the impact of displaying aggregated biofeedback data to the group to enhance empathy and understanding of each other’s emotional states.
- 43. On top of that.. • Multimodal empathy / emotional sharing • Visual + Haptic; Audio + Haptic; Haptic + Haptic etc.. • Expand the haptic modality • Thermal, vibrotactile feedback, pressure, actuator and many more.. • Empathy XR that involves 3 or more users • Group emotions vs individual emotion; selective emotion • Further onto hacking emotions • Embodiment, cognitive load, attention etc.
- 44. Summing Up “Build empathy and better understanding through sharing each other's emotional states to enhance collaboration in Mixed Reality” (Ens et al. 2019 survey toward future MR collaboration topics and directions) • Studying the understanding emotions, particularly empathy in immersive collaboration is still an expanding topic • Positioning emotions closer to yourself or your partner could be helpful and useful for understanding emotions
- 45. Hacking Emotions in VR Collaboration Sharing Cues that Facilitate Emotional Consensus Thank you!