Understanding the Impact of the Reality-Virtuality Continuum on Visual Search using Physiological Measures

Editor's Notes

• #1: Hi everyone I am Francesco and today I am very happy to engage with you in my presentation where we will come through how users can identify relevant information across the MR continuum. This work is a joint effort between LMU Munich, Duisburg Essen University, Lancastar and Alto and I would not be here presenting without the support of my coauthors Uwe, James, Joshua, Changkun, Rulu, Robin and Sven.
• #2: I guess many of you are familiar with Mixed Reality, a term used to describe a spectrum of environments where we interact with everything that falls between reality and virtuality. Following this definition, every MR experience contains physical and virtual components. Here, environments we are presented with a gradient of information that can be progressively virtual. No matter what the nature of such information is, the success of MR interactions depends on how well users can perceive, process, and understand visual information in these complex environments.
• #3: So here is relevant to have a look how previous work investigated information processing in MR. Van Den Oever studied visual search between VR and a Real Environment and found that users were slower in VR, while showing similar accuracy but reporting increased workload.
• #4: In AR only, there are instead contradictory results. Kim et al. (2022) explored visual search tasks in outdoor AR environments. Their work demonstrates that virtual objects are more salient and easier to identify than physical ones, especially in low-light conditions.
• #5: David-John (2021) were able to predict user interaction such as exploratory of target focused visual search using gaze behavior features in VR.
• #7: Here the research questions we want to address in our studies.
• #8: Specifically we first focus on visual search performance and workload.
• #9: Then we focus on the amount of cognitive resources that users need to allocate across the continuum to identify relevant information.
• #10: And lastly, we investigate if the gaze behaviors of users vary across MR conditions.
• #12: We investigate how users identify relevant information and suppress distracting ones across different manifestations of the MR continuum under varying levels of task difficulty. To achieve this, we chose an established paradigm, i.e., visual search task, transferable to different ecological settings situated in MR, such as cognitive training, information retrieval .
• #13: The visual search trial was structured into three phases: Initially, participants were shown a fixation cross for a baseline duration of 1000 ms, followed by an additional, randomly assigned jitter duration of either 250 ms, 750 ms, or 1250 ms. Then, participants had 5000 ms to identify the target among distractors, followed by a 1000 ms interstimulus interval (ISI). Each condition involved 50 trials per participant.
• #14: In the AR conditions, participants execute a visual search task in a real-life 360-video scenario, like if where they are asked to look for virtual objects on a shelf. Virtual objects have typical AR features. Thus, they are see-through, while the surrounding physical environment maintains its natural color and transparency.
• #15: In the AR conditions, participants execute a visual search task in a real-life 360-video scenario, like if where they are asked to look for virtual objects on a shelf. Virtual objects have typical AR features. Thus, they are see-through, while the surrounding physical environment maintains its natural color and transparency.
• #16: In the AV condition, participants perform the visual search task in an AV environment, where there are shared features elements from both AR and VR scenarios, i.e., the floor is virtual and parts of the closet are virtual and physical
• #17: In the AV condition, participants perform the visual search task in an AV environment, where there are shared features elements from both AR and VR scenarios, i.e., the floor is virtual and parts of the closet are virtual and physical
• #18: In the AV condition, participants perform the visual search task in an AV environment, where there are shared features elements from both AR and VR scenarios, i.e., the floor is virtual and parts of the closet are virtual and physical
• #19: In the AV condition, participants perform the visual search task in an AV environment, where there are shared features elements from both AR and VR scenarios, i.e., the floor is virtual and parts of the closet are virtual and physical
• #21: We now gonna proceed presenting the results. We will first describe the results on behavioral results and proceed with our multimodal evaluation.
• #22: accuracy was higher under low perceptual load across all environments, with no significant differences between AR, AV, and VR. However, reaction times were significantly faster in AV and VR compared to AR, particularly under low perceptual load, suggesting that virtual elements facilitate quicker responses in less complex tasks.
• #23: time to last fixation was shorter under low perceptual load, indicating faster target detection. Saccade frequency also decreased, reflecting fewer, more efficient eye movements in when the perceptual load was lower. Lastly, pupil size was largest in AR and decreased from AR to VR, particularly under high perceptual load, signaling reduced cognitive load in virtual environments.
• #24: The Fixation-Related P3 amplitude reflects how the brain allocates attentional resources during visual search, specifically following a fixation. A lower P3 amplitude typically indicates increased attention or cognitive effort required for processing visual stimuli. In this study, the P3 amplitude was significantly higher in AV and VR environments than in AR, suggesting that AV and VR require fewer cognitive processing resources. Notably, no significant differences were found for Perceptual Load conditions, indicating that the main driver of attentional resource allocation was the MR environment itself.
• #25: Subjective results were pretty straightforward and coherent across measures. Subjective reports indicated that AR was perceived as significantly more difficult, distracting, and overwhelming compared to AV and VR, especially under high perceptual load conditions. These findings are mirrored in the NASA TLX scores, which further validate the increased workload perception in AR
• #27: Short-form videos represent multimodal and emotional stimuli, that capture our attentional resources and limit our capacity to remember intentions . This is because such dynamic visual and auditory features require attention focus for effective information processing during video watching. Participants were vulnerable to the interruption and not ready to resume the task, as they were cognitively engaged with the interruption. DDM analysis showed that TikTok interruption slowed information processing efficiency and made harder for participants to make a decision, requiring more time Short-form videos increased the memory demand on participants, making it harder to process each alternative, leading to reduced evidence accumulation rates for making a decision.
• #28: Our study found no significant differences in accuracy across MR environments, but reaction times were faster in AV and VR compared to AR, indicating quicker responses in those settings. From a subjective perspective, perceived workload was rated much higher in AR. The implications for AR are critical, especially in time-sensitive tasks such as healthcare or emergency response, where delayed reactions can have serious consequences. Therefore, it's important to balance virtual content in AR, ensuring we don’t overwhelm users with too much cognitive load. Finally, in critical scenarios, it’s essential to prioritize user safety by minimizing distractions in AR. In less critical tasks, introducing more virtual elements to create an AV-like environment could enhance performance."
• #29: In our second research question, we examined how cognitive resource allocation differs across MR environments. We found that P3 amplitude was higher in AV and VR compared to AR, indicating more efficient attentional processing in these environments. The richer textures and visual noise in AR increased cognitive demands, leading to delayed and diminished ERP responses compared to AV and VR. This suggests that visual complexity in AR can overload cognitive resources. As a result, MR designers should consider reducing visual complexity in AR environments or introduce features like directional cues or contextual filtering to optimize user performance and aid in information processing."
• #30: For RQ3, we focused on how different MR environments impacted visual search efficiency through eye-tracking measures. In VR, we observed shorter fixation durations compared to AV and AR, indicating more efficient visual processing. This suggests that VR helps streamline attention, while AR and AV, with their transparent and see-through objects, increase attentional load due to reduced visual salience. Although pupil size remained consistent across perceptual load conditions, it was largest in AR and progressively decreased from AR to VR, especially under high perceptual load, signaling reduced cognitive load in virtual environments. These findings suggest that in MR design, particularly for AR and AV, reducing visual complexity and enhancing the salience of virtual objects could help improve search efficiency and reduce cognitive load.
• #31: "Our study focused on virtual elements in controlled settings, but future work should incorporate both virtual and physical clutter to better reflect real-world MR environments. In optical see-through AR, ambient lighting affects the visibility of virtual objects, so exploring lighting adjustments could improve coherence. Additionally, investigating how luminance levels impact pupil size can help develop adaptive AR systems that respond to environmental conditions for enhanced user experience.
• #32: Our work contributes to understanding how AR, AV, and VR environments impact cognitive load and visual search performance, with VR showing more efficient processing and AR presenting higher attentional demands. You can access the dataset by scanning the QR code or visiting the link. For any questions or further discussions, feel free to contact me via the email provided. Thank you for your time and interest in our research!