IVE 2024 Short Course - Lecture 2 - Fundamentals of Perception
0 likes201 views
The document outlines the fundamentals of virtual reality (VR) and augmented reality (AR), emphasizing key concepts such as immersion, interaction, and presence. It discusses the sensory input mechanisms crucial for creating realistic experiences in XR environments, including human visual and auditory systems, as well as the challenges related to motion sickness and system latency. Additionally, it explores the technological advancements needed for life-like VR displays and the importance of understanding human perception in XR design.
![Augmented Reality Definition
● Defining Characteristics [Azuma 97]
○ Combines Real and Virtual Images
■ Both can be seen at the same time
○ Interactive in real-time
■ The virtual content can be interacted with
○ Registered in 3D
■ Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.](https://image.slidesharecdn.com/lecture2-fundamentals-240721133043-6dc4e098/85/IVE-2024-Short-Course-Lecture-2-Fundamentals-of-Perception-7-320.jpg)
IVE 2024 Short Course - Lecture 2 - Fundamentals of Perception
- 1. Lecture 2: Fundamentals The Psychology of XR July 15th – 19th 2024
- 2. Virtual Reality (VR) ● Users immersed in Computer Generated environment ○ HMD, gloves, 3D graphics, body tracking
- 3. The First VR Experience … https://www.youtube.com/watch?v=pAC5SeNH8jw
- 4. Virtual Reality Definition ● Defining Characteristics ○ Sense of Immersion ■ User feels immersed in computer generated space ○ Interactive in real-time ■ The virtual content can be interacted with ○ Independence ■ User can have independent view and reaction to environment
- 5. David Zeltzer’s AIP Cube Autonomy – User can to react to events and stimuli. Interaction – User can interact with objects and environment. Presence – User feels immersed through sensory input and output channels Interaction Autonomy Presence VR Zeltzer, D. (1992). Autonomy, interaction, and presence. Presence: Teleoperators & Virtual Environments, 1(1),127-132.
- 6. Augmented Reality (AR) • Virtual Images blended with the real world • See-through HMD, handheld display, viewpoint tracking, etc..
- 7. Augmented Reality Definition ● Defining Characteristics [Azuma 97] ○ Combines Real and Virtual Images ■ Both can be seen at the same time ○ Interactive in real-time ■ The virtual content can be interacted with ○ Registered in 3D ■ Virtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 9. Milgram’s Mixed Reality (MR) Continuum Augmented Reality Virtual Reality Real World Virtual World Mixed Reality "...anywhere between the extrema of the virtuality continuum." P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays Internet of Things
- 10. Apple Vision Pro (2024) ● Transitioning from AR to VR ● Spatial Computing – interface seamlessly blending with real world
- 12. Extended Reality (XR) Augmented Reality Virtual Reality Real World Virtual World Mixed Reality Extended Reality Internet of Things
- 13. Goal of Virtual Reality “.. to make it feel like you’re actually in a place that you are not.” Palmer Luckey Co-founder, Oculus
- 14. Creating a Good VR Experience ● Creating a good experience requires multisensory input ○ Integrating multiple perceptual cues
- 15. Example: Shard VR Slide ● Ride down the Shard at 100 mph - Multi-sensory VR https://www.youtube.com/watch?v=HNXYoEdBtoU
- 16. Creating Illusions ● Virtual Reality ○ You’re immersed in a place ● Augmented Reality ○ Virtual content is in your place ● Mixed Reality ○ Seamless moving from RW in VR
- 17. Perception
- 18. What is Reality?
- 19. How do We Perceive Reality? ● We understand the world through our senses: ○ Sight, Hearing, Touch, Taste, Smell (and others..) ● Two basic processes: ○ Sensation – Gathering information ○ Perception – Interpreting information
- 20. Motivation ● Understand: In order to create a strong sense of Presence we need to understand the Human Perception system ● Stimulate: We need to be able to use technology to provide real world sensory inputs, and create the VR illusion VR Hardware Human Senses
- 21. Senses ● How an organism obtains information for perception: ○ Sensation part of Somatic Division of Peripheral Nervous System ○ Integration and perception requires the Central Nervous System ● Five major senses (but there are more..): ○ Sight (Opthalamoception) ○ Hearing (Audioception) ○ Taste (Gustaoception) ○ Smell (Olfacaoception) ○ Touch (Tactioception)
- 22. Relative Importance of Each Sense ● Percentage of neurons in brain devoted to each sense ○ Sight – 30% ○ Touch – 8% ○ Hearing – 2% ○ Smell - < 1% ● Over 60% of brain involved with vision in some way
- 23. Other Lessor Known Senses.. ● Proprioception = sense of body position ○ what is your body doing right now ● Equilibrium = balance ● Acceleration ● Nociception = sense of pain ● Temperature ● Satiety = state of being fed or gratified to or beyond capacity ● Thirst ● Micturition = amount of CO2 and Na in blood
- 24. Sight
- 25. The Human Visual System ● Purpose is to convert visual input to signals in the brain
- 26. The Human Eye ● Light passes through cornea and lens onto retina ● Photoreceptors in retina convert light into electrochemical signals
- 27. Photoreceptors – Rods and Cones ● Retina photoreceptors come in two types, Rods and Cones ○ Rods – 125 million, periphery of retina, no colour detection, night vision ○ Cones – 4-6 million, center of retina, colour vision, day vision
- 28. Human Horizontal and Vertical FOV ● Humans can see ~135 o vertical (60 o above, 75 o below) ● See up to ~ 210 o horizontal FOV, ~ 115 o stereo overlap ● Colour/stereo in centre, black and white/mono in periphery
- 29. Vergence + Accommodation ● saas
- 31. Visual Acuity Visual Acuity Test Targets ● Ability to resolve details ● Several types of visual acuity ○ detection, separation, etc ● Normal eyesight can see a 50 cent coin at 80m ○ Corresponds to 1 arc min (1/60th of a degree) ○ Max acuity = 0.4 arc min
- 32. Stereo Perception/Stereopsis ● Eyes separated by IPD ○ Inter pupillary distance ○ 5 – 7.5cm (avge. 6.5cm) ● Each eye sees diff. image ○ Separated by image parallax ● Images fused to create 3D stereo view
- 34. Depth Perception ● The visual system uses a range of different Stereoscopic and Monocular cues for depth perception Stereoscopic Monocular eye convergence angle disparity between left and right images diplopia eye accommodation perspective atmospheric artifacts (fog) relative sizes image blur occlusion motion parallax shadows texture Parallax can be more important for depth perception! Stereoscopy is important for size and distance evaluation
- 36. Depth Perception Distances ● i.e. convergence/accommodation used for depth perception < 10m
- 37. Properties of the Human Visual System ● Visual acuity: 20/20 is ~1 arc min ● Field of view: ~200° monocular, ~120° binocular, ~135° vertical ● Resolution of eye: ~576 megapixels ● Temporal resolution: ~60 Hz (depends on contrast, luminance) ● Dynamic range: instantaneous 6.5 f-stops, adapt to 46.5 f-stops ● Colour: everything in CIE xy diagram ● Depth cues in 3D displays: vergence, focus, (dis)comfort ● Accommodation range: ~8cm to ∞, degrades with age
- 38. Creating the Perfect Illusion Cuervo, E., Chintalapudi, K., & Kotaru, M. (2018, February). Creating the perfect illusion: What will it take to create life-like virtual reality headsets?. In Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications (pp. 7-12). ● Technology to create life-like VR HMDs ● Compared to current HMDs ○ 2 − 10× higher pixel density ○ 20 − 30× higher frame rate
- 39. Comparison between Eyes and HMD
- 40. When Will We Achieve Life-like VR Displays? ● Could achieve visual fidelity by 2025 ○ BUT: ■ GPUs not fast enough for high framerate (140 Tflops by 2025, need 10x for 1800Hz) ■ Wireless life like VR requires 2.7Tbps, c.f. wireless HD standard of 25Gbps Display FR and Pixels/degree
- 41. Hearing
- 42. Anatomy of the Ear
- 43. Auditory Thresholds ● Humans hear frequencies from 20 – 22,000 Hz ● Most everyday sounds from 80 – 90 dB
- 44. Sound Localization ● Humans have two ears ○ localize sound in space ● Sound can be localized using 3 coordinates ○ Azimuth, elevation, distance
- 46. Sound Localization (Azimuth Cues) Interaural Time Difference
- 47. HRTF (Elevation Cue) ● Pinna and head shape affect frequency intensities ● Sound intensities measured with microphones in ear and compared to intensities at sound source ○ Difference is HRTF, gives clue as to sound source location
- 48. Accuracy of Sound Localization ● People can locate sound ○ Most accurately in front of them ■ 2-3° error in front of head ○ Least accurately to sides and behind head ■ Up to 20° error to side of head ■ Largest errors occur above/below elevations and behind head ● Front/back confusion is an issue ○ Up to 10% of sounds presented in the front are perceived coming from behind and vice versa (more in headphones) BUTEAN, A., Bălan, O., NEGOI, I., Moldoveanu, F., & Moldoveanu, A. (2015). COMPARATIVE RESEARCH ON SOUND LOCALIZATION ACCURACY IN THE FREE-FIELD AND VIRTUAL AUDITORY DISPLAYS. InConference proceedings of» eLearning and Software for Education «(eLSE)(No. 01, pp. 540-548). Universitatea Nationala de Aparare Carol I.
- 49. Touch
- 50. Haptic Sensation ● Somatosensory System ○ complex system of nerve cells that responds to changes to the surface or internal state of the body ● Skin is the largest organ ○ 1.3-1.7 square m in adults ● Tactile: Surface properties ○ Receptors not evenly spread ○ Most densely populated area is the tongue ● Kinesthetic: Muscles, Tendons, etc. ○ Also known as proprioception
- 51. Cutaneous System ● Skin – heaviest organ in the body ○ Epidermis outer layer, dead skin cells ○ Dermis inner layer, with four kinds of mechanoreceptors
- 52. Mechanoreceptors ● Cells that respond to pressure, stretching, and vibration ○ Slow Acting (SA), Rapidly Acting (RA) ○ Type I at surface – light discriminate touch ○ Type II deep in dermis – heavy and continuous touch Receptor Type Rate of Acting Stimulus Frequency Receptive Field Detection Function Merkel Discs SA-I 0 – 10 Hz Small, well defined Edges, intensity Ruffini corpuscles SA-II 0 – 10 Hz Large, indistinct Static force, skin stretch Meissner corpuscles RA-I 20 – 50 Hz Small, well defined Velocity, edges Pacinian corpuscles RA-II 100 – 300 Hz Large, indistinct Acceleration, vibration
- 53. Spatial Resolution ● Sensitivity varies greatly ○ Two-point discrimination Body Site Threshold Distance Finger 2-3mm Cheek 6mm Nose 7mm Palm 10mm Forehead 15mm Foot 20mm Belly 30mm Forearm 35mm Upper Arm 39mm Back 39mm Shoulder 41mm Thigh 42mm Calf 45mm http://faculty.washington.edu/chudler/chsense.html
- 54. Proprioception/Kinaesthesia ● Proprioception (joint position sense) ○ Awareness of movement and positions of body parts ■ Due to nerve endings and Pacinian and Ruffini corpuscles at joints ○ Enables us to touch nose with eyes closed ○ Joints closer to body more accurately sensed ○ Users know hand position accurate to 8cm without looking at them ● Kinaesthesia (joint movement sense) ○ Sensing muscle contraction or stretching ■ Cutaneous mechanoreceptors measuring skin stretching ○ Helps with force sensation
- 56. Simple Perception Action Model Wickens, C. D., & Carswell, C. M. (2021). Information processing. Handbook of human factors and ergonomics, 114-158. Open Loop Closed Loop
- 58. Human Information Processing Model Wickens, C. D. (1992), Engineering Psychology and Human Performance, 2nd ed., HarperCollins, New York.
- 59. Creating the Illusion of Reality ● Fooling human perception by using technology to generate artificial sensations ○ Computer generated sights, sounds, smell, etc
- 60. Reality vs. Virtual Reality ● In a VR system there are input and output devices between human perception and action
- 61. Using Technology to Stimulate Senses ● Simulate output ○ E.g. simulate real scene ● Map output to devices ○ Graphics to HMD ● Use devices to stimulate the senses ○ HMD stimulates eyes Visual Simulation 3D Graphics HMD Vision System Brain Example: Visual Simulation Human-Machine Interface
- 62. Example Birdly - http://www.somniacs.co/ ● Create illusion of flying like a bird ● Multisensory VR experience ○ Visual, audio, wind, haptic
- 65. HMD Basic Principles ● Use display with optics to create illusion of virtual screen
- 66. Simple Magnifier HMD Design p q Eyepiece (one or more lenses) Display (Image Source) Eye f Virtual Image 1/p + 1/q = 1/f where p = object distance (distance from image source to eyepiece) q = image distance (distance of image from the lens) f = focal length of the lens
- 67. Vergence-Accommodation Conflict ● Looking at real objects, vergence and focal distance match ● In VR, vergence and accommodation can miss-match ○ Focusing on HMD screen, but accommodating for virtual object behind screen
- 68. AR Vergence and Accommodation ● Fixed focal distance for OST displays ● Accommodation conflict between real and virtual object
- 69. AR – Focal Rivalry ● Optical see-through AR displays with fixed focal length ○ E.g. Hololens focal length ~2m ● When real objects < focal length, can’t keep virtual object in focus ○ Either real or virtual become blurry
- 70. Example ● People made errors twice as large on connect the dots task in AR vs. real world ○ Connect virtual numbers either – without AR 0.9 mm average error, using AR 2.3 mm error Focus on Ruler Focus on Virtual Image Condino, S., Carbone, M., Piazza, R., Ferrari, M., & Ferrari, V. (2019). Perceptual limits of optical see-through visors for augmented reality guidance of manual tasks. IEEE Transactions on Biomedical Engineering, 67(2), 411-419.
- 71. Using Multiple Image Planes
- 72. MagicLeap Display ● Optical see through AR display ○ Overlay graphics directly on real world ○ 40o x 30o FOV, 1280 x 960 pixels/eye ● Waveguide based display ○ Holographic optical element ○ Very thin physical display ● Two sets of waveguides ○ Different focal planes ■ Overcomes vergence/accommodation problem ○ Eye tracking for selecting focal plane ● Separate CPU/GPU unit
- 74. Distortion in Lens Optics A rectangle Maps to this
- 75. HTC Vive Optics
- 76. To Correct for Distortion ● Must pre-distort image ● This is a pixel-based distortion ● Use shader programming
- 78. Interpupillary Distance (IPD) n Horizontal distance between a user's eyes n Distance between the two optical axes in a HMD n Typical IPD ~ 63mm
- 79. Field of View Monocular FOV is the angular subtense of the displayed image as measured from the pupil of one eye. Total FOV is the total angular size of the displayed image visible to both eyes. Binocular(or stereoscopic) FOV refers to the part of the displayed image visible to both eyes. FOV may be measured horizontally, vertically or diagonally.
- 80. Typical VR HMD FOV
- 81. Foveated Displays ● Combine high resolution center with low resolution periphery
- 82. Varjo Display Varjo resolution Non-Varjo resolution Focus area (27° x 27°) 70 PPD, 1920 x 1920px 115° FOV 30 PPD 2880 x 2720px ● 1 LCD (wide FOV) ● 1 uOLED panel (centre)
- 83. Varjo XR-3 Demo – Threading a Needle https://www.youtube.com/watch?v=5iEwlOEUQjI
- 84. Perception Based Graphics ● Eye Physiology ○ Rods in eye centre = colour vision, cones in periphery = motion, B+W ● Foveated Rendering ○ Use eye tracking to draw highest resolution where user looking ○ Reduces graphics throughput
- 86. Typical VR Simulation Loop ● User moves head, scene updates, displayed graphics change
- 87. ● Need to synchronize system to reduce delays System Delays
- 88. Typical System Delays ● Total Delay = 50 + 2 + 33 + 17 = 102 ms ○ 1 ms delay = 1/3 mm error for object drawn at arms length ○ So total of 33mm error from when user begins moving to when object drawn Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
- 89. Effects of System Latency ● Degraded Visual Acuity ○ Scene still moving when head stops = motion blur ● Degraded Performance ○ As latency increases it’s difficult to select objects etc. ○ If latency > 120 ms, training doesn’t improve performance ● Breaks-in-Presence ○ If system delay high user doesn’t believe they are in VR ● Negative Training Effects ○ User train to operative in world with delay ● Simulator Sickness ○ Latency is greatest cause of simulator sickness
- 90. Simulator Sickness ● Visual input conflicting with vestibular system
- 91. What Happens When Senses Don’t Match? ● 20-30% VR users experience motion sickness ● Sensory Conflict Theory ○ Visual cues don’t match vestibular cues ■ Eyes – “I’m moving!”, Vestibular – “No, you’re not!”
- 92. Avoiding Motion Sickness ● Better VR experience design ○ More natural movements ● Improved VR system performance ○ Less tracking latency, better graphics frame rate ● Provide a fixed frame of reference ○ Ground plane, vehicle window ● Add a virtual nose ○ Provide peripheral cue ● Eat ginger ○ Reduces upset stomach
- 93. Many Causes of Simulator Sickness ● 25-40% of VR users get Simulator Sickness, due to: ● Latency ○ Major cause of simulator sickness ● Tracking accuracy/precision ○ Seeing world from incorrect position, viewpoint drift ● Field of View ○ Wide field of view creates more periphery vection = sickness ● Refresh Rate/Flicker ○ Flicker/low refresh rate creates eye fatigue ● Vergence/Accommodation Conflict ○ Creates eye strain over time ● Eye separation ○ If IPD not matching to inter-image distance then discomfort
- 95. Applying HIP to XR Design
- 96. Human Information Processing Model ● High level staged model from Wickens and Carswell (1997) ○ Relates perception, cognition, and physical ergonomics Perception Cognition Ergonomics
- 97. Design for Perception ● Need to understand perception to design AR/VR ● Visual perception ○ Many types of visual cues (stereo, oculomotor, etc.) ● Auditory system ○ Binaural cues, vestibular cues ● Somatosensory ○ Haptic, tactile, kinesthetic, proprioceptive cues ● Chemical Sensing System ○ Taste and smell
- 98. Depth Perception Problems ● Without proper depth cues AR interfaces look unreal
- 99. Which of these POI are near or far?
- 100. Types of Depth Cues
- 102. Cutaway Example ● Providing depth perception cues for AR https://www.youtube.com/watch?v=2mXRO48w_E4
- 103. Design for Cognition ● Design for Working and Long-term memory ○ Working memory ■ Short term storage, Limited storage (~5-9 items) ○ Long term memory ■ Memory recall trigger by associative cues ● Situational Awareness ○ Model of current state of user’s environment ■ Used for wayfinding, object interaction, spatial awareness, etc.. ○ Provide cognitive cues to help with situational awareness ■ Landmarks, procedural cues, map knowledge ■ Support both ego-centric and exo-centric views
- 104. Micro-Interactions ▪ Using mobile phones people split their attention between the display and the real world
- 105. Time Looking at Screen Oulasvirta, A. (2005). The fragmentation of attention in mobile interaction, and what to do with it. interactions, 12(6), 16-18.
- 106. Dividing Attention to World ● Number of times looking away from mobile screen
- 107. Design for Micro Interactions ▪ Design interaction for less than a few seconds ○ Tiny bursts of interaction ○ One task per interaction ○ One input per interaction ▪ Benefits ○ Use limited input ○ Minimize interruptions ○ Reduce attention fragmentation
- 108. NHTSA Guidelines - www.nhtsa.gov For technology in cars: • Any task by a driver should be interruptible at any time. • The driver should control the pace of task interactions. • Tasks should be completed with glances away from road <2 seconds • Cumulative time glancing away from the road <=12 secs.
- 109. Make it Glanceable ● Seek to rigorously reduce information density. Successful designs afford for recognition, not reading. Bad Good
- 110. Reduce Information Chunks You are designing for recognition, not reading. Reducing the total # of information chunks will greatly increase the glanceability of your design. 1 2 3 1 2 3 4 5 (6) Eye movements For 1: 1-2 460ms For 2: 1 230ms For 3: 1 230ms ~920ms Eye movements For 1: 1 230ms For 2: 1 230ms For 3: 1 230ms For 4: 3 690ms For 5: 2 460ms ~1,840ms
- 111. Ego-centric and Exo-centric views ● Combining ego-centric and exo-centric cue for better situational awareness
- 112. Cognitive Issues in Mobile AR ● Information Presentation ○ Amount, Representation, Placement, View combination ● Physical Interaction ○ Navigation, Direct manipulation, Content creation ● Shared Experience ○ Social context, Bodily Configuration, Artifact manipulation, Display space Li, N., & Duh, H. B. L. (2013). Cognitive issues in mobile augmented reality: an embodied perspective. In Human factors in augmented reality environments (pp. 109-135). Springer, New York, NY.
- 113. Information Presentation • Consider • The amount of information • Clutter, complexity • The representation of information • Navigation cues, POI representation • The placement of information • Head, body, world stabilized • Using view combinations • Multiple views
- 114. Example: Twitter 360 ● iPhone application ● See geo-located tweets in real world ● Twitter.com supports geo tagging
- 115. But: Information Clutter from Many Tweets Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah
- 118. Outdoor AR: Limited FOV
- 119. ● Show POI outside FOV ● Zooms between map and panorama views Zooming Views
- 121. Design for Physical Ergonomics ● Design for the human motion range ○ Consider human comfort and natural posture ● Design for hand input ○ Coarse and fine scale motions, gripping and grasping ○ Avoid “Gorilla arm syndrome” from holding arm pose
- 122. Gorilla Arm in AR ● Design interface to reduce mid-air gestures
- 123. XRgonomics ● Uses physiological model to calculate ergonomic interaction cost ○ Difficulty of reaching points around the user ○ Customizable for different users ○ Programmable API, Hololens demonstrator ● GitHub Repository ○ https://github.com/joaobelo92/xrgonomics Evangelista Belo, J. M., Feit, A. M., Feuchtner, T., & Grønbæk, K. (2021, May). XRgonomics: Facilitating the Creation of Ergonomic 3D Interfaces. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-11).
- 125. System Design Guidelines - I ● Hardware ○ Choose HMDs with fast pixel response time, no flicker ○ Choose trackers with high update rates, accurate, no drift ○ Choose HMDs that are lightweight, comfortable to wear ○ Use hand controllers with no line-of-sight requirements ● System Calibration ○ Have virtual FOV match actual FOV of HMD ○ Measure and set users IPD ● Latency Reduction ○ Minimize overall end to end system delay ○ Use displays with fast response time and low persistence ○ Use latency compensation to reduce perceived latency Jason Jerald, The VR Book, 2016
- 126. System Design Guidelines - II ● General Design ○ Design for short user experiences ○ Minimize visual stimuli closer to eye (vergence/accommodation) ○ For binocular displays, do not use 2D overlays/HUDs ○ Design for sitting, or provide physical barriers ○ Show virtual warning when user reaches end of tracking area ● Motion Design ○ Move virtual viewpoint with actual motion of the user ○ If latency high, no tasks requiring fast head motion ● Interface Design ○ Design input/interaction for user’s hands at their sides ○ Design interactions to be non-repetitive to reduce strain injuries Jason Jerald, The VR Book, 2016