![Eye Tracking Helps With The Detection of the
Onset of Driver Drowsiness/Fatigue
Driver drowsiness has been widely recognized as a major contributor to
highway crashes:
– 1500 fatalities/year
– 12.5 billion dollars in cost/year
Crashes and near-crashes attributable to driver drowsiness:
– 22 -24% [100-car Naturalistic Driving study, NHTSA]
– 4.4% [2001 Crashworthiness Data System (CDS) data]
– 16- 20% (in England)
– 6% (in Australia)
Driver
Physiological
State
14%
Driving Task
Error
76%
Road Surface
8%
Vehicle Defects
3%
Driver
Physiological
State
14%
Driving Task
Error
76%
Road Surface
8%
Vehicle Defects
3%
Source: NHTSA](https://image.slidesharecdn.com/3-eyeblink-detection-180416100955/85/eyeblink-detection-28-320.jpg)
eyeblink-detection
- 1. Companion Eye Systems for Assistive and Automotive Markets Nov 04, 2013 Dr. P.SUDHAKARA RAO PROFESSOR,ECE, VMTW
- 2. Eye Tracking as a Non-Invasive Tool to Collect Rich Eye Data for Various Applications Eye Tracking Device Operators ALS/CP Patients Web Surfers,… ADS AAC …. ADS: Advanced Driver Support Systems AAC: Augmentative & Alternative Communication Collect Eye Data Interpret Eye Data
- 3. Eye Tracking is a Key Technology in Advanced Driver Support Systems (ADS) Drowsy Driver Detection Driver Distraction Alert Driver Physiological State 14% Driving Task Error 76% Road Surface 8% Vehicle Defects 3% Driver Physiological State 14% Driving Task Error 76% Road Surface 8% Vehicle Defects 3%
- 4. ADS: Visual Distraction Alert Reduces Vehicles Crashes
- 5. AAC Improves Quality of Lives Eye Tracking Technology Allows Disabled People to Communicate » Compose Text Messages » Dial Phone Numbers » Play Games » Drive Power Wheelchair http://www.youtube.com/watch?v=gDKFNqrmtZ4
- 6. Eye Tracking Markets & Differentiators Tobii Smart Eyes Seeing Machines EyeTech Digital System SensoMotoric Instruments GmbH DynaVox Companion Eye Systems Price range Accuracy & Robustness Calibration Head box Power consumption Onboard processing Customer support
- 7. Accuracy Matters! Eye Tracking Vs. Head Tracking Eye Cursor Can Get as Precise as a Mouse Cursor Head Tracker Lacks of Precision but Still Useful for those with Eye Diseases
- 8. Overview of HW and SW of an Eye Tracker Device Eye–Gaze Tracking – Eye detection/Tracking – Gaze measurements form dark pupil & corneal reflections – 3D gaze tracking » System Calibration » Corneal/Pupil centers estimation » Optical axis Vs. Visual axis » User Calibration » Experiments Eye Closure Tracking (EC) – Driver fatigue detection
- 9. Choosing The Right Setup Helps Simplifying the Image Processing Algorithms and Increasing Accuracy Near Infrared Camera – 880 nm » Must respect the MPE threshold (eye safety threshold) – Filter to block ambient lights – >= 15HZ – Global Shutter Off Axis LEDs – dark pupil – Corneal reflexes (glints)
- 10. Eye Tracking Algorithmic Building Blocks Dual corneal ref. centers computation Quality Control tracking recovery Eye corners, iris center detection Point of Gaze on the Screen / World coordinate system Eye Gaze measur. computation in 2D & 3D Data Analysis: saccade, scanning path, fixation 6DOF head pose Area-of-interest 3D Pupil center est. Estimation of the Gaze Mapping function Left & right pupil centers detection in 2D Eye typing, Heat Map, Contingent display, controlled wheelchair, etc. Brow / lips tracking Blink / Eye Closure detection Nose tip tracking Input Video Ctrl/switch LEDs Switch cameras 3D Cornea center estimation Input Video Command PTZ Global-local calibration scheme Gaze Error / Qual. Ass. Calibration auto- correction Camera(s), LEDs & screen Calibration Calculation of the intersection point <LOS & plane> POG mapping from Camera coordinates to screen Pupil/CR Tracking Facial Action Code recognition head pose & eye pose combination <Vis. & Opt.> angle comp. Track left & right eye gaze (2 eyes) Estimation of the correction func. for head mvt Facial detection Face detection/Single Eye region detection smoothing, filtering, validation, history keeping Head motion orientation 3D LOS Pre - pro ces sin g Depth estimation 2D eye socket tracking 2-5-9- 16 pts calibra tion
- 11. Understanding the Eye Anatomy Helps in the Formulation of the Image/Ray Formation Aq. Humor refraction index = 1.3 Distance from corneal center to Pupil center = 4.5mm Radius of corneal sphere = 7.8mm
- 12. www.youtube.com/watch?v=kEfz1fFjU78 Eye Tracking Refers to Tracking All Types of Eye Movements Saccadic: Abruptly Changing Point of Fixation Smooth Pursuit: Closely Following a Moving Target Eye Closure: Going from Open Eye State to Closed Eye State Fixation: Maintaining The Visual Gaze On a Single Location Eye Blinking: Sequence of Blinks Eye Gesture: Sequence of Eye Movements
- 13. Extracting Infrared Eye Signatures for Eye Detection & Tracking Low-pass filter High-pass filter Region growing dot product filter Potential eye candidates Input Image (dark pupil, two glints)
- 14. Learn an Eye/non-Eye Models using Machine Learning to Enhance the Automatic Eye Detection Process Variations of the eye appearance due to lighting changes, eye wear, head pose, eyelid motion and iris motion …
- 15. Filter Eye Candidates using Spatio- Temporal and Appearance Information
- 16. Example of Pupil/Glints Tracking During Fast Head Motion (Cerebral Palsy Subject)
- 17. Example of Pupil/Glints Tracking During Fast Head Motion (Cerebral Palsy Subject)
- 18. Tracking of Facial Features and Eye Wear Increases Efficiency and Allows Dynamic Camera/Illumination Control Iris Upper & lower lids Brow Furrow Eye & Glasses Head Face ellipse Left eye + Right eye
- 19. From eye detection to eye features localization and 2D gaze vector calculation a. Extract left glint and right glint centers in 2D images b. Define corneal region around the two glints to search for the pupil c. Fit an ellipse on the convex- hull of the darkest region near the two glints (segment the region using mean-shift algorithm) d. Compute the center of mass of the pupil in 2D images Gaze vector / 2D gaze measurement in the image space to be mapped to the screen coordinate system Next step: estimate the coefficient of a mapping function during a user calibration session & the system is ready for use!
- 20. User’s Calibration for Eye Gaze Tracking User to look at displayed target on the screen System to collect gaze measurement for that target Repeat for N targets System to learn a bi- quadratic mapping function between the two spaces . . . http://www.ecse.rpi.edu/~qji/Papers/EyeGaze_IEEECVPR_2005.pdf Springer Book: Passive Eye Monitoring Algorithms, Applications and Experiments, 2008
- 21. 3D GazeTracking Allows Free Head Motion Optical axis CC PC Visual axis GT POG OffsetEst POG Estimate corneal center in 3D Estimate pupil center in 3D Construct the 3D line of sight Construct the monitor plane Find the intersection point of the 3D LOS and Monitor plane Compensate for the difference between optical axis and visual axis 3D Pupil center estimation 3D Cornea center estimation Calculation of the LOS & Monitor intersection POG mapping from Camera coordinates to screen Camera(s), light source & screen(s) Calibration
- 22. Imager: Intrinsic, extrinsic parameters LCD: Screen relative to camera LEDs: Point light sources relative to camera top-left corner 3D position: (-cx*3.75*10-3mm, -cy*3.75*10-3mm, (fx+fy)/2*3.75*10-3mm) (Δx, Δy, Δz) = (3.75*10-3mm, 0, 0) if you walk along the column by one pixel Rotation and Translation Matrix + screen width and height(unit:mm) + screen resolution(unit: pixel) 3D Gaze Tracking Requires Camera/System Calibration
- 23. Lighting source (L) 3D Cornea 2D glint center in the captured frame (Gimg) 3D Glint center Point of incidence (G) Cc (O) focal point Image Plane Radius Reflection law: (L1-G1)·(G1-C)/||L1-G1|| = (G1-C)·(O-G1)/||O-G1|| Spherical: |G1 – C| = Rc Co-planarity: (L1 – O) ˣ (C – O) · (Gimg1 – O) = 0 Reflection ray: •Gimg1: 3D position of the glint on the image plane (projected cornea reflection) (known) •L1 : 3D IR light position (known) •O: imager focal point (known) •G1/ G2: 3D position of CR(unkown) •C: Cornea Center (unkown) • Rc: Cornea Radius (known, population average) Construct and Solve a System of Non-Linear Equations to Estimate the 3D Corneal Center Lighting source (R) 9 variables 10 equations
- 24. Input & Output Input: Frame nb, pupil center in 2D image, first glint, second glint, mid-glint point 160 979.534973 336.336365 991.500000 339.500000 978.500000 339.500000 985.000000 339.500000 161 978.229858 336.898865 989.500000 339.500000 977.500000 339.500000 983.500000 339.500000 162 973.933411 336.968689 987.500000 340.500000 974.500000 340.500000 981.000000 340.500000 163 -1 -1 -1 -1 -1 -1 -1 -1 164 975.000000 338.500000 987.500000 341.500000 975.500000 341.500000 981.500000 341.500000 Output : Corneal Center (x, y, z): (-31.85431, 38.07172, 470.4345) Pupil center(x, y, z): (-30.80597, 35.80776, 466.6895)
- 25. POG Estimation Concept: – Estimate the Intersection of Optical Axis and Screen Plane Input: – Estimated Corneal Center 3D Position – Estimated Pupil Center 3D Position – Screen Origin, Screen size – Rotation Matrix in Camera Coordinate Output: POG Position Optical axis CC PC Visual axis GT POG OffsetEst POG
- 26. Input & Output Input: Frame nb, pupil center in 2D image, first glint, second glint, mid-glint point 160 979.534973 336.336365 991.500000 339.500000 978.500000 339.500000 985.000000 339.500000 161 978.229858 336.898865 989.500000 339.500000 977.500000 339.500000 983.500000 339.500000 162 973.933411 336.968689 987.500000 340.500000 974.500000 340.500000 981.000000 340.500000 Output sample: Corneal Center (x, y, z): (-31.85431, 38.07172, 470.4345) Pupil center(x, y, z): (-30.80597, 35.80776, 466.6895) POG(x, y): (148.7627, 635.39)
- 27. 9 Targets POG Estimation Plot – With Glasses 5 pts Calibration 4 pts Test -100 0 100 200 300 400 500 600 700 800 900 -200 0 200 400 600 800 1000 1200 LeftEYE_Glass_5ptsCalib RightEYE_Glass_5ptsCalib TwoEYE_Glass_5ptsCalib GroundTruth Averaging Both Eyes Increases Accuracy
- 28. Eye Tracking Helps With The Detection of the Onset of Driver Drowsiness/Fatigue Driver drowsiness has been widely recognized as a major contributor to highway crashes: – 1500 fatalities/year – 12.5 billion dollars in cost/year Crashes and near-crashes attributable to driver drowsiness: – 22 -24% [100-car Naturalistic Driving study, NHTSA] – 4.4% [2001 Crashworthiness Data System (CDS) data] – 16- 20% (in England) – 6% (in Australia) Driver Physiological State 14% Driving Task Error 76% Road Surface 8% Vehicle Defects 3% Driver Physiological State 14% Driving Task Error 76% Road Surface 8% Vehicle Defects 3% Source: NHTSA
- 29. (1) Shape (2) Pixel- density (3) Eyelids motion & spacing (5) Iris-radius Eye Tracking: Hybrid Recognition Algorithm for Eye Closure Recognition Time (6) Motion-like method (eye dynamic) Velocity curve Eye closure data (7) Slow closure vs. Fast closure
- 30. Participant Metrics Ethnicity Vision Gender Participant volume:113, December 2006 December 2007
- 31. Extended Eye Closure (EEC) Evaluation ♦ EEC accuracy is the same across groups
- 32. Drowsy Driver Detection Demo
- 33. SAfety VEhicle(s) using adaptive Interface Technology (SAVE-IT) program Utilize information about the driver's head pose in order to tailor the warnings to the driver's visual attention. SAVE-IT: 5 year R&D program sponsored by NHTS and administered by Volpe
- 34. Eye Tracking & Head Tracking for Driver Distraction 78 test subjects – Gender – Ethnic diversity – Height (Short(≤ 66”), Tall (> 66”)) – Hair style, – Facial hair, – Eye Wear Status and Type: – No Eye Wear – Eye Glasses – Sunglasses – Age (4 levels) – 20s, 30s, 40s, 50s