eyeTalk - A system for helping people affected by motor neuron problems
1 like576 views
This document summarizes an on-screen T9 keyboard project for people with locked-in syndrome. It introduces the motivation to enable interaction through eye tracking, describes the team and software development process, and outlines the system architecture including a backend, UI, T9 and word prediction module, and analytics module. The team aims to build an easy-to-use communication tool and collect data to improve the system and research.
eyeTalk - A system for helping people affected by motor neuron problems
- 1. Grad Students: Manindra Moharana, Narendran Thangarajan, Soham Shah Undergrads: Cary Cheng, Christine He, Jessica Cho, Joann Kim, Koa Nies, Luke Pickett On-Screen T9 Keyboard for Bob CSE 218
- 3. Motivation Trends in Interest on LIS by countryTrends in Interest on LIS by news headlines Enable people affected by Locked-In Syndrome (LIS) to interact with any touch based display.
- 4. Existing Solutions - Tobii Eye Tracker Pros ● Navigate the web ● Communicate by words ● Skype, emails, music, etc. Cons ● Expensive ● Complicated to use ● Bad tech support
- 5. Existing Solutions - EyeGaze Edge Talker Pros ● Standard keyboard with horizontal and vertical eye tracking ● Text to Speech ● Quick access to frequent phrases Cons ● Difficult to use ● Bob’s use case Image Courtesy : https://www.youtube.com/watch?v=lY22CZ7XP-4, www.eyegaze.com
- 6. Vision 1. Easy to use tool for communication 2. Collect data to improve our tool and foster research.
- 7. Outline ● Requirements ● Product Description ● Team ● Software Process ● Architecture ● System design and implementation ● Post Mortem
- 8. Requirement Methods ● Professor ● Articles on LIS Two types of LIS ○ Classical ○ Total We focus on classical LIS patients
- 9. Product Description ● Vertically T9 on-screen keyboard ● Activate on fixation ● Blink gestures for switching context (keyboard - predictions) ● Wink gesture to send keystrokes to application
- 11. Weekly team meetings Facebook group for communication Google Docs for collaboration Git(Bitbucket) + Eclipse
- 12. Software Process ● SCRUM 1. Product Owner 2. Scrum Masters 3. Development Team ● Sprints 1. Prototype (Madness Demo) 2. UX design, bugfixes, backlog.
- 13. Architecture Hardware ● EyeTribe eye tracker ● Microsoft Surface Pro 3 Software ● Java ● HTML/CSS/JS ● Python/mongoDB Microsoft Surface Internals EyeTribe server eyeTalk application backend eyeTalk application frontend 1 2 3
- 14. System Four main components : 1. eyeTalk Backend 2. eyeTalk UI 3. T9 and word prediction 4. eyeTalk analytics
- 15. 1. eyeTalk backend eyeTalk Backend Image Courtesy : Eyetribe Docs
- 16. 1. eyeTalk backend ● Written in Java ● GazeManager and IGazeListener to communicate with EyeTribe server. ● Technical Challenge : Handle saccades using running average filter ● At any point in time, provides the current smoothened value of X,Y eyeTalk Backend eyeTalk UI Get gaze data points in real-time (polling)
- 17. 2. eyeTalk UI ● Built using Processing graphics library (Java) ● Design focus/constraint - Use only vertical eye movement for control ● HCI design principles applied ● Dwell time (1.5 sec) based button clicks - with progress bar animation ● T9 and Manual input mode supported
- 18. 2. eyeTalk UI ● Modal screen to select from multiple word predictions ● Integrated TTS (CMU Sphinx) ● On-screen keyboard, works without staying in focus ● Long blink to send keystrokes to foreground application (notepad, email client, etc.) ● Customisable UI
- 19. ● T9 ● Word Completion ○ Higher weights for more frequently used words ● Word Prediction ○ Learns commonly used phrases from corpus ● Preprocessing for quick lookups 3. T9 and word prediction
- 21. 4. eyeTalk analytics ● Written in Python, JS. DB : mongoDB ● Technical Challenge : Real-time analytics using map-reduce to calculate heatmap on demand. ● Future: ○ Streaming API support. ○ Real-time rendering using websockets. eyeTalk Backend Push data points map-reduce queries
- 22. How they all fit together eyeTalk Backend eyeTalk UI Get gaze data points in real-time (polling) Get predictionsT9 and word prediction algorithm eyeTalk analytics
- 23. Post-mortem ● Lessons learnt ○ Time management with part-time developers. ○ Precision while assigning tasks. ○ Identifying skillsets and exercising comparative advantage. ● What went right/wrong ○ Team split and task allocation. ○ Productive meetings. ○ Avoiding new tools for project management. ● Sub-teams by expertise vs. interest. ● Evaluation
- 24. Thanks!