Eyeing the Web
Download as PPTX, PDF1 like875 views
This document summarizes Steffen Staab's keynote presentation on eye tracking and web interaction. It discusses how eye tracking can be used to understand how users interact with and understand websites. It presents a framework for discovering active visual stimuli on websites using eye tracking data and machine learning. It also introduces GazeTheWeb, a system that aims to optimize gaze-based interaction with websites by adapting the interaction based on semantic understanding of page elements and dynamics. A lab study found that GazeTheWeb improved task completion times, usability and workload compared to traditional gaze emulation.
Eyeing the Web
- 1. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 1Institute for Web Science and Technologies · University of Koblenz-Landau, Germany Web and Internet Science Group · ECS · University of Southampton, UK & Eyeing the Web Steffen Staab Joint work with Chandan Kumar, Raphael Menges, Korok Sengupta Christoph Schäfer, Tina Walber @ EyeVido
- 2. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 2 Here‘s looking at you, kid
- 3. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 3 Remote Eye Tracking
- 4. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 4 Anatomy of the Human Eye Unavoidable issues wrt eye tracking • accuracy → bias → calibration • precision → variance → focal area • intention → Midas touch
- 5. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 5
- 6. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 6 Web Interaction Experience the Web Manipulate the Web Eyes Ears Hands Mouth Feet Other Voice Haptic smell taste balance Understand Augment
- 7. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 7 Eye & Web: Why should one care? Accessibility Standardization Sound architecture Interaction bandwidth Web community HCI community
- 8. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 8 Eye & Web: Why should one care? If you make it there, You make it everywhere On the Web Accessibility Standardization Sound architecture Interaction bandwidth
- 9. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 9 Web Interaction Part I Part II Eyes Ears Hands Mouth Feet Other Voice Haptic smell taste balance Understand Augment
- 10. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 10 Understanding Users‘ Understanding of Web Sites
- 11. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 11 • Spin-out of WeST in 2015 • Major companies as clients: „If a user is having a problem, it‘s your problem.“ - Steve Jobs -
- 12. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 12 Cloud-Software EYEVIDO Lab . . .. . . . . . . . . . . . . . . Creation and evaluation of usability studies in the web browser Storage and analysis of test data in the cloud Data collection with eye trackers anywhere and anytime Various possibilities for data visualization, analysis, and data export 0100101 0101000 Secure server in Germany, security approved (TÜV label) Eye tracker https://youtu.be/-wbdHx0Idbk
- 13. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 13
- 14. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 14 Multiple users interact with an interface Usability experts want to analyze user behavior!
- 15. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 15 User Semantics&Pragmatics vs Web Site Syntax Usability Objective Understanding the Semantics and Pragmatics of Many Users‘ Interactions Issues with the Web site Dynamics implemented using syntax of • URL change • z-index, or visibility, or opacity, or transform … – In the page, in the style sheet, – in the code
- 16. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 16 Data recording during interaction • Video recording • Mouse data • Eye-tracking data • Voice data… One could watch every video with data, but… How to make the analysis effective and efficient? Research Question
- 17. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 17 Frames from a video are stitched to a single image First Approach: Stitching (Menges et al. 2018)
- 18. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 18 What if the pages look different? Dynamics in the Web Interface Frame n Frame n+1
- 19. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 19 Paradigms of Usability Analysis Traditional • Passive stimuli – Static images Required • Active stimuli – Carousels – Video – Ajax – ... Both • Correlating stimuli and (lack of) user activities – Clicks – Gaze – Voice • Coming up with suggestions for improved usability
- 20. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 20 Discovery of Active Visual Stimuli
- 21. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 21 Novel Framework for Discovering Visual Stimuli
- 22. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 22 Ground Truth of Visual Change Visual Change?No Yes Frame n Frame n+1
- 23. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 23 Ground Truth of Visual Change: Refined
- 24. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 24 Training of the Classifier
- 25. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 25 … compute for each pair of sequential frames. Features for Classifiers of Visual Change
- 26. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 26 • Data recording on 12 Web sites, each 3 pages • 4 participants, total of 155 minutes of videos • In total 23,571 frame pairs have been labeled • 4,446 have been labeled as visually different / changing Ground Truth Data Set Interface for Labeling
- 27. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 27 For each frame pair: Features and Label. Try to predict! Visual Change Classifier: Within Site Same-Site Classifications with 4-fold cross validation *F1-scores everywhere
- 28. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 28 Same-Category Classifications Across-Category Classifications Feature Importances Visual Change Classifier: Across Sites
- 29. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 29 Framework for Visual Stimuli Discovery Remember the proposed framework… This we have covered Splitting Merging
- 30. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 30 Splitting Input Video with frames Output Stimulus shots with stitched frames
- 31. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 31 Merging Stimulus shots with stitched frames of each video Output Stitched frames across videos Input Output of Splitting
- 32. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 32 Input to the framework
- 33. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 33 Output of the framework
- 34. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 34 More Output of the framework
- 35. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 35 • Efficiency for the UI expert? • Better features for the machine learning? • How to handle borderline cases? • How to consider UI expert feedback and personalization? – Not every analysis must have the same purpose! Open questions
- 36. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 36 Web Interaction using your Eyes
- 37. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 37 The Assumption Photo by User Gflores on en.wikipedia - http://www.epa.gov/win/winnews/images05/0510keyboard.gif, Public Domain, https://commons.wikimedia.org/w/index.php?curid=36965777
- 38. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 39 Web Accessibility
- 39. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 40 EU Project MAMEM 2015-2018 https://youtu.be/42yGmr3NE0k
- 40. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 41 GazeTheWeb
- 41. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 42 User Experience: Issues with Emulation Precision, Accuracy, Midas Touch • Filtering, calibration, element size and feedback • Better Pointing and Selection Method (e.g., replace mouse and keyboard) EyeTracker (1) (2) (4) GazeData Cursor / Button Press/ Scroll Wheel Key Press No semantic feedback! Monitor Reflection of user Emulation Software ScreenEmulation Filtering DesktopWeb Browser a) b) KeyboardMouse Pixel Data ? ? (3) (5) (6) Gaze Emulation
- 42. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 43 User Experience: Issues with Emulation User experience depends on application control – Emulation of mouse and keyboard incurs interaction and visual overhead Example: Type a search query
- 43. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 44 Context-adapted Gaze Interaction: GazeTheWeb • Optimize gaze interaction experience – Minimize the required input actions – Minimize inspection overhead • Extract the semantics of the interface – Text input – Hyperlink – Video – … • Adapt the means of interaction – Appropriate input translation – Visual indicators – Contextual Feedback Pixel Data Commands Webpage Elements/ Information Interface Web Engine <ContextualData/> <html> <head> ... </head> <body> ... <input/> ... <a/> ... </body> </html> WebPage Screen EyeTracker (1) (3) (4) (5) (6) (7) (8) GazeData Monitor Reflection of user (2) Filtering Open URL or Bookmark Back Mark Bookmark Remove Tab (b) Tab 1 Tab 2 New Tab Tabs Duckduckgo.com (a) T Back Forward TabsHyperlink Germany | (c) Text Input Scrolling
- 44. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 45 GazeTheWeb: Adaptations
- 45. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 46 GazeTheWeb Video https://www.youtube.com/watch?v=x1ESgaoQR9Y (48s)
- 46. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 47 Web Interface Introspection • Web page semantics – Identification and Tracking of Interaction Elements • Web browser semantics – Page Meta-Information and Control Handles • Dynamics in modern Web – Structure and layout often changes
- 47. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 48 Examples of ideosyncratic engineering Google Search. • Stacks two input fields over each other using CSS z-index – one displays suggestions – the other the actual input – figure out by specific programming which is which Facebook Chat prohibits insertion of text using JavaScript. • We simulate keystrokes through the Web engine YouTube Video • Videos in general: how to operate if no button is visible? – We introduce an invisible button! • YouTube prevents making the video full screen. – Make the 7th parent full screeen. Rule of thumb: Generalizing methods donot work on most sophisticated Web sites
- 48. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 49 Task completion Usability Workload Gaze based emulation GazeTheWeb UX Evaluation: Summative lab study EyeTracker (1) (2) (4) GazeData Cursor / Button Press/ Scroll Wheel Key Press No semantic feedback! Monitor Reflection of user Emulation Software ScreenEmulation Filtering DesktopWeb Browser a) b) KeyboardMouse Pixel Data ? ? (3) (5) (6) Pixel Data Commands Webpage Elements/ Information Interface Web Engine <ContextualData/> <html> <head> ... </head> <body> ... <input/> ... <a/> ... </body> </html> WebPage Screen EyeTracker (1) (3) (4) (5) (6) (7) (8) GazeData Monitor Reflection of user (2) Filtering Open URL or Bookmark Back Mark Bookmark Remove Tab (b) Tab 1 Tab 2 New Tab Tabs Duckduckgo.com (a) T Back Forward TabsHyperlink Germany | (c) Text Input Scrolling
- 49. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 50 UX Evaluation: Results Task completion time Workload –NASA TLX Usability Design Heuristics
- 50. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 51 Task completion Usability Workload Gaze based emulation GazeTheWeb UX Evaluation: Summative lab study EyeTracker (1) (2) (4) GazeData Cursor / Button Press/ Scroll Wheel Key Press No semantic feedback! Monitor Reflection of user Emulation Software ScreenEmulation Filtering DesktopWeb Browser a) b) KeyboardMouse Pixel Data ? ? (3) (5) (6) Pixel Data Commands Webpage Elements/ Information Interface Web Engine <ContextualData/> <html> <head> ... </head> <body> ... <input/> ... <a/> ... </body> </html> WebPage Screen EyeTracker (1) (3) (4) (5) (6) (7) (8) GazeData Monitor Reflection of user (2) Filtering Open URL or Bookmark Back Mark Bookmark Remove Tab (b) Tab 1 Tab 2 New Tab Tabs Duckduckgo.com (a) T Back Forward TabsHyperlink Germany | (c) Text Input Scrolling 👍 👍 👍 👍 👍 👍
- 51. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 52 Trial with Patients • Neuro muscular disease • Spinal cord injury • Parkinson disease
- 52. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 53 MAMEM Trials: Phase I SUS Scores Target Group: 77.36 Control Group: 72.17 P=.2 SUS Scores Phase II: 70 (NMD), 75.5 (PD), 73.3 (SCI)
- 53. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 54 Conclusion
- 54. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 55 It‘s the semantics, of course! Experience the Web Manipulate the Web Eyes Ears Hands Mouth Feet Other Voice Haptic smell taste balance Semantic questions asked by UI expert Semantic comprehension required by UI
- 55. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 56 Discussion Challenges • Re-engineering the Semantics • Semantics-oriented syntax (HTML5 <nav>) not much used Outlook • How to get user interaction semantics annotated? • How to re-engineer the user interaction semantics? How to move away from single-use solutions towards Web way of working?
- 56. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 57 What‘s next?
- 57. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 58 Don‘t touch!
- 58. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 59 Talk-and-Gaze for Text Editing For everyone !
- 59. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 60 Usability Data Eco System Stakeholders • UI Experts • Test subjects • Users Data • UI Experts – Observations – Analyses • Test subjects – Gaze – Mouse – Voice – .... • Users – Mouse – .... feed answers Usability Data Mining
- 60. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 61 Thank you!
- 61. Steffen Staab: Eyeing the Web Keynote at WebIST-2019 62 Control R. Menges, C. Kumar, S, Staab. Improving user experience of eye tracking-based interaction: Introspecting and adapting interfaces. In: ACM Transactions on Computer-Human Interaction. Accepted May 2019. C. Kumar, R. Menges, S. Staab. Eye-Controlled Interfaces for Multimedia Interaction. IEEE MultiMedia 23(4): 6-13, 2016. C. Kumar, D. Akbari, R. Menges, S. MacKenzie, S. Staab. TouchGazePath: Multimodal Interaction with Touch and Gaze Path for Secure Yet Efficient PIN Entry. In: 21st ACM International Conference on Multimodal Interaction (ICMI 2019), Suzhou, Jiangsu, China. October 14-18, 2019. K. Sengupta, S. Bhattarai, S. Sarcar, S. MacKenzie, S. Staab. Leveraging Error Correction in Voice- based Text Entry by Talk-and-Gaze. Tech Report. R. Menges, C. Kumar, S. Staab. Eye tracking for Interaction: Adapting Multimedia Interfaces. Book chapter. IET. To appear 2020. Analysis R. Menges, H. Tamimi, C. Kumar, T. Walber, C. Schäfer, S. Staab. Enhanced Representation of Web Pages for Gaze-based Attention Analysis. In: Proc. of ETRA 2018 ACM Symposium on Eye Tracking Research & Applications. Warsaw, Poland, June 14-17, 2018. R. Menges, C. Kumar, S. Staab. What did my Users Experience? Discovering Visual Stimuli of Dynamic User Interfaces. Tech Report 2019. T. Walber, A. Scherp, S. Staab. Smart Photo Selection: Interpret Gaze as Personal Interest. In: Proc. of 2014 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI ‘14, Toronto, Canada, April 26-May 1, 2014. References
Editor's Notes
- #2: 60 minutes 45 slides For most people, the dominating sense they use to interact with the Web is their eye sight. However, web site providers do neither understand very well what users see when they look at their web site nor can they help the user to exploit her eye sight as an interaction modality. While eye tracking devices have become commercial commodity products, what has been lacking is the contextualized understanding of eye sight. Web browsers are almost unique software applications as they require an (almost) declarative representation of the user interface. We have engineered solutions that map eye traces and fixations onto interface regions and interaction widgets. In the EU project Mamem we have used such mappings to facilitate hands-free Web interaction, especially for motor-impaired individuals. In collaboration with the company EyeVido, we use these mappings to mine user viewing behavior. We envision that eye sight will become a major interaction modality for the Web in the future, but for this we also suggest that Web browsers remain transparent, declarative-oriented applications that are so accessible and modifiable that we can develop methods that involve all our senses.
- #3: Let‘s open a Web browser. What do we do first? We look, we watch, we explore visually. We may succeed with a plan we had or we might fail. Increasingly, therre is commodity technology to observe such eye-based interaction.
- #4: Augmented and virtual reality devices often feature eye tracking natively - Google has a patent on eye tracking for google glass Eye tracking also on smartphones: Not so successful Amazon phone Intriguing research by Gellersen on eye tracking for smart phones
- #8: Why should the Web community Why should AR/VR care? Why should the world care? Accessibilitly Standardization for delineating business logics from presentation Plenty of frameworks (too many, too many bad ones)
- #9: Why should the Web community Why should AR/VR care? Why should the world care? Accessibilitly Standardization for delineating business logics from presentation Plenty of frameworks (too many, too many bad ones) If I can make it there I'll make it anywhere
- #22: Screencast includes logging of all other data (gaze, mouse, etc.)
- #41: On the left Udi. He got involved in a traffic situation where he tried to help and another car hit him leaving him paraplegic a decade ago. He did a second course of study and now works as a designer, using 3D printing, working with a mouth-control. When I met him a year ago he was about to become father. Other people we have been working with suffer from neuro-muscular diseases or Parkinson. The computer is a very important instru