Complexities of Practical Web Automation
0 likes326 views
This document discusses the complexities of developing practical web automation tools based on observations from user studies. It identifies challenges such as dealing with uncertainty due to the lack of standards in web technologies, maximizing user trust by ensuring user control and recoverability, and minimizing cognitive load. An example automation assistant is described that guides users step-by-step through tasks by suggesting actions and automating one action at a time with user confirmation. The conclusion notes some successes but that full automation has not been achieved due to uncertainties from lack of standardization, and that building trust and usability remain key challenges.
Complexities of Practical Web Automation
- 1. Yury Puzis, Yevgen Borodin, I.V. Ramakrishnan Complexities of Practical Web Automation Stony Brook University 2015 NSF Grant No. IIS-1218570
- 2. Contents Goal: help design practical web automation tools by sharing observational experience ❖ Human-Computer Interaction Perspective ❖ Technical Perspective ❖ Example: Automation Assistant ❖ Conclusion
- 3. Why Web Automation? ❖ Problem: non-visual browsing is hard ❖ It is hard (or impossible) to find relevant information and easy to become overwhelmed by what is irrelevant ❖ There are many shortcuts (gestures) to learn and hard (or impossible) to accomplish non-trivial tasks ❖ Web automation has the potential to enable visually impaired users to breeze through Web browsing tasks that beforehand were slow, hard, or even impossible to achieve
- 4. Observation User Environment Web Automation Tool Browsing Actions Events User Environment Web Automation Tool Events Automation Instructions Automation
- 5. Maximizing Trust ❖ Gaining and maintaining user trust is the cornerstone of web automation: even a few disasters is a big problem ❖ The user needs to know and influence what will happen (review, parameterize, choose) and what has happened (review, revert, recover) at all times ❖ Failure is inevitable and has to be graceful: terminate automation, ignore failed action, take corrective action, or suggest the user to take corrective action
- 6. Minimizing End-To-End Cost ❖ Cognitive load and operation time must be end-to-end lower when using automation than otherwise ❖ Web automation costs: managing creation, execution and consequences of automation; context switching ❖ Screen-Reader and browser costs: many and are well known, including the need to plan complex sequences actions by memory or execute exhaustive search and guess, guess, guess ❖ In conflict with the need to maximize trust
- 7. Dealing with Uncertainty Goal: automate user intent without resorting to handcrafting scripts (programming), interpret environment reaction Problem: we can only guess ❖ Semantics of user browsing actions ❖ Semantics of environment events ❖ Semantics of webpage elements
- 8. Making Observations ❖ Goal: make meaningful observations from events ❖ Problem: browsing actions can trigger multiple (including cascading) events, and there are different types of events: e.g., shortcut press -> JavaScript call -> DOM mutation -> virtual cursor movement ❖ Problem: over time, an event may change its semantics (same event - different results) or implementation (different event - same results)
- 9. Addressing Webpage Elements ❖ Goal: identify target webpage element ❖ Problem: most addressing approaches are designed to query DOM for elements at the specified address, but we need to query DOM for address of the specified element ❖ Solutions: sloppy programming, machine learning, etc. but no unbreakable approaches exist
- 10. Detecting Action Completion ❖ Goal: wait for action to complete (succeed or fail) before continuing to interact with the user & the environment ❖ Problem: no standard way to specify action completion; cascading, asynchronous and scheduled JavaScript events make things harder ❖ Solutions: listen to all relevant JavaScript events through callback functions; timeout; wait for predefined DOM mutations / value changes (success or failure)
- 11. Example: Automation Assistant ❖ Observes everything the user is doing (no macros) ❖ Guides the user through browsing tasks step-by-step ❖ suggests several alternative browsing actions based on user’s prior actions ❖ automates only one action at a time ❖ each set of suggestions is explicitly requested, each action is explicitly chosen, each outcome is reviewed ❖ No context switch between automation and screen-reading Puzis Y., Borodin Y., Puzis R., Ramakrishnan I. V., Predictive Web Automation Assistant for people with vision impairments. WWW '13.
- 12. Conclusion ❖ There are some successes but automation is not there yet ❖ The biggest technical challenge is uncertainty which stems from lack standardization ❖ The biggest HCI challenges are building trust and keeping things “cheap” ❖ The HCI aspect of this talk is, to a large extent, applicable to all automation tools, not just web automation. It is also applicable to all users not just the visually impaired users (think handheld, wrist devices)
- 13. Thank You!