3a8 picture driven computing in assistive
1 like492 views
The document discusses using picture-driven computing as an assistive technology to improve accessibility. It describes using visual images instead of text inputs to allow indirect access for users with disabilities. The Sikuli platform is highlighted as a way to automate tasks by programming screenshots of graphical interfaces. Future work areas include more research on picture-driven approaches, evaluating Sikuli, and developing additional assistive technology scripts.
3a8 picture driven computing in assistive
- 1. PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY Silvio Bonfiglio1 - George Kourousias2 1. FIMI, a Barco Company, Saronno (Italy) 2. Synchrotron Radiation Facility, ELETTRA, Trieste (Italy) 1st International AEGIS Conference 7-8 October 2010, University of Seville www.aegis-project.eu
- 2. Domain: Improved Accessibility in ICT Why? It may positively affect and improve our society PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
- 3. Two Paths to Accessibility DIRECT ACCESS Universal Design (UD) – Design for All i.e. a product that – without add-ons – can be experienced with similar ease of use by any user independently by the disability. INDIRECT ACCESS Multi-modal, Personalized Accessible Design i.e. a product that is made accessible for a specific category of users by incorporating as an add on a specific AT component or an extra interaction modality. PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
- 4. Picture Driven Computing We define as Picture-driven Computing the Computing paradigm that has as core Data Abstraction the Visual Information in a 2- dimensional spatial domain. This Visual Information is what intends to be visible on a graphical Output device. •We aim to introduce it as an Assistive Technology •Adding a Modality to the GUI of the computer •Accessibility through Indirect Access PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
- 5. Example: a Picture-Driven Search •Using Picture objects instead of keywords PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
- 6. Sikuli, a Picture-Driven language •Developed recently at MIT by T. Yeh, T.H. Chang and R.C.Miller for the search and automation of GUI elements using screenshots •Based on Sikuli-script , a scripting module that allows programmers to use screenshots of GUI elements to control them programmatically without any API. •Python •Open Source & Multiplatform •We use it as our main paradigm for Picture-Driven Development of AT PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
- 7. Sikuli: How it Works A. a computer vision library B. a mechanism to programmatically “dictate” Input actions C. a mature underlying language (Python) for traditional programatic control PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
- 8. Sikuli: An Example Goal: clicks all the “minimize window” buttons on the screen Picture-Object Input: target image of the button i) then the systems computer vision (A) will locate the x,y position of that on the screen ii) (B) the “mouse control” will “send” the mouse on x,y, and iii) through a for or while loop (C) will click (B) for all the targets that (A) has found. PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
- 9. Sikuli as an Accessibility Design enabler We Identify the following 3 classes: Automation Scripts: where Sikuli script - following a predefined set of mouse and keyboard actions- preforms a task that may be difficult otherwise to the user with a disability. Event Raisers: where the system can keep monitoring the user's screen for specific graphical objects and raise an event/alarm and consequently trigger a set of programmatic actions. Bridge Systems: where applications can communicate with other software even if an API is not provided. + Hybrid approaches PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
- 10. Future work •Further research on Picture-driven Computing. •Comparison of “indirect access” Accessibility Design versus the “direct” one (Universal Design) for the computer as an AT-aid. •Additional development of Sikuli AT Scripts. •Evaluation of the technical issues and performance of Sikuli. PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
- 11. Acknowledgments
- 12. further info: silvio.bonfiglio@barco.com george.kourousias@elettra.trieste.it PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville