Conducting accessibility user research: What’s really needed
- 1. Conducting Accessibility User Research: What’s Really Needed Michele A. Williams, PhD Senior UX Researcher - Accessibility, Pearson Technica11y.org | May 1, 2019
- 2. WHAT’S REALLY NEEDED: 1. Knowing the basics 2. Accessible artifacts 3. Accessible testing 2
- 4. Understand disability Learn the diversity and manifestations of disability Photo Credit: International Labour Organization 4
- 5. Know the laws & standards Web & Applications Section 508 Web Content Accessibility Guidelines (WCAG) Hardware Section 508 Americans with Disabilities Act (ADA) Voice Section 508 21st Century Communications and Video Accessibility Act (CVAA) Note: this information is primarily U.S.-based 5
- 6. Get immersed Assistive Technology Centers Government- sponsored statewide centers focused on matching clients to assistive technology Disability Organizations Disability-specific national & local organizations Local library Meetup.com Email lists/groups Industry Professionals Examples: Orientation & Mobility Specialists Assistive Technology Specialists Rehabilitation Specialists 6
- 7. Know the current state Related Work Search for accessibility publications such as ACM’s ACCESS or higher education’s AHEAD The Competition Note accessibility from similar or competitor products Your Baseline Employ an accessibility consulting firm to perform a formal audit of your product 7
- 8. User testing is not an audit You are responsible for testing accessibility, your participants are testing usability 8
- 10. Research occurs at all project phases 10 Abstract Concrete Ideas ----- Sketches ----- Prototypes ----- Product Project Phases & Artifacts What’s needed to conduct successful inclusive research with these artifacts?
- 11. Considerations for your testing artifact To use your artifact, does it require…? ▸ Clear vision ▸ Sharp hearing ▸ Talking ▸ Walking ▸ Staying focused ▸ Grasping objects If so, start to pivot… 11
- 12. Ideas – No artifacts to test This is more concerned with method which we’ll visit in Section 3 12
- 13. Sketches – Go beyond “paper” Web & Application Leverage accessible technologies such as Microsoft Office Suite to increase compatibility with assistive technology Hardware Use inexpensive 3D objects that users can manipulate and see such as Play- Doh, Styrofoam, blocks, and 3D prints Voice Leverage free text- to-speech technologies to build out your idea 13
- 14. Prototypes – Getting close to real Web & Application Start creating “throw away” code Be careful of “prototyping” tools; they won’t yield accessible products Hardware Use the closest equivalent, e.g., GoPro camera, iPad, computer monitor Voice Leverage devices that create realistic voices Use existing accessible telecommunication devices 14
- 15. Wizard of Oz “Fake it until you make it” testing method Users think the system is real, researchers create the illusion 15
- 16. WOZ Example 1: Telephone IVR User dials a desk phone; designer acts out script Provides interface validation before production 16
- 17. WOZ Example 2: Alexa Skill User talks to speaker; developer plays the files Can determine user grammar and keep study under control 17
- 18. Product – The real, full artifact Web & Application WCAG or Section 508 compliant Hardware ADA compliant Voice CVAA compliant Caveat: If your system is not accessible, work with other methods and an audit until you can get there; be cautious of bringing in users to test broken systems 18
- 19. What the user is testing needs to be accessible Take the extra steps to seek out (or advocate for) accessible prototyping tools 19
- 22. Methods: Biometric & Card Sort Biometric Testing Ensure equally thorough alternative collection for participants who can’t use biometrics Ensure equal representation and triangulation of data from biometrics and other sources Card Sort & Tree Test Use an accessible (WCAG- compliant) tool or leverage accessible drag-and-drop examples to build a system 22
- 23. Methods: Diary & First Click Diary Study Use multiple collection methods – Word doc, email, voicemail, video, etc. Note, blind people can and do take photos First Click Test Needs accessible (WCAG- compliant) webpage, not image of page Must detect keyboard events and mouse clicks from assistive technology such as voice dictation software 23
- 24. Methods: Focus Group & Observations Focus Group Ensure this is appropriate for the participant group and everyone can contribute Consider alternatives such as live chat or group forum for those with speech impairments, delays, or anxieties Observations If following along in public, note you will be mistaken for an aide Consider instrumenting participant with video rather than being present 24
- 25. Methods: Participatory Design & Online Surveys Participatory Design If creating artifact on behalf of participant, get explicit confirmation idea was properly conveyed Consider leaving artifact with participant then interviewing them after time to reflect Online Surveys Use accessible (WCAG- compliant) survey tool Minimize use of complex question types Carefully word demographic questions; not everyone identifies as having a “disability” 25
- 26. Methods: Usability Testing Usability Testing Be mindful that participants don’t want to look helpless Unmoderated testing is feasible but leaves out valuable data that can be educational for teams, e.g., recording troubleshooting techniques and general assistive technology use Time on task may not be applicable for a diverse population, but may apply if using an exclusive population SUS and Ease of Use measures may need to be separate; for instance, a system may be difficult for blind users but not others 26 Synopsis of Research Methods from User Zoom
- 27. A Few Notes About Communication Use Interpreters if Needed Determine if sign language interpreters are needed; if so, ask for recommendations and offer to pay Determine if a caregiver can relay messages from someone with a speech impairment Be Prepared to Type Determine if typed versus spoken interactions will be easier or necessary 27
- 28. Recruiting Techniques for finding the right participants
- 29. Two ways to recruit Relationship Building Use same resources for learning accessibility (Slide 6) to get to know the communities they serve Downside: May limit pool of participants Recruiting Firm Contract with professionals or disability organizations that have a database of users, e.g., Knowbility’s Access Works Downside: May not have a specific demographic 29
- 30. Reality of recruiting Minority Group Implications Ideal participants may not be involved in organizations or signed up for research; try reaching out to key stakeholders, though this will take more time People are still breaking barriers in some areas (e.g., Haben Girma) Use Proxies Consider using older adults as proxies for your studies Use people who meet the disability requirements and can sit in for other requirements; example, participants who are not currently enrolled in school to test college-level course material 30
- 31. Logistics Consider details from start to finish
- 32. Paperwork & Payments Consent Form Use email or verbal consent, do not require a signature Ensure the form is readable with assistive technology Ensure the wording is not too legal or hard to understand Ensure participants can give their own consent Payment System Ensure compensation systems (such as gift card payment systems) are accessible Attempt to compensate logically (that is, no payments to services participants can’t use) 32
- 33. Technical Considerations Tech Check For remote testing, have participants join the remote system days prior to testing and simulate computer use to work out interruptions Recommend using Zoom as many user testing tools will not be accessible Equipment Needs Determine if participants will bring and use their own equipment or will you supply it (recommend they use their own if possible) 33
- 34. Physical Spaces Meeting in a Lab Provide street-to-door directions Ensure space is ADA- compliant & transit-friendly (or provide compensation for transportation) Communicate exact meeting point Ensure space will not cause participant anxiety Meeting in Public or Home Ensure and agree the public space is conducive to research Determine and communicate boundaries if conducting study in-home Compromise: Meet at common space such as public library where go for other meetings 34
- 36. Conduct Appropriate Analysis Interpret Findings Need to understand how participants intend to use the system, especially if using assistive devices or technology (AT) E.g., web issues can be due to the design, novice experience with AT, incorrect code markup, an AT defect, a browser bug, or operating system issue Situate Findings Participants with disabilities will likely be a smaller group in inclusive studies, especially when segmented by disability Determine the weight and significance to give to findings and validate them with additional studies or related work as appropriate 36
- 37. Provide Mindful Reporting Add Disability Details Include background on accessibility to level-set reviewers, including videos Add applicable standards to reports (e.g., WCAG) Ensure the report is accessible, use it as an example for others Report to All Stakeholders Findings will relate to designers, developers, engineers, testers, product managers, project managers, everyone! The entire team must be involved in making an accessible system and thus share your findings and experiences with everyone 37
- 38. Use Research to Help the Team Research is Education Use research to educate the entire team/company on accessibility through first-person interactions Create powerful videos from the research to help points resonate with stakeholders Directly Connect Stakeholders Bring users in for large group testing (“user nights” concept) or take teams on field trips to connect them to users with disabilities directly Invest in training to ensure stakeholders know how to act on what you’re sharing 38
- 39. Accessibility user testing needs accessible tests The protocol and facilitation details will make or break the data collection 39
- 40. Closing Thoughts
- 41. In Summary ▸ Inclusive testing takes an ecosystem: ▹ Designers and Developers must know how to build accessible systems (e.g., WCAG- compliant) ▹ Researchers must know how to facilitate research and interpret findings ▹ Testers must know how to test for accessibility and how users use assistive technology ▹ Product Managers need to prioritize accessibility ▸ Ultimately to be successful, everything the participant interacts with needs to be accessible: consent form, location, tools for study, artifacts being tested, payment 41
- 42. Related Work
- 43. Additional Resources ▸ Prior Research found under “Publications” on LinkedIn ▹ "Pray before you step out": Describing personal and situational blind navigation behaviors ▹ Reading, writing arithmetic: Experiences of US students with disabilities ▹ The question still remains: How do I know you know accessibility? ▹ What not to wearable: Using participatory workshops to explore wearable device form factors for blind users ▹ Collaboratively designing assistive technology ▸ User Research with People with Disabilities: What you Need to Know (Peter McNally; UXPA Boston) ▸ A large user pool for accessibility research with representative users (Marianne Dee, Vicki Hanson; ACM ASSETS) ▸ Knowbility Access Works participants 43
- 44. Presentation Template “Arviragus” from SlidesCarnival 44