Benefits of Concurrent Cognitive and Usability Testing
Download as PPT, PDF1 like901 views
The document discusses the benefits of concurrent cognitive and usability testing using case studies from the U.S. 2010 Census, highlighting separate and joint testing methods. It outlines the effectiveness of different pretesting approaches to improve questionnaire design, respondent understanding, and interviewer performance. Recommendations emphasize the importance of conducting these tests simultaneously to enhance question wording, visual design, and training for interviewers.
Benefits of Concurrent Cognitive and Usability Testing
- 1. Benefits of Concurrent Cognitive and Usability Testing Jennifer Childs, Jennifer Romano, Erica Olmsted-Hawala, and Elizabeth Murphy U.S. Census Bureau Paper presented at the Questionnaire Evaluation Standards Conference Bergen, Norway May 17-20, 2009 1
- 2. Overview • Background – Cognitive and Usability Labs • Case Study – U.S. 2010 Census – Example 1 – Separate Cognitive and Usability Testing – Example 2 – Joint Cognitive and Usability Testing • Conclusions 2
- 3. Description of Pretesting Methods • Cognitive Testing • Usability Testing – Focus on respondent’s – Focus on user’s ability to understanding of complete the survey questions • User = Interviewer – Some focus on • User = Respondent navigation – Focus on users’ – Often used to interaction with determine final questionnaire and on question wording effects of visual design – Used to improve visual design and navigational controls 3
- 4. U.S. Census Bureau Lab Structure • Cognitive Lab • Usability Lab – Psychologists, – Psychologists, human sociologists, factors and usability anthropologists, specialists demographers, etc. – Based on principles of – Based on CASM and user-centered design Tourangeau’s 4 stages (UCD) – Measures – Measures accuracy, comprehension, efficiency (time to accuracy and ability and complete tasks), and willingness to respond satisfaction 4
- 5. Census Bureau Lab Techniques • Cognitive Lab • Usability Lab – Concurrent think aloud – Random assignment (as – Concurrent and/or appropriate) retrospective probing – Scenarios (hypothetical – Retrospective situations) debriefing – Concurrent think aloud – Emergent and – Concurrent and/or expansive probing retrospective probing – Vignettes (hypothetical – Eye tracking situations) – Satisfaction – In-depth ethnographic questionnaire interviews – Retrospective debriefing 5
- 6. Case Studies: United States 2010 Census
- 7. U.S. 2010 Census • Approximately 10 questions – Names, relationships, ages, sex, race and Hispanic origin for each household member • Mailed forms to most households • Non-response follow-up by personal visit 7
- 8. U.S. Census Nonresponse Followup (NRFU) • Developed and tested CAPI instrument – Usability and Cognitive testing independently conducted – Example 1 • Turned to PAPI data collection – Usability and Cognitive testing conducted concurrently – Example 2 8
- 9. Example 1: CAPI Instrument Testing • Usability testing and cognitive testing conducted separately
- 10. CAPI Usability Testing Methods • Early version of software • 2 rounds • Round 1 – 6 users (4 English and 2 Spanish speakers) • Round 2 – 5 users (4 English and 1 Spanish speaker)
- 11. CAPI Cognitive Testing Methods • 4 Rounds (2 English and 2 Spanish) • English Round 1 – 14 interviews – Paper script • English Round 2 – 16 interviews – Computer • Spanish Rounds 1 and 2 – 30 interviews total – Paper script
- 12. CAPI Usability Findings (1) • Data Entry – The way the cursor moves between entry boxes • Navigation issues – Next and Back button navigated between “questions” not “screens”
- 13. CAPI Usability Findings (2) • Question Text – Repeating questions for each household member – Unclear whether or not to read examples associated with question text – Unclear if verification for sex was ok – Identified instances of problematic question wording • Interview Tasks – Administering flashcard was difficult for interviewers – Difficult fills – e.g., Is this (house/apartment/mobile home)
- 14. CAPI Cognitive Findings • Navigation Issues – “Back” and “Next” buttons • Question Text – Repetitive to read each question for each person – Reference period unclear – Long, complex questions – Problematic question wording • Interviewer Tasks – Difficulty administering flashcard – Difficult fill, e.g., (house/apartment/mobile home)
- 15. Separate Usability and Cognitive Testing Conclusions • Separate – Unique results – Common results – Areas of expertise and knowledge of relevant research • Together – Fully understand how to best resolve problems
- 16. Example 2: Joint Cognitive and Usability PAPI Testing • Paper and Pencil Instrument • 40 cognitive interviews • 20 usability sessions • Concurrent testing led to development of joint recommendations 16
- 17. Methods • Cognitive Testing • Retrospective probing • Comprehension, accuracy, ability to answer questions given personal situation • Usability Testing • Shortened interviewer training • Test scenarios • Accuracy, satisfaction and ease of use 17
- 19. Information Sheet 19
- 20. Information Sheet Findings Cognitive Findings Respondents understood and were able to use the Information Sheet for all relevant questions Usability Findings Interviewers were successful in administering the information sheet and associated questions 20
- 21. Hispanic Origin Question 21
- 22. Hispanic Origin Findings • Cognitive Findings – A lot of difficulty for Hispanic respondents • Responding “Latino” or “Spanish”– without a country of origin • Describing race and origin • Unable to respond unassisted – With probing by interviewer, able to provide a code- able answer (note: the dash is yellow- I must have added it last time- so check the other slides too…) • Usability Findings Dominican Puerto Scenario Columbian Cambodian Mexican Republic Rican Success 95% 95% 100% 95% 100% Rate 22
- 23. Hispanic Origin Discussion and Recommendations • Respondents had some difficulty, but… • Interviewers in usability testing were able to successfully navigate the question. • Recommendations focused on training 23
- 24. Conclusions from Joint Cognitive and Usability Studies • Understanding of how: – Respondents react to questions – Interviewers react to questionnaire – Interviewers react to respondent situations • Recommendations: – Improve the form – question wording, visual design, and/or navigational instructions – Improve interviewer training 24
- 25. General Recommendations Conduct cognitive and usability testing concurrently with early versions of the questionnaire, with time to change: – Question wording – Visual Design (i.e., format and layout of the questionnaire) – Navigational strategies – Interviewer training Conduct iterative testing 25
- 26. Future Research • Cognitive and Usability testing of American Community Survey Internet data collection instrument – Early in development cycle – Iterative testing 26
- 27. Thank you! Questions or Comments? E-mail: jennifer.hunter.childs@census.gov 27
Editor's Notes
- #4: Under Usability Testing, it could be argued that the focus is on both the “look and feel” of the form, i.e., the visual design and the ways in which the various controls behave.
- #5: Note that I added some text under Usability Lab for you to consider. We do not measure how “intuitive” a user-interface is because there is no defined unit of intuitiveness. We measure accuracy, efficiency (time to complete tasks), and satisfaction with the user interface. Accuracy and efficiency together represent the user’s success in completing the survey. So we measure success and satisfaction, and we diagnose what is was about the design that led to something less than success and satisfaction. We find that a problem with Tourangeau’s model is that people don’t always read the question. They often look first at the response options, and they infer the question. People’s goal with an online survey is to finish it as quickly as possible and get back to their lives. Usability recommendations are grounded in what we know about human capabilities and limitations in attention, perception, memory, learning, expectations, problem solving, decision making, i.e., everything we know about human cognition.
- #20: If this is where the lists came from, it should be presented earlier. I was thinking the lists were on flash cards.