Improving Your Surveys and Questionnaires with Cognitive Interviewing
Download as PPTX, PDF2 likes747 views
The document discusses cognitive interviewing (CI) as a methodology to improve surveys and questionnaires, highlighting its similarities and differences with usability testing (UT). CI helps identify how respondents understand questions and the challenges they face, ensuring that survey questions accurately capture intended information. The paper emphasizes the importance of testing questions and provides guidelines on incorporating CI into the research process for user experience professionals.
Improving Your Surveys and Questionnaires with Cognitive Interviewing
- 1. 1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Improving Your Surveys and Questionnaires with Cognitive Interviewing Jean E. Fox Scott S. Fricker Jennifer Edgar U.S. Bureau of Labor Statistics UXPA 2018 June 28, 2018 All views expressed in this paper are those of the authors and do not reflect the views or policies of BLS.
- 2. 2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Thank you Scott Fricker Jennifer Edgar
- 3. 3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Agenda Surveys in UX work Why we need Cognitive Interviewing (CI) Background on CI How CI is similar to and different from Usability Testing (UT) So you can get started with CI More ways that CI can help!
- 4. 4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov What is a Survey? A series of questions given to respondents to collect information from them “Questionnaires” and “polls” have same purpose Systematic Ask the same thing to all participants Can get data from many people easily Self- or Interviewer-administered
- 5. 5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Surveys in UX Work We can use surveys On their own With other methods – Usability Testing – Card Sorting – Field Observations
- 6. 6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Standard UX Surveys Known UX surveys that have been tested Reliable Valid Sensitive Examples System Usability Scale (SUS) (Brooke) SUPR-Q (Sauro) SUMI, WAMMI (Kirakowski) QUIS (Norman)
- 7. 7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov BUT… If standard surveys can’t address our issues We write our own.
- 8. 8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Home-Grown Surveys Benefits— Target the information we want Easy to write (right??) BUT… Are you sure you are measuring what you want? How do you know? AND You know how to design interfaces, but you still run usability tests And you still find problems.
- 9. 9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Cognitive Interviewing Make sure you are measuring what you want to measure
- 10. 10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov So, What is Cognitive Interviewing? It’s kind of like Usability Testing for surveys. Participants answer survey questions You talk to them about their response process Identify problems with the survey But there are some important differences.
- 11. 11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Background on Cognitive Interviewing Developed in the field of Survey Methodology Goal is to learn how respondents: Understand the question Retrieve the information Decide what answer to give Respond to the question Problems can occur in any of these stages
- 12. 12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Types of Problems CI Can Uncover Complicated Instructions Confusing or vague question wording Jargon Inappropriate assumptions (e.g., the question doesn’t apply to your respondent) Unexpectedly sensitive topics Questions that are difficult to answer Answers that are difficult to recall
- 13. 13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Just to Give You an Idea… How many UX professionals are in your group? What is your answer? How did you determine your answer? Who did you include as a “UX Professional”? Exclude? How did you define your “group”? What does it mean to be “in” your group? What time frame did you include?
- 14. 14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Testing Overview – Both UT and CI Define research goals, create materials, etc. Recruit participants Conduct study Informed consent Complete task Debriefing Incentives Analyze data Make recommendations
- 15. 15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Testing Goals UT Uncover usability problems and potential solutions CI Ensure the question/survey is capturing the intended information
- 16. 16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Moderators UT & CI: Similar skills and training Work comfortably with all participants Interact in an unbiased manner React and adjust the test session as needed Analyze and interpret data
- 17. 17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov When to Test UT & CI Ideally, start early and continue through development stages Goals are different at different stages – Early – Test wireframe / concepts – Midway – Test features / questions – Late – Test overall product / survey
- 18. 18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Participants UT and CI Target specific populations, determined by research goals Typically use convenience samples UT Often 5-10 (per user group) Or more for a quantitative study CI Typically 20-30
- 19. 19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Moderating Sessions UT In-person or remote Moderated or unmoderated CI In-person or over the phone Almost always moderated
- 20. 20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Tasks Type of Tasks UT: Complete tasks using the product/website CI: Answer survey questions Scenarios/vignettes to provide context UT: You are a school counselor, and you want to know the education requirements for accountants. CI: Pretend this is your high school transcript, and answer these questions about your courses.
- 21. 21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Thinking Aloud UT and CI Information about participant’s processes, thoughts, and reactions Pros and cons, decide based on research goals
- 22. 22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Debriefing Questions / Probes To learn about participants’ thoughts and reactions Helpful for UT Essence of CI
- 23. 23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Debriefing Questions / Probes Can be done During the task – Get thoughts as participants are working – But this can interrupt their work Post-Task / Post-question(s) – Get feedback on specific tasks/questions Post-Test / Post-survey – Get overall impressions
- 24. 24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Goals of Questions / Probes During the Task UT and CI Explore unexpected behaviors, responses, and reactions Understand why participants are struggling
- 25. 25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Sample Questions / Probes During the Task UT and CI I noticed you hesitated, can you tell me what you are you thinking? What are you looking for?
- 26. 26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Goals of Post-Task Debriefing Questions / Probes UT Why participants did what they did during an individual task How they felt about their experience Scripted or spontaneous CI What participants were thinking as they answered a specific question or group of questions How they decided what to do Scripted or spontaneous
- 27. 27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Sample Post-Task Debriefing Questions / Probes UT How easy or difficult was the task? How confident are you that you completed the task? CI In your own words, what was the question asking? What does <term or phrase> mean to you? How did you decide on your response?
- 28. 28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Goals of Post-Test Debriefing Questions / Probes UT and CI Overall impressions about the product / survey What participants liked and didn’t like Any recommendations they have Scripted or spontaneous questions
- 29. 29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Data Collected Data Usability Testing Cognitive Interviewing Qualitative • Observations • Comments • Debriefing responses Quantitative • Task times • Success rates • Satisfaction ratings • Etc • Actual responses to survey questions, in context of the debriefing information
- 30. 30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Analysis for Both UT and CI Qualitative data – look for patterns and themes across participants Quantitative data UT: look at means by user group, use statistics as appropriate CI: review responses in context of the answers to the probes
- 31. 31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Outcome of Analysis UT Identify usability problems – Effectiveness • Tasks that are difficult to complete • Features that are hard to use – Efficiency • Activities that took too long to complete – Satisfaction Make recommendations – Ways to fix the usability problems
- 32. 32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Outcome of Analysis CI Identify problems with – Understanding the questions – Retrieving the information – Deciding what answer to give – Responding to the question Make recommendations – Rewording of questions – Rework skip patterns – Provide context for interpreting survey results.
- 33. 33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Getting Started with CI Whenever you are asking users questions, consider testing the questions first. For usability testing, use Cognitive Interviewing to evaluate your questions during the pilot test. For other surveys, consider starting with a review by your UX colleagues or actual respondents You want to know – Do they interpret the questions the way you want? – Can they answer the questions? – Do the response options make sense?
- 34. 34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov CI Can Do Even More for UX work! Evaluate web content, instructions, or other text products Can users understand it? Does it mean what you think it means? Evaluate other types of forms Do people know what goes in each field?
- 35. 35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Conclusions It’s important to be sure that our surveys are asking what we want them to ask. CI can help you do that What you know about UT already may make it easy for you to incorporate CI into your research To learn more about CI, see: Willis, G. B. (2004). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage Publications.
- 36. Contact Information 36 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Jean E. Fox Bureau of Labor Statistics Fox.Jean@bls.gov
- 37. 37 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Usability Testing Cognitive Interviewing What are the objectives of the methods? Identify usability problems and potential solutions Understand how respondents interpret survey questions and the challenges they face completing a survey When in development would you use the method? Anytime in development Who conducts the tests? Typically User Experience professionals, possibly with a formal coursework related to usability testing Typically Survey Methodologists, often with informal training in cognitive interviewing Who participates in the tests? Participants are from the target population(s) Smaller sample sizes for qualitative tests, larger for quantitative tests Generally aim for 20-30 participants
- 38. 38 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Usability Testing Cognitive Interviewing What methods are used? Participants complete tasks (answer survey questions) while their experience is observed in some way Participants complete tasks assigned to them, often with scenarios providing context for the task Participants complete surveys with their own responses In-person or remote testing Usually in-person testing Sometimes moderated Always moderated Can be done in lab or at respondent’s location May use think aloud What data are collected? Quantitative and/or qualitative data Generally qualitative data only Focus is on performance and user satisfaction Focus is on cognitive processes while completing the survey