A best practices approach to cognitive interviewing

Editor's Notes

• #5: These unique attributes pose a challenge. The challenge is…
• #6: How do we integrate these unique facets of CI with a quality approach to the research design in order to arrive at valid data? And that is important because I would suggest that CI ultimately mitigates survey measurement error to the extent that it is done well.
• #7: What I am suggesting is not new.
• #8: What I am presenting today is the TQF It is not prescriptive but rather a way to…. Today, I am asking you to think about many different issues relevant to data integrity & the validity of your outcomes. I realize that some of these best practices may fall outside the scope of your reality when conducting CI BUT they are something to think about – to question yourself – related to these issues of quality. My comments today relate to the face-to-face (in-person or webcam) CI mode, not web probing.
• #11: 9
• #12: One of the considerations within Scope: How many interviews should I complete? I know, CI typically uses small samples (5-15 or more) & are conducted in a short time frame…HOWEVER, these Qs GO BEYOND WHETHER YOU ARE IDENTIFYING NEW PROBS.
• #13: 11
• #14: RESEARCHER & PARTICIPANT EFFECTS are mitigated by using a qualitative interviewer – not survey field interviewer – who is trained on interviewing skills such as
• #15: Mitigating RESEARCHER & PARTICIPANT EFFECTS also has to do with the interviewer’s probing skills. Probing uncovers the underlying meaning of the participant’s reactions/comments. In the broadest, simplest sense, probing skills boil down to: When How and What WHEN: Concurrent, in-the-moment probing is a best practice because it mitigates the participant burden & error associated with memory/recall, and is more conversational which helps build rapport which is a key ingredient in the participant-researcher relationship mentioned earlier HOW: Proactive probes + with scripted AND spontaneous questions are a best practice. + scripted probes bring a systematic approach to CI which adds consistency across interviews which fosters an apples-to-apples comparison at the end. - But error because the anticipated probes may miss important input. + spontaneous probes account for unanticipated behavior or comments, gives interviewer FLEXIBILITY WHAT Comprehension: “WHAT IS THIS Q ASKING?” What is your understanding of this Q? We are asking the participant to paraphrase. Be sure interviewer understands the meaning of the paraphrase & how participant derives meaning, e.g., Q wording, structure (format), & ordering (context). Recall: The probe “Is this Q easy or difficult to answer?” is NOT the right probe. That is NOT what you want to know. The participant’s response is based on an unintended understanding of the Q. Learn from “How did you arrive at your answer?”
• #16: And be careful how you write probing Qs….
• #17: Another important skill of the qual interviewer is the ability to recognize & handle a variety of participant types….. This last bit re: the think-aloud technique speaks to the importance of allowing practice prior to the onset of the CI to reduce participant burden. The shy participant situation also highlights the power imbalance that can occur in the one-on-one interview environment. In addition, there is also the participant who becomes fixated on giving an answer rather than their understanding of what the Q is asking, needing the interviewer to rebrief or redirect the P.
• #19: Resources = Time & money Best approach = Is it the most efficient approach? What will be lost? Gained? Will you lose context & meaning? Making sense–Have we identified all of the problematic areas in the items being tested AND the root causes of problems Across Qs: Not always testing a full set of questions,. May be more evident in piloting? However, in the Debt Stress Index questions we tested two orders – asking about Debt first, and then Debt Stress, and asking about Debt Stress first and then Debt (as we knew from the survey work that people had different acknowledgement of debt depending on the order.
• #20: 18
• #22: Resources may preclude the use of thick description – and it may not be needed – BUT a best practice is to provide details of the CI process so that users of the CI results can understand what you did and determine their confidence in the results and therefore its usefulness and its applicability to other similar survey work. BTW, having clients/sponsors actually attend and observe the CIs can be effective in helping them understand what you communicate in the report.
• #23: It will differ for each study & situation. Failure to interview all sampled: Replace with similar because the sample is so small there is too much of risk we miss stuff by not having them included
• #24: All of this serves to define the Usefulness of your research.
• #25: Does my research..... Help to reduce survey measurement error Confirm or deny what we already knew about the strengths/weaknesses of the survey design & participants Allow us to use this new knowledge in other survey contexts & comparable participants
• #27: To meet this challenge, THE TQF IS NOT PRESCRIPTIVE, RATHER IT FOSTERS ..... For each phase of the research process – Data collection, analysis, reporting, & outcomes