Rss Oct 2011 Mixed Modes Pres1
- 1. How and when does the mode of data collection affect survey measurement? Annette Jäckle (University of Essex) Pamela Campanelli (Independent Survey Methods Consultant) Peter Lynn (University of Essex) Gerry Nicolaas (National Center for Social Research) Steven Hope (University College London) Alita Nandi (University of Essex) RSS 11 Oct 2011
- 2. Background Existing guidelines for designing questionnaires for use in different modes US Census Bureau 2007 Dillman 2000
- 3. E.g. Census Bureau guidelines (2007) Aim meaning and intent of the question and response options must be consistent across modes. Some differences between modes in question wording may be necessary to collect equivalent information But differences should be minimized and tested
- 4. Census Bureau guidelines (2007) ctd. 30 specific guidelines regarding Q wording and instructions Examples Response categories Formatting of answer spaces Visual design elements Question order and grouping Showcards Prompts and help
- 5. Census Bureau guidelines (2007) ctd. Examples: 1. Maintain the same wording of questions across modes. 6. The underlying response task posed by a question should be consistent across modes 12. Use identical response categories ...
- 6. Census Bureau guidelines (2007) ctd. But: Even if Q wording, response options, task etc are the same across modes Mode effects sometimes occur Why? Which questions at risk?
- 7. Project aims Effect of mode on measurement Which survey Qs are at risk? generalisable features of questions / modes variable specific mechanisms Practical advice for improving question portability across modes Funded by ESRC-SDMI
- 8. Today Background 1. Framework, design of experiments 2. Cognitive interviewing follow-up study Empirical results: the role of.... 3. ...visual and aural stimuli 4. ...the interviewer 5. ...mode specific question formats Implications 6. Questionnaire design for mixed mode surveys
- 9. Framework: causes of mode effects Reporting situation Perception of risk: Extent of Interviewer Privacy of reporting Legitimacy of survey Willingness to disclose? involvement ► Social desirability bias I-R interaction Comprehension Retrieval Judgment Response Depth of cognitive processing: Information available to R: Sufficient effort? How is Q processed? ►Satisficing ►Response effects Respondent Respondent motivation Task difficulty Context information ability x Importance of survey x Interviewer characteristics Reporting situation Cognitive Non-verbal communication Non-verbal communication •Time pressure demands Visual layout Interviewer probing •R distraction of Q Sequential/simultaneous Qs Extent of Interviewer Respondent control involvement Visual/aural stimulus Q format over Qaire
- 10. Hypotheses: survey-level Interviewers motivate respondents to make required effort ⇒ Less satisficing Interviewers can help with difficult tasks ⇒ Less confusion / fewer errors Visual stimulus helps cognitive processing ⇒ Less satisficing / less confusion with visual Format effects versus mode effects ⇒ No mode differences if same format
- 11. Hypotheses: item-level Extent and nature of item-specific mode effect depends on Q characteristics: Question format Inherent difficulty (content, wording) Sensitivity Type (attitude, behaviour, other factual, ...) Level of measurement (ordinal, nominal, ...) Response format (yes/no, frequency, ...) Specific hypotheses in papers 3, 4, 5
- 12. Experimental design I Split ballot question format experiments Long vs. short list Agree/disagree vs. forced choice Ranking vs. rating Full vs. end labels Showcard vs. no showcard Branching vs. no branching Yes/no vs. code all that apply
- 13. Experimental design II Each Q format experiment with several items which varied by Inherent Q difficulty (content, wording) Sensitivity Type (satisfaction, other attitude, behaviour, other factual) Measurement level (ordinal, nominal)
- 14. Experimental design III Follow-up to NatCen Omnibus and BHPS Great Britain population aged 16+ Restricted to respondents with web access Random allocation to CAPI, CATI, CAWI
- 15. Response rates and sample sizes Sample sizes after exclusion of non-internet access or use cases from CAPI and CATI samples. * Not available yet.
- 16. Today Background 1. Framework, design of experiments 2. Cognitive interviewing follow-up study Empirical results: the role of.... 3. ...visual and aural stimuli 4. ...the interviewer 5. ...mode specific question formats Implications 6. Questionnaire design for mixed mode surveys