Surveys with Mixed Samples and Modes - David Dutwin, Ph.D.
1 like128 views
This document discusses the benefits and challenges of combining multiple samples and modes in survey research. It notes that while telephone surveys appear resistant to bias despite low response rates, costs continue rising and response rates dropping. Common reasons for mixed method designs include using appended samples, listed samples with incomplete coverage, or selecting specific geographic subsamples. However, combining probability and nonprobability samples can increase systematic nonresponse and variability in bias. Using multiple modes also poses issues as different modes may produce different levels of measurement error. The document discusses various weighting and calibration methods that can help adjust for sample and nonresponse errors when combining samples and modes.
Surveys with Mixed Samples and Modes - David Dutwin, Ph.D.
- 2. research. refined.Sometimes Hybrids Work…
- 3. research. refined.…and Sometimes They Don’t
- 4. research. refined. • Telephone surveys appear remarkably resistant to systematic bias, despite low response rates. • However, costs continue to rise and response rates continue to drop. Why Talk About Mixed Samples & Modes?
- 5. research. refined.Typical Drivers of Mixed Sample/Mode Designs
- 6. research. refined. • CDSF file • Enumerated/supplemented • Empaneled • Single/Dual Frame • Telephone Recontacts • Empaneled • Full Universe Samples • Internet Empaneled • In-Person w/ w/o Interviewer • Mail • Telephone w/ w/o Interviewer • Internet Samples & Modes
- 7. research. refined.Options Within Samples • Appended Sample from available sources • Listed sample of any kind with incomplete coverage • High incidence areas or other geographically selected subsamples • Internet could/should be considered incomplete at least due to coverage area • Etc.
- 8. research. refined. • Can have any combination of inbound phones, outbound phones, mail, and internet • A single sample frame Popular Mixed Samples/Modes Designs • 2 samples • 2 modes • 2 samples • 1 mode • 2 samples • 1 mode • multiple samples • 1 mode • multiple samples • 2 modes
- 9. research. refined.Problems of Combining Multiple Samples The weight of academic research continues to find in nonprobability samples 1) greater systematic non-response (bias) and 2) greater variability in the size of bias than in probability samples. One mode has more measurement error than another on variables of interest. However, in a single sample study, mode becomes a sample type, potentially confounding with measurement error.
- 10. research. refined.Problem 1: Probability & Nonprobability
- 11. research. refined.Problem 1: Probability & Nonprobability 0 5 10 15 20 Luth Web Unweighted Research Now Web Unweighted SSRS Omnibus Cell Phones… SSRS Telephone Omnibus… SportsPoll Telephone Unweighted NHIS Unweighted Luth Web Raked Research Now Web Raked Research Now Propensity Weighted Research Now Propensity Weighted… Research Now Matched Research Now Matched and Raked Luth Web Matched Luth Web Matched and Raked Cell Phones Raked SSRS Omnibus Telephone Raked SportsPoll Raked NHIS Weighted Race within Education Age within Education Region within Education Race within Age Education within Age Region within Age Age within Race Region within Race Education within Race Race within Region Age within Region Education within Region
- 12. research. refined.Problem 2: Multiple Modes • Two samples, different modes in each • One sample, sequential or concurrent mixed-mode offering • Multiple time points via panel or cross-sectional 1) Media-related factors: who has control over the Q and A 2) Factors influencing the information transmission: visual/auditory, verbal, nonverbal, paralinguistic differences and visual/graphical/layout 3) Interviewer effects: Privacy, locus of control, social desirability, etc. 1) Less social desirability 2) More missing data 3) Less extremeness (evidence weak) 4) Inconsistent effects with regard to recency/primacy
- 14. research. refined.Correction of Sample & Nonresponse Error
- 15. research. refined.Typical Problems to Solve • • • • • • • •
- 16. research. refined. Percent of Jews Percent of Sample Weight Client List 40% 65% 0.62 Name Targeted 10% 15% 0.67 Residual 50% 20% 2.50 • • • Standard Base Weighting for Selected & Residual Designs
- 18. research. refined.Propensity Modeling & Weighting
- 19. research. refined.Propensity Modeling & Weighting
- 20. research. refined.Propensity Modeling & Weighting
- 22. research. refined.Propensity Stratification Weighting
- 26. research. refined.The Alternative: Calibration • •
- 27. research. refined.Calibration Applications - DiSogra, 2013
- 28. research. refined.Calibration Applications - DiSogra, 2013
- 29. research. refined.Case Study 1: Centris Survey
- 30. research. refined.Case Study 1: Centris Survey
- 31. research. refined.Case Study 1: Centris Survey
- 32. research. refined.Case Study 1: Centris Survey
- 33. research. refined.Case Study 1: Centris Survey
- 34. research. refined.Case Study 1: Centris Survey
- 35. research. refined.Case Study 2: Adjusting for Multiple Waves/Recontact Samples
- 36. research. refined.Case Study 2: Adjusting for Multiple Waves or Recontact Sample
- 37. research. refined.Problem 3: Mode Effects
- 38. research. refined.Problem 3: Mode Effects • • •
- 40. thank you for your time