Using the web to survey to survey the general population
- 1. 1 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Paste co- brand logo here Gerry Nicolaas ESSnet workshop, 4-5 September 2014, Wiesbaden, Germany “Data Collection for Social Surveys using Multiple Modes” 05/09/2014 Using the web to survey the general population: Conclusions from the GenPopWeb network
- 2. 2 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereContents • What is GenPopWeb? • The UK context • Barriers to using web for random probability surveys • A Research Agenda • GenPopWeb recommendations for the UK
- 3. 3 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here 3 What is GenPopWeb? • GenPopWeb is a network of survey researchers, practitioners and commissioners who explore how the web can be used to survey the general population • The network aims to: • exchange knowledge, • generate new thinking, and • work towards resolving the challenge. • The 2013 events of this network were funded by the National Centre for Research Methods. • In September 2013, the network was integrated within the European network WEBDATANET (COST Action IS1004)
- 4. 4 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here 4 Acknowledgements • The investigators and their institutions: Lisa Calderwood (CLS), Peter Lynn (ISER), Gerry Nicolaas (NatCen), Caroline Roberts (University of Lausanne) • All authors, presenters and event participants • The UK Core Group: Bill Blyth (TNS Global), Mario Callegaro (Google UK), Ed Dunn & Laura Wilson (ONS), Rory Fitzgerald (City University London), Joanna Lake (ESRC), Carli Lessof & Joel Williams (TNS BMRB), Nick Moon (GfK NOP), Patten Smith (Ipsos MORI), Professor Patrick Sturgis (NCRM), Joe Twyman & Michael Wagstaff (YouGov UK)
- 5. 5 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereWhat has GenPopWeb done? • Events: • Opening conference in February 2013 • Workshop in June 2013 • Three sessions on this topic at ESRA conference, Ljubljana, July 2013 • WEBDATANET meeting, Reykjavik, September 2013 • Final meeting with UK Core Group in January 2014 • Outputs: • 3 synthesis papers on (1) coverage & sampling, (2) participation & engagement, and (3) measurement issues • Various presentations from UK & international experts • An overview of omnibus surveys & web panels in UK (Carlos Lagorio) • Final report & recommendations for the NCRM • Social and digital media: website, twitter, NCRM podcast
- 6. 6 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here 6 Paste co- brand logo here www.natcenweb.co.uk/genpopweb/ @GenPopWeb
- 7. 7 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereThe UK context No population register Dominant data collection mode for high quality surveys in the UK is face-to-face Financial challenges have made web a desirable mode for surveys Internet access • 84% of households have access to the internet (ONS, 2014) • Web is becoming mobile • 58% of UK adults access the internet on their mobile phone (ONS, 2014) • 44% of UK households own at least one tablet (Ofcom, 2014) Rapid growth of web panels • None of which use random probability sampling methods Limited use of web for random probability samples of general population • Mainly longitudinal surveys (e.g. birth cohort studies, LSYPE) • Some experimentation on x-sectional surveys (e.g. LFS W1, ESS, Community Life)
- 8. 8 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereBarriers to using web for random prob surveys of gen pop 1. Coverage of general population 2. Sampling 3. Participation 4. Measurement 5. Other
- 9. 9 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereBarriers to using web for random prob surveys of gen pop 1. Coverage of general population • Exclusion of offline households (16%) • Under-representation of older age groups • Under-representation of lower socio-economic groups • Possible solutions • Provide internet access (costly) • Mixed mode: offer alternative mode (e.g. concurrent, sequential, adaptive/responsive) • Exclude off-liners
- 10. 10 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereBarriers to using web for random prob surveys of gen pop 2. Sampling • Dependency on Postcode Address File (PAF) for sampling of general population • No names, telephone numbers or email addresses • No information on number of households and individuals at address • 1st contact has to be either face-to-face (costly) or postal (self-selection bias, low rr) • On the whole Random Digit Dialling is not considered for high quality surveys in UK • Difficult & costly to include mobile-only (15% in UK) • Low response rates
- 11. 11 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereBarriers to using web for random prob surveys of gen pop 3. Participation • Web response rates are much lower than comparable surveys using other modes: • ESS experiment: 21% for web compared to 55% for F2F (with incentives) • Community Life experiment: 19% for web compared to 60% for F2F (no incentives) • Possible explanations for lower response rates for web: • Additional effort required of the respondent to go online after contact made in other mode • Lack of ability to complete survey online • Technical difficulties • Concerns about confidentiality • Greater likelihood of break-offs • Response rates decrease when mixing web with traditional modes: • Innovation panel expt: 74% for mixed mode with web compared to 78% for F2F only • Possible explanations for lower response rates for web & mixed mode with web: • Unclear (Operational difficulties? Paradox of choice? Easy step from non-participation in web to refusal in any mode?)
- 12. 12 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereBarriers to using web for random prob surveys of gen pop 4. Measurement • Scope to improve measurement but replication of measurement remains an issue • Using web is not a major barrier for new surveys • A lot is known about causes of measurement differences & how to reduce this risk • New research on how to detect & possibly adjust for remaining measurement differences • Switching to web in existing surveys (incl mixed mode) remains problematic • Many existing and standard questions on high quality surveys in the UK have been designed for F2F interviewing and are therefore unsuitable for web • Particularly problematic for surveys that contribute data to important time series • Rapid uptake of mobile web • To allow or not to allow survey completion on smart phones? • Location, length, question design, visual layout?
- 13. 13 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereBarriers to using web for random prob surveys of gen pop 5. Other barriers: • Over-reliance on response rate as indicator of data quality • Speed of technological & societal changes
- 14. 14 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo herePossible solutions being explored for the UK 1. Web data collection after making contact using traditional modes • Community Life Survey 2. Mixed mode data collection • For longitudinal and cross-sectional surveys 3. Probability-based web panels • UK Research Council (ESRC) call for proposals for feasibility work • Could it be a solution for some Official Statistics?
- 15. 15 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereResearch Agenda Coverage & Sampling Participation & Engagement Measurement Issues Sampling methods: • enhancing PAF • enhancing existing individual-based frames • feasibility of UK pop register • within household selection Recruitment protocols: • making contact • use of incentives • number/timing of reminders Mode effects: • Impact in relation to other sources of error, costs, purpose & type of analysis • Guidance on assessing mode effects • Correction techniques Under-coverage: • ability to use as well as have access to the web • when can it be ignored? Engaging & motivating respondents: • minimising break-offs • how to engage online • length of questionnaires Technological advances: • Devices for internet access (e.g. mobile!) • Interactive media design • Capacity building in web survey programming Minimising nonresponse bias: • better knowledge about representativeness of web surveys
- 16. 16 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereGenPopWeb recommendations for the ESRC National expert centre or network for research and development A probability-based web panel for the UK Encourage & support experimentation with web and mixed mode on existing surveys Verify whether there is a business case Support innovations/changes in survey design, methods & practice Further developmental work; e.g. how to optimise the procedures involved in setting up and maintaining a prob-based web panel Provide grant funding that encourages collaboration between academics, survey organisations & government departments If feasible, the national expert centre/network could be responsible for setting up the panel Conduct methodological research and testing The panel should be a vehicle for developing and testing new methods & practices as well as a resource for collecting substantive data Provide a knowledge sharing system that is open, fast, and online
- 17. 17 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereSome general discussion points: Tolerance for error versus cost in Official Statistics Should we rethink how we design and conduct surveys? Are we focusing too much on replication rather than improvement of measurement with web data collection? A probability-based web panel for Official Statistics?
- 18. 18 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Paste co- brand logo here Thank you Gerry.Nicolaas@ipsos.com | 020 3059 5492 05/09/2014 © Ipsos MORI This work was carried out in accordance with the requirements of the international quality standard for market research, ISO 20252:2006 and with the Ipsos MORI Terms and Conditions which can be found here www.natcenweb.co.uk/genpopweb/ @GenPopWeb