Addressing the challenge of using web in cross-sectional surveys of the general population
- 1. 1 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Paste co- brand logo here Gerry Nicolaas (Ipsos MORI) and Ana Villar (City University London) WEBDATANET conference, 26-28 May 2015, Salamanca, Spain 05/09/2014 Addressing the challenge of using web in cross-sectional surveys of the general population WEBDATANET Task Force 21: GenPopWeb
- 2. 2 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereContents 1. Overview of Task Force 21: GenPopWeb • What is it? • Summary of conclusions • Research Agenda • Recommendations for the UK 2. Using web in cross-sectional surveys of the general population • What are the issues? • Three recent UK examples • Conclusions • What next?
- 3. 3 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Task Force 21: GenPopWeb
- 4. 4 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here 4 What is GenPopWeb? • GenPopWeb is a network of survey researchers, practitioners and commissioners who explore how the web can be used to: • survey the general population • while maintaining population representativeness and preserving data quality. • The 2013 events of this network were funded by the UK National Centre for Research Methods (NCRM). • In September 2013, the network was integrated within the European network WEBDATANET (COST Action IS1004)
- 5. 5 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereSummary of GenPopWeb conclusions Coverage & Sampling Participation & Engagement Measurement Issues Internet access is rapidly increasing and will soon reach universal coverage: • ignore non-coverage • web + traditional modes • provide internet access Dependency on traditional modes of contact which has impact on: • cost • response rate Differences in measurement between web and other modes: • not a big issue for most new surveys • a challenge for existing surveys (e.g. time series) Dependency on conventional sampling frames & methods for web data collection Response rates to web surveys are lower than comparable surveys using other modes: • dependency on response rate as an indicator of quality • marginal non-response bias? Rapid uptake of mobile web is a challenge
- 6. 6 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereResearch Agenda Coverage & Sampling Participation & Engagement Measurement Issues Sampling methods: • enhancing PAF • enhancing existing individual-based frames • feasibility of UK pop register • within household selection Recruitment protocols: • making contact • use of incentives • number/timing of reminders Mode effects: • Impact in relation to other sources of error, costs, purpose & type of analysis • Guidance on assessing mode effects • Correction techniques Under-coverage: • ability to use as well as have access to the web • when can it be ignored? Engaging & motivating respondents: • minimising break-offs • how to engage online • length of questionnaires Technological advances: • Devices for internet access (e.g. mobile!) • Interactive media design • Capacity building in web survey programming Minimising nonresponse bias: • better knowledge about representativeness of web surveys
- 7. 7 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereGenPopWeb recommendations for the UK National expert centre or network for research and development A probability-based web panel for the UK Encourage & support experimentation with web and mixed mode on existing surveys Verify whether there is a business case CURRENTLY IN PROGRESS Support innovations/changes in survey design, methods & practice Further developmental work; e.g. how to optimise the procedures involved in setting up and maintaining a prob-based web panel Provide grant funding that encourages collaboration between academics, survey organisations & government departments If feasible, the national expert centre/network could be responsible for setting up the panel Conduct methodological research and testing The panel should be a vehicle for developing and testing new methods & practices as well as a resource for collecting substantive data Provide a knowledge sharing system that is open, fast, and online
- 8. 8 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Using web in cross-sectional surveys
- 9. 9 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here The main barriers for using web in cross-sectional surveys of the general population: 1. Dependency on conventional sampling frames and methods such as population registers, RDD and address lists 2. Dependency on using traditional modes of contact such as telephone, postal or even face-to-face: • Implications for response rates • Implications for cost
- 10. 10 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereThe UK context No population register Dominant data collection mode for high quality surveys in the UK is face-to-face Random Digit Dialling is seldom used Financial challenges have made web a desirable mode for surveys Internet access • 84% of households have access to the internet (ONS, 2014) • Web is becoming mobile • 58% of UK adults access the internet on their mobile phone (ONS, 2014) • 44% of UK households own at least one tablet (Ofcom, 2014) Rapid growth of web panels • None of which use random probability sampling methods
- 11. 11 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereUK General Election 2015: pre-election polls Source: Chris Hanretty (@chrishanretty) | Twitter
- 12. 12 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereThe UK context No population register Dominant data collection mode for high quality surveys in the UK is face-to-face Financial challenges have made web a desirable mode for surveys Internet access • 84% of households have access to the internet (ONS, 2014) • Web is becoming mobile • 58% of UK adults access the internet on their mobile phone (ONS, 2014) • 44% of UK households own at least one tablet (Ofcom, 2014) Rapid growth of web panels • None of which use random probability sampling methods Limited use of web for random probability samples of general population • Mainly longitudinal surveys (e.g. birth cohort studies, LSYPE) • Some experimentation on other surveys (e.g. LFS W1, ESS, Community Life) • One-off survey on flood insurance
- 13. 13 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Using web in cross-sectional surveys: 3 recent UK examples
- 14. 14 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here 3 UK examples using random sample of addresses and postal contact UK ESS web experiment Community Life Survey web experiment Flood Insurance Web Survey NatCen on behalf of City University London TNS BMRB on behalf of the Cabinet Office (Govt) Ipsos MORI on behalf of the Dept of Environment, Food & Rural Affairs (Govt) Main survey=CAPI, parallel web Main survey=CAPI, parallel web New one-off survey Random prob address sample Random prob address sample Random prob address sample Respondent = experiment with Last/Next Bday and online KISH Respondent = Last/Next Bday Respondent = decision-maker on home insurance Mailings addressed to resident Mailings addressed to resident 1st sample sent to ‘resident’ & 2nd sample sent to matched name from Electoral Register 3 mailings = pre-notice, invitation, reminder 3 mailings = invitation & up to 2 reminders 3 mailings = invitation & up to 2 reminders Experiments with incentives Experiments with incentives £5 conditional in flood areas, £10 conditional in other areas
- 15. 15 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereResponse rates 1. Web response rates are considerably lower than equivalent CAPI survey, ranging from about 14% to about 25% (depending on topic, type & value of incentive)
- 16. 16 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereCommunity Life: CAPI & web response rates by incentive 60% 16% 19% 22% 25% 0% 10% 20% 30% 40% 50% 60% 70% F2F with conditional £5 shopping voucher Web - no incentive Web - cond £5 e- voucher Web - cond £10 e- voucher Web - uncond £5 shop voucher ■CAPI ■Web Source: TNS BMRB (2013). Community Life Survey: Summary of web experiments. Report prepared for the Cabinet Office.
- 17. 17 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereESS web incentive experiment • Unconditional £5 shopping voucher sent with invitation letter to all • Additional incentive experiment (N=2,000) • £15 conditional on web completion, web response = 22.0% • £35 conditional on web completion, web response = 23.3% (ESS CAPI response rate = 55%) Source: City University London (forthcoming). Feasibility Report – costs, response rates, design effects & quality of alternative data collection. ESS-DACE, Deliverable 7.4
- 18. 18 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereFlood Insurance Study (no experiment) • Conditional £5 shopping voucher for those living in flood areas (N=4,047) • Final response rate = 13.1% of all issued addresses • Adjusted response rate = 14.2% • Conditional £10 shopping voucher for those not living in flood areas (N=3,400) • Final response rate = 16.2% of all issued addresses • Adjusted response rate = 17.8% • 1 in 4 who didn’t respond after 1st reminder didn’t receive 2nd reminder Source: Ipsos MORI (2015). The Flood Insurance Study. A review of the online survey response rates. A report prepared for the Department of Environment and Rural Affairs.
- 19. 19 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereResponse rates 1. Web response rates are considerably lower than equivalent CAPI survey, ranging from about 14% to about 25% (depending on topic, type & value of incentive) 2. Following up web non-respondents with another mode increases response rates but not to the same level as the equivalent CAPI survey.
- 20. 20 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Community Life Survey experiment: postal questionnaire sent with second reminder 60% 27% 31% 35% 39% 0% 10% 20% 30% 40% 50% 60% 70% F2F with conditional £5 shopping voucher Web+Mail no incentive Web+Mail cond £5 evoucher Web+Mail cond £10 evoucher Web+Mail uncond £5 shop voucher Source: TNS BMRB (2013). Community Life Survey: Summary of web experiments. Report prepared for the Cabinet Office. ■CAPI ■Web + postal
- 21. 21 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here ESS web experiment (N=500): Web non-respondents followed up by F2F interviewer 55% 25% 38% 0% 10% 20% 30% 40% 50% 60% 70% ESS main Web Web + F2F ■CAPI ■Web + face-to-face Source: City University London (forthcoming). Feasibility Report – costs, response rates, design effects & quality of alternative data collection. ESS-DACE, Deliverable 7.4
- 22. 22 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereSample profiles 1. Differences in sample profile between web and CAPI • ESS – in web more middle-aged, more higher education, more high earners, more likely to be employed, fewer politically interested. • Comm Life – in web more middle aged, more higher education, more high earners, more home owning, more likely to be living as a couple, more native English speakers, more everyday internet users 2. Following up web non-respondents with another mode does not necessarily improve sample profile: • ESS – CAPI for web non-respondents improved profile but still under- represented youngest & oldest age groups, and over-represented people whose main activity is looking after the home to a much greater extent than the CAPI-only sample (external population data). • Comm Life – adding postal questionnaire brought into the sample more middle-aged people making the age profile worse. Slight improvement on other sample characteristics.
- 23. 23 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereWithin household random selection of respondent • Random sample of addresses – no names and the number of eligible household members is unknown. • Random selection using Last/Next Birthday commonly used for postal and web surveys: • ESS – about 20% of respondents were not the randomly selected respondents. • Comm Life – about 25% of respondents were not the randomly selected respondent • Possible solution? • ESS experiment showed no improvement when using online KISH method • Community Life Survey plans to ask all eligible household members to complete an online questionnaire in the next round of the survey
- 24. 24 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereCosts Expectations of cutting costs considerably by using web data collection in cross-sectional surveys are not being met: • Use of traditional modes to make contact • Extra efforts to push respondents online (reminders, incentives) • Increase response and reduce risk of non-response bias by following-up web non-respondents with another more expensive mode.
- 25. 25 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Conclusions
- 26. 26 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo hereConclusions: 1. Two main barriers for using web in UK cross-sectional surveys of the general population: • Risk of selection bias with address sample & postal contact • Risk of marginal non-response bias 2. Expectations of cutting costs considerably by using web data collection in cross-sectional surveys are currently not being met.
- 27. 27 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Using web in cross-sectional surveys: Looking for solutions
- 28. 28 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo herePossible solutions to the two main barriers Risk of selection bias Risk of non-response bias Postal request for online registration & household enumeration followed up with email invitation for randomly selected household member to complete online questionnaire Multiple and varied contact attempts • Including different modes of contact when possible Postal request for all (adult) household members to complete online questionnaire Effective incentives • Essential to maximise web response rate but still considerably lower than equivalent face- to-face survey Abandon sample of addresses and piggy-back on existing surveys using traditional modes with relatively high response rates and collection of names and email addresses Follow-up web non-respondents with alternative mode of data collection Probability-based web panels Focus on risk of nonresponse bias rather than magnitude of nonresponse
- 29. 29 Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Paste co- brand logo here Paste co- brand logo here Thank you Gerry.Nicolaas@ipsos.com | 020 3059 5492 05/09/2014 © Ipsos MORI This work was carried out in accordance with the requirements of the international quality standard for market research, ISO 20252:2006 and with the Ipsos MORI Terms and Conditions which can be found here GenPopWeb final report at: http://eprints.ncrm.ac.uk/3309/3/GenPopWeb.pdf @GenPopWeb
Editor's Notes
- #8: National expert centre or network for research and development – strategic and scientific leadership
- #12: 1992 methodological changes that ICM’s Nick Sparrow implemented: a switch to telephone polling, weighting by recall of past voting, turnout weighting and ‘adjustment’ of the data to reflect the proportions of people who told us what they did at the last election, but ‘Don’t Know’ or refused to tell us what they would do this time around – our ‘partial refusers’. A nagging fear that has been present for some time: whether or not telephone (but this also very much applies to online) data collection techniques remain capable of delivering a representative sample which can be modelled into an accurate election prediction. The problems with access to people over landline phones are well documented, and are true, and the insertion of random mobile sample into sample frames has clearly been mostly ineffective.
- #15: National expert centre or network for research and development – strategic and scientific leadership
- #29: National expert centre or network for research and development – strategic and scientific leadership