Improving inferences from poor quality samples
1 like604 views
The document outlines a workshop held by the Social Research Centre on improving survey methodologies and addressing issues such as declining response rates. It discusses the Total Survey Quality framework and highlights challenges in obtaining accurate data through probability and non-probability sampling, with specific reference to trends in Australia and the U.S. It emphasizes the need for well-designed surveys to mitigate non-response bias and ensure the reliability of survey results.
![The general decline in response rates is evident
across nearly all types of surveys, in the United
States and abroad. At the same time, greater effort
and expense are required to achieve even the
diminished response rates of today.
These challenges have led many to question
whether surveys are still providing accurate and
unbiased information [and whether or not
probabilistic surveys are better than the
alternatives?]
Pew Research Center, 2012
10](https://image.slidesharecdn.com/improvinginferencesfrompoorqualitysamples-170825020618/85/Improving-inferences-from-poor-quality-samples-10-320.jpg)
![www.srcentre.com.au
A Word of Warning on Probability Sampling
Rivers again ….
[Probability surveys work] if the nonresponse rate is small. Cochran
(1977, p. 363), in his classic text, concluded that the upper limit is
approximately 10 percent nonresponse, which is difficult to achieve today in
even the best funded surveys.
Lohr (2010, p. 355) warns that “many examples exist of surveys with a 70%
response rate whose results are flawed.”
(Doug Rivers, Comments on the “2013 AAPOR Taskforce Report on Non-
probability Sampling” (Douglas Rivers; Comment. J Surv Stat
Methodol 2013; 1 (2): 111-117. doi: 10.1093/jssam/smt009)
25](https://image.slidesharecdn.com/improvinginferencesfrompoorqualitysamples-170825020618/85/Improving-inferences-from-poor-quality-samples-22-320.jpg)
Improving inferences from poor quality samples
- 1. www.srcentre.com.au Improving inferences from poor quality samples Social Research Centre / Centre for Social Research and Methods Workshop 16 August, 2017
- 2. www.srcentre.com.au Presenters Social Research Centre Pty Ltd Darren W. Pennay Dina Neiger, PhD Paul J. Lavrakas, PhD Darren Pennay is an Adjunct Senior Research Fellow with the ANU Centre for Social Research and Methods and an Adjunct Professor with the Institute for Social Science Research at the University of Queensland Dina Neiger is a Centre Visitor with the ANU Centre for Social Research and Methods Paul Lavrakas is a Senior Fellow, NORC at the University of Chicago; Senior Research Fellow, Office of Survey Research, Michigan State University; and Senior Methodological Advisor for the ANU’s Social Research Centre 2
- 3. www.srcentre.com.au The Social Research Centre ➢ Based in Melbourne, Australia o owned by the Australian National University in Canberra o operates as a for-profit research services company. o profits returned to ANU o Co-founders of the ANU Centre for Social Research and Methods ➢ Conduct social and public policy research for government, not-for-profit and academic organisations ➢ Primary data collection involving survey research, qualitative research, statistical methods, data processing and analytics, survey methodology ➢ 65 staff, plus 120-seat CATI call centre 3
- 4. www.srcentre.com.au What is Total Survey Quality? ➢ Many national statistical offices, including Australia, Canada, New Zealand and USA, operate within a Total Survey Quality (TSQ) framework to determine whether or not a survey is “fit-for-purpose”. ➢ Accuracy is not enough. 4 Dimensions of TSQ Framework Dimension Description Accuracy Total survey error is minimised Credibility Data are considered trustworthy by the survey user communities Comparability Consistent with past studies in terms of demographic, spatial, and temporal comparisons Usability / Interpretability Documentation is clear and metadata are well organised Relevance Data satisfy user needs Accessibility Access to the data is user friendly Timeliness / Punctuality Data deliverables adhere to schedules Completeness Data are rich enough to satisfy the analysis objectives without undue burden on respondents Coherence Estimates from different sources can be reliably combined Source: Biemer, P. (2010) Public Opinion Quarterly 74 (5): p.819.
- 5. www.srcentre.com.au REPRESENTATION MEASUREMENT Survey Cycle from a Design Perspective 5 Final Results & Conclusions Target Population Sampling Frame Designated Sample Final sample Final Dataset Response Specification Measurement
- 6. www.srcentre.com.au Total Survey Error Framework 6 Final Results & Conclusions Sampling Error Nonresponse Error Adjustment Error ERRORS OF REPRESENTATION ERRORS OF MEASUREMENT Coverage Error Processing Error Specification Error Measurement Error Inferential Error Target Population Sampling Frame Designated Sample Final sample Final Dataset Response Specification Measurement
- 7. www.srcentre.com.au What is a probability sample? Textbook Definition: A probability sample is one in which every unit in the population of interest has a known, non-zero probability of being selected for the sample. Key elements which will differentiate the process from a non-probability sampling process: o Selection into the sample is via a random process. o Every unit in the population of interest has a chance of being sampled for the research. o The probability of selection is known. These design features enable us to: o Calculate standard errors o Calculate confidence intervals o Generalise to the target population of interest 7
- 10. The general decline in response rates is evident across nearly all types of surveys, in the United States and abroad. At the same time, greater effort and expense are required to achieve even the diminished response rates of today. These challenges have led many to question whether surveys are still providing accurate and unbiased information [and whether or not probabilistic surveys are better than the alternatives?] Pew Research Center, 2012 10
- 11. www.srcentre.com.au Trends in response rates (U.S. data) There is no sign of an increase in nonresponse bias since 2012. On 13 demographic, lifestyle, and health questions that were compared with benchmarks from high response rate federal surveys, estimates from phone polls are just as accurate, on average, in 2016 as they were in 2012. The average (absolute) difference between the Center telephone estimates and the benchmark survey estimates was 2.7 percentage points in 2016, compared with 2.8 points in 2012. 11
- 12. www.srcentre.com.au Trends in response rates (U.S. data) 12
- 13. www.srcentre.com.au Impact of declining response rates on telephone survey estimates in the U.S. 13
- 14. www.srcentre.com.au Impact of low response rates on survey estimates Pew conclusions 2012 ➢ Despite the growing difficulties in obtaining a high level of participation in most surveys, well-designed telephone polls that include landlines and cell phones reach a cross- section of adults that mirrors the American public, both demographically and in many social behaviours. Pew conclusions 2016 ➢ Telephone polls still provide accurate data on a wide range of social, demographic and political variables, but some persistent weaknesses persist. 14
- 15. www.srcentre.com.au What is the situation in Australia? A literature review and industry consultation commissioned by RICA and undertaken by Bednall et al. (2013) concluded the following: ➢ Telephone response rates: o As far as the telephone is concerned response rates have been in a gradual decline the last decade. o Among cold-calling general community surveys, telephone response rates are typically below 10%. o Co-operation rates, (the ratio of obtained interviews to refusals) are typically below 0.2 (that is below one interview to five refusals). o Telephone interviews from client lists have a higher response rate – typically above 20% with co-operation rates above 1.0. o It would appear that some topics, such as financial services, may induce a lower level of co- operation. o Government sponsored surveys have higher response rates, at times over 50%, but even here a sharp decline in response rates over time for one long running monitor was observed. Co- operation rates were also higher in government sponsored surveys. 15
- 16. www.srcentre.com.au What is the situation in Australia – SRC Landline surveys 17 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 2011 2012 2013 2014 2015 Non-contactsas%usablesample Survey year Non-contacts as % usable sample - various projects Landline sample 1- National Landline sample 2 - National Landline sample 3 - Victoria Landline sample 4 - Victoria Landline sample 5 - National
- 17. www.srcentre.com.au What is the situation in Australia – SRC Landline surveys 18 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 2011 2012 2013 2014 2015 Refusalsasa%ofcontacts Survey year Refusals as % contacts - various projects Landline sample 1- National Landline sample 2 - National Landline sample 3 - Victoria Landline sample 4 - Victoria Landline sample 5 - National
- 18. www.srcentre.com.au What is the situation in Australia – SRC Landline surveys 19 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 2011 2012 2013 2014 2015 Interviewsasa%ofinterviews+refusals Survey year Interviews as % interviews and refusals - various projects Landline sample 1- National Landline sample 2 - National Landline sample 3 - Victoria Landline sample 4 - Victoria Landline sample 5 - National
- 19. www.srcentre.com.au What is the situation in Australia (cont.)? ! Not saying that low response rates are inconsequential! ! Poorly designed and poorly executed surveys, especially those with very limited call routines (e.g. polls/trackers enumerated over one or two evenings only), those with non-coverage errors (e.g. landline telephone surveys), and those that use non-probability sampling methods (e.g. online panels) are much more likely to produce biased results. ! HOWEVER, well designed and well executed probability-based surveys still produce estimates that can be relied on in many situations (and with known standard errors and confidence intervals!) 22
- 20. www.srcentre.com.au This appears to be the main situation regarding non-response 23
- 21. www.srcentre.com.au The sky is not falling in
- 22. www.srcentre.com.au A Word of Warning on Probability Sampling Rivers again …. [Probability surveys work] if the nonresponse rate is small. Cochran (1977, p. 363), in his classic text, concluded that the upper limit is approximately 10 percent nonresponse, which is difficult to achieve today in even the best funded surveys. Lohr (2010, p. 355) warns that “many examples exist of surveys with a 70% response rate whose results are flawed.” (Doug Rivers, Comments on the “2013 AAPOR Taskforce Report on Non- probability Sampling” (Douglas Rivers; Comment. J Surv Stat Methodol 2013; 1 (2): 111-117. doi: 10.1093/jssam/smt009) 25
- 23. www.srcentre.com.au The rise of non-probability online panels
- 24. www.srcentre.com.au The take-up of non-probability online panels The situation in Australia ➢ In 2014-15, 86% of Australian households had access to the internet at home (ABS). ➢ Online research is continuing to grow in popularity domestically and internationally. 28 24 39 34 34 40 26 28 25 31 12 0 10 20 30 40 50 2009 2010 2012 2013 2015 Online CATI Source: ESOMAR / Research Industry Council of Australia Percent of research industry turnover Online research globally – 28% in 2014 ➢ 50+ commercial ‘online research panels’. ➢ First of these established in the late 1990’s. ➢ All ‘until now’ use non- probability sampling methods.
- 25. www.srcentre.com.au The rise of non-probability panels - globally $0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000 $7,000 $8,000 '99 '00 '01 '02 '03 '04 '05 '06 '07 '08 '09 '10 '11 '12 '13 '14 '15E '16E Revenuein$USM Year Online Research Revenues 1999-2016E US Europe Rest of World 29
- 26. www.srcentre.com.au Cons ➢ Non-coverage ➢ Self-selection ➢ Reliance on the computer-literate respondents ➢ Non-probability sampling o Calculating standard errors o Calculating confidence intervals o Generalising to the target population of interest Pros ➢ Reduced cost ➢ Improved timeliness ➢ Respondent convenience ➢ Reduced social desirability bias ➢ Can target ‘hard to reach’ populations ➢ Multimedia functionality ➢ Computerised questionnaire scripts 30 Pros and cons of non-probability panels
- 27. www.srcentre.com.au How do they compare – the SRC Online Panels Benchmarking Survey ➢ Three surveys based on probability samples of the Australian population aged 18 years and over and five surveys of persons aged 18 years and over administered to members of non-probability online panels. ➢ Survey questionnaire included a range of demographic questions and questions about health, wellbeing and use of technology for which high-quality population benchmarks were available. ➢ Same questions were used across the eight surveys. ➢ 9 minutes average interview length for online and telephone. o 12 page booklet for the hard copy version. ➢ Fieldwork Oct – Dec 2015. ➢ Data and documentation available from the Australian Data Archive. https://www.ada.edu.au/ada/01329 31
- 28. www.srcentre.com.au Results – substantive health characteristics 32 Substantive variables Benchmark value Distance from benchmarks Probability Non-probability ABS ANU Poll RDD P1 P2 P3 P4 P5 Life satisfaction (8 out of 10) Percentage point error 32.6 -2.0 -2.0 1.9 -11.9 -11.6 -4.5 -9.2 -7.9 Psychological distress - Kessler 6 (Low) Percentage point error 82.2 -10.6 -11.6 -8.1 -25.9 -23.5 -22.2 -25.0 -23.2 General Health Status (SF1) (Very good) Percentage point error 36.2 0.4 -2.0 -2.6 -4.1 -5.8 -5.3 -5.0 1.5 Private Health Insurance Percentage point error 57.1 3.4 1.9 3.3 -8.9 -12.5 -3.7 -0.6 -2.6 Daily smoker Percentage point error 13.5 -4.1 3.5 1.6 9.8 6.7 3.9 2.7 4.3 Consumed alcohol in the last 12 months Percentage point error 81.9 3.6 -2.8 -4.0 2.4 5.3 3.9 4.2 1.5
- 29. www.srcentre.com.au Results – substantive health characteristics Average error across six substantive measures 33 Probability Surveys Non-probability Panels ABS ANU Poll RDD P1 P2 P3 P4 P5 Avge. error 4.02 3.98 3.58 10.5 10.9 7.24 7.78 6.83 Largest absolute error 10.59 11.57 8.08 25.86 23.52 22.20 24.96 23.20 No. of significant differences from benchmarks (out of 6) 2 3 2 4 6 3 3 3
- 30. www.srcentre.com.au Two questions … 1. How do we reduce non-response bias in low response rate probability surveys 2. How do we reduce bias in survey estimates based on non- probability online panels 34
- 31. www.srcentre.com.au Instructors Dina Neiger, PhD ➢ Dina is a professional statistician with over 20 years experience and a track record of achievement in leadership and technical roles. ➢ Social Research Centre, Monash University, Australian Bureau of Statistics, Peter MacCallum Cancer Centre. ➢ 1st Class Honours degree in Statistics and PhD in Business Systems from Monash University with an emphasis in applied Operations Research and Process Engineering. ➢ Accredited Statistician (AStat) and member of the Statistical Society of Australia (SSA). ➢ Full member of the Australian Society for Operations Research (ASOR). ➢ Calibration and blending methods to improve accuracy of the non-probability samples, establishment and maintenance of the first Australian Online Probability panel and complex business survey design and weighting. 35
- 32. www.srcentre.com.au Instructors Paul J. Lavrakas, PhD ➢ Research psychologist, research methodologist, and prolific author. ➢ Since 2007, Independent Consultant with clients in the USA, Australia, Belgium, Japan and Canada; and Senior Fellow at NORC (U-Chicago) and Office of Survey Research (Michigan State U.) . ➢ From 2000-2007, Chief Research Methodologist for Nielson Media Research. ➢ 1978-2000 - Professor at Northwestern U. and Ohio State U. and founding faculty director of a survey research center at each university. ➢ Australian roles - Senior Methodological Adviser, the Social Research Centre. Member of the Scientific Advisory Board for the Centre for Social Research and Methods, ANU. ➢ President of the American Association for Public Opinion Research (2011-2014) and continues to serve on a volunteer basis on many AAPOR task forces and committees. 36
- 33. www.srcentre.com.au Investigating Nonresponse Bias by Level of Effort Paul J. Lavrakas
- 34. www.srcentre.com.au Investigating Nonresponse Bias ➢ Methods have evolved in the past decade regarding how NR Bias can be investigated (Groves and Brick, 2007; Olson and Montaquilla, 2012) ➢ One approach is to conduct analyses comparing the difference between early responders and middle responders vs. late responders on key measures in a survey with the hypothesis/premise being that late responders are more like nonrespondents than are early and middle responders ➢ However, if no differences are found between late responders vs. early and middle responders, that is NOT taken as evidence that there is no differences between respondents and nonrespondents. 38
- 35. www.srcentre.com.au Case Study 1: University Faculty Health Benefits Survey Study Background ➢ The purpose of this survey was to provide reliable and valid information about the opinions of faculty towards the prospect of the university creating a new health care service for faculty working on the main campus. ➢ Approximately 800 of 5,000 members of the faculty were randomly sampled, and a telephone survey was conducted by the university’s survey research center, which yielded 47% completion rate. ➢ Nonresponse bias was investigated using multiple methods, with one being a Level of Effort analysis. 39
- 36. www.srcentre.com.au Level of Effort Analyses ➢ Level of Effort was operationalized as the number of calls made to a given sampled respondent who completed the interview ➢ Analyses were conducted to investigate whether the effort it took to complete an interview was related to any of the substantive questions that were asked in the questionnaire. ➢ From these analyses it was learned than none of the key questionnaire items had a statistically significant (p < .05) association with this effort measure ➢ For one of the questionnaire items – asking whether the respondent had heard about the possible new health service prior to being interviewed – there was a marginally significant correlation (p < .07) with effort, but the size of this correlation was extremely small (r = 0.09). ➢ Thus, it was judged that there as no reliable evidence that any nonignorable nonresponse bias was present in the findings reported in the main body of the report. 40
- 37. www.srcentre.com.au Case Study 2: Voter Identification Survey Study Background ➢ In 2012, there were several states in the USA whose state governments were controlled by conservative party members who passed legislation changing the requirements for the identification that a voter must show to prove that he or she were eligible to vote on the day of the Presidential Election. ➢ The purpose of the survey, which was funded by large labor unions that were known to support liberal candidates/policies, was to identify how many and which types of residents in the state were most likely to be disenfranchised by the new Voter ID legislation. 41
- 38. www.srcentre.com.au Level of Effort Analysis 1 ➢ One approach was to compare key data provided by those who initially refused, but later agreed to complete the questionnaire after being recontacted with a “refusal conversion” protocol, with data provided by the cohort that never refused. ➢ In this analysis it was found that for neither of the two of the key statistics measured by the survey – awareness of the new Voter ID legislation; whether someone had a valid photo ID to vote at their local polling place on November 6, 2012 – were there statistically significant differences (p < .05) between those who never refused and those who initially refused but later were converted. ➢ Thus, there was no evidence found to suggest that refusing nonrespondents who were not converted would have given materially different answers to these key questions than did the respondents. 42
- 39. www.srcentre.com.au Level of Effort Analysis 2 ➢ A second approach was to take into account the number of call attempts it took to complete an interview, and investigate if answers to the same two key variables correlated with the level of effort expended. ➢ In this analysis it was found that as more effort was expended to reach a respondent, the likelihood the respondent had a valid photo ID for voting purposes increased at a statistically significant level (p < .007). o Thus, the findings suggested that the unweighted survey data were somewhat biased in the direction of underestimating the proportion of people with a valid photo ID 43
- 40. www.srcentre.com.au Level of Effort Analysis 2 (cont.) ➢ However, since the level of effort to achieve a completion correlated with various demographic characteristics of respondents (e.g., it take less effort to reach elderly respondents and more effort to reach young adults) and given that key demographic characteristics were taken into account in the weighting process, it was judged to be unlikely that the findings reported about the proportion of registered voters without a valid photo ID were subject to nonignorable nonresponse bias. ➢ Concerning awareness of the legislation, there was no statistically significant difference associated with the level of effort expended to gain a completion. o Thus, there was no evidence found to suggest that uncontacted nonrespondents would have given materially different answers to this key questions than did the respondents. 44
- 41. www.srcentre.com.au Non-Response Follow-up Studies Paul J. Lavrakas
- 42. www.srcentre.com.au Case Study: IMLS Nonresponse Follow-Up (NRFU) Study ➢ Another preferred method to investigate Nonresponse Bias is to conduct a follow-up survey of a sample of the original survey’s nonresponders. ➢ Little has been reported regarding how to use the data from such follow-up studies ➢ Please note, that my slides today are extracted from a paper presentation that I made in 2015 at the 70th annual conference of the American Association for Public Opinion Research (AAPOR), “Studying Nonresponse Bias with a Follow-up Survey of Initial Nonresponders in a Dual Frame RDD Survey” 46
- 43. www.srcentre.com.au 2013 Public Needs for Library and Museum Services (PNLMS) Survey ➢ Sponsored by the Institute of Museum and Library Service (IMLS), a federal agency in the USA ➢ Conducted by M. Davis and Company, Inc., Philadelphia PA ➢ National dual-frame RDD survey of the general population of the USA o Data gathered September-November 2013 o 3,537 interviews completed; 2,506 from LL frame; 1,031 from CP frame • Landline sample AAPOR RR3 was 25% • Cell Phone sample AAPOR RR3 was 10% 47
- 44. www.srcentre.com.au 2014 IMLS Nonresponse Follow-up (NRFU) Survey ➢ Created shortened questionnaire with key variables ➢ Used a noncontingent and a contingent incentive protocol ➢ Used only best calibre interviewers ➢ 201 interviews were completed in January-February 2014 with a random sample of nonresponders to the main survey; 100 from LL frame, 101 from CP frame o Landline follow-up sample AAPOR RR3 was 32% o Cell Phone follow-up sample AAPOR RR3 was 16% ➢ The original sample and the follow-up samples had remarkably similar demographic characteristics with the exception that the follow-up sample had proportionally more young adults aged 18-24 years and fewer older adults aged 65+ year (p < .03) than the original survey. 48
- 45. www.srcentre.com.au Analytic Approach: Phase 1 ➢ Combining the two surveys via Weighting, in two Phases o Phase 1: Begin by assigning weights, separately, to each of four groups • Landline RDD sample (original study) • Landline NR follow-up sample • Cell phone RDD sample (original study) • Cell phone NR follow-up sample 49
- 46. www.srcentre.com.au Analytic Approach: Phase 1 ➢ For each of the four groups, Phase 1 including the following seven steps conducted for each US Census region within each sample type: 1. Probability of Selection and Design Weight 2. Nonresponse Follow-up Adjustment 3. Unknown Eligibility Adjustment 4. Removal of Known Ineligible 5. Unit Nonresponse Adjustment 6. Adult Subsampling Adjustment 7. Multiple Phone Line Adjustment ➢ Steps 1 and 2 created the weights to start the weighting process ➢ Steps 3-7 were sequential adjustments to the starting weights 50
- 47. www.srcentre.com.au 1. Probability of Selection and Design Weight 51 For each of the four samples: ➢ Using information from the sampling frame, the probability of selection was calculated as the number of released telephone numbers in each Census region divided by the total number of telephone numbers in the census region ➢ The design weight was the inverse of the probability of selection for the released telephone numbers and zero for the non-released telephone numbers
- 48. www.srcentre.com.au 2. Nonresponse Follow-up Adjustment 52 For each of the two NRFU samples: ➢ For the telephone numbers that were eligible for the nonresponse follow-up study, a further adjustment to the design weight was required to account for the subsampling of telephone numbers that were called from among all the eligible nonresponse follow-up numbers ➢ This adjustment to the design weight was calculated as the number of follow-up eligible telephone numbers divided by the number of follow-up eligible called telephone numbers
- 49. www.srcentre.com.au 3. Unknown Eligibility Adjustment As in any DFRDD survey, a considerable portion of the sampled numbers ended the original field period with their eligibility unresolved For each of the four samples: ➢ Using standard AAPOR final disposition codes, two groups of telephone numbers were created in each Census region: (1) unknown eligibility telephone numbers and (2) known eligibility telephone numbers ➢ Unknown eligibility adjustment factor was the sum of the follow-up adjusted design weights for all telephone numbers divided by the sum of the follow-up adjusted design weights for the known eligibility telephone numbers ➢ The unknown eligibility adjusted weight is the product of the unknown eligibility adjustment factor and the follow-up adjusted design weight for the known eligibility telephone numbers and zero (0.0) for the unknown eligibility telephone numbers 53
- 50. www.srcentre.com.au 4. Removal of Known Ineligible As with all telephone surveys of the general public, many of the telephone numbers that were released were found to be ineligible for various reasons, including various non-working numbers (e.g., disconnected or temporarily out of service, technical problems, etc.), and numbers that were not part of the target population, e.g., fax, business, or other nonresidential numbers. For each of the four samples: ➢ Using AAPOR final disposition codes, these known ineligible telephone numbers were identified and removed from the weighting process 54
- 51. www.srcentre.com.au 5. Unit Nonresponse Adjustment 55 As with all telephone surveys of the general public, for many of the telephone numbers for which contact was made, it was determined that there was at least one eligible adult but no data were collected For each of the four samples: ➢ Using the final disposition codes for the eligible telephone numbers produced two groups of eligible telephone numbers in each Census region, (1) eligible responding telephone numbers and (2) eligible non-responding telephone numbers ➢ The unit nonresponse adjustment shifts the weights from the eligible nonrespondents to the eligible respondents ➢ The nonresponse adjustment factor was the sum of the unknown eligibility adjusted weights for all eligible telephone numbers divided by the sum of the unknown eligibility adjusted weights for the responding telephone numbers ➢ The nonresponse adjusted weight was the product of the nonresponse adjustment factor and the unknown eligibility adjusted weight for respondents, and zero (0.0) for eligible nonrespondents
- 52. www.srcentre.com.au 6. Landline Adult Subsampling Adjustment 56 Within eligible landline households there may have been more than one eligible adult in residence; this was determined in the questionnaire For each of the four samples: ➢ When there were two or more eligible adults associated with an landline telephone number, the adult subsampling adjustment factor was capped at two (2.0) ➢ The adult subsampling adjusted weight was the product of the adult subsampling adjustment factor and the nonresponse adjusted weight.
- 53. www.srcentre.com.au 7. Multiple Phone Line Adjustment Most adults can be sampled on more than one phone line. For each of the four samples: ➢ A multiplicity adjustment was implemented when there were two or more telephone numbers, either landline or cell phone, at which the responding adult could have been contacted; this multiplicity adjustment factor was capped at two (2.0), otherwise, the multiplicity adjustment factor was one (1.0) This completed Phase 1 of the separate weighting of each of the four groups 57
- 54. www.srcentre.com.au Phase 2 Combining Samples: Composite Weights 58 For each of the four samples: ➢ The composite weighting adjusted the multiplicity adjusted weight from the landline respondents and cell phone respondents to account for overlap in the two samples ➢ If the person could have been reached by both frames the composite weight was 0.5; if the person could only be reached by one frame it was 1.0
- 55. www.srcentre.com.au Phase 2 Combining Samples: Calibration For each of the four samples: ➢ The final adjustment forced the weight totals from the survey data using the composite weight to match external population control totals; the external control totals were based on the following characteristics: o An extrapolation of Blumberg and Luke’s 2013 NHIS findings for telephone service ownership so that 50% were both landline and cell phone, 41% cell phone only, and 9% landline only o Socio-demographic characteristics from the most recent American Community Survey • Gender (Male, Female) • Age group (18-44, 45-64, 64+) • Marital status (Never been Married, Married, Separated/Divorced/Widow) • Hispanicity/Race (Hispanic, Non-Hispanic African American, Non-Hispanic White, Non-Hispanic Other) • Education (Less High School/High School, Some College, Associate or Bachelor Degree, Advanced Degree) • Presence or absence of children (No, Yes) 59
- 56. www.srcentre.com.au Phase 2 Combining Samples: Calibration (cont.) ➢ The calibration methodology that was used was Iterative Proportional Fitting, i.e., raking or sample balancing; ➢ This method forces all of the different characteristics to simultaneously match the control totals o Each of the socio-demographic characteristics was used as a separate dimension in the raking process 60
- 57. www.srcentre.com.au Using the Final Combined Data from the Four Samples ➢ After all this was done we had a weighted dataset containing all the interviews from the original study and from the follow-up survey of nonresponders (n= 3,738) ➢ This allowed us to generate two types of population estimates for key behavioral variables in the study o Estimates based on the original survey only o Estimates based on the combined surveys ➢ We then compared the differences between the two estimates to determine whether or not a statistic was materially different when the data from the nonresponse follow-up survey was taken into account 61
- 58. www.srcentre.com.au An Example of a Material Difference in a Key Statistic ➢ Percent of parents/guardians reporting having a child who visited a Zoo/Aquarium in the past 30 days 62 Original Survey Yes, Did Visit 19.2% Est. # in USA 14.45M
- 59. www.srcentre.com.au An Example of a Material Difference in a Key Statistic ➢ Percent of parents/guardians reporting having a child who visited a Zoo/Aquarium in the past 30 days ➢ 8.1 percentage point difference ➢ Estimated 6.1M difference 63 Original Survey Combined Surveys Yes, Did Visit 19.2% 27.3% Est. # in USA 14.45M 20.55M
- 60. www.srcentre.com.au The Extent of Differences Identified by the NR Follow-up Study ➢ For 10 of the 28 behavioral measures, the percentage found in the original survey differed by more than two percentage points (2 pp) from the percentage found in the combined survey dataset ➢ Of these, six of the behavioral measures differed by more than five percentage points (5 pp) ➢ All of these measures were for estimates associated with the reported behavior of children (where the adult respondent served as a proxy for her/his child) and none of them were for estimates of the reported behavior of the adult respondent Nonresponse Bias exists at the level of the individual measure, and here we found evidence of several measures that likely were highly biased 64
- 61. www.srcentre.com.au Response Propensity Modelling – Non-response Adjustments Dina Neiger
- 62. www.srcentre.com.au Acknowledgement Analysis by Andrew Ward, Principal Statistician, Social Research Centre Thanks to Dr Siu-Ming Tam and Mr Paul Schubert, ABS, for their collaboration on this work and their kind permission to use the survey data for this presentation. Slides based on presentation by Andrew Ward at the Australian Statistical Conference in December 2016. 66
- 63. www.srcentre.com.au Community Trust in Statistics Survey Determine public awareness and trust of official statistics Dual frame phone survey Also available for respondents and non-respondents: part-of- state (based on the landline prefix) or mobile Collected info from ~727 refusals – age, sex, awareness, trust 67
- 64. www.srcentre.com.au Challenge ➢ Incorporation of refusal information in non-response adjustment Probability of response derived from propensity model Base weight = Design weight / Probability of response Limited auxiliary information available for respondents and non- respondents 68
- 65. www.srcentre.com.au Non-response adjustment Awareness / Trust Non-respondents (%) Respondents (%) Have heard of and trust a great deal 12.4 20.5 Have heard of and tend to trust 28.5 54.6 Have heard of and tend to distrust 6.2 10.5 Have heard of and distrust a great deal 4.8 2.2 Have heard of but DK / Refused trust 26.3 2.9 Total awareness 78.2 90.7 Have not heard of 17.1 9.0 Don’t know / Refused 4.8 0.4 69
- 66. www.srcentre.com.au Gives boost to respondents most like non-respondents Location, Age, Sex, Awareness of official statistics, Trust in official statistics Logistic regression model predicts response probability from information available for both respondents and non-respondents Non-response adjustment (cont’d) 70
- 67. www.srcentre.com.au Non-response adjustment (cont’d) Awareness / Trust Unweighted (%) Without propensity weight (%) With propensity weight (%) With % - Without % Have heard of and trust a great deal 20.5 15.3 14.3 -1 Have heard of and tend to trust 54.6 52.6 47.5 -5.1 Have heard of and tend to distrust 10.5 11.1 9.9 -1.2 Have heard of and distrust a great deal 2.2 2.4 2.9 0.5 Have heard of but DK / Refused trust 2.9 3.4 7.5 4.1 Total awareness 90.7 84.8 82.1 -2.7 Have not heard of 9.0 14.5 17.1 2.6 Don’t know / Refused 0.4 0.7 0.9 0.2 71
- 68. www.srcentre.com.au Non-response adjustment (cont’d) Assumptions Refusals who provided basic information are representative of all non-contacts Refusals and respondents answer non- response items in the same way 72
- 69. www.srcentre.com.au Non-response adjustment (cont’d) Has face-validity, though… For example, persons with university education are typically over- represented in RDD surveys Therefore awareness estimates may have been inflated 73
- 70. www.srcentre.com.au Techniques to reduce bias in non- probability samples Dina Neiger
- 71. www.srcentre.com.au Data The Social Research Centre Cancer Council Victoria Slides and Ideas Darren Pennay Andrew C Ward Paul J Lavrakas Tina Petroulias Sebastian Misson Charles DiSogra Inferences from Non Probability Samples Workshop participants, Paris 2017 75 Acknowledgments
- 72. www.srcentre.com.au Key background references ➢ Yeager, D. S., Krosnick, J. A., Chang, L., Javitz, H. S., Levendusky, M. S., Simpser, A., (2011) o Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples. Public Opinion Quarterly ➢ DiSogra, C., Cobb C., Chan E., Dennis J. M. (2011) o Calibrating Non-Probability Internet Samples with Probability Samples Using Early Adopter Characteristics. Section on Survey Research Methods – JSM 2011 ➢ Valliant, R., Dever, J. A. (2011) o Estimating propensity adjustments for volunteer web surveys, Sociological Methods & Research, 40(1), pp. 105-137) ➢ Terhnian, G., Bremer, J., Haney C., (2014) o A model based approach for achieving a representative sample ➢ Fahimi, M., F. M. Barlas, R. K. Thomas and N. Buttermore (2015) o Scientific Surveys Based on Incomplete Sampling Frames and High Rates of Nonresponse. Survey Practice. 8 (5) 76
- 73. www.srcentre.com.au Usual starting point ➢ Design weight – inverse of probability of selection o Probability sample: number of people in the household, number of landlines, etc. o Non-probability sample: unknown, usually given design weight=1 ➢ Calibration (post-stratification, raking, RIM) o Uses external measures as “benchmarks” to adjust/weight (calibrate) the data to improve accuracy and force estimates to be consistent with other data sources o Applied to both probability and non-probability samples for key population demographics e.g. gender, age, education to reflect population distribution o If non-probability sample uses quotas for the same demographics then post- stratification will have minimal impact 77
- 74. www.srcentre.com.au Selecting benchmarks Should be a known source of non-response bias and likely to have an effect on survey estimates. Must have information for each sample member individually and the population as a whole ➢ Identical or as close to as possible (e.g. the same survey question and the census) Stratified sample: ➢ Unequal probability of selection within population subgroups (markets / strata) ➢ Population totals within each subgroup/stratum must be included as part of the weights Common variables in general population surveys: ➢ Telephone status (for dual frame surveys) ➢ Gender ➢ Age by education ➢ Country of birth (Australia V Other English Speaking Country V Non-English Speaking Countries) ➢ Geography (State, Metro V regional) 78
- 75. www.srcentre.com.au Is your adjustment working? Key considerations: ➢ Variance (probability samples) and bias Variance: ➢ One sample of many possible samples, how different would your result be if you happened to select a different random sample? ➢ If use too many benchmarks (or data is severely biased), weights will become extreme increasing variance of your estimate and decreasing your confidence in the accuracy of your result Bias: ➢ Difference from your survey result to the true value ➢ If ignore a known source of non-response bias then survey results will be further away from the truth ➢ Need to know the truth in order to measure ➢ Bias measurement dilemma - if already know the truth why bother with the survey? What about weighting efficiency? ➢ A measure of how much work a weight is doing ➢ What is an acceptable minimum? No standard ➢ Can be misleading especially in non-probability or highly biased samples (e.g. weight=1) 79
- 76. www.srcentre.com.au Case study 1: The Online Panels Benchmarking Study ➢ Three surveys based on probability samples of the Australian population aged 18 years and over ➢ Five surveys of persons aged 18 years and over administered to members of non- probability online panels ➢ Survey questionnaire included a range of demographic questions and questions about health, wellbeing and use of technology ➢ Same questions were used across the eight surveys (Unified approach to questionnaire design to try and minimise mode effects) ➢ 9 minutes average interview length for online and telephone (12 page booklet) ➢ Fieldwork: Oct – Dec 2015 ➢ Data and documentation available from the Australian Data Archive https://www.ada.edu.au/ada/01329 80
- 77. www.srcentre.com.au Results – substantive health characteristics (modal response) 81 Substantive variables Benchmark value Distance from benchmarks Probability Non-probability ABS ANU Poll DF RDD P1 P2 P3 P4 P5 Life satisfaction (8 out of 10) Percentage point error 32.6 -2.0 -2.0 1.9 -11.9 -11.6 -4.5 -9.2 -7.9 Psychological distress - Kessler 6 (Low) Percentage point error 82.2 -10.6 -11.6 -8.1 -25.9 -23.5 -22.2 -25.0 -23.2 General Health Status (SF1) (Very good) Percentage point error 36.2 0.4 -2.0 -2.6 -4.1 -5.8 -5.3 -5.0 1.5 Private Health Insurance Percentage point error 57.1 3.4 1.9 3.3 -8.9 -12.5 -3.7 -0.6 -2.6 Daily smoker Percentage point error 13.5 -4.1 3.5 1.6 9.8 6.7 3.9 2.7 4.3 Consumed alcohol in the last 12 months Percentage point error 81.9 3.6 -2.8 -4.0 2.4 5.3 3.9 4.2 1.5
- 78. www.srcentre.com.au Impact on weighting Impact of weighting on average absolute error: The average difference (percentage points) across all benchmarks between the official statistics and the survey estimates. 82 Probability Surveys Non-probability Panels ABS ANU Poll RDD P1 P2 P3 P4 P5 Unweighted (avge. error) 4.28 3.68 4.63 9.34 10.35 7.28 7.20 6.41 Weighted (avge. error) 4.02 3.98 3.58 10.5 10.9 7.24 7.78 6.83 Impact ✓ ✓ -
- 80. www.srcentre.com.au Design weight: Non-probability sample ➢ Current approaches: o Best case scenario • Model based selection to address known biases (e.g. quotas) o Worst case scenario: • Participation is a combination of idiosyncratic contacts, low response rates, and non-coverage. • This is not the kind of design one would choose if there were an affordable alternative. • (Doug Rivers, Comments on the 2013 AAPOR Taskforce Report on Non-probability Sampling” (Douglas Rivers; Comment. J Surv Stat Methodol 2013; 1 (2): 111-117. doi: 10.1093/jssam/smt009) ➢ Possible alternative: o Adapt propensity response model to model likelihood of being part of the non-probability sample o Need probability reference sample 84
- 81. www.srcentre.com.au Probability reference sample ➢ Gold standard that we’d like non-probability sample to resemble as closely as possible o Known to produce accurate estimates for key outcomes o Ideally includes data that can be compared to independent benchmarks ➢ Needs o Common data items with the non-probability estimate • As a minimum: demographic and attitudinal and outcomes of interest o Comparable data • Mode (depending on the questionnaire) • Question wording • Reference timeframe 85
- 82. www.srcentre.com.au Case study 2 LinA as a reference sample for OPBS ➢ Life in Australia o http://www.srcentre.com.au/our-research#life-in-aus o Australia’s first and only probability-based online panel o Launched in December 2016 o 3,300 adults from across Australia randomly recruited via a dedicated dual-frame telephone survey, Aged 18 years and over, Online and offline population o Replicated online panel benchmarking study in February 2016 ➢ OPBS non-probability panels reweighted o Design weight=1 o Post-stratification weight matched to LinA weighting o Add enrolment to vote to the substantive characteristics 86
- 83. www.srcentre.com.au LinA as a reference sample 87 Substantive variables Benchmark value (%) Distance from benchmarks (percentage point difference from benchmark) LinA Non-probability Panels 1 2 3 4 5 Life satisfaction (8 out of 10) 32.6 -1.4 -11.1 -13.6 -5.9 -10.3 -8.0 Psychological distress - Kessler 6 (Low) 82.2 -21.3 -26.5 -25.5 -22.7 -25.1 -23.4 General Health Status - SF1 (Very good) 36.2 -3.8 -4.2 -4.2 -4.0 -5.3 0.1 Private Health Insurance 57.1 2.6 -8.0 -9.8 -3.9 0.1 -2.0 Daily smoker 13.5 -1.0 9.0 6.0 3.5 2.2 3.5 Consumed alcohol in the last 12 months 81.9 2.7 -1.1 -6.4 -3.6 -4.9 -1.7 Enrolled to vote 78.5 9.0 8.4 7.6 10.7 8.8 13.1 Average absolute difference 5.23 8.54 9.14 6.79 7.09 6.48
- 84. www.srcentre.com.au Back to model-based design weights ➢ Use reference sample to calculate o Propensity scores – conditional probability that a respondent in the non-probability sample rather than in a probability sample given observed characteristics of the respondent • Logistic model (R PracTools package, Valliant et al. 2015) • Non-probability cases=1, reference cases=0 • Design weight for non-probability sample - inverse of estimated probability of inclusion in non- probability sample given weighted reference sample • Beware: extreme weights, size and quality of the reference sample ➢ Result: o Probability-based design weights for non-probability sample o Now ready to look at calibration/weighting adjustment methods 88
- 86. www.srcentre.com.au No agreed approach – work in progress ➢ Starting point – known biases in the non-probability sample ➢ For example, for online panels o Heavier internet users o Heavier media users o More interested in technology o Early adopters skew o More health care card holders ➢ Common theme o Standard demographics are not enough o High quality “official” benchmarks not available ➢ Probability reference sample to the rescue! o As long as items that are known/suspected biases in the non-probability sample are included 90
- 87. www.srcentre.com.au Cast study 2 ➢ Significant differentiators o Early adopter variables o Internet usage variables o Income o Employment o Remoteness o Home ownership o Media consumption variables were not included in the survey ➢ Incorporate one additional variable at a time compare with benchmarks to evaluate impact on bias 91
- 88. www.srcentre.com.au Options for EA inclusion in calibration ➢ Each EA variable is included as a 0/1 variables where o 1 means agreed or strongly agreed with that statement and o 0 all else (disagreed or strongly disagreed, did not respond) ➢ Scale derived using Rasch model o Bond, T. G. and C. M. Fox (2007). Applying the Rasch model: Fundamental measurement in the human sciences. (2nd ed.) Mahwah, N.J.: Erlbaum. ➢ Categorical scale o None – did not agree or strongly agree to any of 5 statements o Some – agreed or strongly agreed with at least one and no more than 2 out of 5 statements o High – agreed or strongly agreed with 3 or more out of 5 statements ➢ Agreement total calculated as the number of statements (min 0, max 5) that the respondent agreed or strongly agreed with 92
- 89. www.srcentre.com.au Options for EA inclusion in calibration Composite is better than individual variables 93 6.2 2.5 -0.4 -1.6 2 -1.5 -4.4 -5.5 7.8 -0.3 -1.5 -2.4 -8 -6 -4 -2 0 2 4 6 8 10 EA1-EA4 EA1-EA5 Rasch Scale EA1-EA5 Categorical Scale EA1-EA5 Agreement total %change Impact of EA variables on bias (% change) Ave abs error compared to unweighted Ave abs error compared to std adjustment Ave RMSE compared to std adjustment
- 90. www.srcentre.com.au Internet usage measures Not useful when calibrating online to online 94 2.6 5 4.5 5.5 -1.5 0.9 0.3 1.3 0.1 2 6.4 1.9 -2 -1 0 1 2 3 4 5 6 7 Look for information Access at home, Look for information Look for information, Post to blogs etc, Financial transactions, Social Media Access at home, Frequency of use %change Impact of Internet Usage measures on bias (% change) Ave abs error compared to unweighted Ave abs error compared to std adjustment Ave RMSE compared to std adjustment
- 91. www.srcentre.com.au Socio-economic variables in calibration Income – single most influential variable to reduce bias and RMSE Benefit from including both: income and employment 95 -7.3 -10.3 -7.3 -5.9 -11.0 -13.9 -10.9 -9.6 -8.5 -10.6 -5.8 -4.6 -16.0 -14.0 -12.0 -10.0 -8.0 -6.0 -4.0 -2.0 0.0 Income Income, Employment Income, Employment, Home Ownership Income, Employment, Home Ownership, Remoteness %change Impact of Income, Employment and Home Ownership variables on bias (% change) Ave abs error compared to unweighted Ave abs error compared to std adjustment Ave RMSE compared to std adjustment
- 93. www.srcentre.com.au Impact of calibration (LinA: Income, Employment, EA total agreement score) 97 Benefit of adjustment is sample dependent! Standard adjustments do not help with bias reductions!! Average Absolute Error % Change in bias Unweighted Std adjustments Design weights and key differentiators From unweighted From std adjustment Panel 1 9.2 9.8 7.8 -15.1 -19.8 Panel 2 10.0 10.4 9.3 -7.2 -10.9 Panel 3 7.7 7.7 5.3 -31.0 -31.5 Panel 4 7.4 8.1 7.1 -3.3 -11.9 Panel 5 7.4 7.4 7.8 5.4 5.7 All Panels 8.3 8.7 7.5 -10.4 -13.9
- 94. www.srcentre.com.au Comparison with independent benchmarks vs reference sample benchmarks 98 Mostly change in bias is consistent but there can be large differences for some samples -15.1 -7.2 -31.0 -3.3 5.4 -26.9 5.7 -31.7 28.7 1.0 -40.0 -30.0 -20.0 -10.0 0.0 10.0 20.0 30.0 40.0 Panel 1 Panel 2 Panel 3 Panel 4 Panel 5 % Change in bias from unweighted Independent Benchmarks Reference Sample -19.8 -10.9 -31.5 -11.9 5.7 -30.6 0.6 -31.5 -12.4 -8.3 -50.0 -40.0 -30.0 -20.0 -10.0 0.0 10.0 20.0 Panel 1 Panel 2 Panel 3 Panel 4 Panel 5 % Change in bias from std adjustment Independent Benchmarks Reference Sample
- 95. www.srcentre.com.au What about blending? ➢ Adding probability sample to non-probability sample will further decrease the bias ➢ Confirmed that o Combining probability and non-probability data greatly reduces variability across panels and has much larger impact than calibration alone o Use the best probability sample to combine with non probability regardless of mode and response rate ➢ But… ➢ Costs and feasibility of running parallel probability sample o Push to web – costs (incentives, follow up), timeliness o Listed vs RDD mobile – option for blending 99