Jeffrey henning april lecture series - 2014

Jeffrey Henning, Researchscape, April Lecture Series 2014
Improving the Representativeness
of Online Surveys
Jeffrey Henning
Researchscape International
Event
Sponsors

Respondent Selection Issues
Sampling
Error
Coverage
Error
Nonrespons
e Error at
Unit
Response Accuracy Issues
Nonresponse
Error at Item
Measurement
Error due to
Respondents
Measurement
Error due to
Interviewers
Survey Administration Issues
Post-Survey
Error
Mode
Effects
Comparability
Effects
Total Survey Error

Niche
Survey
Topline
Survey
Probability
Survey
Mode Online Online Telephone
Target > 5%
incidence
> 20%
incidence
General
population
Respondents 100 400 400
Length 15 questions 25
questions
5 minutes
Cost/response $5 $5 $20
Price $495 $1,995 $7,995
Comparing Prices

Probability
sampling
Probability
online panels
Open online
panels
Weighting
Quota
sampling
Sample
matching
River
sampling
Intercept
samples
Practical
ramifications
Agenda

2.0%
3.3% 4.1% 4.7% 5.0% 5.3% 6.4% 6.4%
12.0%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
A B C D E F G H I
Average Absolute Errors
Source: Yeager, Krosnick, et al, 2011
Probability Non-probability

6.0
3.6
2.9
2.6 2.4 2.3
1.9 1.9
1.0
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
A B C D E F G H I
Accuracy = Value
Base = worst performing survey’s average absolute error

9.6%
11.7%
13.2% 13.7%
15.3% 15.6% 16.0%
18.0%
35.5%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
A B E F H G D C I
Largest Absolute Errors

Key Elements of
Probability Sampling
Coverage
• Known non-
zero chance of
selecting any
member of the
target
population
External
selection
• Random
selection of
members of
the population
to participate
in the survey

Robustness?
Any method with a low response rate is not a
random probability sample. We can’t assume a
known and non-zero chance of selection. This
is true of telephone, so for most studies the
gold standard is not a practical option, even if
money were no object. – Ray Poynter, director,
Vision Critical, 2013
What about the vast majority of research that
has 90% opt-out rates? Do we decide that
those people weren’t part of the population to
begin with? ...I’m just having a hard time
understanding the ongoing push to prove we
are using probability samples. – Annie Pettit,
Research Now, 2010

Robustness?
Any method with a low response rate is not a random
probability sample. We can’t assume a known and non-zero
chance of selection. This is true of telephone, so for most
studies the gold standard is not a practical option, even if
money were no object. – Ray Poynter, director, Vision
Critical, 2013
What about the vast majority of research that has 90% opt-
out rates? Do we decide that those people weren’t part of
the population to begin with? ...I’m just having a hard time
understanding the ongoing push to prove we are using
probability samples. – Annie Pettit, Research Now, 2010
Response rates were positively associated with
demographic representativeness, but only very
weakly... In general population RDD telephone
surveys, lower response rates do not notably reduce
the quality of survey demographic estimates.
– Holbrook, Krosnick, Pfent, 2008

Probability Online Panels
• Build a large panel using Address Based Sampling
– Relentlessly invite candidates to join the panel
– Provide computers or tablets and Internet
connectivity if needed
• Consistently perform as well as RDD
– Transitive property of probability sampling: a random
sample of a random sample is highly accurate even
though net response rates are low
– $900 per question from Knowledge Networks
– Perhaps the rise of smartphones will lead to new
mobile probability panels that hit a lower price point

Impractical for Low Incidence
Mothers of children 4 and under
Families with chronically ill members
Women who do yoga workouts
Adventure racing enthusiasts
Video game players
Board and card game purchasers
Purchasers of apps for smartphones and tablets
E-book purchasers
Purchasers of self-help books
Golfers
Small-business owners
Middle managers

Bye, Bye, Probability
But where randomized treatments
are not possible... we must do the
best we can with what is available
to us. - Donald T. Campbell, social
scientist, 1969

Open Online Panels
• Anyone can join the panel
– Panelists join for cash or prizes
– Many field surveys through web intercepts,
collecting responses from non-panelists
• Inconsistent results
– A random sample of a convenience sample is still
a convenience sample
– Random sampling does produce greater
consistency for longitudinal studies

Examples of Online Panel Results
17%
69%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Panel U.S. Census
35+
18-34

Bias of People Not on Internet

Weighting
• Post-stratification weighting viewed as a
common solution to removing sampling bias
from convenience samples
• Often misrepresented as a simple process of
arithmetic

Cell Weighting
Men Women
18 to 54
79,184,164
169 responses
469K weight
79,017,200
199 responses
397K weight
55+
36,301,576
15 responses
2,420K weight
43,154,705
17 responses
2,539K weight

Rim Weighting / Raking
Age
Gender
Region
Race/ethnicity
Education
level
Household
income
Proprietary
measure

Weighting
• Implicit assumption is people we did survey in a
particular demographic group are representative
of the people we did not survey in that group
• Many researchers weight convenience samples...
– In the hope it does no harm
– In the belief it improves quality
– For the fact it redistributes demographics to match
target population

Wait, Wait
Waiting until the weighting stage to adjust is
too late. The combination of coverage error and
nonresponse in online panels generally creates
a sample that is beyond fixing post hoc. We
need to do more at the selection stage.
- Reg Baker, former president and COO of
Market Strategies, 2013

Quota Sampling
Men Women
18 to 54 79,184,164
169 133 responses
595K weight
79,017,200
199 132 responses
599K weight
55+ 36,301,576
15 61 responses
595K weight
43,154,705
17 72 responses
599K weight

• Divide the sample into cells and recruit to fill
those cells
• Once 51% of respondents are women, stop
accepting responses from women
• Each quota increases price:
– $1,000 for no quota
– $1,500 for 3 quota cells
– $2,000 for 12 quota cells
• Bad reputation among public opinion
researchers, good reputation among corporate
researchers
Quota Sampling

Quota Sampling
• Good reputation among corporate researchers
• Bad reputation among public opinion
researchers

Quota Sampling

Sample Matching
• Rim weighting : cell weighting = Sample
matching : quota sampling
• Imagine trying to fill 400 cells:
– 57-year old African American woman with
associates degree in Newton Falls, OH
– 21-year old white male high school graduate in
Worcester, MA
• Where do the “cells” come from?

YouGov Model of U.S. Population
• 2010 American Community Survey
– Age
– Gender
– Race
– Education
– Region
• Imputation from Registration and Voting Supplements
– Voter registration
• Imputation from Pew Religion in American Life Survey
– Religion
– Political interest
– Minor party identification
– Non-placement on an ideology scale

Sample Matching
• Finding 57-year old African American woman
with associates degree in Newton Falls, OH
• Proximity function tests all members of panel,
calculating distance from target (distance in
age, gender, physical location, etc.)
• Invite 59-year old African American woman
with GED in Warren, OH

River Sampling

Pros & Cons of Steady Panel Participation
Practice Effects (Major)
• Regularly answering
surveys may improve
accuracy of responses
• Panel members may
become more
introspective and self-
aware, improving their
reporting
• Respondents’ answers
to attitudinal questions
improve with practice
Panel Conditioning (Minor)
• “Stimulus hypothesis”
that acting about future
activity prompts that
activity
• Past surveys makes
panelists less like general
population
• Panelist attrition
nonrandomly affects
panel representativeness
Source: Chang & Krosnick, 2008

70% of NPD Panelists are
Introverts vs. 50% in U.S.
[Diligent panelists are] high on introversion,
have a high need for cognition, enjoy thinking,
and prefer complex to simple problems, and
they like surveys – they find surveys
worthwhile. - Inna Burdein, direct of panel
analytics for NPD, 2013

Intercept Surveys

Key Elements of Probability Sampling?
Coverage
• Known non-
zero chance of
selecting any
member of the
target
population
External
selection
• Random
selection of
members of
the population
to participate
in the survey

Mimicking Probability Sampling
Coverage
• Known non-
zero chance of
selecting any
member of
the target
population
External
selection
• Random
selection of
members of
the population
to participate
in the survey
Sample matching –
Random selection of
members of the
population to match
in the panel
Probability panel –
Random selection of
randomly recruited
panelists
Weighting –
Correcting for
demographic
underrepresentation
Margin of error –
AAPOR is against
reporting margin of
error for non-
probability samples
Open panel –
Random selection of
panelists

Recommendations
• When sourcing sample, ask for steps taken
to minimize sampling bias
• When evaluating panels, ask how they
select respondents for a given study
• Don’t use weighting if sample was
significantly demographically unbalanced
• Don’t report sampling error but do
consider reporting de factor error ranges

For Further Reading
Free 125-page report from the
American Association for Public
Opinion Research:
http://bit.ly/AAPOR2013

Thank you!
Jeffrey Henning

The Sponsors for this Event
If you are interested in sponsoring a future NewMR event
Email Michele.Poynter@TheFuturePlace.com
Event
Sponsors

Q & A
Ray Poynter
The Future Place
Jeffrey Henning
Researchscape

Jeffrey Henning, PRC
Up-to-date Research on the
Changing Consumer
Toll Free: +1 (888) 983-1675 x 701
jhenning@researchscape.com
http://www.researchscape.com/
http://www.twitter.com/jhenning

Jeffrey henning april lecture series - 2014

More Related Content

Viewers also liked (18)

Similar to Jeffrey henning april lecture series - 2014 (20)

More from Ray Poynter (20)

Recently uploaded (20)

Jeffrey henning april lecture series - 2014