FedCASIC 2019: Designing, implementing, and analyzing Leverage Saliency Theory experiments

icf.com
Designing, implementing, and
analyzing Leverage Saliency
Theory experiments
Wed, April 17, 2018
FedCASIC 2019 | Washington, DC
Matt Jans, Julia Sumner, Rachael Forando, John
Boyle, James Dayton, Lew Berman
ICF

ICF proprietary and confidential. Do not copy, distribute, or disclose.
What is Leverage Salience(y) Theory?
24/17/19 - FedCASIC 2019 - BLS

Leverage Saliency Theory (LST) Principals
(Groves, Singer, & Corning, 2000)
 Participation decisions are a combination of three things
1. How salient a specific survey design feature is (i.e., whether it’s noticed…kind of odd use of “salience”)
2. How much weight a potential respondent puts on that feature (i.e., how important it is to them)
3. And the valence of that weight (i.e., does it nudge them toward or away from participation?)
34/17/19 - FedCASIC 2019 - BLS

Salience and Leverage for a Respondent and
Nonrespondent
44/17/19 - FedCASIC 2019 - BLS

LST Example
 Respondents who are interested in birding will be more likely to
participate in a survey than people who are not
 And more likely when the survey is titled “Birders’ Survey” than “Survey of Hobbies”
54/17/19 - FedCASIC 2019 - BLS

LST Example
 The stated survey topic (a design feature) influences participation and
thus, birding activities (and related characteristics) are over-represented
64/17/19 - FedCASIC 2019 - BLS

LST Example
 Further, incentives will have less of an effect of people with high topic
interest…more of an effect on people with low topic interest
 Thus incentives help minimize nonresponse bias by increasing response from those with low topic interest
74/17/19 - FedCASIC 2019 - BLS

LST Example
 Further, incentives will have less of an effect of people with high topic
interest…more of an effect on people with low topic interest
 Thus incentives help minimize nonresponse bias by increasing response from those with low topic interest
 Interest in a topic leads people to think it’s important that they respond
84/17/19 - FedCASIC 2019 - BLS

Our Research Questions
94/17/19 - FedCASIC 2019 - BLS

LST Research Questions in the Dimensions of
Survey Participation Study
RQ #1: Incentive: Increasing incentive amount from $25 to $50 will lead to an
increase in stated willingness to participate?
104/17/19 - FedCASIC 2019 - BLS

RQ #2: Duration: Increasing duration form 1 hour to 2 hours will reduce stated
willingness to participate?
114/17/19 - FedCASIC 2019 - BLS

RQ #3: Interactions (moderation): Do incentive and duration interact to predict
participation willingness?
124/17/19 - FedCASIC 2019 - BLS

RQ #3: Interactions (moderation): Do incentive and duration interact to predict
participation willingness?
RQ #4: Generalizability of effects (LST): Do these main effects vary across …
4a: Survey topics and topic importance
4b: Types of respondents
- Demographics
- Survey experience
- Civic involvement and general interests
134/17/19 - FedCASIC 2019 - BLS

Principle #1: Start with problem/question/theory-
based research questions
 In this case, LST is the theory
 Write out problem statement (i.e., research questions) clearly
 State as hypotheses if possible
 Directional if possible
144/17/19 - FedCASIC 2019 - BLS

Designing our Experiment
154/17/19 - FedCASIC 2019 - BLS

Principal #2: Experimental factors flow from
research questions
 All respondents asked about…
 Importance/Unimportance of several topics (RQ #4b)
 Respondents randomized factorially to…
 Incentive level $25 v. $50 (RQ #1)
 Interview duration 1 hr. v. 2 hrs (RQ #2)
 Tabular example of the design
 Unimportant survey topic (RQ #4b)
 Interesting survey topic (RQ #4b)
164/17/19 - FedCASIC 2019 - BLS
$25 $50
1 hour % willing to participate % willing to participate
2 hours % willing to participate % willing to participate
$25 $50
1 hour % willing to participate % willing to participate
2 hours % willing to participate % willing to participate

Topic Importance
 Q39 “If you were asked to participate in government surveys on the
following topics, how important do you think it would be for you to
participate?”
 Very important
 Somewhat important
 Not too important
 Not important at all
Topics
 Education and schools
 Child care and problems of parents
 Medicare and aging
 Voting and elections
 Health and disease prevalence
 Nutrition and physical activity
 Issues facing the nation
174/17/19 - FedCASIC 2019 - BLS

Participation likelihood/willingness
 “Imagine you were contacted this week to participate in a government survey
about [INSERT “VERY IMPORTANT” OR “SOMEWHAT IMPORTANT” TOPIC
FROM Q39] that was conducted face to face in your home at a time of your
choice, took one hour [OR two hours], and you received $25 [OR $50],
 How likely would you be to participate in a survey like this?”
 Definitely would
 Probably would
 Might or might not
 Probably would not
 Definitely would not
184/17/19 - FedCASIC 2019 - BLS

Participation likelihood/willingness
 “Imagine you were contacted this week to participate in a government survey
about [INSERT “NOT TOO IMPORTANT” OR “NOT IMPORTANT” TOPIC
FROM Q39] that was conducted face to face in your home at a time of your
choice, took one hour [OR two hours], and you received $25 [OR $50],
 How likely would you be to participate in a survey like this?”
 Definitely would
 Probably would
 Might or might not
 Probably would not
 Definitely would not
194/17/19 - FedCASIC 2019 - BLS
Previous question with unimportant
topics filled.

Analyzing Results
204/17/19 - FedCASIC 2019 - BLS

Principal #3: Consider (i.e., plan) the analysis
approach before collecting data
 Easy for experimental studies
214/17/19 - FedCASIC 2019 - BLS

 First explore experimental factors through ANOVA or linear regression
(continuous outcome) or logistic regression (binary outcome)
 Code multi-category responses into binary ones (e.g., combining very likely and somewhat likely)
 Conduct statistical tests
– Significance of main effects and interaction
 Presentation visualization
– Bar charts of %s (binary outcome)
– Box and whisker plots (continuous outcomes)
– For interactions, “cross-over plots” are VERY helpful
224/17/19 - FedCASIC 2019 - BLS

 Second, compare significance and form of interactions between key
groups
 Respondents interested in topic should be less influenced by incentive and duration than those who aren’t
234/17/19 - FedCASIC 2019 - BLS

 Second, compare significance and form of interactions between key
groups
 Respondents interested in topic should be less influenced by incentive and duration than those who aren’t
 Third, explore whether adding additional predictors (e.g., R
characteristics change nature of interaction)
244/17/19 - FedCASIC 2019 - BLS

Analysis methods for our LST experiment
svy:logistic Q40_important_will_bin ///
int_dur_bin##inc_amt_bin
contrast int_dur_bin##inc_amt_bin
margins int_dur_bin#inc_amt_bin
marginsplot…
254/17/19 - FedCASIC 2019 - BLS
Incentive main effect
(F = 11.35, p = .0008)
Duration main effect
(F = 3.68, p = .0551)
Interaction btwn
Incentive and Duration
(F = 5.86, p = .0155)

Exploring the effect of R characteristics – Residence
location
svy:logistic Q40_important_will_bin ///
int_dur_bin##inc_amt_bin##town_large_city_bin
contrast int_dur_bin##inc_amt_bin@town_large_city_bin
margins int_dur_bin#inc_amt_bin#town_large_city_bin
marginsplot…
264/17/19 - FedCASIC 2019 - BLS
(F = 8.68, p = .0033)
Duration main effect
(F = 3.54, p = .0602
(F = 3.57, p = .0591)
Interaction btwn
incentive and duration
(F = 6.38, p = .0116)

Exploring the effect of R characteristics - Age
274/17/19 - FedCASIC 2019 - BLS

LST Design and Analysis Checklist
284/17/19 - FedCASIC 2019 - BLS

Checklists flow from the framework
 Principal #1: Start with problem/question/theory-based research
questions
 Define the problem statement or hypothesis you want to test
 Which design features do you want to manipulate? How many levels (and what values) will they have?
 Will you use a “captive audience“ and hypothetical response, or “field survey”
 What respondent (or sample member) sub-groups do you want to compare? What indicators of topic interest
do you have?
 Write specific research questions or hypotheses, including the direction of expected effect
294/17/19 - FedCASIC 2019 - BLS

 Principal #2: Experimental factors flow from research questions
Define the factors you want to study (i.e., comparisons you want to make)
 Consider counter-factuals (i.e., comparison and control groups)
 If you want to make causal inferences, make them experimental manipulations via random assignment
 Confirm that the experimental factors will answer your research questions (or adjust research questions)
304/17/19 - FedCASIC 2019 - BLS

 Principal #3: Consider (i.e., plan) the analysis approach before collecting
data
Choose modeling method (e.g., ANOVA, linear regression, logistic regression)
 Define how effects and interactions will be tested and reported (e.g., odds ratios v. coefficients)
 Determine visualizations that you will create before or during modeling and testing
314/17/19 - FedCASIC 2019 - BLS

ICF proprietary and confidential. Do not copy, distribute, or disclose. 32
Matt Jans: matt.jans@icf.com
4/17/19 - FedCASIC 2019 - BLS

FedCASIC 2019: Designing, implementing, and analyzing Leverage Saliency Theory experiments

More Related Content

Similar to FedCASIC 2019: Designing, implementing, and analyzing Leverage Saliency Theory experiments (12)

More from Lew Berman (14)

Recently uploaded (20)

FedCASIC 2019: Designing, implementing, and analyzing Leverage Saliency Theory experiments