Experimental Evaluation Methods

Course Overview
1. What is evaluation?
2. Measuring impacts (outcomes, indicators)
3. Why randomize?
4. How to randomize?
5. Sampling and sample size
6. Threats and Analysis
7. Scaling Up
8. Project from Start to Finish

Lecture Overview
 Unit and method of randomization
 Real-world constraints
 Revisiting unit and method
 Variations on simple treatment-control

UNIT AND METHOD OF
RANDOMIZATION

Unit of Randomization: Options
1. Randomizing at the individual level
2. Randomizing at the group level- “Cluster Randomized Trial”
Which level to randomize?

Unit of Randomization: Considerations
 What unit does the program target for treatment?
• Groups in microfinance.
• Political constituencies for governance projects.
• Schools for education projects.
 What is the “unit” of analysis?
• What are the outcomes we care about?
• At what level are we able to measure them?
• Examples-
 Test scores for school children.
 Health outcomes for individuals.
 Vote shares for politicians.

Unit of Randomization: Individual?

Unit of Randomization: Clusters?
“Groups of individuals”: Cluster Randomized Trial

Unit of Randomization: School?

How to Choose the Level
 Nature of the Treatment
• How is the intervention administered?
• What is the catchment area of each “unit of intervention”?
• How wide is the potential impact?
 Aggregation level of available data
 Power requirements
 Generally, best to randomize at the level at which the treatment is
administered.

Constraints: Political
 Not as severe as often claimed
 Lotteries are simple, common and transparent
 Randomly chosen from applicant pool
 Participants know the “winners” and “losers”
 Simple lottery is useful when there is no a priori reason to
discriminate
 Perceived as fair and transparent

Constraints: Resources
 Most programs have limited resources
• Vouchers, Farmer Training Programs
 Results in more eligible recipients than resources will allow services
for.
 More often than not, a lot of resources are spent on programs that
are never evaluated.
 Randomized experiments are usually no more costly than other
methods.

Constraints: Contamination Spillovers/Crossovers
 Remember the counterfactual!
 If control group is different from the counterfactual, our results can
be biased
 Can occur due to-
• Spillovers
• Crossovers

Constraints: Logistics
 Need to recognize logistical constraints in research designs.
 For example- Individual de-worming treatment by health
workers
• Many responsibilities. Not just de-worming.
• Serve members from both T/C groups
• Different procedures for different groups?

Constraints: Fairness
Is the randomization “fair”?
 Randomizing at the child-level within classes
 Randomizing at the class-level within schools
 Randomizing at the community-level

Constraints: Sample Size
 The program has limited scale
 Primarily an issue of statistical power
 Will be addressed tomorrow

What would you do?
An intervention proposes to study the
impact of regular washing of hands by
children on their attendance in school.
What would be your unit of
randomization?
A. Child level
B. Household level
C. Classroom level
D. School level
E. Village level
F. Don’t know
0% 0% 0% 0% 0% 0%
Child level
Household level
Classroom level
Village level
School level
Don’t know

Phase-in: Takes Advantage of Expansion
 Extremely useful for experiments where everyone will / should get
the treatment
 Natural approach when expanding program faces resource
constraints
 What determines which schools, branches, etc. will be covered in
which year?

Phase-in Design
Round 3
Treatment: 3/3
Control: 0
1
1
3 3
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3 3
3
3 3
3
3
3 3
3
Round 1
Treatment: 1/3
Control: 2/3
Round 2
Treatment: 2/3
Control: 1/3
Randomized
evaluation
ends

Phase-in Designs
 Advantages
• Everyone gets the treatment eventually
• Provides incentives to maintain contact
 Concerns
• Can complicate estimating long-run effects
• Care required with phase-in windows
• Do expectations change actions today?

Rotation Design
 Groups get treatment in turns:
• Group A benefits from the program in period 1 but not in in period
2
• Group B does not have the program in period 1 but receives it in
period 2

Rotation Design
Round 1
Treatment: 1/2
Control: 1/2
Round 2
Treatment from
Round 1 
Control
——————————————————————————
Control from
Round 1 
Treatment

Rotation Design
 Advantages
• Perceived as fairer
• Easier to get accepted (at least initially).
 Concerns
• Control group anticipates future eligibility
• AND Treatment group anticipates the loss of eligibility
• Impossible to estimate a long term impact.

Encouragement Design: What to do When you Can’t Randomize
Access
 Sometimes it’s practically or ethically impossible to randomize
program access
 But most programs have less than 100% take-up
 Randomize encouragement to receive treatment

What is “Encouragement”?
 Something that makes some folks more likely to use program than
others
 Not itself a “treatment”
 For whom are we estimating the treatment effect?
 Think about who responds to encouragement

Which two groups would you compare in an
encouragement design?
A. Encouraged vs. Not encouraged
B. Participants vs. Non-participants
C. Compliers vs. Non-compliers
D. Don’t know
0% 0% 0% 0%
Participants vs. Non-part...
Encouraged vs. Not enco...
Compliers vs. Non-compliers
Don’t know

Encouragement Design
Encourage
Do not encourage
Participated
Did not participate
Complying
Not complying
Compare encouraged to
not encouraged
These must be correlated
Do not compare
participants to non-participants
Adjust for non-compliance in
analysis phase

Randomization « In the Bubble »
 Sometimes there are as many eligible as treated individuals
• Suppose there are 2000 applicants
• Screening of applications produces 500 « worthy » candidates
• There are 500 slots: you can’t do a simple lottery.
 Consider the screening rules
• Selection procedures may only exist to reduce eligible
candidates in order to meet a capacity constraint
• It may be interesting to test the relevance of the selection
procedure and evaluate the program at the same time.
 Randomization in the bubble:
• Among individuals who just below the threshold

Randomization in “The Bubble”
Within the
bubble, compare
treatment to
control
Participants
(scores > 700)
Non-participants
(scores < 500)
Treatment
Control

To Summarize: Possible designs
 Simple lottery
 Randomized phase-in
 Rotation
 Encouragement design
• Note: These are not mutually exclusive.
 Randomization in the “bubble”
• Partial lottery with screening

Methods of Randomization: Recap
Design Most useful
when…
Advantages Disadvantages
Basic
Lottery
 Program
over
subscribed
 Familiar
 Easy to
understand
 Easy to
implement
 Can be
implemented in
public
 Control group
may not
cooperate
 Differential
attrition

Design Most useful
when…
Phase-In
 Expanding
over time
 Everyone
must receive
treatment
eventually
 Easy to
understand
 Constraint is
easy to
explain
 Control
group
complies
because they
expect to
benefit later
 Anticipation of
treatment may
impact short-run
behavior
 Difficult to
measure long-term
impact

Design Most useful
when…
Rotation
 Everyone
must receive
something at
some point
 Not enough
resources per
given time
period for all
 More data
points than
phase-in
 Difficult to
measure long-term
impact

Methods of randomization: Recap
Design Most useful
when…
Encouragement
 Program
has to be
open to all
comers
 When take-up
is low,
but can be
easily
improved
with an
incentive
 Can
randomize at
individual
level even
when the
program is
not
administered
at that level
 Measures impact
of those who
respond to the
incentive
 Need large
enough
inducement to
improve take-up
 Encouragement
itself may have
direct effect

What randomization method would you choose if your partner
requires that everyone receives treatment at some point in time?
A. Basic lottery
B. Phase-in
C. Rotation
D. Randomization in the bubble
E. Encouragement
F. Don’t know
0% 0% 0% 0% 0% 0%
Basic lottery
Randomization in the b...
Phase-in design
Rotation design
Don’t know
Encouragement

VARIATIONS ON SIMPLE
TREATMENT-CONTROL

Multiple Treatments
 Sometimes core question is deciding among different possible
interventions
 You can randomize these programs
 Does this teach us about the benefit of any one intervention?
 Do you have a control group?

Multiple Treatments
Control
Treatment 1
Treatment 2

Cross-cutting Treatments
 Test different components of treatment in different combinations
 Test whether components serve as substitutes or complements
 What is most cost-effective combination?
 Advantage: win-win for operations, can help answer questions for
them, beyond simple “impact”!
 An example?

Varying Levels of Treatment
 Some villages are assigned full treatment
• All households get double fortified salt
 Some villages are assigned partial treatment
• 50% of households get double fortified salt
 Testing subsidies, prices and cost-effectiveness!

Stratification
 Objective: balancing your sample when you have a small sample
 What is it?
• Dividing the sample into different subgroups
• Assign treatment and control within each subgroup

When to Stratify
 Stratify on variables that could have important impact on outcome
variable
 Stratify on subgroups that you are particularly interested in (where
may think impact of program may be different)
 Stratification more important when small data set
 Can get complex to stratify on too many variables
 Makes the draw less transparent the more you stratify

Mechanics of Randomization
 Need sample frame
 Pull out of a hat/bucket
 Use random number
generator in spreadsheet
program to order
observations randomly
 Stata program code
 What if no existing list?

Experimental Evaluation Methods

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Experimental Evaluation Methods (20)

Recently uploaded (20)

Experimental Evaluation Methods