Evaluación de impacto en microeconomía y finanzas

THE EVALUATION PROBLEM
Module ECO00074M, Evaluation of Health Policy
Spring term, session 2017/18
ANDREW M. JONES
DEPARTMENT OF ECONOMICS AND RELATED STUDIES

The evaluation problem
A fundamental problem arises in attempting to derive inference of a causal
relationship between a treatment, denoted d, and an outcome, denoted y.
The treatment effect of interest,  , is the change in potential outcome for
individual, i , when exposed to the intervention compared to an alternative (referred
to as the control) and can be defined as:
0
1
i
i
i y
y 


The evaluation problem is that an individual cannot be observed to be under
treatment and under the control at the same time. At any particular point in time
only one of the potential outcomes can be observed (Neyman, 1923; Roy 1951; Rubin
1974).

Average treatment effects
A common approach to addressing the evaluation problem is to focus on
average treatment effects (ATE). For example the population average
treatment effect (PATE) is the difference in the average potential outcomes
in the treated and control groups for the population as a whole:
1 0 1 0
PATE E y y E y E y
     
   
     
For a particular sample of data the analogue of the PATE is the sample
average treatment effect (SATE):
1 0
1 1
SATE y y
n n
 
 

ATTs
More often, the relevant concept is the treatment effect for the subset of the
population who would actually be assigned to treatment (e.g., Heckman, LaLonde
and Smith 1999). This is the treatment effect on the treated (ATT, or sometimes ATET
or TT).
The population average treatment effect on the treated (PATT) is:
1 0 1 0
| 1 | 1 | 1
PATT E y y d E y d E y d
     
      
     
The PATT represents the expected gain from the intervention for an individual
randomly selected from the treated population, rather than for any individual in the
general population. The sample analogue is the sample average treatment effect on
the treated (SATT).

Selection bias
Consider the difference between the population means of y for the treated
and controls, which can be decomposed as follows (see Heckman, Ichimura,
Smith and Todd 1998):
   
 
1 0
1 0 0 0
| 1 | 0
| 1 | 0
| 1 | 1 | 0
E y d E y d
E y d E y d
E y y d E y d E y d
PATT Bias
  
   
   
   
     
      
     
 

Sources of bias
The bias term can be further decomposed (Heckman, Ichimura, Smith and
Todd 1998; King and Zeng 2007):
 Failing to control for relevant confounders (omitted variable bias);
 Due to inclusion of covariates that are themselves affected by the
treatment (post-treatment bias);
 Due to failure to adequately control for covariates within the observed
range of data, for example when applying a linear model to a nonlinear
relationship (interpolation bias);
 Or, due to failure to adequately control for covariates when extrapolating
to areas outside the observed range of the data, for example if a linear
approximation holds within the sample but not beyond it (extrapolation
bias).

Cross section: no effect
Treatment Observation
t
y0 ( = y1)
Outcome - y
time

Cross section: genuine effect
t
y0
Outcome - y
time
y1

Cross section:
heterogeneous treatment effects
t
y(0)
y0
Outcome - y
time
y1

Cross section: selection bias - no effect
t
y0
Outcome - y
time
y1

Cross section: selection bias - genuine
effect
t
y0
Outcome - y
time
y1

Before and after: no selection bias, no
trend - no effect
t
y0 = y1
Outcome - y
time
t-1

Before and after: no selection bias,
with trend - no effect, biased
t
Outcome - y
time
t-1

Before and after: no selection bias,
with trend - biased estimate
t
Outcome - y
time
t-1

Difference in differences: control
group - no effect
t
y0
Outcome - y
time
y1
t-1

Difference in differences: control
group - genuine effect
t
y0
Outcome - y
time
y1
t-1

Methods for impact evaluation
- (Randomised) social experiments
- “Natural” or quasi-experiments – actual policy reforms with
controls (linked to instrumental variables, regression
discontinuity & diff-in-diffs)
- “selection on observables” - adjusting for covariates using
regression or matching approach
- “selection on unobservables” – instrumental variables,
selection models
- Structural simulation approach (ex ante evaluation)

Identification: Sources of variation
The following examples (among others) are used in:
Andrew M. Jones (2009) “Panel Data Methods and
Applications to Health Economics”, The Palgrave Handbook
of Econometrics Volume II: Applied Econometrics, Terence
C. Mills & Kerry Patterson (eds.), Basingstoke: Palgrave
MacMillan.

Randomised social experiments
RAND HIE
“Worms” : Miguel & Kremer (Ecta, 2004, 72:159-217)
Oregon HIE
PROGRESA: Gertler (AER, 2004, 94: 336-341)

PROGRESA
- The Mexican government’s PROGRESA programme, which was initiated in
1997, has received considerable attention and has influenced policy
throughout Latin America.
- The programme relies on conditional cash transfers that are designed to
influence the use of health and welfare services for children in poor
families and covers 2.6 million families in 50,000 rural villages.
- The programme focuses on health, hygiene and nutrition. It links
substantial cash transfers, on average amounting to 20-30% of household
income, to the use of prenatal care, well-baby care and immunization,
nutrition monitoring and supplementation, preventive check-ups and
participation in educational programmes.

- PROGRESA works by first selecting whole communities to participate in
the scheme and then selecting households within those communities that
satisfy the eligibility criteria to receive the benefits of the scheme.
- Financial constraints on the implementation of PROGRESA meant that its
introduction was phased. To make the implementation equitable
communities were selected randomly to receive the benefits immediately
or with a delay.
- The random phasing provides researchers with an ideal opportunity to
use a randomised design in the evaluation of the impact of the
programme. Of the communities selected for the programme 320 were
randomly selected to receive the intervention in August-September 1998
with the remaining 185 delayed for two years. The communities in the
control group were not informed that they would eventually receive the
programme.

Gertler (2004, AER)
- Gertler (2004) focuses on health outcomes among children. These include self-reported
morbidity, measured by illnesses in the past month as reported by the child’s mother, and
objective measures including anthropometric measures of height and stunting and a
biomarker for anaemia (haemoglobin levels).
- The analysis is restricted to those households, in both the treatment and control groups,
that satisfy the eligibility criteria for PROGRESA.
- Although the data is randomised multivariate regression models are used to control for
observed covariates, reducing idiosyncratic variation, and individual and village random
effects are included, the latter to allow for the clustered sampling.
- The results show significant improvements in both self-reported and objective measures
of health and the impact increases with length of exposure to the programme.
- Gertler (2004) is careful to note that the comparison of treated and controls does not
explain the mechanism behind this effect: for example, it is not possible to say whether an
unconditional transfer would have had the same effect as the conditional one.

Natural experiments: health shocks
- Almond (2006, JPE) makes use of the 1918 influenza pandemic as a natural
experiment for the 'fetal origins hypothesis'
- Cohorts that were in utero during the pandemic, between the Fall of 1918 and
January 1919, are shown to have poorer outcomes: lower educational
attainment, more disability, lower income, lower socioeconomic status and
higher transfer payments.
- Has the potential to be used as a natural experiment: it was unanticipated, the
period of exposure was short and the impact varied systematically across States.
- Uses discontinuity across birth cohorts to identify the long-term effects, drawing
on data from the 1960, 70 and 80 US Census microdata (which identify quarter of
birth).
- Geographic variation is also exploited, based on the 'laggard' States where the
epidemic had less pronounced long-term effects, although this reduces the
sample size available.

Economic shocks: Evans & Lien (2005, JEcts)
- Use of the 1992 Port Authority Transit (PAT) strike in Allegheny County, Pennsylvania as a
source of independent variation in access to prenatal care.
- Prenatal visits were affected most for black women and city residents (in Pittsburgh) and
the results show that, for these groups, missing visits early in pregnancy had a detrimental
effect but missing those later in the pregnancy did not.
- A control group of counties that were not affected by the strike are selected on the basis
of regression analyses.
- The use of prenatal care by women who were pregnant at the time of the strike is
included in regression equations for birth weight, gestation, maternal weight gain and
maternal smoking.
- Models are estimated by OLS and 2SLS, the latter using the strike as an instrument,
produce similar results, suggesting that selection bias is not a problem.
- The clearest effect of prenatal care is on maternal smoking.
- The robustness of the findings is tested by checking for general a decline in earnings or
employment coincident with the strike and for evidence of increases in abortions or
'unwanted' births.

Lindahl (2005, JHR)
- Lottery winnings can provide one source of 'exogenous variation' in income, in an attempt
to overcome the selection biases inherent in disentangling the socioeconomic gradient in
health.
- There is a statistically significant effect of income on morbidity and mortality and the
magnitude of this effect is largely unchanged when lottery winnings are used as an
instrument, although the estimates are less precise.
- Data from the Swedish Level of Living Surveys (SLLS) for 1968, 74 and 81 are matched with
register data on income and deaths up to 1997. Morbidity is measured by combining 48
symptoms into a standardised measure and mortality is measured as death within 5 or 10
years of the surveys.
- Lottery winnings are treated as a source of exogenous variation in income: assuming that
the variation is independent of health.
- Models are estimated with lottery winnings included directly. Then OLS and instrumental
variable estimates (using winnings as the instrument) are compared for the sample of
individuals who are identified as 'players'. The magnitudes of the income effects are
similar although standard errors are inflated when IV is used.

Natural controls: Siblings
- Holmund (2005, JHR) uses variation within biological sisters can be used to assess
the long-term consequences of teenage pregnancy for educational outcomes.
- The siblings approach and standard cross section methods produce similar
results so long as heterogeneity within the family is controlled for.
- The potential for selection bias is that teenage mothers may have family
backgrounds that would lead to poorer outcomes irrespective of an early
pregnancy. Using variation within biological sisters can be used to control for
these 'family effects'.
- However within-sibling variation will not deal with heterogeneity within the
family and the study controls for observable pre-motherhood school
performance, measured by the GPA from primary school, to try and control for
this.
- Data are taken from a 20 per cent sample of each cohort born in Sweden
between 1974 and 1977, with the population register used to identify siblings.
This is linked to census data on outcomes.

Twins: Black, Devereux and Salvanes (2007, QJE)
- Use Norwegian registry data to extend the use of twins studies to investigate the
impact of low birth weight on long-term socioeconomic outcomes rather than
just short-run outcomes.
- Within-twins fixed effects estimates are shown to be significant and similar to
standard least squares estimates for long-run outcomes, such as height, IQ,
earnings and education, while the estimates for short-run outcomes are smaller
for the twins data.
- The analysis is made possible by the richness of the data which uses personal
identifiers to link all Norwegian births between 1967-97, as recorded in the
Medical Birth Registry, with other registry data for those aged 16-74 in the
period 1986-2002. The register data is augmented with military records and a
survey of twins that identifies zygocity.
- Within-twin variation is used to capture unobservable socioeconomic and
genetic factors that may confound the causal effect of birth weight. This means
that identification stems from differences in nutrition in utero (resulting from

different placentas for fraternal twins and different positioning on the placenta
for monozygotic). Birth order is included as a control.
- The robustness of the findings is assessed by separate analyses for mothers who
have more than one singelton birth, allowing for mother fixed effects. To assess
the role of zygocity the sample is restricted to same-sex twins. Also the sub-
sample where there is survey data on zygocity is used.
- The findings are robust but reveal interesting evidence that those who
participate in twins studies are a selected sample.
- It should be borne in mind that selection into the sample of registry data for
long-run outcomes may be effected by infant mortality.
- Also there are substantial differences between twins and singletons in terms of
factors such as gestation and age of mothers and twins usually appear in the
lower part of the distribution of birth weights.

Anti tests & sensitivity/robustness
- Galiani et al’s (2005, JPE) evaluation of the impact of the privatisation of local
water services on child mortality in Argentina.
- They adopt two strategies for assessing the reliability of their difference-in-
differences approach that can both be interpreted as anti- or placebo tests.
- A placebo regression: the model of interest is estimated using only data from the
pre-treatment period, but including an indicator of those cases that will go on to
be treated. Tests the “parallel trends” assumption required for difference-in-
differences analysis is not valid.
- As well as measuring deaths from infectious and parasitic diseases they include
measures of deaths from causes unrelated to water quality. The fact that they
detect a reduction for the former but not for the latter creates confidence in
their difference-in-differences identification strategy.

Evaluación de impacto en microeconomía y finanzas

More Related Content

Similar to Evaluación de impacto en microeconomía y finanzas (20)

Recently uploaded (20)

Evaluación de impacto en microeconomía y finanzas