Potential Solutions to the Fundamental Problem of Causal Inference: An Overview
- 1. Potential Solutions to the Fundamental Problem of Causal Inference: An Overview Day 1, Lecture 2 By Caroline Krafft Training on Applied Micro-Econometrics and Public Policy Evaluation July 25-27, 2016 Economic Research Forum
- 2. The fundamental problem • In program evaluation, want to know the impact of the program (“treatment”) on participant outcomes • In the real world, participation in programs and the impact of public policies is difficult to identify • Participation is likely to be related to characteristics that also affect outcomes • Endogeneity: assignment to treatment is not random • Not only depends on observables, but may also depend on unobservables • Both observables and unobservables may affect the outcome 2
- 3. Solutions • Random assignment • Quasi-experimental solutions • Type I: Conditional exogeneity of placement • Difference-in-difference • Panel data (fixed and random effects) • Propensity score matching • Type II: Rules or instruments of placement • Control function and instrumental variables techniques • Regression discontinuity design 3
- 5. Random experiments • Random experiments are often referred to as Randomized Controlled Trials (RCTs) • Random allocation of intervention to program beneficiaries such that all units (within a defined set) have equal chance ex ante of receiving the treatment • Assignment process creates treatment and control groups that are directly comparable • Should not have any observable or unobservable differences • By eliminating selection bias, randomization allows direct comparison of participants and non-participants to detect impact of program • Observed ex post differences in mean outcomes between treatment and control group can be attributed to program 5
- 6. Problems with Experimental Designs • Ethical and political obstacles • Difficult to randomize at level of individual beneficiaries • Those assigned to treatment group may decline to participate or participate in a partial manner • This is referred to as selective compliance • Selection bias gets introduced through this self-selection process • Those not selected and assigned to the control group may try to find alternative ways to get benefit of program • “Contamination” of control group 6
- 7. Case Study: The Labor Market Impact of Youth Training in the Dominican Republic: Evidence from a Randomized Evaluation Card et al. (2007) 7
- 8. Case study of a randomized evaluation • From 2001 to 2005 the government of the Dominican Republic operated a subsidized training program, Juventud y Empleo (JE) • Targeted low-income youth (18-29) with less than a secondary education in urban areas • Several weeks of classroom training (basic skills & vocational skills) by private training institutions • Followed by an internship at a private sector firm • Program was evaluated in: • Card, David, Pablo Ibarraran, Ferdinando Regalia, David Rosas, Yuri Soares (2007). “The Labor Market Impact of Youth Training in the Dominican Republic: Evidence from a Randomized Evaluation.” National Bureau of Economic Research Working Paper 12883. 8
- 9. Structure of the evaluation • JE program was unique in incorporating a randomized design • Each time 30 eligible applicants were recruited, 20 of the 30 were assigned to training (treatment), 10 to control. • Up to 5 individuals from control could be re-assigned to treatment if those assigned to treatment failed to show up for training (no- shows) or dropped out in the first two weeks (dropouts) • Evaluation looks at the second cohort of the JE program • Trained in early 2004 • Baseline data from registration form (prior to randomization) • Follow up survey in summer 2005 (~1 year after training) 9
- 10. Sample of the evaluation • Second cohort consisted of 8,391 eligible applicants • 5,802 (69.1%) assigned to treatment • 1,011 dropouts or no shows • 2,589 (30.9%) controls • 966 reassigned • Led to realized treatment group of 5,757 and realized control group of 1,623 • Only these groups have follow-up data • Evaluation based on stratified sampling of realized treatment and control • Problem of missing post-program data on no-shows and dropouts • Will bias results if this group is non-random. 10
- 11. Outcomes • Labor market outcomes examined: • Employment • Hours of work • Hourly wages • Job with health insurance 11
- 12. • Table 1: comparison of mean characteristics of realized treatment and control groups • Compared to labor force survey of 2004 data for comparable sample • Some differences in education 12
- 13. 13 • Check the initial assignment and re-assignment to see if realized status is “as good as random” • Multinomial logit model for being in each group shows some significant age effects but small explanatory power • Present re- weighted (“balanced”) results as well as unadjusted
- 14. • Examine employment rates • See no impact on participant employment rate • 57% of treatment v. 56% of controls • No differences among sub-groups 14
- 15. 15 • No impact on employment, hours of work, some differences in monthly earnings • Some marginally significant impact on hourly wages of about 10% • No significant differences in health insurance
- 16. Lessons from Card et al. 2007 • Randomization is the “gold standard” but reality of randomization usually less than perfect • Imperfect compliance • “Contamination” (reassignment) of controls • Potential selection bias due to no-shows and drop-outs • Potential dilution of impact for partial participation • Still have to check assumptions and correct for selection in many randomized evaluations. 16
- 18. Causal Effects in Non- Experimental Evaluations • We want to identify the causal impact of a program or policy • Typically we do not have experimental data (undertaking a non- experimental evaluation) • Referred to as quasi-experiments • To estimate a causal effect in non-experimental evaluations we need “identifying assumptions” • Non-experimental methods can be classified into two types depending on the identification assumptions they make. • Type I: “conditional exogeneity of placement” or “conditional exogeneity of placement to changes in outcomes” • Type II: instrumental variables or discontinuities that can explain placement can be found 18
- 19. Non-Experimental Methods Type I Non-Experimental Methods • 1- Regression Methods • 2- Propensity Score Methods • 3- Difference in Difference Methods • 4- Panel data (fixed or random effects) models • Type II Non-Experimental Methods • 4- Instrumental Variable Methods • 5- Regression Discontinuity Design Methods (RD or RDD) 19
- 20. Causal Inference in Type I Non- Experimental Methods • Type I non-experimental methods make the following identification assumptions: • Conditional exogeneity of placement (i.e. that placement only depends on exogenous observable characteristics X and not on unobservables) • Often referred to as “selection on observables” OR • Exogeneity of placement with respect to changes in outcomes (i.e. that unobservable factors affecting changes in outcomes do not affect the probability of placement) • Unobservables that determine placement can affect initial conditions but are assumed not to affect changes in outcomes over time 20
- 21. Type I Methods: First and second differences • Under “conditional exogeneity of placement”, all we need to do is compare outcomes for a treatment and control group at one point in time controlling for observables X – This is called a first difference approach (see D(X) estimator) • Under the weaker “exogeneity of placement to changes in outcomes” we need to compare the difference from before and after the program for a treatment group to the same difference for a control group. – This is called difference-in-difference or a second difference approach 21
- 22. Type I Methods: Propensity Score Matching & Weighting • Assumes conditional exogeneity of placement (selection on observables) • Models that selection process with a probit or logit model to predict the probability of participation, Pr(T=1) based on observable characteristics (X) • Creates “matched” treatment and control groups • After matching or weighting, no observable differences between groups • Can then estimate program impacts by looking at mean differences (ATT) between matched/weighted T and C groups 22
- 23. Type I Methods: Random and Fixed Effects • Often concerned about unobservables that are going to be related to an observable unit (school, family, city) • Panel data models assume that after controlling for the effect of that unit, the remainder of selection is fully observable • Random effects (RE) models assume the unobservable effects have some underlying (normal) distribution • REs assumed to be unrelated to observable X • Fixed effects (FE) models do not require parametric assumptions • FEs can be related to observable X 23
- 24. Causal Inference in Type II methods • Identifying assumptions: – There exists at least one (instrumental) variable (IV) that affects participation (placement) but that does not affect the outcome conditional on participation and other covariates (X)) – i.e. that the IV can be excluded from the outcome regression without causing omitted variable bias. This is called an “identifying restriction” – To be valid this IV must be exogenous – This called the instrumental variables approach • Regression discontinuity design (RD or RDD) is based on a similar assumption. • The instrument is some cutoff for eligibility/participation in the program • RD focuses on the differences in outcomes around that cutoff to model program impacts 24