IFPRI - The Problem of Impact Evaluation
2 likes6,415 views
The document discusses the challenges of impact evaluation in programs, focusing on confounding factors and selection biases that affect the reliability of assessments. It outlines methods to compare treatment and control groups, emphasizing the need for comprehensive data collection, including both qualitative and quantitative approaches. The proposed evaluation method, using double difference techniques, aims to accurately measure program impacts while accounting for various influencing factors.
IFPRI - The Problem of Impact Evaluation
- 1. Tuesday, April 01, 2014 Problems of Impact evaluation Confounding factors and selection biases
- 2. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 2 Objective of Impact Evaluation Measure the effect of the program on its beneficiaries (and eventually on its non-beneficiaries) by answering the counterfactual question: • How would individuals who participated in a program have fared in the absence of the program? • How would those who were not exposed to the program have fared in the presence of the program? Two main problems arise: confounding factors and selection biases.
- 3. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 3 Comparing averages • Individual-level measure of impact : what would be the outcome (e.g. purchase patterns) had he/she not participated to the program (in our case the treatment? • Compare the individual with the program, to the same individual without the program, at the same time ? Pb: can never observe both, missing data problem. • Instead: Average impact on given groups of individuals • Compare mean outcome in group of participants (Treatment group) to mean outcome in similar group of non-participants (Control group) • Average Treatment effect on the treated (ATT):
- 4. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 4 Building a control group • Compare what is comparable. • Treatment” and “Control” groups must look the same if there was no program. • But: very often, those individuals who benefit from the program initially differ from those who don’t. • External selection: programs are explicitly targeted (Particular areas, Particular individuals). • Self selection: the decision to participate is voluntary. Pb with comparing beneficiaries and non-beneficiaries: the difference can be attributed to both the impact or the original differences. • SELECTION BIAS when individuals or groups are selected for treatment on characteristics that may also affect their outcomes.
- 5. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 5 Initial Population Selection Treatment Group (receives procedure X) Impact = Y Exp – Y Control Quintile I (Poorer) Quintile II Quintile III Quintile IV QuintileV (Richer) Program selection does not lead to selection bias (from Bernard 2006) Control group (does not receives procedure X)
- 6. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 6 Initial Population Quintile I (Poorer) Quintile II Quintile III Quintile IV QuintileV (Richer) Control group (does not receives procedure X) Treatment Group (receives procedure X) Program selection leads to selection bias Selection Impact ≠ Y Exp – Y Control
- 7. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 7 “Sign” of the selection bias (1) Program targeted on “worse-off” households Treatment Control Observed difference is negative Actual impact
- 8. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 8 Treatment Control Observed difference is very large Actual impact “Sign” of the selection bias (2) Program targeted on “better-off” households
- 9. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 9 Exercise 1. Detail how confounding factors may be an issue in evaluating the impact of your project. 2. Suppose that you were to compare households in communities were the project was implemented to households in the neighboring communities were the project was not implemented. - What would be the likely sign of the selection bias? 3. Suppose that you were to compare, within the communities were the project is implemented, households who have decided to use the project (e.g. drink water from the tap or build stone bunds in their field), to the ones who have decided not to use it. - What would be the likely sign of the selection bias?
- 10. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Step 3: What data to collect -Collect qualitative data and quantitative data on both treatment and control households in the baseline • Qualitative data-key supplement to quantitative IE providing complementary perspectives on program’s performance. • Approaches include FGD, expert elicitation, key informant interviews • Useful 1. Can use to develop hypotheses as to how and why the program would work • 2. Before quantitative IE results are out, qualitative work can provide quick insights on happenings in the program. • 3. In the analysis stage, it can provide context and explanations for the quantitative results Page 10
- 11. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Mixed methods- Quantitative and qualitative : • Possible rationale: • Triangulation: to cross-check and compare results and offset any weaknesses in one method by the strengths of another; • Complementarities: examining overlapping and different facets of a phenomenon by using several approaches and tools; • Initiation: discovering paradoxes, identifying contradictions, or obtaining fresh perspectives that relate to the topic of investigation; • Development: using quantitative and qualitative methods sequentially, such that results from the first method inform the use of the second method and vice versa; and • Expansion: adding breadth and scope to a project to convey findings and recommendations to audiences with different capabilities and interests. Page 11
- 12. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Step 3 linked to Step 2: Focusing on quantitative methods- Propose to execute double difference methods • Central feature of the method is use of longitudinal data to use “difference-in- differences” or “double difference”. • Method relies on baseline data collected before the project implementation and follow-up data after it starts to develop a “before/after” comparison. • Data collected from households receiving the program and those that do not (“with the program” / “without the program”). Page 12
- 13. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Double difference methods: continued • Why both “before/after” and “with/without” data are necessary ? • Suppose only collected data from beneficiaries. • Suppose between the baseline and follow-up, some adverse event occurs. • —the benefits of the program being more than offset by the damage from bad event. These effects would show up in the difference over time in the intervention group, in addition to the effects attributable to the program. • More generally, restricting the evaluation to only “before/after” comparisons makes it impossible to separate program impacts from the influence of other events that affect beneficiary households. • To guard against this add a second dimension to evaluation design that includes data on households “with” and “without” the program. Page 13
- 14. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Summary of the method and its application • The approach- By comparing changes in selected outcome indicators between treatment group and the comparable control group, the project impact is estimated quantitatively. • Approach can also be applied to measure spillover effect from the treated to the non-treated famers in the treated areas. • examined by comparing the outcomes between non-treated households in treatment areas and households in control areas. • Moreover, impact heterogeneity across population sub-groups can be investigated. • The sub-groups can be defined based on caste, gender, agro- ecological zones etc. • Such information will be collected in the baseline survey. Page 14