External Validity and Mechanism Mapping
0 likes87 views
The document discusses the importance of context and external validity in impact evaluation, addressing how scientific findings can be generalized across different populations and settings. It outlines methods for testing external validity, including meta-analysis and structural modeling, while emphasizing the need to adapt policies based on contextual differences. Additionally, it introduces mechanism mapping as a framework for understanding the relationship between program inputs, activities, outputs, and final outcomes.
External Validity and Mechanism Mapping
- 1. Context and external validity in impact evaluation
- 2. Michael-Paul James Student led discussion on Beyond ‘context matters’: Context and external validity in impact evaluation Martin J. Williams In service of PPOL 8630: Advanced Program Evaluation Dr. Jacqueline Chattopadhyay 2
- 3. Threats 3 Internal Validity ✘ History ✘ Selection ✘ Maturation ✘ Statistical Regression ✘ Mortality ✘ Testing Effects ✘ Instrumentation ✘ Contamination Different People Different Places Across Different Times ExternalValidity? ExternalValidity?
- 4. External Validity The extent to which one can apply the findings of a scientific study to the world at large. 4 Can sample be generalized to population?
- 6. The Problem ✘ Constructability ○ What is a framework for understanding how context matters for policy effectiveness? ✘ Generalizability ○ Will the impacts of a policy evaluated in a specific context be the same in other, unspecified, contexts? ✘ Applicability ○ How do we apply evidence gathered from another context to our specific context? 6 ?
- 7. Existing Methods to Testing External Validity ✘ Various combinations of meta-analysis ○ A statistical process that synthesizes findings from many studies ✘ Extrapolation based on structural modelling ○ Estimating while assuming that existing trends will continue ✘ Exploiting selection and non-compliance within RCTs ○ Identifies a range of heterogeneous treatment effects. ○ Local Average Treatment Effects (LATE) ✘ Principal stratum analysis ○ Compute causal effects only within identified strata (race, gender, etc) ✘ Reweighting based on observable characteristics to make predictions about a policy's effectiveness in other contexts 7
- 8. Local Average Treatment Effects (LATE) the average potential outcome for comPliers when they do receive treatment T T 8 T D D D D T Always Takers Compliers Never Takers Defiers Assigned to Treatment* Assigned to Control T T D D T T D D Compliers Never Takers¹ Complier/ Never Takers² N/A Average outcome in control group² Ave complier outcome without treatment % of Compliers from ATT* Ave Never Taker outcome with no treatment¹ % of Never Takers from ATT* = x + x
- 9. Linear program theory model 9 Inputs Activities Outputs Initial Outcomes intermediate Outcomes Final Outcomes Adapted from Donaldson, S. I. (2007). Program theory-driven evaluation science. New York, NY: Lawrence Erlbaum, p. 25.
- 10. Mechanism Mapping Method 1. Map out the policies intended Theory of Change 2. Map the contextual assumptions that must hold for the Theory of Change to work 3. Map the actual characteristics of the context and compare them to the assumptions 4. Adapt the policy to eliminate these mismatches between assumptions and characteristics 10
- 11. Funds, potential workers Procure food, hire and train workers Gov’t provides supplies, food, and nutrition advice to mothers Mother’s nutrition awareness improves Mothers decide to use food for selves and infants Mother and infant nutrition improves Funds will be dedicated and suitable workers available Adequate procurement, logistics, training processes Food and advice delivered to target mothers in timely fashion Mothers able to attend sessions, trust advice Mothers control household food allocation More food will improve nutrition Funds will be dedicated and suitable workers available Adequate procurement, logistics, training processes Food and advice delivered to target mothers in timely fashion Mothers able to attend sessions, trust advice Mothers do not control household food allocation More food will improve nutrition 11 Inputs Activities Outputs Intermediate Outcomes 1 Intermediate Outcomes 2 Final Outcomes Theoryof Change Contextual Assumptions Actual Context MechanismMappingExample Tamil Nadu Integrated Nutrition Programme v. Bangladesh Integrated Nutrition Programme
- 12. Funds, potential workers + extra funds for home visits Procure food, hire and train workers + Different types of workers for home visits Gov’t provides supplies, food, and nutrition advice to mothers, husbands, and mothers in-law through home visits Mother’s, husband’s, and mothers in-law’s nutrition awareness improves Mothers, husbands, and mothers in-law decide to use food for selves and infants Mother and infant nutrition improves Funds will be dedicated and suitable workers available + Funds available for home visit cost Adequate procurement, logistics, training processes + Suitable home visit workers hired Food and advice delivered to target mothers in timely fashion + Homes are accessible for visits Mothers able to attend sessions + Husbands & Mothers in-law Reached through home Visits, trust advice Better awareness by mothers, husbands, and mothers in-law will improve household food allocation More food will improve nutrition 12 Inputs Activities Outputs Intermediate Outcomes 1 Intermediate Outcomes 2 Final Outcomes AdaptedTheoryof Change AdaptedContextual Assumptions MechanismMappingExample Tamil Nadu Integrated Nutrition Programme v. Bangladesh Integrated Nutrition Programme
- 13. 1. Mechanism Mapping is arguably already informally executed in the rational process by practitioners and theorists. What is the value add of the additional step of charting it? 13
- 14. 2. Mechanism Mapping greatly oversimplifies behavior. What are some of the risks in the process of simplification? 14 Original Target Population New Target Population Total Population
- 15. 3. Mechanism Mapping visualizes behavior through a linear theory of change. What are some of the risks of thinking about behavioral change linearly? 15 activities outputs outcomeinputs
- 16. 4. Mechanism Mapping relies on judgement and predispositions. What are some of the dangers or potential downfalls of personal judgements in anticipating inputs, activities, outputs, and outcomes? 16
- 17. 5. Mechanism Mapping is an intuitive technique for analyzing the differences between similar programs exhibiting different results. Is there a value in mapping when only a single study is available? 17
- 18. You are Amazing! Ask me all the questions you desire. I will do my best to answer honestly and strive to grasp your intent and creativity. 18