An Introduction to the Manual: How Do We Know if a Program Made a Difference? A Guide to Statistical Methods for Program Impact Evaluation
- 1. An Introduction to the Manual: How Do We Know if a Program Made a Difference? A Guide to Statistical Methods for Program Impact Evaluation Peter M. Lance MEASURE Evaluation September 25, 2014
- 2. Where it all began….
- 5. Approachable But Basic Sophisticated But Unapproachable
- 6. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
- 7. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
- 8. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
- 9. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
- 10. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
- 11. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
- 12. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
- 13. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
- 14. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
- 15. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
- 16. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
- 17. A Tale of Two Manuals: Manual 1 Manual 2 푌1 푌1 The Outcome if Individual i participates 푃 푇 Program Participation/Treatment Exposure of individual i 푇 푡 Time 푁 푛 Number of observations 푃 푁 Size of the population Δ푥 푥 Change in 푥
- 18. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
- 19. Obviously leads to 풏≈ 풁ퟏ−휶 ퟐ ퟐ ∙풑∙ퟏ−풑 풅ퟐ ퟏ+ 풁ퟏ−휶 ퟐ ퟐ ∙풑∙ퟏ−풑 풅ퟐ 푵 Obviously…. Totally 푃푟−푑≤푝−푃≤푑|푁≥1−훼 The Inequality
- 20. A B C D E F G
- 21. A B C D E F G
- 22. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
- 23. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
- 25. Key Guiding Features: 1. Consistent Notation 2. Little Prior Knowledge Assumed 3. Explicit Derivation of All Results in the Manual with verbal/narrative explanation 4. Reliance on a single “behavioral model” woven through the discussion 4. Explicit discussion of the practical estimation
- 26. 5. Equal coverage of major IE traditions 6. Practical examples of how to estimate the IE models and their applied performance in STATA and
- 27. 5. Equal coverage of major IE traditions 6. Practical examples of how to estimate the IE models and their applied performance in STATA and …. All Statistical Programs for them Released with Manual!!!
- 28. Topics Covered 1. Introduction to the basic challenge of IE 2. RCTs 3. Selection on Observables 4. “Within” Estimators 5. Instrumental Variables (including Regression Discontinuity)
- 40. Homer Simpson
- 42. Homer Simpson homer@email.deepthinkingu.edu The University of Deep Thinking
- 43. Homer Simpson homer@email.deepthinkingu.edu The University of Deep Thinking Dear Authors, Wow this manual is boring. Too much blah blah blah and not enough cartoons. Yours, Homer
- 44. Homer Simpson homer@email.deepthinkingu.edu The University of Deep Thinking Dear Authors, Wow this manual is boring. Too much blah blah blah and not enough cartoons. Yours, Homer
- 45. The Future 1. Continued revisions to the first edition of the first volume 2. A second edition of the first volume: -Expanded Empirical Examples (adding in new STATA treatment effects commands, examples in R, etc.); -Separate chapter for Regression Discontinuity; -Extending behavioral model to RCTs; -What else?
- 46. The Future 3. A Potential Second Volume -Hybrid models; -Deeply structural models; -Sample selection/control function rooted models; -Aggregate data (e.g. time series basics, synthetic cohort models, etc.); -What else?
- 47. MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) and implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group International, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University. Views expressed in this presentation do not necessarily reflect the views of USAID or the U.S. government. MEASURE Evaluation is the USAID Global Health Bureau's primary vehicle for supporting improvements in monitoring and evaluation in population, health and nutrition worldwide.