Probabilistic decision tools for evidence based impact evaluation
Download as PPTX, PDF1 like356 views
This document discusses using probabilistic decision tools and modeling approaches for evidence-based impact evaluation and development policy decisions. It argues that traditional scientific methods are not well-suited for studying complex, real-world agricultural systems due to their interdisciplinary nature and many uncertainties. Instead, it advocates adopting a decision analysis approach which (1) incorporates all important system aspects, (2) models the system using all available information including uncertainties, (3) identifies key uncertainties, and (4) recommends options based on expected outcomes to support decisions that must be made without perfect information. The document provides a case study on using decision analysis to evaluate options for reducing reservoir sedimentation in Burkina Faso.
Probabilistic decision tools for evidence based impact evaluation
- 1. Probabilistic Decision Tools for Evidence-Based Impact Evaluation Modeling real-world complexities to support development policy decisions. Caroline Muchiri, Eike Luedeling, Keith Shepherd and Yvonne Tamba
- 2. Modeling the impact pathway of agricultural development research • Aimed at supporting development decisions • Contribute to development impact • Therefore this requires our models to adequately reflect the reality of agricultural systems Agricultural model Support decisions Improved decisions Development impact
- 3. Agriculture is inherently complex and interdisciplinary Reproduced from: McConnell and Dillon, 1997. Farm Management for Asia: A Systems Approach. FAO Farm Systems Management Series 13 (http://www.fao.org/docrep/w7365e/w7365e00.htm#Contents). Managerial subsystem Technical subsystem Informal structural subsystem Goals and values subsystem Organizational structural subsystem Environmental suprasystem • Culture • Technology • Education • Political setting • Legal setting • Demography • Sociology • Climate • Economics System behavior arises from a host of factors and interactions How can we study the behavior of the whole system? Can our usual research and modeling approaches address this? Input-output flow of material, energy, information and influence Decision-makers need tools that allow forecasts for the whole system
- 4. Our usual approach to knowledge generation The scientific method Inspired by https://commons.wikimedia.org/w/index.php?curid=42164616 Formulate hypotheses Make observations & think of interesting questions Develop general theories Develop testable predictions Gather data to test predictions Refine, alter, expand or reject hypotheses • Seeks for objective, widely valid system behavior rules • Reduces complexity to allow hypothesis testing • Dissects the system into researchable fragments • Struggles where many disciplines are involved and rules are complex
- 5. System models from solid building blocks? Can we ‘stack’ study results obtained with the scientific method to gain system understanding? Study 1 Study 2 Study 3 Study 1 Doesn’t work if the blocks don’t fit Insufficient knowledge for decision support Or if we can’t get enough blocks (gaps)
- 6. How to do a small study on agriculture This is usually preferable, but how can it be done? Biophysical aspects Economic aspects Social aspects Environmental aspects Cultural aspects Biophysical aspects Biophysical aspects Economic aspects Social aspects Environmental aspects Cultural aspects An agricultural system Given limited resources for research, how should we study this? By addressing one aspect in great detail? Are our results still relevant for the agricultural system? By doing a coarser assessment that includes all essential dimensions? OR
- 7. How do good decision-makers generate knowledge? Structured decision analysis Frame entire decision context Model the situation using all available information Quantitatively project decision outcomes Characterize all risks and uncertainties Identify key uncertainties Measure where information value is high Recommend preferable option • For systems that are too complex to fully understand (with available resources) • For supporting decisions that MUST be made without perfect information • “What’s the best option according to our limited understanding?”
- 8. Key principles of decision analysis Shepherd et al., 2015. Nature 523, 152-154. Luedeling and Shepherd, 2016. Solutions 7(5), 46-54. 1. Incorporate all important aspects into the model. 2. Model the system using all sources of information, including local and expert knowledge 3. Explicitly consider uncertainties about inputs, processes and outputs (probabilistic models) 4. Identify key uncertainties for measurement using ‘Value of Information’ analysis 5. Update model, when new information becomes available
- 9. The decision analysis process Adopted from Luedeling and Shepherd, 2016. Solutions 7(5), 46-54
- 10. Probabilistic simulation Normal model Precise numbers as input 42 Precise number as output Probabilistic model Distributions as input, because precise values are unknown Distribution as output • Allows working with variables that we don’t have perfect knowledge on • Requires characterizing and quantifying our uncertainty about them • Common methods are Monte Carlo simulation and Bayesian Networks
- 11. A case study - Decision analysis in agricultural development research Reservoir Protection in Burkina Faso • How can sedimentation be prevented? (dredging, check dams, buffer strips) • Are the options available cost-effective?
- 12. Reservoir protection from sedimentation in Burkina Faso Participatory assessment What management options are available? What are the risks, costs and benefits for each option? What is known about them? What is the plausible range of outcomes? What additional information do decision-makers need? What is the most promising and cost- effective course of action?
- 13. Reservoir protection in Burkina Faso Lanzanova et al., in press.
- 14. Reservoir protection in Burkina Faso Profit margin of vegetable production • Best option is combination of dredging, check dams and buffer strips Lanzanova et al., in preparation. • Probably positive outcome, but small risk of net losses • Additional information on profitability of vegetable production would facilitate decision Net Present Value (NPV) for combined intervention (in thousands of USD) Expected Value of Perfect Information (EVPI; in thousands of USD)
- 15. Contrasting research paradigms ? ? ? Knowledge generation Problem-solving Scientific method Decision scienceWhere is agricultural research? Agricultural Scientists’ methodological comfort zone How far can we stretch it? It may be easier to reach our objectives from here Relatively simple Predictable Replicable Disciplinary Generalizable Follows a small number of rules Complex Stochastic/unpredictable No replication possible Inter-/transdisciplinary Behavior is context-dependent Complex and unclear behavior rules ? Main purpose System characteristics ?
- 16. Conclusions • Decision sciences are geared towards solving problems • They don’t aim at precision or ultimate answers, but at offering comprehensive advice for decisions and system management • If solving problems, supporting decisions and facilitating development in complex systems is our goal… Decision sciences offer a more fitting paradigm for agricultural modeling The scientific method is a poor fit for decision-oriented agricultural models • Most agricultural models aim to solve problems and support decisions • This is (usually) not a quest for basic laws of nature! • We need research approaches that can deal with the complexity and uncertainty of the systems we work on • We need pragmatic ways of supporting decision processes
- 17. …we should adopt decision analysis thinking as our research paradigm Thanks for your attention! k.shepherd@cgiar.org
Editor's Notes
- #6: Gaps such as Fundamental uncertainties in the future (climate changes, conflicts, price fluctuations)
- #9: Point 2: Quickens gathering knowledge from different sources but allows for comprehensive assessments Develops transdisciplinary ‘holistic’ models that fully acknowledge errors, risks and uncertainties Produces actionable decision-oriented information