Workshop: Monitoring, evaluation and impact assessment
Download as PPT, PDF4 likes2,517 views
The document introduces monitoring and evaluation in results-based management and discusses key concepts like logic models and theories of change. It provides 3 key points: 1) Results-based management focuses on achieving important organizational changes and improvements in performance through defining expected results, monitoring progress, reporting on performance, and learning lessons. 2) Logic models graphically illustrate program components and how activities lead to outputs, outcomes and impact. Theories of change explain the underlying assumptions and causal pathways of change. 3) Evaluations are used to assess what was implemented, the strength of causal models, intended outcomes, and ultimately the impacts of interventions. Different evaluation strategies are suited to simple, complicated and complex interventions.
Workshop: Monitoring, evaluation and impact assessment
- 1. Monitoring, evaluation and impact assessment C. Crissman Science Week Workshop 22 July 2011, Penang Photo: A. Gordon
- 2. Workshop objectives Introduce Monitoring and Evaluation in Results Based Management Introduce the identification of outcomes in projects
- 3. What is Results Based Management? goal of RBM : “…a management strategy aimed at achieving important changes in the way organizations operate, with improving performance in terms of results …” purpose is “…to improve efficiency and effectiveness through organizational learning , and secondly to fulfill accountability obligations through performance reporting.” key success factors in RBM are: “ the involvement of stakeholders throughout the management lifecycle in defining realistic expected results, assessing risk, monitoring progress, reporting on performance and integrating lessons learned into management decisions.” Source: Meyer 2003
- 4. RBM rests on four pillars Planning : defining clear and measurable results and indicators, based on a logic model or framework. Monitoring: measuring and describing progress towards results, and the resources consumed, using appropriate indicators. Reporting, internally and externally, on progress towards results. Managing: using results information (and evaluation) for lesson-learning and management decision making. Source: Meyer, 2003
- 5. “… is deliberately results driven, meaning that the drivers for planning will be real-world impacts—the outcomes that make a difference to global development goals…” “… is a results-oriented research system —in contrast with, for instance, a results oriented development program…” The CG Consortium Strategy and Results Framework
- 6. CG use of RBM tools in Planning, Monitoring and Evaluation Strategic Plan Medium-term Plan Review of MTP before implementation Performance measurement after implementation Performance Measurement System Research Indicators Outputs (Output targets, publications) Outcomes Impact Center Commissioned External Review (a program-level evaluation) EPMR (a center-level evaluation)
- 7. The results chain Source: ADB, 2006
- 8. WorldFish Strategic Plan Research in Development Reduce poverty and vulnerability through fisheries and aquaculture. Increase food and nutrition security through fisheries and aquaculture. Our Strategic Results
- 9. Multiple dimensions of poverty Certain groups are systematically disadvantaged due to discrimination based on: gender, ethnicity, race, religion, caste, age, HIV status, migrant status People’s exposure to risks; sensitivity of livelihoods to risks; capacity to use assets and capabilities to cope and adapt Little access to means to make a decent standard of living A complex problem… Vulnerability Income and asset poverty Marginalization
- 10. Interventions to reduce poverty Organisational development, labour rights, migrant’s rights, gender equity Improve access to health services, secure land rights, aquatic property rights Diversification, microfinance, education & skills … requiring complex solutions Vulnerability Income and asset poverty Marginalization
- 11. Selecting paradigms for implementation AR4D
- 12. Strategic Plan – Implementation strategy
- 13. Outputs - change in knowledge - change in capacity - change in technology - change in materials - change in policy options - change in awareness/understanding Research - recognition/appreciation of research knowledge Outcomes - use of knowledge by partners - mobilisation of new capacity - extension of technology/materials - change in policy environment Impacts - change in problem - change in opportunities Development - change in actions/behaviour of stakeholders Outcomes - change in productivity - change in equity/empowerment - change in market conditions - change in investments - change in security of assets/habitats
- 14. Ex-post Impact Assessment Measuring ‘the change in the problem’ Reduce poverty and vulnerability through fisheries and aquaculture. Increase food and nutrition security through fisheries and aquaculture.
- 15. Did we make it to happen?
- 16. Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Every child is unique Uncertainty of outcome remains Complicated Complex The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Simple (Diagram from Zimmerman 2003)
- 17. What should be monitored? SIMPLE COMPLICATED COMPLEX What works? What works in what contexts? (implementation environments and participant characteristics) What works here and now? What do we mean by ‘works’?
- 18. Types of interventions Simple intervention Complicated or complex intervention Single causal strand. Intervention is sufficient to produce the impacts Multiple simultaneous causal strands required to produce the impacts Universal mechanism Intervention is necessary to produce the impacts Different causal mechanisms operating in different contexts Linear causality, proportional impact Recursive, with feedback loops, leading to disproportionate impact at critical levels Pre-identified outcomes Emergent outcomes
- 19. Challenges for impact assessment Simple Complicated Complex Deciding impacts Likely to be agreed Likely to differ, reflecting different agendas May be emergent Describing impacts More likely to have standardised measures developed Evidence needed about multiple components Harder to plan for given emergence Analysing cause Likely to be clear counter-factual Causal packages and non-linearity Unique, highly contingent causality Reporting Clear messages Complicated message Uptake requires further adaptation
- 20. M&E system interaction with project implementation Monitoring captures what happened Evaluation explains why
- 22. Evaluations and the R4D Results Chain Scale Pilot / Small Global Research Unit of Impact Analysis Project System Program Outcome evaluation studies that measure the effect size Outcome evaluation that measures the scale of output adoption/ uptake Time Input Output Outcome Impact Objectives Goals Program M&E, Impact pathway analysis, Adoption constraints analysis Ex post Impact Assessment as a function of (effect size * scale) Source: taken from a presentation by M. Maredia
- 23. Evaluations Process implementation evaluation What did or did not get implemented as planned? Rapid Appraisals provide timely, relevant information to decision-makers on pressing issues they face in the project and program setting. The aim of applied research is . . . to facilitate a more rational decision-making process in real-life circumstances Impact Evaluation … the classic evaluation that attempts to find out the changes that occurred, and to what they can be attributed Performance Logic Chain Assessment The performance logic chain assessment evaluation strategy is used to determine the strength and logic of the causal model behind the policy, program, or project. Pre-implementation assessment Are the objectives well defined so that outcomes can be stated in measurable terms? Is there a coherent and credible implementation plan that provides clear evidence of how implementation is to proceed and how successful implementation can be distinguished from poor implementation? Is the rationale for the deployment of resources clear and commensurate with the requirements for achieving the stated outcomes? Case Studies … use when a manager needs in-depth information to understand more clearly what happened with a policy, program, or project
- 24. Logic Models and Theory of Change Photo: World Bank
- 25. Conventional logic for achieving results Inspired by Jeff Conklin, cognexus.org Are we efficient? Are we effective? Source: cognexus.org ACTIVITIES OUTPUTS OUTCOMES IMPACT INPUTS Time
- 26. Works well for outputs Inspired by Jeff Conklin, cognexus.org Workshops, training manuals, research and assessment reports, guidelines and action plans, strategies, and technical assistance packages, amongst others. Source: cognexus.org ACTIVITIES OUTPUTS OUTCOMES IMPACT INPUTS Time
- 27. But not so well for outcomes and impact Source: cognexus.org Multiple pathways of changes in behavior of different, interacting actors; this is where we have the possibility of collecting and making sense of evidence that sustains the impact we are aiming to contribute to. Social change is long-term, complex and is the result of what many actors do (their actions and interactions) Vision and Mission Intentional design ACTIVITIES OUTPUTS OUTCOMES IMPACT INPUTS Time
- 28. Logic Models and Theory of Change Terms often used interchangeably Confusion by funders and grantees about expectations Limited knowledge on how to use TOC and LMs can “blend” into each other
- 29. Logic Models 30 year history Clear identification of goals (outcomes) First widespread attempt to depict program components so that activities matched outcomes
- 30. What is a logic model? Basic United Way format, 1996 Inputs Activities Outputs Inter-mediate Outcomes Long-term Outcomes
- 31. Theories of Change Popularized in 1990s to capture complex initiatives Outcomes-based Causal model Articulate underlying assumptions
- 32. What is a Theory of Change? Long-term Outcome Necessary Pre- condition Necessary Pre- condition Necessary Pre- condition Necessary Pre- condition Necessary Pre- condition All outcomes that must be achieved BEFORE long-term Explain WHY here Show activities here also
- 33. How are they different? Logic models graphically illustrate program components, and creating one helps stakeholders clearly identify outcomes, inputs and activities Theories of Change link outcomes and activities to explain HOW and WHY the desired change is expected to come about
- 34. How are they different? (1) Logic Models usually start with a program and illustrate its components Theories of Change may start with a program, but are best when starting with a goal, before deciding what programmatic approaches are needed
- 35. How are they different? (2) Logic Models require identifying program components , so you can see at a glance if outcomes are out of sync with inputs and activities, but they don’t show WHY activities are expected to produce outcomes Theories of Change also require justifications at each step – you have to articulate the hypothesis about why something will cause something else
- 36. How are they different? Summary Logic Models Theories of Change Representation List of Components Descriptive Critical Thinking Pathway of Change Explanatory
- 37. When to Use? Logic Models when you need to: Show someone something they can understand at a glance Demonstrate you have identified the basic inputs, outputs and outcomes for your work Summarize a complex theory into basic categories
- 38. When to Use? Theories of Change when you need to: Design a complex initiative and want to have a rigorous plan for success Evaluate appropriate outcomes at the right time and the right sequence Explain why an initiative worked or did not work, and what exactly went wrong
Editor's Notes
- #26: OM was developed by IDRC, based in Ottawa, as a response to some of the challenges they were facing when using a more conventional results chain. Projects were having difficulty with the linear characteristic of this logic As well as the it’s cause-and-effect nature
- #27: OM was developed by IDRC, based in Ottawa, as a response to some of the challenges they were facing when using a more conventional results chain. Projects were having difficulty with the linear characteristic of this logic As well as the it’s cause-and-effect nature
- #28: OM was developed by IDRC, based in Ottawa, as a response to some of the challenges they were facing when using a more conventional results chain. Projects were having difficulty with the linear characteristic of this logic As well as the it’s cause-and-effect nature