Learning Approaches
- 2. Why do ‘we’ want to learn? To improve an innovation To replicate elsewhere To ‘scale-up’
- 3. The Standard Theory Innovation design Implementation – monitoring Outcomes – evaluation Evidence-based learning Innovation design revised Impact evaluation ‘ What works?’
- 4. The Experience Innovation design & implementation often inextricably combined – crucial decisions often taken by local agents. Unpredicted effects (good and bad) the rule rather than the exception. ‘ Other things’ almost never equal – context (location, key actors, population, institutions) usually highly relevant.
- 5. Traditional M&E M&E language derives from project management - originally mainly from construction projects (roads, bridges, schools, hospitals, dams, etc.) Characteristics: Simple objective Implementation process is well understood M&E resources minor part of total budget
- 6. M&E in social sector projects Typically complex innovations Multiple objectives Multiple beneficiary groups (gender, age, poverty, ethnicity, etc.) Complex environment Multiple providers and products Multiple interested stakeholders Resources required for effective M&E much higher relative to total budget And rarely allocated
- 7. The evidence base – what works? In medical research the ‘gold standard’ is the randomised controlled trial (RCT) Aim: attribution of effects to treatment There is a current debate between: Those arguing that the RCT approach (or something similar) should be used wherever possible. Those arguing that the RCT approach is usually not appropriate to test social sector innovations.
- 8. Typical requirements for RCTs Standardised and measurable treatment Defined and measurable effects Sample can be seen as representative of some much larger population Number of treatments and controls sufficient to test for statistical significance
- 9. “ It’s the way that you do it” ? Specific context and the detailed implementation process are typically as important as the nature of the intervention in terms of successful outcomes of social sector innovations. This is problematic for a traditional RCT approach, quasi-experimental designs and similar attempts to mimic scientific trials
- 10. Moving from M&E to learning Aims: improve innovation, ‘scale-up’ Need to determine not simply whether an innovation ‘succeeds’ or ‘fails’, but Why and under what conditions different outcomes are more or less likely How resultant costs and benefits will be distributed across a range of stakeholders Predetermined framework of indicators & targets remain an essential starting point - but of limited value without in-depth appreciation of the complexities of the implementation process and context.
- 11. Some useful principles: 1 Time series information - analysed and interpreted as it becomes available. Multi-level rather than aggregate analysis: implementation sites, communities within sites, service delivery units Measure differences both in terms of overall performance and the extent to which benefits and costs are distributed across different stakeholders. How do differences vary over time? Can these variations be interpreted?
- 12. Some useful principles: 2 To the extent practicable, engage multiple stakeholders (users, providers, officials, etc.) in gathering, analysis and interpretation of monitoring and evaluation data Explicitly investigate unexpected issues – positive & negative – as they arise. Include explicit monitoring of policy, economic and social context within which innovation is implemented. Address implications for innovation assessment of changes in this context as they arise.
- 13. Methods Be creative: routine information systems; financial records; rapid surveys of providers and users; case-studies; facility assessments; key informants; focus groups; participatory activities; etc. The key question is how to allocate available resources, especially human resources, across these potential sources of information.
- 14. What’s the story? Filling in the ‘missing middle’ – recursive, detailed explanations of the causal links in the chain linking inputs and outcomes. Use empirical evidence to continuously challenge and modify these explanations. Involve multiple stakeholders to document diverse interpretations Try to determine: Winners and losers What they win or lose (if possible how much) Why (in a given context)
- 15. Tensions Innovation and risk versus fixed indicators and milestones ‘ Quick wins’ versus long run change Learning and adaptation versus audit Accountability to multiple stakeholders versus flexibility Independence versus transparency Resources for learning versus resources for doing