What is impact?

Editor's Notes

• #2: We hear impact a lot these days: collective impact, social impact, impact investing, research impact. In the field of International development also, Impact has become somewhat of an obsession. There are many reasons for this but it's in part due to a confluence of decreasing or retargeting aid budgets in many donor agencies, greater public sensitivity to overseas spending as austerity measures have begun to set in and increasing momentum in the evidence based policy arena. So donors and their programmes are focussing much more on demonstrating value for money and tangible stories of their impact. We certainly saw this when we started engaging with programmes for the Methods Lab research. They felt they needed an impact evaluation but they weren’t really sure what they meant by impact.
• #3: Yet, our instincts told us that the way impact is defined has a big effect on programmes:  In programme design, discussions about impact set the aspirations for a programme and help to build cooperation, inform investment decisions, and are used to identify risks.  During or after a programme, the way impact is defined will affect how its success or failure is perceived, and who takes credit or blame.  And this then affects what we learn from the programme to help adapt it or contribute to other programmes. So we decided to look into it a bit and see if we could find a way of bringing greater clarity. We started by looking at the definitions that exist. In aid, there are two  predominant positions:
• #4: The most common definition is from the OECD DAC – “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended” this is extremely open and only bounded by the requirement to be long-term and to be produced by the intervention.
• #5: Another common definition comes from the domains of economics and statistics and is used as the basis for many impact evaluations, this comes from a 3ie paper – “Impact is defined as the difference in the indicator of interest (Y) with the intervention (Y1) and without the intervention (Y0). That is, impact = Y1 – Y0” – which looks completely different to the first definition.
• #6: Many agencies and organisations have their own understanding of impact which operationalise these two definitions.  The results-chain metaphor is used frequently to define impact in relation to other kinds of results – most commonly outputs and outcomes.  Many agencies also define impact as change at the level of end-beneficiaries, who are usually understood as ordinary people and communities in the countries where the programme is running.
• #7: Then there’s the elephant in the room. In some disciplines, impact refers to environmental impact. Which you’d expect to play a bigger part of our definition of impact, given the increasing focus on climate change.
• #8: So how can we make sense these definitions? One way is to take the broadest definition – the OECD-DAC – and break it down. We can see six different kinds of impact…  E.g. youth unemployment.§  Decrease in unemployment rates§  Programme finds that students also need literacy skills§  Outside investment in local business increases§  Students start to mentor each other§  Students migrate§  There is conflict between programme and non-programme students  By looking at these different kinds of impact we can decide where our programme needs to focus. E.g. if a programme has already been developed and just needs testing then perhaps the top left box is sufficient. If there are clearly many risks predicted then include the top right box. If the programme is likely to be adaptive and evolve over time then the bottom row should be considered.
• #9: Another way to understand impact we suggest is to develop a common language around the characteristics of impact which we need to consider in each context.  Use – When we use the term impact: are we describing the intended or desirable change; are we forecasting what kind of effects might be expected; or are we describing actual observed or measured change?  Focus – Are we narrowing our description of impact to one or a few pre-defined variables (economic growth, household income, school test scores) or are we taking an open perspective of impact and looking at as many variables as makes sense in order to make a useful judgement?  Level of change – Where are we looking for impact: at the level of ordinary citizens, households, sub-groups or communities, at organisations or institutions, at sectors, at the environment?  Degrees of separation – How direct is the link between the intervention and impact as you define it: Can impact occur close to the intervention or is it always several steps removed? How certain is the causal link: Does there have to be existing theory or empirical evidence for causal links or can impact occur in completely unexpected places?  Change trajectory – Is our understanding of impact static (this project had this much impact) or can impact vary over time? And does it always take a long time before impact can be observed?  Homogeneity of benefits – Is impact a uniform or ‘average’ property or does it vary across population groups or institutions with different experiences of benefits? We hope that by thinking across these six dimensions of impact we can begin to find a middle ground between the econometric definition of impact and the very broad OECD DAC definition – and be able to defend a contextually appropriate understanding of impact