Reflections from a realist evaluation in progress: Scaling ladders and stitching theory_21 April 2016

Editor's Notes

• #2: Intro – who we are CDI practice paper out yesterday EMAIL PRESENTATION TO INKA!!
• #3: This is the starting assumption behind the Building Capacity to Use Research Evidence Programme (BCURE). Our session today will reflect on the first year of the three year realist evaluation of the programme. The evaluation is asking: is this assumption true? And if research and evidence is crucial to making policy more successful – how and why does this happen?
• #5: Funded by the UK Department for International Development (DFID), BCURE works with policy makers in low and middle-income countries to develop skills, knowledge and systems in order to improve the use of evidence in decision making. £13 million, launched in 2013. BCURE consists of six separate projects, implemented by six different partners, across 11 countries in Africa and Asia. The evaluation team are visiting six countries to collect primary data, and are drawing on monitoring data from the rest. Each project has its own design, although all work with government stakeholders with the aim of building capacity for evidence-informed policy making. The projects involve a range of interventions, designed and combined in different ways by different partners. These include: Training on how to access, appraise and use evidence in policy making (online and face-to-face; in-workplace and residential; shorter and longer in duration), mentoring Practical workshops where policy makers are helped to apply evidence into a particular policy process Facilitating online and face-to-face networks between policy makers and researchers Developing tools, systems and manuals to embed evidence use at an organisational level – like evidence guidelines And the expected outcomes of these interventions is to improve capacity to use evidence, to actually increase the use of evidence, and ultimately to make policy better.
• #7: The purpose of the evaluation – this is what the ToRs said.
• #8: Donors are increasingly commissioning evaluations to answer these questions – Why does my intervention work? Can we scale the intervention up? Can we roll it out and apply it somewhere else? These are the sorts of questions that RE can help to answer. And these were the questions that DFID wanted the evaluation to answer
• #9: Wanted to take a few minutes to give an introduction to realist evaluation for any non-experts in the room. Just a headline! In summary, realist evaluation is a theory based evaluation method - it sits alongside methods like contribution analysis, process tracing, and theory-of-change based evaluations. A realist evaluation does not ask ‘what works’ but rather ‘what works for whom in what circumstances and why’. It does this through opening up the black box between an intervention and an outcome.  And it does this through developing and testing theories about how and why the resources provided by the intervention in particular contexts spark certain mechanisms to lead to certain outcomes.  And realist evaluation has a particular analytical frame for this theorising.  RE theorise about the mechanisms sparked by the intervention, which operate in particular contexts to generate specific outcomes.  The shorthand for this is context-mechanism-outcome configurations, or CMOs. These are the core analytical units of RE.
• #10: So to take a BCURE intervention – a training course, that teaches policy makers about how to access, appraise and apply evidence in their day to day work. How might this lead to change? One theory is that the training might increase the self-efficacy of participants – their confidence and belief in their own ability to use evidence appropriately, which results in them using evidence more effectively in their day job. But this mechanism of self-efficacy will only fire if the participants have sufficient pre-existing knowledge and skills to know what the trainer is on about, to use a computer to access the training materials etc… Or the training might result in this outcome in a very differet way – perhaps at the training people meet their colleagues and form informal peer networks, where people support one another to introduce new practices at work – but this might only happen in a context where the group get on well, where participants are on the same level of the hierarchy. The crucial thing here is that the mechanism is NOT the training course. The mechanism is the causal force, or process, or power, that is sparked by the resources provided by the training course, to lead to the outcome. The mechanism is hidden – it happens inside people’s heads. And crucially, this mechanism will ONLY SPARK if the contextual conditions are right. A couple of other examples loosely based on our stage one data: Perhaps in a context where participants at the training are in the middle of developing a new policy, attending this training course sparks an ‘aha moment’ – they suddenly realise, wow this is really relevant to what I’m trying to do, and go away from the course and immediately put their learning into practice, because they can see how it will be useful and solve some of their problems. Or maybe the training course doesn’t result in behaviour change, but actually just crystallises people’s general awareness of ‘this thing called EIPM’ – allowing them to talk more knowledgeably about the need for evidence, or allowing them to put a label to what they’re already doing - for example because the people attending are less intrinsically motivated but are just there because they need to be to be promoted. This awareness may translate into behaviour change later down the road; or it may not result in behaviour change at all but just more policy makers who can make the right noises. The two headline messages to take away are: ‘Mechanisms’ are NOT interventions, but rather they are the forces or powers or processes that generate outcomes as a result of the resources provided by an intervention to an outcome. They are usually hidden, and happen inside people’s heads. And mechanisms will only ‘fire’ given the right contextual conditions.
• #11: Realist evaluation is underpinned by specific assumptions about the nature of reality, causality, and how we can gain knowledge about these things. Realism sits between and draws from both positivism and constructivism. What this means is that it assumes that the material and social worlds are real - social forces exist outside of our perception of them, and cause change to happen. However, realism assumes that our knowledge of the world is accumulated and interpreted  through our imperfect human senses and biased human brains, filtered through our language, culture and past experience, and the world is a complex place in which reality is not static but constantly shifting and changing. This means we can acquire true knowledge, but it will always be partial and incomplete. Realist evaluation also assumes a ‘generative’ model of causality, in which causal links are demonstrated through a fine-grained explanation of what happens between cause and effect to explain why a certain effect occurred. This is a different kind of causal framework to those used in other types of evaluation. These three types of assumptions have big implications for how the researcher goes about trying to prove a causal link between a programme and an outcome, and the tools and methods they use to do it.
• #12: One big implication of this is that realist evaluation does not require a control group. Your analysis is intra-programme: you explain different patterns of outcomes for different groups of people within the same programme.
• #13: This question comes up a lot. One of the reasons we chose RE as an approach is because DFID wanted to learn from BCURE in order to decide whether to scale up and roll out the approach elsewhere, What is the external validity, and how do you generalise from realist evaluation findings? The ability to generalise from realist evaluation findings comes from these theoretical assumptions. Causal mechanisms are real forces or processes that exist in the world – the social world as well as the physical world. And just as in the physical world, laws and regularities like gravity explain all kinds of different events….in the social world the same mechanisms are present in very different situations and explain why very different strategies or programmes result in change. For example, the mechanism of deterrence underpins all kinds of crime reduction interventions, from burglar alarms to CCTV cameras to neighbourhood watch schemes to steering wheel locks. But it also underpins all kinds of broader strategies too– from nuclear weapons to Santa’s naughty list. So in realist evaluation, you generalise based on mechanisms. The theories you develop, in the form of CMO configurations, are PORTABLE. If a training course in South Africa can be shown to work through sparking an ‘aha moment’ in the minds of training participants, then this gives us a degree of confidence that a training intervention in Ghana might spark the same mechanism, when implemented in a certain way and when the contextual conditions are right. If policy makers understand what it is about the training that sparks which sorts of mechanisms, and what it is about the context is needed to make this happen, then they can make good decisions about how to design and modify the intervention for a new context. There are of course no guarantees – the same intervention in a new context, even if intelligently designed based on lessons from elsewhere, might still not work. There may be new mechanisms at play, interacting with new features of context you haven’t come across before. BUT by investigating what happens in Ghana, you can continue to test and refine your theory, which adds to our collective knowledge about how these kinds of programmes work in different contexts.
• #14: There is no step by step guide to conducting a realist evaluation – it is an overarching approach and way of thinking rather than a prescribed methodology. But RE has 3 broad stages – developing, testing, refining theory, which are iterative and overlap somewhat. In BCURE, we started by developing a common theory of change to explain how we thought all the different BCURE programmes were going to lead to the expected outcomes.  This theory of change helped shape and structure a literature review, and in turn the literature review helped test and refine the theory of change.  Out of the literature review also came a set of initial, more detailed theories – our starting CMOs.  These CMOs structured our sampling and instruments for Stage 1 data collection, which aimed to start testing and refining these theories through primary data collection. At synthesis stage, the data from across the programmes is analysed and this is used to further develop and refine the theories.   At Stages 2 and 3, we’ll test our refined theory again, with the aim of coming out with a more refined, better evidenced set of theories about how and why capacity building interventions work to promote EIPM. The overall aim is to generate a refined and evidenced set of theories about what works to build capacity for EIPM, for whom, in what circumstances and why. 
• #15: First step in realist evaluation is to develop your theory about how and why the programme is expected to work, for whom, in what contexts.
• #16: Going to be refined at Stage 2 Learning curve throughout the process of theory development!
• #17: Four levels of change which interrelate and reinforce one another.
• #18: Theory of change created the framework for the literature review: which aimed to examine, further articulate and start to refine the theory. When we started the literature review, we originally intended to keep focussed fairly narrowly on empirical evidence from other capacity building interventions. This is partly with this aim of it being a practical resource rather than a theoretical paper. But when we started to look into the evidence base more thoroughly, it became clear that theory had to play a central role within the review. Why? Firstly, once we started to examine this concept of EIPM and cpacaity building for EIPM, it became clear that these things are underpinned by a huge number of theories and assumptions. Just some of the questions that fall out of this: What is ‘evidence’? What makes it ‘good quality’? What is ‘policy’ and what makes THAT good quality? What role does evidence play in policy processes anyway? What does ‘capacity’ to do EIPM even look like, and how do you build it? There are a huge number of assumptions underpinning the concept of EIPM. All programmes aiming to promote EIPM are making certain assumptions, whether they acknowledge them or not. There are also several voices in the literature we reviewed which criticise the evidence base on EIPM for failing to make use of the rich theory available, and for failing to make their assumptions about these questions explicit. So this led us into the realm of a huge range of interdisciplinary theories: from the huge diversity of theories from political science on how policy making works, to psychological theories about how people cognitively process and use new evidence, to adult learning theories about how people learn. Lots written within international development, and also a huge amount within the health field, where the field of Evidence Based Medicine is very well established. What the literature review does is provide a very surface level summary of some of these theories and how they relate to EIPM. These theories help provide starting points to investigate this question: what works for who in what circumstances and why? We’re not starting from scratch: we’re building on existing theories in this literature, as unpacked and explored through the literature review.
• #19: The literature review helped us develop a clear and critical understanding of this concept of ‘EIPM’ – enabling us to take a critical stance on the assumption underpinning BCURE that EIPM is a ‘good thing’ Some of the insights from the theoretical literature… Literature review is now available online
• #20: In BCURE, we talk about ICMOs – we’ve added intervention features to the basic CMO framework. We found this helped us to separate out features of the intervention which are under the control of the programme (such as how training is designed) from features of the context which are not. There are lots of debates in the literature about how to formulate and conceptualise CMOs – this is just one way, not necessarily the best way in all cases! Pawson emphasised the importance of catchy names for your CMOs…helping to get the research team on the same page and make theories more intuitive.
• #21: Complexity of programme Literature – skills and knowedge not enough
• #22: Next step in realist evaluation – once you have developed your initial theory – is to test it.
• #23: In realist evaluation, once you have your initial theory, you test it. The aim is to collect evidence that allows you to confirm, reject, refine or nuance your theory. In BCURE, there are three rounds of data collection and analysis, in 2015, 2016 and 2017. At each stage, there are six country visits, involving around 30 qualitative interivews per country. Evaluators speak to project staff, intervention participants, high level government informants and other people who can give an insight into how and why the BCURE interventions are influencing (or not influencing) the use of evidence in government systems. On top of this, we conduct a review of all the monitoring data collected by the programmes. Interviews are our main source of data for BCURE. Partly the nature of the programme – it is difficult to collect quantitative data that provides evidence of the sort of change and processes we are looking for. Difficult to access documentary evidence. Stage 2 data collection starts next month.
• #24: This leads us to our third challenge – mastering the art of the realist interview. The aim of a realist interview: I’ll show you my theory if you show me yours. A realist interview is theory driven. The subject matter is the researcher’s theory, and the respondent is there to confirm, falsify and refine it. This is done through explicitly presenting theory to respondents and asking them to comment on it, confirm it, refine it or refute it. You say ‘I’ve seen this elsewhere, do you think that is happening here?’ Or ‘I’m wondering if this is happening because of this – what do you think’? However, our team were worried about this approach at stage 1, and whether it might result in confirmation bias – our respondents simply agreeing with what they think we want to hear. We felt this might be a particular issue in an international development context – where in some countries it is considered rude to disagree with someone considered to be an expert. Or where there might be various power dynamics due to the desire of government officials to receive continued donor funding. We’ve had various conversations with realist colleagues about this, and our strategies for stage 2 are: Training our team more on the principles of conducting a realist interview Asking for examples. If someone agrees with a theory, ask them to explain and give an example of this in action. Or asking people to adjudicate between rival theories. Do you think it happens like this, or like this? We are going to try these strategies out during our stage 2 data collection, happening over the next couple of months.
• #25: The next step was working out how to analyse our qualitative data – over 100 interviews from across six countries. We originally wanted to use qualitative analysis software to help us manage the dataset. However, we worked out a slightly lower-resource alternative that didn’t require expensive licenses or induction of all the team members in how to use the software. We reviewed the interview transcripts for insights into WHAT OUTCOMES WERE OBSERVED, or anticipated, and WHY the respondent thought those outcomes came about. What features of the intervention, implemented in which contexts, sparked which mechanisms to generate these outcomes. We entered this information into rows in Excel. The advantage of this is that it ensures that CMOs are considered IN CONFIGURATION – not isolated and disaggregated from one another. It also gives us a systematic record of outcomes, and will allow us to make judgements about the strength of evidence behind particular outcomes, and particular CMOs, over time. Needed detailed interview transcripts in order to do this This does take time! The team struggled with this – on how to do it and the time needed – this year investing more support time.
• #26: Once you have collected data to test your theory, the next step is to refine it – with the aim of ending up with a well-evidenced, tested and refined theory that tells you about how and why the programme works for different groups in different circumstances.
• #27: The data from across the six programmes was then synthesised, in order to draw out lessons and refine both the ICMO configurations and our ToC. We used a metaphor (the ladder of abstraction) and elements of a method (meta-ethnography) to help us do this in a clear, transparent and systematic way. Ladder of abstraction - ‘there is always more than one correct way to describe what caused a result or justifies a prediction. Some of these ways will generalise across a number of cases, others across very few.” We synthesised data from across our cases through moving from specific findings at the bottom of the ladder (for example a statement made by an interview respondent), up to more general explanations several rungs up (ICMOs) which encompassed findings from across different respondents and country settings. We also used elements of the synthesis method meta-ethnography to help us make sense of the process - Happy to go into this in more detail afterwards for anyone interested. Team workshop to start synthesis process. ‘is this an example of something we have seen elsewhere? Is there a common concept we can use to explain these things?’ ‘does this apply in your BCURE context? Are there any nuances from interviews with respondents in your setting?’ Followed by systematic review of dataset by two team members. Tables in Word were used to help group data by ICMO, and reviewing this data allowed us to further refine the ICMO configurations. At the end of the synthesis process, we had a revised set of ICMO configurations representing our ‘best guesses’ at the end of Stage 1 about how BCURE interventions are leading to change. These provided new insights into how elements of our ToC lead to and reinforce other elements, used to refine our ToC through nuancing expected outcomes and adjusting the anticipated links between them. T
• #28: This is an example of what a BCURE CMO looks like. We call this the ‘aha moment’ theory. Training can result in behaviour change through sparking an ‘aha moment’ – this is where participants in the training recognise the relevance of the training to their own work and put it into practice accordingly. Aha moments happen when training is practical, possibly participatory, linked to needs assessments, and when it targets people who can directly put their skills into practice in their own work. There’s some evidence to suggest that this will only happen in contexts where there are external motivations to apply training, or where participants already have internal motivation for EIPM.
• #29: Capacity building is a complex process, EIPM is also a complex process, can’t reduce things to component parts. Realist evaluation was designed to examine complexity – ‘natural bedfellows’ according to Gill. But we struggled a bit because ICMOs are a very linear construction. We have sometimes felt trapped by the CMO analytical frame, and have struggled with how to encompass complexity and feedback loops into our theories. What helped was: Thinking of mechanisms in terms of levels of a system. For example, a mechanism might be conceptualised at the level of an individual, a family, a community, or a society. An evaluator can ‘layer’ theories, with the outcome at one level becoming the context at the next level up, creating a ripple effect up the chain CMOs interacting with each other on different levels. Feedback loops operating between them. Stage 2 – starting to explore dynamics and interrelationships
• #30: Our data from Stage 1 indeed suggests that ICMO configurations at different levels of our ToC (individual, interpersonal, organisational and institutional – see Box 3) are interconnected, with the outcome at one level becoming the context at another level up or down the system, and with feedback loops operating (processes leading to outcomes, which reinforce or further catalyse the process, resulting in more substantial outcomes).
• #31: Widespread recognition that understanding context and navigating complexity is crucial to successfully promote change. Realist evaluation provides a systematic (although certainly not easy!) way to examine how context affects the way people respond to the resources provided by programmes in a complex environment, and how this influences programme outcomes. Findings can provide a very practical steer on what types and features of interventions work best in a given context, and what practitioners need to think about when scaling up or rolling out a programme. DFID really like them. However, there is an art in communicating realist evaluation findings! Making use of metaphors has been invaluable. Helps to talk about our ICMO configurations (e.g. ‘eye openers’). Framing ICMO configurations as sentences and largely avoiding the term ‘ICMO’ altogether when talking to practitioners (focussing instead on the content of the theory) has also been very useful. BUT There are tensions between the theory-driven nature of realist evaluation, and the structures and incentives of the international development sector. Flexibility crucial in RE – iterative, theory driven. Sample and methods may change over the course of the evaluation in order to test new and refined theory. However, evaluation commissioners’ internal systems are not always conducive to flexibility – including procurement and programme management systems that require detailed up-front work-plans and top-down budgeting and planning. Makes it essential to engage evaluation commissioners up front and ensure they are on board.