Using realist evaluation with vulnerable young people and the services that support them:
Download as PPTX, PDF2 likes1,002 views
The document outlines the application of realist evaluation in understanding the experiences of vulnerable young people and the services supporting them. It discusses the importance of context, mechanisms, and outcomes in evaluations, emphasizing the need for a nuanced approach that considers various factors influencing change. The conclusion highlights how realist evaluation offers a framework to navigate the complex realities in child safeguarding and invites reflection on its utility in practice.
Using realist evaluation with vulnerable young people and the services that support them:
- 1. Using realist evaluation with vulnerable young people and the services that support them: examples from research Not to be reproduced without permission from the authors Dr Julie Harris, Dr Debi Roker, Dr Lucie Shuker University of Bedfordshire
- 2. The International Centre • Committed to increasing understanding of, and improving responses to, child sexual exploitation, violence and trafficking in local, national and international contexts • Achieved through: – academic rigour and research excellence – collaborative and partnership based approaches to applied social research – meaningful and ethical engagement of children and young people – active dissemination and evidence-based engagement in theory, policy and practice
- 3. Workshop outline • Reflect on multiple ways evaluations are used • Realist cycle of evaluation (including theory of change) • Taking account of context • Understanding how change happens –mechanisms • Conclusions/reflections
- 4. To begin… Think about an evaluation that you have participated in…. Discuss the following with a partner 1) What was the purpose of the evaluation? 2) How were the findings used?
- 5. Four uses of evaluation • Symbolic use – providing the rationale for action or inaction, reinforce a policy position or reassure commissioners (Mark and Henry, 2004) • Instrumental use - where action occurs directly as a result of evaluation e.g. regarding continuation of a programme, expansion, revision or termination (Preskill & Caracelli, 1997)
- 6. Four uses of evaluation cont.. • Conceptual use – learning that emerges from a programme e.g.… regarding its participants, its operation or its outcomes (Weiss, 1977) • Process use – the learning or activities that may occur through the engagement of participants in the research process (rather than from the findings per se) (Patton, 1997)
- 7. An example of process use • Piloting a co-determined approach to outcomes for young people leaving care: - how did the process of involving young people in outcome setting and reviewing improve their projection and planning skills? - did the process in itself contribute to improved outcomes e.g. by increasing their sense of control through the leaving care journey?
- 8. An example of conceptual use • What does this approach to measuring outcomes tell us about the population of care leavers and the needs of the different groups within it? - young people who are ‘moving on’ - young people who are ‘surviving’ - young people who are ‘struggling’ (Stein, 2006)
- 9. Making evaluation meaningful • Informing practice • Being attentive to the experiences of those at the heart of the system “Use is the link between the day-to-day work of evaluation, on the one hand and those activities that could actually improve the lives of programme participants and society on the other.” Mark and Henry, 2004. p.35.
- 10. The Realist Evaluation Cycle … how does it work?
- 11. The realist evaluation cycle Literature review & stakeholder interviews Formulate initial theories Collect data to test theories Further interviews with stakeholders Refine theories Adapted from Introduction to Realist Evaluation - workshop presentation Joanne Greenalgh 2012
- 12. Characteristics of the Realist Evaluation cycle • It starts with theory • It focusses on practitioner wisdom and the experiences of young people and service users • It is a respectful, honest, and collaborative approach
- 13. Realist evaluation methods and approaches • Mixed methods • Triangulation of data • Focus on testing and refining theories • Aims to identify CMOs (contexts, mechanisms and outcomes – more in a minute!) • …. to identify what works, for whom, in what circumstances, and why (Pawson and Tilley, 1997)
- 14. As an example …. • Evaluation of the ‘Hub and Spoke’ model for addressing CSE • Different stages, reflecting the realist evaluation approach • Different sources of data collection, reflecting the realist evaluation approach
- 15. … over to Julie and Lucie … … for more detail on context, mechanisms and outcomes (CMOs)
- 16. Context Think about a new initiative that was introduced into your organization. How did it play out? (e.g. Embraced? Resisted? Ignored?) What kinds of contextual factors were influencing the way that you/others responded to the change being introduced?
- 17. The importance of context “Programmes or interventions are never introduced into neutral spaces – context is key” (Pawson and Tilley, 1997)
- 18. • All programmes have an idea about how change is meant to happen, but whether programmes are successful in activating ‘change mechanisms’ depends on contexts (C) • Pre-existing features of localities into which programmes are introduced • Some will support programme theory and others won’t – so there will always be winners and losers Contexts
- 19. Surrounding the black box: Contexts Contexts are always multiple: • People Institutions • Place Economies • Period Cultures • Processes Society Adapted from Introduction to Realist Evaluation - workshop presentation Joanne Greenalgh 2012
- 20. Example: Safe Accommodation Project • Carers’ capacity to provide appropriate care to high risk young person (individual) • Multi-agency knowledge of CSE • Length of placement/commissioned support • Location of placement relative to risky areas/people • Young person’s willingness/capacity to engage in placement • Young person’s strength of attachment to friends/family/communities that pose a risk
- 21. Context feeds into refined theories • Placements most effective when YP ready/wiling to engage – whatever risk level (YP capacity to engage) • Placements have to counteract the ‘pull’ away by giving reasons to stay (YP attachment to FFC) • Trusting relationships underpin other interventions (carer capacity to build relationship) • Takes time to go from physical to relational to psychological security (length of placement) • Multi-agency shared understanding is key (MA knowledge of CSE)
- 22. • Describe what it is about programmes and interventions that bring about any effects. • Programmes usually offer new resources – material, social, emotional etc. • How people act on these is known as the programme ‘mechanism’ • Mechanisms often work behind the scenes • The concept of mechanism is the pivot around which realist research revolves. Mechanisms
- 23. • Programme resources are always interpreted in different ways by different programme subjects- triggering different causal mechanisms. Subjects ‘ponder’, ‘enter’, ‘support’, ‘go along with’, ‘undermine’, ‘sabotage’, ‘hide behind’ etc. Mechanisms are always multiple M 1 M 2 M 3 M 4 M 5 Adapted from Introduction to Realist Evaluation - workshop presentation Joanne Greenalgh 2012
- 24. “It is not programmes that work or don’t work…rather it is the subjects choosing to act on these resources that determine whether the programme works” (Pawson and Tilley, 1997) The importance of volition
- 25. Understanding mechanisms in the context of child protection Examples: • Reluctance of young people affected by CSE to engage in support services, or • Non recognition of abuse
- 26. Key mechanisms: • Persistence of project workers (for example, through assertive outreach) – eventually persuades young people that someone is investing in them. • The recognition of their peers as being exploited (for example, through group work) – may help young people to better understand their own situation.
- 27. Summary • If ‘What works?’ is the wrong question, RE suggests we ask ‘What works for whom, in what circumstances?’ • Assumes that because of context there will always be winners and losers. Change mechanisms will only be activated for some people • We need to look for a complex footprint of outcomes – some intended, some not.
- 28. ‘CMO configuration’ CONTEXT MECHANISM OUTCOME YP poor previous experience of care YP complex emotional needs Carer experienced with YP Carer confidence, commitment, compassion and ability to cope Building relationship through doing things together Positive relationship built with carer(s) + =
- 29. The realist evaluation cycle Literature review & stakeholder interviews Formulate initial theories Collect data to test theories Further interviews with stakeholders Refine theories Adapted from Introduction to Realist Evaluation - workshop presentation Joanne Greenalgh 2012
- 30. Conclusion • Realist evaluation provides a framework to account for the complex realities in which we try to safeguard children • It combines the importance of listening to the wisdom of children and practitioners, with a more rigorous understanding of causality • It invites us to think more carefully about how change happens, and why
- 31. Finally …. • What does all this mean for you? • How might a realist evaluation approach be useful for you, in your work? • Questions and discussion
- 32. References • Mark M and Henry G (2004) The mechanisms and outcomes of evaluation influence. Evaluation 10:35. Sage Publications • Preskill H and Caracelli V J (1997) Current and developing conceptions of use: Evaluation use topical interest group survey results. Evaluation Practice, 18, 209-225 • Patton M Q (1997) Utilization-focussed Evaluation: The New Century Text. Thousand Oaks, CA: Sage • Pawson R and Tilley N (1997) Realistic Evaluation. London: Sage Publications • Stein M (2006) Research review: Young people leaving care. Child and Family Social Work, 11, pp. 273-279 • Weiss C (1987) Using Social Research in Public Policy Making. Lexington, MA: Lexington Books
- 33. Julie.harris@beds.ac.uk For more information on our work including research publications, short films and outputs from young people please visit www.beds.ac.uk/ic