![As far as I am aware, you cannot go and ‘pay and
play’ in the [CLC] sports facilities, so there’s this
lovely gym there and the kids and adults have got
their noses pressed against the window saying ‘can
we use this great gym?’. But you can’t just go and
use it, what you can do is hire the whole thing and
then use it. With great respect, not even the local
football team is that well organised and well
resourced to go and hire the whole of the gym to do
a fitness session. So we’ve got this thing that’s right
there in the community and for six months it hasn’t
been used by any member of the community.](https://image.slidesharecdn.com/multi-strandinitiativesfinaltoc-170726092505/85/Multi-strand-initiatives-using-theory-of-change-evaluations-33-320.jpg)
Multi-strand initiatives: using theory of change evaluations
- 1. Multi-strand initiatives: using theory of change evaluations Alan Dyson: D.A.Dyson@manchester.ac.uk Kirstin Kerr: Kirstin.Kerr@manchester.ac.uk Karen Laing: k.j.c.laing@newcastle.ac.uk Liz Todd: liz.todd@newcastle.ac.uk Michael Wigelsworth: michael.wigelsworth@manchester.ac.uk
- 2. Introduction • Aims • The team – Alan Dyson, Kirstin Kerr, Karen Laing, Liz Todd, Michael Wigelsworth • Programme: – 10.00 Introduction to theory of change evaluation – 11.15 Coffee – 11.30 Theory of change in action – 12.00 Practical activity – 12.30 Lunch – 1.15 Using evidence – 1.45 Practical activity – 2.30 Coffee – 3.00 Key issues: quantification; working with users; drawing conclusions – 4.00 Finish
- 3. What is ‘theory of change’ approach to evaluation? Professor Liz Todd Liz.todd@ncl.ac.uk
- 4. Dyson & Kerr Cummings, Dyson, Muijs, Papps, Pearson, Raffo, Tiplady & Todd Cummings Dyson & Todd Full service extended schools national evaluation Colleen Cummings, Alan Dyson, Ivy Papps, Daniel Muijs, Diana Pearson, Carlo Ruffo, Lucy Tiplady, Liz Todd: Newcastle University, University of Manchester, Tecis Ltd http://www.ncl.ac.uk/cflat/ESnetwork.htm
- 5. Theory of Change A systematic and cumulative study of the links between activities, outcomes and context of the initiative Fullbright-Anderson, Kubisch and Connell, 1998: 16
- 6. 6 Theories of Change • Popularized in 1990s to capture complex initiatives • Outcomes-based • Causal model • Articulate underlying assumptions
- 7. 7 ..and for the history buff: Logic Models • 30 year history • Clear identification of goals (outcomes) • First widespread attempt to depict program components so that activities matched outcomes
- 8. 8 What is a logic model? Inputs Activities Outputs Inter- mediate Outcomes Long-term Outcomes Basic United Way format, 1996
- 9. Realistic evaluation • Social programmes aim for some kind of social change • Programmes ‘work’ by enabling participants to make different choices through the programme program ‘mechanism’ • Making and sustaining different choices requires a change in participant’s reasoning (for example, values, beliefs, attitudes, or the logic they apply to a particular situation) and/or the resources (eg information, skills, material resources, support) they have available to them. This combination of ‘reasoning and resources’ is what enables the program to ‘work’ and is known as a programme ‘mechanism’. • The contexts in which programmes operate make a difference to the outcomes they achieve • Programmes work differently in different contexts and through different change mechanisms therefore programmes cannot simply be replicated from one context to another and automatically achieve the same outcomes. • Good understandings about ‘what works for whom, in what contexts, and how’ are, however, portable • One of the tasks of evaluation is to learn more about ‘what works for whom’, ‘in which contexts particular programs do and don’t work’, and ‘what mechanisms are triggered by what programmes in what contexts’ Pawson, R. and Tilley, N. (1997) Realistic Evaluation Sage
- 10. 10 How are they different? Logic models graphically illustrate program components, and creating one helps stakeholders clearly identify outcomes, inputs and activities Realistic evaluation emphasises outcomes produced via a mechanis within a context Context + Mechanism = Outcome Theories of Change link outcomes and activities to explain HOW and WHY the desired change is expected to come about
- 11. What is involved in theory of change…. • Qualitative and quantitative data collected over time • Identify early, intermediate and long-term outcomes • Theorise retrospectively and prospectively
- 12. Children’s Aid Society Community Schools in New York …community schools are complex systems making fundamental institutional changes, and the means that events occur in many ways and on many levels. – It is, therefore …necessary to look beyond standardized test scores to understand the impact of community schools. Clark & Grimaldi, 2005
- 13. Chicago FSES Evaluation Many efforts were underway in each school during the FSSI period to improve student outcomes, and no simple causal links can be drawn between FSSI and improvement at the three schools. Whalen, 2002
- 14. Constructing a theory of change
- 15. The starting situation What it’s like now – and why Strands of action What we are going to do about it Intended outputs What we will need to do differently Steps of change How things will change (for beneficiaries) Intended outcomes How the starting situation will change for CYP
- 16. Steps of change
- 17. Lessons from experience • The process is iterative…and human • Start where the users are (situation, action, outcomes…) • Keep it simple • Act as a critical friend – Help to clarify – Test the logic – Test the evidence
- 18. Theory of change in action
- 21. Starting situation – High levels of disadvantage on a wide range of social indicators. – A small local authority - can lack staff capacity to support service reform/delivery. – Communities which are locally very cohesive, but which are also inward looking and ‘territorial’ – acts as barriers to accessing services. – A growing number of third generation unemployed, with this helping to establish and perpetuate a non-aspirational, non-learning, low-skills and workless culture. – New Community Learning Centres – Full range of extended services
- 22. Asking the right questions • What is the situation you face? • What needs to change? • How will these changes be made? • What actions will you take? • What effect will those actions have? On whom? • How will you know if change is happening? • What will happen for person A, person B etc
- 23. Community Learning Centres: a theory of change for their community role • The CLCs will develop a full range of extended services – either provided on site, or facilitating access to services off site. (There will be sufficient capacity within the voluntary and community sector, and within other services, to support this.) • The CLCs will attract existing community groups to use their resources, and will be approached by other services and organisations which want to develop new activities (or continue established activities) • The CLCs’ extended activities will be accessible and affordable to community members. • Community members will engage in extended activities and use the CLCs’ facilities. • The CLCs will become viable as not-for-profit organisations with regard to extended service provision. • Through engaging with activities either directly provided by, or co-ordinated or commissioned by the CLCs, outcomes for families within the borough will improve.
- 24. Construct your own theory • Identify a situation in which someone is taking purposeful action… • …or use ‘coming to a training day’, ‘getting married’, ‘welfare reform’ • Build a possible theory of change for this action
- 25. Using evidence
- 31. Some questions evidence is needed to answer • CLC programme of activities (& what commissioned?) • What community-based activities are out there? • Which services/community groups are booking rooms? • Which activities are new? Which are established and moving from elsewhere? • How much do activities cost? Who attends? Where do they live? • What revenue does the centre make from room hire? What – if any – shortfall is there? • What outcomes are there for children and families (what outcomes are expected?)
- 32. I think there are some mixed messages about is it really for the community? I mean it’s physically in the middle of it, but there’s a great concern that the community will be priced out because I reckon each centre needs to take £70,000 before it even breaks even. Now £70,000 is a lot of bookings for squash and badminton…
- 33. As far as I am aware, you cannot go and ‘pay and play’ in the [CLC] sports facilities, so there’s this lovely gym there and the kids and adults have got their noses pressed against the window saying ‘can we use this great gym?’. But you can’t just go and use it, what you can do is hire the whole thing and then use it. With great respect, not even the local football team is that well organised and well resourced to go and hire the whole of the gym to do a fitness session. So we’ve got this thing that’s right there in the community and for six months it hasn’t been used by any member of the community.
- 34. We were having trouble with the all weather pitches at the school with anti-social behaviour because the people who were using it were the people who could pay to use the outdoor sports facility and some of the young people were throwing stones over and some of the comments were ‘you don’t live round here, this is ours and if we can’t afford to go on here or we’re being kept out, you’re not playing on it either’.
- 35. Populating your own theory with evidence • What evidence could you use to populate your theory of change?
- 37. Theory of Change - Quantification Dr. Michael Wigelsworth Manchester Institute of Education University of Manchester
- 38. Hello! • Dr. Michael Wigelsworth • Lecturer: – M.Ed. Psychology of Education – M.Sc Educational Research • Special education and additional needs (SEAN) – Mental health and wellbeing – Social and emotional learning • Programme Evaluation - Outcomes and impact – Multi-component / complex school based programmes – SEAL (primary and secondary)/ Achievement For All / PATHS to Success • Pragmatic researcher • Not a statistician
- 39. Quantification of ToC • What? • Why? • How? • Where /When?
- 40. What is quantification? • The use of numerical data / quantitative evidence to assess / evaluate a ToC • A quantified theory of change becomes a logic model
- 41. Quantification of ToC • ToC Logic model • The theory of change is specified to the point that individual components of the model can be evidenced, and the casual chains investigated • To operationalise diffuse concepts into definable factors - should be measurable
- 42. Same stages… • Define input • Explain effects • Define outcomes • Identifying moderators / mediators
- 43. ToC /Logic model Effects OutcomesInput Moderating effects? Mediating effects?
- 44. Improved pupils mental health An example: Mental health in schools Improved attainment Mental health programme
- 45. Improved pupils mental health Taught curricula Model behaviour by teachers Parent materials Reduction in in- class exclusions Improved pupil’s self reported “readiness to learn” Improved KS scores Proximal (short term) outcomes Distal (long term) An example: Mental health in schools Consideration of other factors
- 46. Why?(?) • Makes ToC explicit / observable as an intervention – Demonstrates impact – this is because the outcome is for everybody – Move to transferability (poss. Replication) • Demonstrate impact of the ToC specifically – Examine the unique constructs (e.g. perceptions vs. outcomes) – Provide empirical evidence of theoretical assumptions / chains • Can test the relative strength of each causal chain (think baby and bathwater) • Identifies key effect mediators or moderators (may be relevant to subgroups) • Specifies differences between proximal and distal effects – manages expectations of changes in outcomes
- 47. Why not? • May be hard to capture “the effect” • May be very hard to account for all important moderating / mediating factors • Requires accountability • possibility of null or iatrogenic effects • Requires levels of technical expertise
- 48. Where / When? • A logic model can (should?) be applied where measureable outcomes (i.e. impact) is expected or needs to be demonstrated • This can apply (lesser /greater extent) at all stages of research – Theory Long term implementation
- 49. Campbell, M., Fitzpatrick, R., Haines, A., Kinmouth, A., Sandersock, P., Spiegelhalter, D. & Tyrer, P. (2000). Framework for design and evaluation of complex interventions to improve health. British Medical Journal, 321, 694–696.
- 50. How? • Dependent on quality… – Full scale testing of a logic model for a complex is a big ask – But can you afford not to? • Dependent on audience – What do you want to be able to say? – To who?
- 51. Take home • A important question to ask is: – How well specified is my ToC? • How clearly established are my outcomes? – Do I need to demonstrate impact? • Is there value in investigating chains of evidence? – E.g. how did we arrive (not arrive) at impact?
- 52. Key issues: relationships with users
- 53. Multi-strand projects mean you are likely to need to build relationships with many different people • Strategic leads • Operational staff • External agencies/partners • Communities • Families and young people
- 54. How to build good relationships • Understand the context people are working in • Be aware of other demands on their time • Understand the relationships they have with each other • Explore their hopes and fears about research • Value their input and ideas
- 55. How a Theory of Change approach can help • Working together • It is dialogical, involving conversation and negotiation • Enables continuous feedback • Can feed into project planning • Provides guidance about data collection methods and self evaluation
- 56. Top tips! • Be knowledgeable and respectful of those you work with • Expect trust to build gradually – earn it, not expect it • Be willing to fit in with them – don’t expect them to work to your demands! • Factor in a ‘getting to know you’ visit • Share information • Theories are often implicit – projects need time and support to work through this without feeling inadequate or pressured
- 57. Impact on users is evidenced by a deputy head of a primary school in Manchester, who said: ‘We’ve used your change document… it’s helped us to ensure the impact of things in a way that we hadn’t thought of’
- 58. Key issues: drawing conclusions • Formative: – ‘Initiative X is triggering changes a.b.c….which are likely/unlikely to lead to outcomes 1, 2, 3….’ • Summative: – ‘Initiative X triggered changes a.b.c….leading to outcomes 1, 2, 3….’ – ‘Initiative X also triggered changes l, m, n….which may yet lead to outcomes 1, 2, 3….’ • But: • ‘Initiative X# may not trigger changes a.b.c….or lead to outcomes 1, 2, 3….’
- 59. Drawing conclusions • Building a theory: – Initiatives X, X#, X##...triggered similar changes leading to similar outcomes’ – Initiatives of type X are likely to lead to similar outcomes • ToC and ‘what works’ • Using a D & R approach