Effectiveness: Improve It Framework - Ben Jackson, Bond
2 likes1,284 views
The document introduces the Improve It Framework, which aims to help UK NGOs better measure, manage, and demonstrate their social impact through common outcome indicators, data collection tools, and quality standards. It describes key elements of the framework, including the Organizational Health Check tool and NGO Evidence Principles. The framework is intended to support greater consistency and harmonization in effectiveness measurement while allowing for context-specific approaches. It also discusses how the different strands of the Consortium Effectiveness Programme fit together and provides timelines for upcoming initiatives related to the framework.
Effectiveness: Improve It Framework - Ben Jackson, Bond
- 1. Introduction to the Improve It Framework - Effectiveness: Improving and Proving It Conference Edinburgh November 2012 Ben Jackson Bjackson@bond.org.uk Chief Executive, Bond bond.org.uk
- 2. Context
- 3. Consortium Effectiveness Programme …supports UK NGOs in improving how they measure, manage and demonstrate their contribution to social change 1. Developing agreement and supporting implementation of a sector wide framework of common outcome indicators and data collection tools (Improve it Framework); and online organisational self- assessment and resource portal (Organisational Health Check; NIDOS Effectiveness Tool); and agreed quality standards regarding NGO evidence. 2. Building knowledge and skills to support members to measure and manage effectiveness and value for money 3. Creating an enabling environment that encourages and supports organisations to deliver improvements in their effectiveness bond.org.uk 3
- 4. Consortium Effectiveness Programme: Key elements • NIDOSEffectiveness Tool / Bond Organisational Health Check • Value for Money • NGO Evidence Principles • Improve it Framework bond.org.uk 4
- 5. Contributors to the programme Consortium partners: bond.org.uk 5
- 6. How the strands of programme fit together Development effectiveness Robust, Appropriate practical approaches to approaches to managing cost measuring efficiencies & effectiveness effectiveness Strong systems, culture, leadership etc Bond Organisational Health Check/ NIDOS Effectiveness Tool bond.org.uk 6
- 8. The Improve it Framework What is it? • A tool for improving how UK NGOs measure, learn from and demonstrate outcomes and impact • A platform for sharing information on outcomes, indicators and data collection tools Why are we developing it? • Developing robust methods for measuring effectiveness is a collective challenge for UK NGOs • UK NGOs need to start telling a more robust story at sector level as well as at organisational level about their added value bond.org.uk 8
- 9. Improve it Framework: what it is and is not It is not… It is… • …about creating a single way of • …about providing a collective resource assessing effectiveness –it’s about that UK NGOs can draw on when encouraging greater harmonisation developing their own context specific and consistency where appropriate M&E frameworks • …going to offer an off the shelf • …about developing shared answer to measuring effectiveness, approaches where we can but it will provide a common starting • ….about providing organisations with point the tools to be able to tell a more • …going to produce an encyclopaedia robust story of how they are of indicators and tools. There will be contributing to change an element of prioritisation. • …going to be practical & simple • …an organisation specific tool; • …going to be dynamic & will continue seeing 60% of your organisation in it to evolve is good enough bond.org.uk 9
- 10. Where the Improve it Framework can help in your work Focus of Improve it Framework 1 2 3 4 5 6 Identify who Identify the Collect, Identify the Articulate you will work Identify what evidence analyze and problem & your vision with & you want to that you will draw underlying of change influence & change collect and conclusions causes how how for the data Outcomes Indicators & Principles 7 8 tools Reflect and learn from Communicate the data results bond.org.uk
- 11. In practice
- 12. Improve it Framework Capacity development
- 13. Domains of Change Framework for Education Outcomes a) Laws, policies and practices b) Adequate resources are allocated c) Performance in are in place that support quality and adequate teachers recruited and education is monitored education that is accessible for trained for the provision of quality at a national and local all children education for all level Outcomes Outcomes 2 a) School management Mothers, fathers and other a) Governments and committees exist and manage caregivers support all their other mandated bodies schools effectively children to access education ensure all children b) School managers access, b) Communities actively support access quality manage, and deploy resources and participate in children’s education in an effective and transparent education way 6 3 c) CSOs support and advocate for c) Teachers, resources and improved education for all infrastructure are in place to Communities actively Schools are well support education support the education managed, safe and well of all children d) Schools are safe and resourced healthy environments for girls 1) All children and boys access quality education where Outcomes they learn useful knowledge and a) Teaching is high quality, skills child-centred and child- Outcomes friendly a) Children participate in a b) Teachers deliver a full meaningful way in decisions that 5 4 allocation of lessons affect their education c) Teachers are trained and b) Children participate in their All children participate All children receive supported to deliver high own and others’ learning in their education and their full allocation of quality teaching in school life high quality teaching d) The curriculum is relevant c) Children have a positive and appropriate attitude towards education e) Teaching supports equality and empowerment
- 14. Domains of Change Framework for Education Outcomes a) Laws, policies and practices b) Adequate resources are allocated c) Performance in are in place that support quality and adequate teachers recruited and education is monitored education that is accessible for trained for the provision of quality at a national and local all children education for all level Outcomes Outcomes 2 a) School management Mothers, fathers and other a) Governments and committees exist and manage caregivers support all their other mandated bodies schools effectively children to access education ensure all children b) School managers access, b) Communities actively support access quality manage, and deploy resources and participate in children’s education in an effective and transparent education way 6 3 c) CSOs support and advocate for c) Teachers, resources and improved education for all infrastructure are in place to Communities actively Schools are well support education support the education managed, safe and well of all children d) Schools are safe and resourced healthy environments for girls 1) All children and boys access quality education where Outcomes they learn useful knowledge and a) Teaching is high quality, skills child-centred and child- Outcomes friendly a) Children participate in a b) Teachers deliver a full meaningful way in decisions that 5 4 allocation of lessons affect their education c) Teachers are trained and b) Children participate in their All children participate All children receive supported to deliver high own and others’ learning in their education and their full allocation of quality teaching in school life high quality teaching d) The curriculum is relevant c) Children have a positive and appropriate attitude towards education e) Teaching supports equality and empowerment
- 15. Domains of Change Framework for Education Outcomes a) Laws, policies and practices b) Adequate resources are allocated c) Performance in are in place that support quality and adequate teachers recruited and education is monitored education that is accessible for trained for the provision of quality at a national and local all children education for all level Outcomes Outcomes 2 a) School management Mothers, fathers and other a) Governments and committees exist and manage caregivers support all their other mandated bodies schools effectively children to access education ensure all children b) School managers access, b) Communities actively support access quality manage, and deploy resources and participate in children’s education in an effective and transparent education way 6 3 c) CSOs support and advocate for c) Teachers, resources and improved education for all infrastructure are in place to Communities actively Schools are well support education support the education managed, safe and well of all children d) Schools are safe and resourced healthy environments for girls 1) All children and boys access quality education where Outcomes they learn useful knowledge and a) Teaching is high quality, skills child-centred and child- Outcomes friendly a) Children participate in a b) Teachers deliver a full meaningful way in decisions that 5 4 allocation of lessons affect their education c) Teachers are trained and b) Children participate in their All children participate All children receive supported to deliver high own and others’ learning in their education and their full allocation of quality teaching in school life high quality teaching d) The curriculum is relevant c) Children have a positive and appropriate attitude towards education e) Teaching supports equality and empowerment
- 16. School management committees exist and manage schools effectively Indicators Tools • # and % of schools with a Parent Teacher The Right to Education indicators and form Association (PTA) for section 9: right to transparent and accountable schools. • Level of activity of the Parent Teacher The Plan UK child friendly tool Section 1: Association (PTA) School management • # and % of schools with a School The VSO quality scale for education services management committee (SMC) quality areas on education management and community engagement • Level of activity of the School management committee (SMC) bond.org.uk 16
- 21. Timelines
- 22. What to look forward to • Organisational Health Check pilot starts (Summer 2012) • Launch of pilot Evidence Principles (October 2012) • Beta version of the Improve it Framework (December 2012) • Comic Relief, Big Lottery Fund, DFID, Bond research into the cost of measuring effectiveness (Spring 2013) • Guide to demystifying theories of change (Spring 2013) bond.org.uk 22
- 23. Thank you Contact: Ben Jackson Chief Executive bjackson@bond.org.uk Regent’s Wharf • 8 All Saints Street • London • N1 9RL • United Kingdom +44 (0)20 7837 8344 • bond@bond.org.uk • bond.org.uk bond.org.uk 23
- 24. Organisational Health Check What is it? • An online self-assessment that: • Identifies strengths and weaknesses in organisational capacity / functionality • Facilitates benchmarking with peers • Sign posts to other initiatives, tools, training and support Why are we developing it? • Strong systems, leadership etc are key building blocks for delivering good development outcomes • A common framework can help facilitate structured peer learning and the sharing of best practice • An online self-assessment enables bond to provide more tailored capacity support bond.org.uk 24
- 25. Value for Money (VfM) What are we doing? • Providing common tools and frameworks for NGOs to navigate the VfM agenda • Bringing NGOs together to share emerging experience • Ensuring constructive dialogue with DfID Why are we doing it? • At the heart of the VfM agenda are important questions about how UK NGOs use their resources • VfM is a part of the wider results agenda • Want to ensure VfM remains appropriate and proportionate to NGO operations and programmes bond.org.uk 25
- 26. NGO Evidence Principles (pilot to May 2013) 1. Voice and Inclusion: We listen to beneficiaries’ perspectives on the effects of the intervention, and can identify who has been affected and how. 2. Appropriateness: We use methods for data collection and analysis that are justifiable given the nature of the intervention and purpose of the assessment 3. Triangulation: We make conclusions about the intervention’s effects by using a mix of methods, data sources, and perspectives 4. Contribution: We show how change happened, explore a range of explanations for why it happened, test our theory of how we thought it would happen, and explain how we contributed 5. Transparency: We are open about the data sources and methods used, the results achieved, and the strengths and limitations of the evidence bond.org.uk 26
- 27. NGO Evidence Principles – the checklist A NGO checklist for assessing the quality of evidence (DRAFT) Piece of evidence being assessed: Name of assessor: Principle Criteria 1 2 3 4 NA Comments / evidence 1a. Are the perspectives of beneficiaries included in the 1) Voice and Inclusion evidence? We listen to 1b. Are the perspectives of the most excluded and beneficiaries’ marginalised groups included in the evidence? perspectives on the effects of the 1c. Are the findings disaggregated according to sex, intervention, and can disability and other relevant social differences? identify who has been affected and how 1d. Do beneficiaries play an active role in designing the assessment process and/or analysing data? Score for voice and inclusion (average of criteria scores): 2a. Are the methods used to collect data appropriate given 2) Appropriateness the nature of the intervention and the purpose of the assessment? We use methods that 2b. Is the size and composition of the group from which are justifiable given data is collected appropriate given the nature of the the nature of the intervention and purpose of the assessment? intervention and 2c. Is the composition of the team collecting and analysing purpose of the the data appropriate given the nature of the intervention assessment and purpose of the assessment? 2d. Are the methods used to analyse the data appropriate given the nature of the intervention and purpose of the assessment? Score for voice and inclusion (average of criteria scores): bond.org.uk
- 28. NGO Evidence Principles – the results • A quality assurance framework Voice & Inclusion Appropriateness 13/16 7/16 • A kite mark / seal of approval Transparency Triangula on 9/16 12/16 Contribu on • A tool for identifying skill 4/16 gaps and building capacity bond.org.uk 28