Moving from evaluation to learning peter york
Download as PPTX, PDF1 like818 views
This document discusses the importance of evaluative learning over traditional program evaluation for non-profits. It argues that evaluation focuses on determining a program's overall impact, while evaluative learning focuses on understanding how program components work in order to improve practices. The document provides six key practices of evaluative learning, including measuring achievable outcomes and gathering data from recipients. It outlines guiding steps for conducting evaluative learning, such as identifying achievable outcomes and analyzing patterns of cause-and-effect. Non-profits that engage in evaluative learning behaviors are over twice as likely to experience sustainable growth compared to those who do not.
Moving from evaluation to learning peter york
- 1. Program Evaluation: Moving from Evaluation to Learning
- 2. 2 Session Agenda • Introductions • The Need for Learning • Why Nonprofits Aren’t Learning • Evaluation vs. Evaluative Learning • Nonprofit Evaluative Learning – Steps, Processes & Tools
- 3. The Need for Learning
- 4. 4 Leadership = Success Leadership is key to: Growth Sustainability Lifecycle Advancement Effectiveness
- 5. 5 There’s More to Leadership… • Advancing effective communication of mission and vision to internal and external stakeholders • Engaging internal and external stakeholders in planning • Taking decisive action when faced with challenges • Making decisions anchored in cost-effectiveness • Demanding accountability that includes demonstrated success with those being served/targeted Only one in four nonprofit organizations is well led…
- 6. 6 The Key to Leading Is Learning Only one in four nonprofit organizations is an effective learner… The following organizational learning behaviors are significantly and singly the biggest predictor of organizational leadership: • Creating sophisticated financial, operational, programmatic and environmental data-gathering and learning processes • Spending time leveraging program data for making meaning, decisions, designs and plans, not accountability or validation • Infusing learning into ALL planning, anchored in program success • Taking immediate and decisive action, particularly at a human resource level, as indicated by strategy research findings
- 7. Why Aren’t Nonprofits Learning?
- 8. 8 The Investment Model Is All Wrong • There’s no “outcome synchronicity” between the investor and investee • Effectiveness and accountability must be viewed through the measurement of proximate effect • Proximate cause-and-effect is the only way! • Learning requires understanding the cause, not the effect • There’s no Research & Development for programs/initiatives
- 9. 9 What Results Do Investors, Business Leaders, & Consumers Want? Example: In-Home Support Services to New Parents, For-Profits vs. Nonprofits Investors (Funders) Business Leaders Consumers For-Profit Results: Just Give Me (or Show Me) the Direct Results, Please… Nonprofit Accountability: Just Give Me the Direct Results, But Somehow Prove to “Them” That We Can Do Much More… Recuperation, parent-child bonding, healthy adjustment to family change, tools for care and feeding, stress-reducing routines and habits, better communication skills with providers • Same Direct Results as the For-Profit Business • Child Development, School Readiness, Crime Reduction • Parental Reduction in Child Abuse & Neglect, Maternal Health Nonprofit Business Leaders & Clients Funders (Investors)
- 10. Why Evaluation Doesn’t Always Lead to Learning
- 11. 11 Why Evaluation Isn’t Achieving Learning 1. Makes uncontrollable community impact indicators the metric of accountability 2. Evaluates the “whole” program/strategic effort to determine if it worked, rather than the combinations of program components to determine how it works 3. Aspires to a scientific research design ideal that seeks widespread generalizability, rather than contextual applications 4. Gathers data from the wrong source
- 12. Evaluation vs. Evaluative Learning
- 13. 13 The Difference Between Evaluation and Evaluative Learning Evaluation: the owner determines success by overall wins, losses and trophies. Evaluative Learning: the coach/manager figures out how to improve the odds of success for the next game by analyzing the play of the previous ones.
- 14. 14 Resources Whole Program Short-Term Outcomes Long-Term Outcomes Community / Social Impact Evaluation Determining a “Whole” Program’s Long-Term Impact to Judge Social Value
- 15. 15 Program Resources Program Elements Achievable Outcomes/ Next Actions Studies say… Community / Social Impact Evaluative Learning Measuring Program Success for “Best Practice” Improvement
- 16. 16 Six Learning Practices that Facilitate Sustainable Growth Organizations are significantly more likely to sustain and grow if they engage in the following evaluative learning behaviors: NOT evaluating to decide if a whole program works; rather, figuring out when and how it works. Gathering data from recipients. Determining outcomes by listening to recipient changes and next actions. Engaging people in making meaning from data. Developing performance management processes, tools and systems. Prioritizing cost-per-outcome.
- 17. 17 Evaluative Learning Facilitates Sustainable Growth 41% of nonprofits grow faster than the rate of inflation over a three year period Nonprofits where leaders engage in these behaviors are 2.5 times more likely to grow faster than the rate of inflation. 7% 5% 4% 2% Conducting All R&D Behaviors (0% of all NPs) Conducting Many R&D Behaviors (5%) Conducting Some R&D Behaviors (45%) Conducing Very Few R&D Behaviors (25%) Conducting NO R&D Behaviors (25%) AverageAnnualGrowthRate Average Annual Growth Rate (Mean Per Year, Based on Three Consecutive Years of Data)
- 18. 18 Guiding Steps for Conducting Evaluative Learning 1. Measure achievable outcomes, next actions, direct results. 2. Gather lots of data QUICKLY, by quantifying the qualitative. 3. Ask the participant/recipient. 4. Analyze data for patterns of cause-and-effect. 5. Gather a few key illustrative stories. 6. Spend time making meaning and making improvements.
- 19. 19 Step One: Identify Your Achievable Outcomes • It’s all about the path to behavior change: – Awareness + – Knowledge + – Attitude + – Motivation + – Skills + – Opportunity + – Behaviors = ___________________ Habits
- 20. 20 Evaluative Learning Processes to Try 1. Positive Deviant Experiment 2. Literature Review to Make the Case for Proximate Outcomes 3. Proximate Logic Model Creation 4. Survey Data Project 5. Qualitative Data Project 6. Creation of Ongoing R&D and Performance Management Tools
- 21. Thanks!