D6 e6 outcome based measurement from theory to implementation
1 like609 views
The document outlines an agenda for a workshop on outcome-based measurement. The workshop objectives are to examine developing and implementing outcome-based measurement, reflect on embedding theory of change in program planning, and explore how measurement can build an evaluative culture. The agenda includes presentations on measurement, theory of change, and building culture. It also has group discussions on these topics. Measurement is defined as collecting regular data on outputs and outcomes, while evaluation establishes causal relationships through research methods. Key questions for different levels of measurement are provided.
D6 e6 outcome based measurement from theory to implementation
- 1. Outcome-Based Measurement From Theory to Implementation Ghebray Consulting 647-823-5402 ghebrayconsulting@rogers.com
- 2. Workshop Objectives 1. Examine the process of developing & implementing outcome-based measurement 2. Reflect on the value of embedding Theory of Change in developing a program theory or logic model 3. Explore how the process of developing & implementing outcome-based measurement can be a vehicle for building evaluative culture
- 3. Agenda 1. Presentation: An overview of measurement & evaluation (15 minutes) 2. Discussion (10 minutes) 3. Presentation: walking through the program planning cycle – embedding theory of change (15 minutes) 4. Break (10 minutes) 5. Small group discussion (50 minutes) 6. Presentation: Measurement plan (15 minutes) 7. Facilitated discussion: Building evaluative culture (25 minutes) 8. Final thoughts & reflections (10 minutes)
- 5. Measurement •Measurement is the process of systematically and regularly collecting data on program quality, outputs, and outcomes for program participants (Not intended to establish causal relationship) Evaluation •Evaluation is a form of research method –in- depth and rigorous effort to measure program impact and uses scientific research methods to compare outcomes with what would happen in absence of strategies or intervention(s)
- 6. INPUTS STRATEGIES OUTCOMES What was Which What was Which Did the resources the strategies participant quantity & were most quantity were most s change? quality of important & quality important If so, how resources for of the for much and used to providing strategies achieving in what implement “high provided? the desired ways? strategies? quality” outcomes? strategy?
- 7. INPUTS OUTPUTS OUTCOMES IMPACT MONITORING PERFORMANCE OUTCOME IMPACT STUDY MEASUREMENT MEASUREMENT What is the What is the What are the What are quantity & quantity results for the quality of and quality participants impacts resources we of the and other that can be have invested program or stakeholders? attributed in this services we to the program? Are have program or they aligned delivered? services? with program goals?
- 8. Monitoring Questions • How many clients are in the program? • What are the socio-demographic characteristic of those in the program? • Are program participants part of the intended or target population? • How are program funds being used?
- 9. Performance Measurement Questions • How well was the program marketed or promoted? • Did the program offer adequate and high quality strategy or intervention? • How do clients perceive the program? • Are clients satisfied with the service and their encounters with service providers?
- 10. Outcome Measurement Questions •Did clients attain the intended outcomes? •What did clients learn? Impact Study Questions •Have clients’ quality of life improved following program experience? Is the program responsible for their improved quality of life? •Is client participation in the program “responsible” for their improvement?
- 11. INPUTS OUTPUTS OUTCOMES IMPACT MONITORING PERFORMANCE OUTCOME IMPACT STUDY MEASUREMENT MEASUREMENT DOCUMENTING DOCUMENTING RESEARCH TRACKING INVESTMENT PROGRESS CAUSALITY SOME EVALUATION EXPERIENCE & HIGH EVALUATION EXPERTISE REQUIRED EXPERIENCE & EXPERTISE REQUIRED LOW/REASONABLE COST HIGH COST SHORTER TIME LONGER TIME
- 13. Before Implementing Outcome-based Measurement… 1. Thorough program description • Program rationale, definition of the problem & feasibility of strategies or interventions • Clarity of causal assumptions - between resources strategies or interventions and expected outcomes as well as performance measures or indicators 2. Capacity & experience of program leaders to facilitate and act on findings & lessons learned
- 14. Community Design Analysis, Design M & E Plan Asset & Needs Program Assessment Develop Engage Key Decision Data Collection Stakeholders Making Tools Share Data Collect Implement Findings & Analysis Data Program Lessons Learned
- 15. CONTEXTUAL ASSUMPTIONS INPUTS STRATEGIES OUTPUTS OUTCOMES ANALYSIS The The Resources What Direct Benefits Reason implicit dedicated program products for clients behind theory to the does of during & the or belief program with program after the need for why the inputs strategies program the program program will work
- 16. Employment Program Example CONTEXTUAL STRATEGIES OUTPUTS ASSUMPTIONS INPUTS -# of classes ANALYSIS -Weekly soft -The absence -2 F.T.E Staff & mentorship -People in skills classes of a job history sessions our community: perpetuates -Program & -Match clients & offered Unemployment counseling -Have few job mentors space -# & type of skills, bad or no -Job search skills jobs & limited - 8 months of clients training helps -Program job histories supervised job served with job supplies & training & Readiness equipment -Have few job mentoring training & job sessions -Job mentorship placement can enhance on IMPACT OUTCOME opportunities the job learning -Clients find -Clients learn soft jobs, remain skills & job strategies -Job placement employed & helps establish establish job -Clients establish supportive job history histories professional relationships with mentors
- 18. Using Program Theory to Plan and implement Outcome-based Measurement
- 19. What does “success” look like? According to Mark Friedman (2005), three basic questions can help us assess program success 1.How much did we do? 2.How well did we do it? 3.Is anyone better off?
- 20. Outcome Indicator (Is anyone better off?) • Outcome indicators measure how much progress was made towards achieving the outcome - For an outcome “Newcomers are job ready to compete in Canadian job market” Outcome indicators include: • Rate of successful training & job placement completion (i.e. ability to prepare resume, positive feedback on mock interview & job placement • Maintaining professional relationship with mentors 6 month after end of program
- 21. Process Indicators (How well did we do it?) • Process indicators measure ways in which program services are delivered – For “Provide high quality service” Process indicators include: • Qualification & experience of mentors • Suitability of client/mentor match & job placement • Rate of client satisfaction
- 22. Output Indicators (How much did we do?) • Output indicators measure the quantity or the volume of services produced - For “Provide adequate support to targeted population” Output indicators include: - # & types services offered (i.e. learning sessions, employment counseling, mentorship etc.) - # and characteristics of clients served - # of mentors recruited & matched
- 23. Measuring Program “Success” # % 1. Meet outreach target 150 100 2. Attendance during orientation, intake & level of 75 50 registration in program (# & type of clients served) 3. High attendance, participation, & rate of satisfaction 65 43 4. Knowledge of job search strategies, Canadian labor 50 33 market & work place culture 5. Successful completion of program (including job 45 30 placement) 7. #s employed in profession related at ($17.00/hour) 26 17 8. Favorable Job assessment (tracked quarterly) 20 13 9. Job retention (for at least one year) 15 10
- 24. The Measurement & Evaluation Plan Program Indicators Data Data Who When Who Needs the Component Sources Collection Collects to Findings & to be Methods Data Collect Lessons Learned Measured Data did we do? How much Scope & Reach we do it it? How well did Program or Service Quality better off? Is anyone Outcome for Participants
- 26. Building Evaluative Culture 1. How do you create a shared understanding of measurement and evaluation? How do you make measurement & evaluation everyone’s business? 2. What does measurement & evaluation mean to your board, staff etc? 3. How do you balance between learning and accountability? How do you manage the conversation about measurement & evaluation with funders/staff? How do you negotiate what can and cannot be delivered?
- 27. References • Crawford, P. & Bryce, P. (2002). Project monitoring and evaluation: A method for enhancing the efficiency and effectiveness of aid project implementation • Friedman, M. (2005). Trying hard is not good enough: How to produce measurable improvements for customers and communities. • Penna, R. & Phillips, W. (2004). Outcome Frameworks: An overview for practitioners • Smith, M. (2010). Handbook of program evaluation for social work and health professionals • www.tccgrp.com