2.12 Creating a Yardstick: Developing a Performance Measurement System
Download as PPT, PDF1 like2,376 views
The document discusses performance measurement for homeless services. It defines performance measurement and outlines ingredients for successful implementation, including establishing strategic objectives and metrics. It notes challenges like pressure on staff to lie about data and discomfort with data analysis. Approaches discussed include prioritizing clients, tracking outcomes over just outputs, and coordinating measurement across service sectors rather than just programs.
![Interactive Options @orgcode [email_address]](https://image.slidesharecdn.com/2-12-creatingayardstickdevelopingaperformancemeasurementsystem-110719124551-phpapp02/85/2-12-Creating-a-Yardstick-Developing-a-Performance-Measurement-System-4-320.jpg)
![Iain De Jong OrgCode Consulting Inc. 416-432-0410 [email_address] @orgcode www.orgcode.com Susan McGee Homeward Trust 780-496-6035 [email_address] @susanmcgee www.homewardtrust.ca Kim Walker Center for Capacity Building National Alliance to End Homelessness 202-942-8292 [email_address] www.naeh.org Amanda Sternberg Homeless Action Network Detroit 313-964-3666 x4201 [email_address] www.handetroit.org](https://image.slidesharecdn.com/2-12-creatingayardstickdevelopingaperformancemeasurementsystem-110719124551-phpapp02/85/2-12-Creating-a-Yardstick-Developing-a-Performance-Measurement-System-52-320.jpg)
2.12 Creating a Yardstick: Developing a Performance Measurement System
- 1. Performance Measurement Some Handy Thoughts Iain De Jong OrgCode Consulting, Inc.
- 2. Agenda Intro/Definition of Performance Measurement Ingredients for Successful Performance Measurement When we know we need to focus on performance measurement Approaches to performance measurement; moving from program to system level Priority setting, coordinating performance measurement across the system; metrics to look at Performance measurement and HEARTH, PM Tools Q & A/Session Close
- 3. To Get Started If you want copies of the presentations, send an email (contact details for each presenter are at the end of this presentation). At some point presentations will be on the NAEH website – www.naeh.org - as well as oodles of other cool resources. The OrgCode website - www.orgcode.com - will also contain some of the presentations and other materials you may find useful.
- 4. Interactive Options @orgcode [email_address]
- 5. Intro/Definitions of Performance Measurement
- 6. Definitions of Performance Measurement Performance measurement is the process by which criteria is established to determine the quality of activities based on organizational goals using quantitative evidence. A simple, effective system for determining whether an organization is meeting objectives.
- 7. What Performance Measurement is Not It is NOT a waste of time. It is NOT your HMIS. That is a data retention, management and report generating system – not a performance measurement system. It is NOT something you consider once per year when doing a funding application. It is NOT a collection of anecdotes. It is NOT something you look at with passing interest and do nothing about.
- 8. 6 Things That Performance Measurement Does (When Done Right) Ensures that service and system requirements are met Sets transparent sensible objectives Provides internal and external standards for comparison Provides a “scoreboard” for people to monitor their performance level Highlights problems that deserve priority attention Provide feedback to drive the improvement efforts
- 9. 4 Purposes to Performance Measurement at the System Level Identify chronic underperformers Identify those organizations where, with coaching and extra attention, performance standards can be met Ensure prudent financial stewardship Acknowledge (reward?) strong performers
- 10. From the Trenches Between 1 in 5 and 1 in 3 organizations report that staff have felt pressure to lie about their data at least once in the past year 3 out of 5 organizations that use an HMIS also use one or more other approaches to collecting information on service Almost half of all staff report they need help analyzing data
- 11. From the Trenches Between 40-50% of frontline staff are uncomfortable with data and performance measurement 3 out of the top 5 ways in which organizations use data pertains to funding (applications, soliciting donations/fundraising, and meetings with funders) and performance measurement tends to track less than 50%
- 12. From the Trenches 4 out of 5 organizations report that data is consistent with their organization’s Mission, but less than 2 out of 5 report that the data adequately demonstrates the efforts that go into the work 7 out of 10 service providers do not have a process for assessing acuity of clients to point them in the direction of the resources best able to end their homelessness
- 13. From the Trenches Less than half of service providers have logic models for each homeless or housing program Of those that have a logic model, only two-thirds regularly update and measure against the logic model
- 14. Ingredients for Successful Performance Measurement
- 15. Intensive Supports Mid-level support Little If Any Support Homeless Population – Not Homogeneous “ Funnel” Of Homeless Services Intake & Assessment – Acuity Determined
- 16. Driving Change Through Performance Measurement 2009 2019
- 17. For Successful Operational Performance The intake into the system should create service pathways for clients based upon their needs. Prioritize those most in need of service first. Support plans should be informed by information gleaned at intake – seamlessly. Track how the acuity of clients change over time. Look beyond outputs to outcomes. Have a logic model, follow it and update against it.
- 18. Getting Started Define your strategic objectives. Define your strategic metrics/key performance indicators. Set performance targets. Establish a baseline. Determine when you need to look at which metrics to determine performance against targets.
- 19. To Be Successful In Getting Started See if people share the same perceptions and values across services. Look at the system as a whole and the place of every organization within it. Review the quality of your existing HMIS data and other data that may sit with other sources. If you need to refresh/update your plan to end homelessness, this is a good time to do so.
- 20. Want the Simplest Approach? Make sure your data and your approach is meaningful, unambiguous and widely understood Spend time on working with all data collecting and measuring organizations so that they “own” and manage the process Up front work on ensuring there is a high level of data integrity Embed data collection as part of “normal” duties instead of something people do in addition to “real” work Take action based upon that which is learned in measurement Link it all to your strategic objectives
- 21. A Good Performance Measurement Approach Looks at the system as a whole as well as individual agents/performers Focuses the attention on what matters most, not everything that is possible to measure Moves from data collection to the creation of a narrative based upon that which is collected, when, by whom, for what purpose and with which degrees of accuracy Ensures measurement of the right things Embraces positive deviants Creates a common language for an organization Is independently verifiable
- 22. Performance Measurement Is Only As Good As the Data Consistent definitions Timely entry by staff Non-duplication of entries (including aliases) Agreement on data updates by program Exits are duly noted Information is complete
- 23. " You get what you measure. Measure the wrong thing and you get the wrong behaviors.” - John H. Lingle
- 24. High Performing Organizations Know exactly which individuals/families they have the expertise to serve and which programs to provide. They are NOT all things to all people. Prioritize which individuals they serve. They are NOT first come, first serve. Internally monitor their performance and are dedicated to being reflective practitioners. Possess a simple, transparent and operational logic model for each program that they run. Have internal performance standards and operational standards that they measure against.
- 25. Signs of System Level Performance Measurement Failure Funders receive information and do not de-fund chronic underperformers Failure to learn and improve (excuse culture instead of a solution focused culture) Delay in reporting out results Continuing to function as a collection of projects rather than the system as a whole Anecdotal tampering of findings
- 26. Process Define Measure Learn Improve
- 27. When We Know We Need to Focus on Performance Measurement
- 28. Approaches to Performance Measurement Moving from the Program to System Level
- 29. “ A zebra does not change its spots.” - Al Gore
- 30. Advice for Success Begin the design of your performance measurement system with the end in mind. Have solid strategic objectives and test against the change that you think needs to happen is happening. Create a data loving culture. Ensure the data collection and performance measurement has value Focus on getting better and better and better.
- 31. Advice for Good Strategic Objectives 1. Is it measurable or verifiable? 2. Is it achievable or feasible? 3. Is it flexible or adaptable? 4. Is it consistent with the rest of your strategic plan? 5. Does it stretch your people without breaking them? 6. Is it clear, easy to understand, and inviting to achieve?
- 32. Performance Measurement & Your Funding Programs While meeting reporting expectations for funding, the key to success in thinking like a system is to find commonality across requirements and structuring community activities into sectors of service rather than funding programs.
- 33. Sectors of Service Approach 1 Prevention Outreach Emergency Shelter Transitional Housing Rapid Re-housing Permanent Supportive Housing Approach 2 Prevention Outreach Shelter Drop-ins Housing Supports
- 34. Example of Performance Indicators by Program Type Prevention Outreach Emergency Shelter Transitional Housing Rapid Re-Housing Permanent Supportive Housing Number of Unique Individuals Served Successful Housing Outcomes Average Length of Stay Recidivism Successful Income Outcomes Direct Client Assistance Utilization Occupancy
- 35. Example of Sector of Service Outreach Strategic Objective Decrease Street Homelessness by 50% or more by the next Point in Time Count by housing people, helping them access shelter and/or reuniting with family/friends, focusing on those with highest acuity first. Performance Indicators # of unique individuals served successful housing outputs successful shelter outputs successful family/friend reunification recidivism Performance Targets each outreach team to work with no more than 25 unique individuals per month 4 successful housing outputs per outreach team per month (at least 2 higher acuity per month) 8 successful shelter outputs per outreach team per month (at least 4 higher acuity per month) 1 successful family/friend reunification per outreach team per month
- 36. To Achieve Excellence in Sectors of Service Create and publish standards or core service expectations for each sector of service. Ensure that all Strategic Objectives are SMART! Set targets that are achievable, not just an aspiration. Define the service pathways and linkages between the sectors of service to allow for unambiguous service navigation.
- 37. The Data Analysis Plan Programs Populations Data Elements Data Calculations Frequency Reporting Format Report Audience
- 38. Data Analysis Plan: Programs By individual program Across all programs (duplicate users across programs; individual program users; average number of users per program)
- 39. Data Analysis Plan: Populations Specific groups to potentially investigate (examples): By gender By age group(s) By military service By presence of a specific diagnosis or life issue By level of acuity By length of homelessness By location within city By indoor or outdoor Etc.
- 40. Data Analysis Plan: Data Elements You can use HMIS measures
- 41. Data Analysis Plan: Data Calculations Need to write out how exactly the conclusions were reached, especially when moving beyond descriptive statistics
- 42. Data Analysis Plan: Frequency Need to pre-determine which data will be analyzed to measure performance Need to determine what the data ranges are for the analysis: Intake date Program enrollment date Assessment dates Exit date
- 43. Data Analysis Plan: Reporting Format Product Considerations Web-update Paper report Annual Report Report to the Board Report to Funder Style Considerations Table Graphs Context Analysis Outcome Impacts
- 44. Data Analysis Plan: Report Audience Board Funder General public Staff Supervisory Team Senior Management Team Executive Director Partner agencies 10 Year Plan Committee Elected Officials
- 45. Quality Assurance " When dealing with numerical data, approximately right is better than precisely wrong." - Carl G. Thor
- 46. Quarterly Report Considerations Target(s) for the quarter Outputs for the quarter Target(s) year to date Outputs year to date Quarter over quarter Year over year Outcomes to date
- 47. Canaries in Coal Mines http://popwatch.ew.com/2008/12/22/site-of-the-14//
- 48. Considerations for Process Review at the System Level Evidence of client prioritization Evidence of seamless linking of clients to programs that meet their needs Funding expensed by quarter Percentage of sector target achieved by organization Site visits
- 49. Priority setting, coordinating performance measurement across the system; metrics to look at
- 50. Performance Measurement & HEARTH, PM TOOLS
- 51. Q & A
- 52. Iain De Jong OrgCode Consulting Inc. 416-432-0410 [email_address] @orgcode www.orgcode.com Susan McGee Homeward Trust 780-496-6035 [email_address] @susanmcgee www.homewardtrust.ca Kim Walker Center for Capacity Building National Alliance to End Homelessness 202-942-8292 [email_address] www.naeh.org Amanda Sternberg Homeless Action Network Detroit 313-964-3666 x4201 [email_address] www.handetroit.org
Editor's Notes
- #11: Comes from surveys we have done in communities that are prepping for HEARTH. This may be different in your community, but we have certainly seen consistency. When we review these with senior leaders in communities – even Executive Directors, HMIS administrators and the like – I think some people shake their head in agreement, while others are a bit shocked at times.
- #19: The key is to do this for the SYSTEM AS A WHOLE, as well as individual organizations within the system. No point everyone going rogue and doing their own thing. Kim is going to talk more about thinking like a service system in the context of HEARTH and moving beyond just HUD-funded programs.
- #24: Lingle wrote Bullseye!
- #31: Too often organizations or communities have been collecting oodles of data without knowing why they are doing it or what it is going to tell them. I say articulate in one sentence what your organization intends to do and the change it seeks to create. If you have good objectives and good measures for those objectives then you are actually testing whether the change that you want to see (say, ending chronic homelessness) is actually happening. Believe it or not, with enough cheerleading data collection, analysis and use to improve performance can be motivating and – dare I say it – fun! I look for data to be used and performance measurement to occur top to bottom and bottom to top within an organization. I love seeing Boards use data to make decisions and staff wrestle with what information is saying to that they drive improvement. . If frontline staff don’t see themselves reflected in the system, they simply won’t use it or they will lie or a combination of both. Recent surveys we have done show that at least 1 in 5 organizations have had staff that have felt this way. Oh, and do clients see value in having their information collected and analyzed? A solid incremental approach is better than starting with a monstrous performance measurement design that fails because of its own weight. Building over time and getting better over time is entirely acceptable – and encouraged.
- #33: " If you can't describe what you are doing as a process, you don't know what you are doing." - W. Edwards Deming "In God we trust, all others bring data." - W. Edwards Deming
- #35: " An acre of performance is worth a whole world of promise." - William Dean Howells