Performance_Measurement_Modules..of total quality management

Editor's Notes

• #2: Welcome to Module 1, Overview of Performance Measurement. This module is intended for grantees and sponsors of all Corporation for National and Community Service (CNCS) national service programs. This introduction provides valuable information to Programs serving in all Focus Areas, using Agency-Wide Priority Measures, Complementary Program Measures, and other performance measures.
• #4: By the end of the module, you will be able to describe what performance measurement is in the context of CNCS programming, and how it can be useful in program management. You will also be able to describe how CNCS approaches performance measurement; and how performance measures will be used to tell the National Service story.
• #5: This module will cover: What is performance measurement? What is its purpose, and how can it be used in program management? What does CNCS hope to achieve? How does CNCS approach performance measurement? And, How will it be used going forward?
• #6: This module will not include information on specific requirements for performance measurement for each CNCS Program. That information is available through Program specific Notices of Funds Opportunity/Availability (NOFO/A’s) at www.nationalservice.gov, and on the Resource Center CNCS Performance Measurement pages at www.nationalservice.gov/resources/npm/home.
• #7: Before we talk about performance measurement, a quick word about some of the terms used in this module: The terms “program” and “project” are used interchangeably throughout. The term “intervention” refers to the service activity conducted by CNCS-supported organizations or National Service Participants.
• #8: So what is performance measurement? Performance measurement is the ongoing, systematic process of tracking your program or project’s outputs and outcomes. Outputs are the amount of service provided. They measure the completion of activities and document the fact that individuals received services, products were created, or programs were developed. They answer the question, “How much service did we perform?” or “What products did we develop?” They do not answer the question, “What changed as a result of the service provided or product developed?”
• #9: Outcomes reflect the changes or benefits that occur. Outcomes can reflect changes in individuals, organizations, communities, or the environment. This may include changes in attitudes, knowledge, behavior, or condition. For example, changes within an organization may include an enhanced system to better serve community members. Outcomes answer the question, “What difference did our service make for beneficiaries?” or “How did the new system or product enhance the capacity of the organization to serve the community?” In general, the most compelling outcomes address changes in behavior or conditions. However, what makes a change compelling is not just the type of change but the magnitude or amount of change. And, of course, the change should reflect the activity’s intended purpose. In summary, performance measurement can show whether a change occurred.  To find out whether the change occurred because of the intervention, you would use other approaches, such as evaluation methodologies.
• #10: Let’s talk a little about the types of outcomes we might see, including changes in attitude or belief, knowledge or skill, behavior, or condition. Outcomes regarding attitude and belief involve a change in thought or feeling. For example, a caregiver feels she has more social support after receiving respite services. Outcomes measuring knowledge and skill involve a change in understanding or ability - what a person learns. For example, a student improves reading ability after attending tutoring. Outcomes addressing behavior involve changes in actions, such as conduct or habits, often in a specific context. For example, a student improves class participation, according to teachers, after participating in a school mentoring program. Outcomes regarding condition involve a change in situation or circumstance. For example, a family is transitioned into safe, affordable housing after receiving referrals and counseling; or a community receives a new after-school program because a youth organization has been able to build its capacity to provide services.
• #11: Let’s explore examples of outcomes in certain focus areas. In the area of education, an example of a change in attitude or belief is an increased interest in school; a change in knowledge or skills is improved math ability; a change in behavior is increased school attendance; and a change in condition is the successful completion of high school.
• #12: In the area of health, an example of a change in attitude is an increased desire to adapt good nutrition habits; a change in knowledge or skills is the ability to prepare healthy meals on a small budget; a change in behavior is increased healthy food intake; and a change in condition is improved household food security, in other words, a stable and adequate supply of food.
• #13: In the area of capacity building… When your aim is to build capacity, the most relevant type of outcome is likely to be a change in condition. For example, National Service participants implement a new training curriculum for community volunteers interested in helping out in the event of disaster. This leads to a cadre of trained volunteers that can be called upon on short notice – a change in condition. Another example: National Service participants organize regular fundraising events for an organization’s afterschool program. These events increase community awareness about the program, and result in measurable outcomes such as an increase in dollars, donations, repeat donors, and numbers of volunteers. An increase in resources allows the organization to expand its afterschool services to two new locations; this represents a change in condition. Or National Service participants assist organizations to develop partnerships with other service providers working with the same population. The goal is to improve efficiency of services as measured by reduced waiting time and cost for clients, an outcome that represents a change in condition. You will need to choose meaningful, but also realistic and measureable outcomes for your program. In a few moments, we will look at some of the national performance measures that CNCS Programs will be using.
• #14: So why do we measure performance? One reason is accountability. Performance measurement satisfies the need of funders and stakeholders (including CNCS participants, board members, community members, staff and clients) to see that the program or project is getting results. Performance measurement helps you communicate achievements in a way that funders and stakeholders will find meaningful and compelling. Another reason to measure performance is to determine if the change you thought would happen with your intervention is actually occurring. Performance measurement is a systematic way to collect reliable information about the intervention’s implementation and progress toward outcomes.
• #15: You can also use performance measurement data to spot and correct problems. Are you reaching the population you intended? Do some people require more or less intensive service to show a positive outcome? Even when an intervention is implemented exactly as planned, performance measurement can help you find ways to strengthen the intervention to make it more effective. Performance measurement is a way to get feedback so you know if your intervention is making the expected difference, and helps you decide how to make the most effective use of your limited resources.
• #16: Let’s talk about when you might use performance measurement. Performance measurement is more than a check-off item for an annual report, it is an integral part of program development, implementation, and assessment. Performance measurement can help you with program planning and management throughout the year. For example, at the beginning or before beginning service, performance measurement can be used to help clarify what, specifically, you hope to achieve at different time periods. If you are working with other sites, performance measurement can help you decide what you will need from partners as you begin developing site applications and your Memoranda of Understanding. 
• #17: If you recruit and select sites, performance measurement can help you not only define your expectations for sites, but also gives you another way to assess which sites will best be able to implement the selected intervention and use resources wisely.
• #18: Third, as you implement the program, performance measurement can serve as a focal point for conversations with staff, National Service Participants, partner sites and agencies. To ensure mutual understanding, you can reiterate why you are undertaking the effort (the need or problem you are addressing), what you are trying to achieve (the outcomes), and how you intend to achieve them (the intervention). Use performance measurement as you communicate with sites or other partners to make sure they are collecting the right data and staying on schedule. Performance measures identify what data to collect, how data are to be collected (methods and instruments), and benchmarks for success (targets). Regular feedback from sites also provides an ongoing “reality check” for your performance measurement tools and data collection processes.
• #19: As you engage in site monitoring throughout the year, you will be able to spot problems and address training and technical assistance needs so sites can be successful in collecting the right information. You can use performance measurement data for continuous program improvement. You can review the data to see where you are having the most success – how different beneficiaries respond to your intervention, for example. You can compare data before and after modifications to your intervention to see what works best. These adjustments can help your program be more efficient and can contribute to greater impact; that is, higher success rates for beneficiaries, and outcomes that are more likely to be sustainable.
• #20: Performance measurement results feed into reports you do for CNCS and for other stakeholders. Results can be used to promote your program or project, maintain or strengthen buy-in from local stakeholders, and to make the case for continued support or new resources. Be sure to celebrate your successes with your staff, National Service Participants, and local sites and partner agencies. Performance measurement results can be an important part of this celebration.
• #21: We talked about performance measurement in general; let’s talk about how CNCS approaches performance measurement and hopes to achieve. CNCS grantees and sponsors will contribute to the National Service story by selecting outputs and outcomes from the national performance measures. These will include both Agency-Wide Priority Measures and Complementary Program Measures. Agency-Wide Priority Measures will aggregate data from CNCS Programs; Complementary Program Measures have been customized to focus on interventions of particular import for specific CNCS Programs.
• #22: CNCS has established a set of national performance measures that reflect its Strategic Plan and programming priorities. These include: Disaster Services, Economic Opportunity, Education, Environmental Stewardship, Healthy Futures, Veterans and Military Families, and Capacity Building.
• #23: National performance measures means that everyone will use common terms, definitions, and approaches to measurement. This will make it possible to combine data from programs and projects with confidence. Although we will all be “speaking the same language” and contributing data to the National Service story, each of the CNCS Programs will have specific performance measurement requirements and guidance. That information is available through Program specific Notices of Funds Opportunity/Availability (NOFO/A’s) at www.nationalservice.gov, and on the Resource Center CNCS Performance Measurement pages at www.nationalservice.gov/resources/npm/home.
• #24: What are the benefits of using national performance measures? CNCS can focus efforts on national priorities - areas of need where National Service has had success and experience. National performance measures provide the data for a strong agency-wide narrative to tell the National Service story. CNCS will collect high quality data and aggregate the results of similar interventions to demonstrate the powerful impact of National Service. To ensure that CNCS can aggregate the data and report results with confidence, guidance for the national performance measures includes definitions for the intervention, target population, output and outcome; and information on collecting data and calculating the measure. Let’s look at a few examples of national performance measures.
• #25: In this example, National Service Participants provide support to homebound adults, older adults, and individuals with disabilities to help them continue living independently. Support services include regular friendly visits to socialize, and help with light housekeeping, food shopping, and running errands. In addition, participants escort these individuals to social events to keep them engaged and connected to the community. The output is that individuals receive services to help them live independently. The program tracks and reports on the number of people who receive those services. The outcome is that the individuals served will report increased social ties and/or perceived social support. The program is looking at a change in condition (increased social ties), and a change in attitude (perceived social support), that helps these individuals remain living independently.
• #26: Here is another example. National Service Participants provide training on nutrition designed for families on a low budget. This includes healthy meal preparation, stretching the grocery budget, and using local resources such as community gardens. They also provide counseling and referrals to other nutrition assistance programs. The output is that individuals receive services – such as training on nutrition and referrals - to alleviate long-term hunger. The program tracks and reports on the number of individuals served. The outcome is that these individuals report increased food security for themselves and their children. This represents a positive change in these families’ circumstances, a change in condition.
• #27: Here is one more example. National Service Participants pilot new volunteer management practices to improve support to volunteer mentors at a youth organization. These practices include training mentors with a new curriculum, meeting with them one-on-one to address issues that come up, and organizing periodic social events for mentors and youth to enhance camaraderie. The output is that National Service Participants manage the community volunteers. The program tracks and reports on the number of volunteers managed. The outcome is that the youth organization will incorporate at least three effective practices into their volunteer support system. This represents a positive change in the condition of volunteer support at the organization. These are just a few of the national performance measures.
• #28: Here is a summary of the key points covered in this module. Outputs are the amount of service provided. This could be numbers of people served, products or programs created. Outcomes are changes that occur for individuals, communities, organizations, or the environment. This could be changes in attitude or beliefs, knowledge or skills, behavior, or condition.
• #29: In addition, Performance measurement is integral to program management over the year; for example, for strategy and planning, site recruitment and selection, program or project implementation, and reporting to stakeholders.
• #30: National performance measures reflect the CNCS 2011-2015 Strategic Plan priorities. These include: Disaster Services, Economic Opportunity, Education, Environmental Stewardship, Healthy Futures, Veterans and Military Families, and Capacity Building. National performance measures will strengthen the National Service Story. We will be combining results from across the country. To ensure we are combining “apples and apples,” terms and definitions will be uniform, and grantees and sponsors will be required to collect data using the same approach.
• #31: This concludes the Overview on Performance Measurement. For more information on CNCS national performance measures, go to the Resource Center CNCS Performance Measurement pages at: www.nationalservice.gov/resources/npm/home. Also see www.nationalservice.gov for more information on Program-specific Notices of Funding Opportunities and application instructions.