Using Outcome Information
0 likes125 views
Nonprofit organizations can use outcome data in several ways: 1. To detect needed improvements by identifying outcomes, client groups, services, or policies that need attention. 2. To motivate and help staff and volunteers by communicating results, holding program reviews, identifying training needs, and recognizing good outcomes. 3. For other internal uses like identifying successful practices, testing program changes, assisting with planning and budgeting, and motivating clients. 4. For external reporting to board members, funders, and the community to demonstrate accountability and the value of the organization's work.
Using Outcome Information
- 1. Using Outcome Information MAKING DATA PAY OFF SERIES ON OUTCOME MANAGEMENT FOR NONPROFIT ORGANIZATIONS The Urban Institute
- 3. Using Outcome Information MAKING DATA PAY OFF This guide is part of a series on outcome management for non- profit organizations. Other guide topics include keys steps in outcome following up with former management clients The Urban surveying clients developing community-wide Institute analyzing outcome information indicators
- 4. Copyright © 2004. The Urban Institute. Permission is granted for reproduction of this document, with attribution to the Urban Institute. The nonpartisan Urban Institute publishes studies, reports, and books on timely topics worthy of public consideration. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.
- 5. Contents Preface v Acknowledgments vii Introduction ix Completing the Data Analyses ix Seeking Explanations xii Formatting the Report xiv About This Guidebook xv SECTION I Detecting Needed Improvements 1 Use 1: Identify Outcomes That Need Attention 1 Use 2: Identify Client Groups That Need Attention 3 Use 3: Identify Service Procedures and Policies That Need Improvement 5 Use 4: Identify Possible Improvements in Service Delivery 7 SECTION II Motivating and Helping Staff and Volunteers 11 Use 5: Communicate Program Results 11 Use 6: Hold Regular Program Reviews 12 Use 7: Identify Training and Technical Assistance Needs 13 Use 8: Recognize Staff and Volunteers for Good Outcomes 15 SECTION III Other Internal Uses 17 Use 9: Identify Successful Practices 17 Use 10: Test Program Changes or New Programs 18 Use 11: Help Planning and Budgeting 19 Use 12: Motivate Clients 21 SECTION IV Reporting to Others 23 Use 13: Inform Board Members 23 Use 14: Inform Current and Potential Funders 24 Use 15: Report to the Community 25 SECTION V Cautions and Limitations 27
- 6. iv Using Outcome Information Exhibits 1. Uses for Outcome Information x 2. Seeking Explanations for Unusual Outcomes xiii 3. Sample Comparison over Time: Adoption Program 2 4. Sample Comparison of Actual Values to Targets: Community Arts Program 3 5. Sample Comparison of Multiple Breakout Characteristics 4 6. Sample Two-Characteristic Breakout 6 7. “How Are We Doing?” Meetings 14 8. Steps for Comparing Alternative Program Approaches 20
- 7. Preface “Program outcome measurement helps organizations increase the effectiveness of their programs and communicate the value of what they do.” The above quote is a common mantra in the outcome measurement field. However, after investing a great deal of effort in identifying and measuring outcomes, analyzing data, and reporting results to the funders that often prompted the effort in the first place, many nonprofit organizations do not take the next steps that will make the effort truly pay off. The emphasis on collecting the data needs to be complemented by a commitment to using it. This final guide in the Urban Institute’s series on outcome management for nonprofit organizations shows how to put the data to use. Systematic use of outcome data pays off. In an independent survey of nearly 400 health and human service organizations, program directors agreed or strongly agreed that implementing program outcome measurement had helped their programs focus staff on shared goals (88%); communicate results to stakeholders (88%); clarify program purpose (86%); identify effective practices (84%); compete for resources (83%); enhance record keeping (80%); and improve service delivery (76%). Based on their experiences, 89 percent of these directors would recommend program outcome measurement to a colleague. Such benefits do not just happen, of course. These organizations set up proce- dures and set aside time to review and discuss their outcome findings regularly, making sure they received a return on the investment that outcome measurement represents for any group that pursues it. In the process, these organizations moved from passively measuring their outcomes to actively managing them, using the data to learn, communicate, and improve. This guide offers practical advice to help other nonprofits take full advantage of outcome data, identifying a variety of ways to use the data and describing specific methods for pursuing each use. It is a valuable tool for ensuring that outcome meas- urement fulfills the promise of helping organizations increase their effectiveness and communicate their value. Margaret C. Plantz Director, Community Impact Knowledge Development United Way of America
- 9. Acknowledgments This report was written by Elaine Morley and Linda M. Lampkin. The report benefited greatly from the assistance, comments, and suggestions of Margaret C. Plantz of United Way of America and Harry P. Hatry of the Urban Institute. In addition, Patrick Corvington of Innovation Network provided initial guidance on the content of the report. The editors of the series are Harry P. Hatry and Linda M. Lampkin. We are grate- ful to the David and Lucile Packard Foundation for its support. We also thank the staff of the following nonprofit organizations, whose uses of outcome information are included in this report: Big Brothers Big Sisters of Central Maryland Boys and Girls Club of Annapolis and Anne Arundel County Community Shelter Board Crossway Community Jewish Social Service Agency KCMC Child Development Corporation Northern Virginia Family Services Northern Virginia Urban League United Community Ministries Volunteers of America Examples of outcome information use by most of these organizations are drawn from Making Use of Outcome Information for Improving Services: Recommenda- tions for Nonprofit Organizations (Washington, DC: The Urban Institute, 2002) and How and Why Nonprofits Use Outcome Information (Washington, DC: The Urban Institute, 2003). Both publications provide additional examples and detail.
- 11. Introduction The outcome data have been collected and the analyses completed—so, what is the next step? Now is the opportunity to benefit from the effort it took to get to this stage by moving from outcome measurement to outcome management. This next step occurs when an organization uses outcome information to improve services. This guide provides ideas on various ways to use outcome information; others in this series provide help in selecting outcome indicators, collecting data, and completing analyses regularly.1 Exhibit 1 summarizes the various uses for outcome information. Outcome data are most often used internally by nonprofit program managers. However, organizations have other important potential users, such as board members and direct service personnel. In addition, there are several potential external users, including clients, funders, volunteers, community members, and other nonprofit organizations providing similar services. Nonprofit managers find outcome data most valuable after comparisons and analy- ses are completed, and possible explanations for unexpected findings are explored. Once these steps are taken, a report that clearly communicates the findings should be prepared for internal use within the organization. Several audiences, including both management and program staff, can use the information for many purposes. While this guide focuses on internal uses of outcome data, it also reviews some important external uses—informing clients, volunteers, board members, services users, or donors and funders. After all the collecting and analyzing of outcome information, the rewards are at hand. Nonprofits can now use that information to help improve programs and services, and provide better information to external stakeholders. This guide is designed to help nonprofits cross into performance management by fully using their outcome information. Completing the Data Analyses The raw data must be combined and summarized for use and interpretation, typ- ically involving comparisons over time or with targets. Often, information from 1 See the rest of the Urban Institute series on outcome management for nonprofit organizations—Key Steps in Outcome Management, Developing Community-wide Outcome Indicators for Specific Services, Surveying Clients about Outcomes, Finding Out What Happens to Former Clients, and Analyzing Outcome Information—all available at http://www.urban.org.
- 12. x Using Outcome Information EXHIBIT 1 Uses for Outcome Information SECTION Detecting Needed Improvements I Use 1. Identify outcomes that need attention Use 2. Identify client groups that need attention Use 3. Identify service procedures and policies that need improvement Use 4. Identify possible improvements in service delivery SECTION II Motivating and Helping Staff and Volunteers Use 5. Communicate program results Use 6. Hold regular program reviews Use 7. Identify training and technical assistance needs Use 8. Recognize staff and volunteers for good outcomes SECTION III Other Internal Uses Use 9. Identify successful practices Use 10. Test program changes or new programs Use 11. Help planning and budgeting Use 12. Motivate clients SECTION Reporting to Others IV Use 13. Inform board members Use 14. Inform current and potential funders Use 15. Report to the community
- 13. Introduction xi basic, easily calculated analysis can help program managers and staff draw conclusions and guide improvement actions. However, drawing conclusions from the data requires judgment and often depends on experience or an intuitive response to the analysis. For example, is a decrease in teen mother attendance at prenatal visits from one quarter to the next the same across categories of clients and for different staff mem- bers? It may be important to identify results by client characteristics (ethnicity; age; income level; living situation—with parents, on her own, or with the father of her child; employment or educational status), as such characteristics may be associated with different outcomes. It is also important to assess the difficulty of helping different types of clients; the level of improvement is related to the challenges of a particular client group. Similarly, a review of outcomes for clients by individual staff member might help identify successful practices or a need for additional training or supervision. Summarized data will mask the fact that some types of clients have sub- stantially poorer outcomes than others. If the types of clients with relatively poor outcomes are identified, then steps to address problems can be devised. Again, staff reviews must account for the difficulty of serving the client, or there may be an incentive to improve performance by avoiding hard-to-serve clients. Typical analyses that can indicate how well individual programs are performing, provide information on areas needing improvement, and identify programs or staff meriting recognition include the following: Comparing recent outcomes with those from previous reporting periods; Comparing outcomes to targets, if the program has set such targets; Comparing client outcomes grouped by client characteristics, such as age, gender, race and ethnicity, education level, family income, public assistance status, household size, and so on; Comparing client outcomes grouped by various service characteristics, such as amount of service provided, location/office/facility at which the service was provided, program content, or the particular staff providing the service to the client; and Comparing outcomes with outcomes of similar programs in other organiza- tions, if available. If clients are surveyed, an “open-ended” question is often very useful. Client reasons for poor ratings or suggestions to improve services can be grouped and sum- marized to help program managers and staff members pinpoint service areas to improve. For example, one employment training program surveyed employers of its clients to identify specific areas in which clients needed training. Based on the employer responses, the program added a customer service component to its training. All these analyses convert the data collected into useful information.2 2 This guide reproduces several exhibits from Analyzing Outcome Information to illustrate how nonprofits can use data that have been analyzed in various ways. Using Outcome Information focuses on using the information, while the previous guide focused on analyzing it.
- 14. xii Using Outcome Information Seeking Explanations A major function of outcome measurement is raising questions. A review of the outcome data report should lead to discussions about the program. Whether out- comes appear very good or poor, nonprofits need to seek explanations before taking action, as these explanations can provide important guidance on what needs to be done. External factors sometimes affect program outcomes. Changes in local economic conditions can affect employment opportunities for graduates of a job training pro- gram, or changes in the mix of clients entering a program (usually not controlled by the nonprofit) can affect program results. Internal factors, such as personnel turnover, facility conditions, and changes in program funding, could also affect outcomes. Exhibit 2 suggests ways to search for explanations. Only the last suggestion, an external in-depth evaluation, is likely to incur significant added expense. Typically, program personnel are in the best position to understand why the out- come data are the way they are. For example, service workers in a teen mother pro- gram may know that their clients’ typical reasons for missing prenatal visits are forgetting they had an appointment and difficulty obtaining transportation. Participants at regular performance review meetings (such as “How Are We Doing?” sessions) can help identify why problems have occurred. These meetings can also be used to generate suggestions for improvements and specific actions to improve future outcomes. If more time is needed, an organization can form a working group of staff members. For example, if unsatisfactory outcomes on delivery times and meal quality for a senior meals program were concentrated in one area, a working group might explore factors such as traffic congestion, patterns of one-way streets, or housing characteristics (such as needing to use stairs or elevators to deliver meals). The group could also develop rec- ommendations that could lead to improvements, such as modifying a delivery route. Sometimes, clients are asked to rate specific characteristics of services, such as waiting times; helpfulness or knowledge of staff; adequacy of information on the program; and location and accessibility of the facility (including proximity to pub- lic transit). Their responses may hold clues to why outcomes are not as expected. Any open-ended comments, suggestions for improvements, or ratings on specific aspects of a program on a survey can provide further guidance. Holding focus groups with clients or other stakeholders is another way to obtain helpful information for identifying the causes of certain outcomes. Participants can also provide suggestions for improvements. Typically, a small number (about 6–12) of current or former clients are invited to a 90–120 minute session to discuss the issue at a time and place convenient for them.3 3 Another guide in this series, Key Steps in Outcome Management, covers focus groups in more detail.
- 15. Introduction xiii EXHIBIT 2 Seeking Explanations for Unusual Outcomes Talk with individual service delivery personnel and supervisors Hold “How Are We Doing?” meetings (group discussions with personnel) to review results Form a working group of staff, and perhaps volunteers, to examine the problem Review client responses to survey questions that asked clients to rate specific service characteristics Review responses to open-ended questions in client surveys and those that probe spe- cific aspects of services Hold focus group sessions with samples of clients to obtain their input Consider performing an in-depth evaluation, if findings are particularly important and resources are available
- 16. xiv Using Outcome Information In some cases, focus groups may reveal that the true nature of the problem dif- fers somewhat from what is indicated by the outcome data. A focus group with sen- ior citizens participating in a meals program might reveal that bland food was a problem, and that spicing up the food a bit could increase client satisfaction. An outside, in-depth evaluation to identify reasons for a problem outcome will take the most time and require special funding unless volunteers or free assistance from a local college are available. This approach can also provide the most infor- mation, but may only be needed for major problems or if funds are available to sup- port an independent evaluation. Formatting the Report Creating useful displays of the analyses of the outcome data is a key, but often poorly performed, next step. While special care needs to be taken with reports for external use, it is also essential to provide an easily understood version for internal use. The goal is a report that is clear, comprehensive, and concise, often a difficult balance. Users should be allowed to develop conclusions about performance without feeling overwhelmed with data. Here are some tips to help present the information effectively: Keep it simple. Include a summary of major points. Don’t crowd too much on a page. Avoid technical jargon and define any unfamiliar terms. Define each outcome indicator. Highlight points of interest on tables with bold type, circles, or arrows. If feasible, use color to help highlight and distinguish key findings. Label charts and tables clearly—titles, rows, columns, axes, and so on. Identify source and date of the data presented and note limitations. Provide context (perhaps a history or comparisons with other organizations or the community). Add variety to data presentation by using bar or pie charts to illustrate points. The first report should be for internal use and typically will be considerably more detailed than reports provided to external stakeholders. Nevertheless, much care needs to be taken to make this report fully clear and useful. After input from those in the organization, a version of the report for public use may be required, after con- sidering the needs of those outside audiences.
- 17. Introduction xv About This Guidebook This guide identifies basic uses of outcome information grouped into four sec- tions. A fifth section focuses on some limitations of outcome measurement. SECTION I Detecting Needed Improvements presents ways to use the outcome information with staff members (and volunteers, where applicable) to identify where improve- ment is needed and ways to improve services, and thus outcomes, for clients. SECTION II Motivating and Helping Staff and Volunteers suggests uses aimed at encouraging staff and volunteers to focus on achieving better service outcomes and helping them improve outcomes. SECTION III Other Internal Uses covers additional ideas on how to use outcome data to improve service delivery procedures, provide feedback to assist in planning and budgeting, strengthen the organization’s ability to sustain itself, and, over the long run, improve its service effectiveness. SECTION IV Reporting to Others provides ways to use reports on outcomes to inform other stakeholders, including donors, funders, volunteers, board members, service users and clients, and community members. SECTION V Cautions and Limitations identifies some issues to consider before changing pro- grams based on outcome information.
- 19. SECTION I Detecting Needed Improvements Like the score of a baseball game, outcome information by itself rarely identifies specific improvement actions, but it can indicate when improvement is needed. Different ways of presenting and analyzing outcome data can provide important clues about where improvements are needed and catalyze identifying improvement actions. Discussing and seeking explanations for the findings with the people responsible for providing services is key to identifying how to improve services and outcomes. Use 1: Identify Outcomes That Need Attention Typically, organizations should collect data on more than one outcome indicator so they have enough information on service outcomes (including the quality of the service delivery) to base proposals for change. Often the latest findings reported for each outcome indicator are compared with results from previous periods and targets set by the program. Exhibit 3 shows an outcome report covering five outcomes for an adoption pro- gram. Results for the latest quarter are compared with those for the same quarter in the previous year, and indicate considerable recent success with adoptions of younger children but a substantial decline in the adoption rate for older children. While it is not surprising that the adoption rate for older children is lower, the decline in the adoption rate for older children appears to require special attention by the pro- gram. Had the adoption rate of younger children also declined substantially, it would have been a basis for concern.
- 20. 2 Using Outcome Information EXHIBIT 3 Sample Comparison over Time: Adoption Program Difference (percentage Oct.–Dec. 2002 Oct.–Dec. 2003 points) Percent of children under age 10 adopted 75% 92% +17 within 180 days of becoming available for adoption Percent of children 10 or older adopted 66% 52% -14 within 180 days of becoming available for adoption Percent of adoptions in good standing 92% 99% +7 12 months after adoption Percent of families reporting satisfaction 95% 95% -0 with the adoption program services Total number of children adopted during 96 88 -8 the time period The program has successfully The length of time older children decreased the length of time remain available for adoption has younger children wait to get adopt- increased. Can this outcome ed and reduced adoptions with improve? Could the actions used to problems. It might be commended reduce the time to adopt younger for these successes. children also work for adoptions of older children? Exhibit 4 compares four outcomes of a community arts program for the last quar- ter with the targets set for that quarter. The only significant problem this quarter is that the target for new memberships has been badly missed, indicating a possible need to revise the program’s current strategy for securing new members. These exhibits indicate how outcome reports can help identify if any issues need special attention. The actual choice of what actions, if any, to take depends on what explanations are found for these unexpected outcomes. Suggestions for finding explanations, a major element of any outcome management system, are described in the introduction.
- 21. Detecting Needed Improvements 3 EXHIBIT 4 Sample Comparison of Actual Values to Targets: Community Arts Program Actual Target Difference between Oct.–Dec. 2003 Oct.–Dec. 2003 Actual and Target Notes New memberships 100 250 –150 during this period Members up for 96% 100% –4 200 members up renewal that renewed for renewal during membership this period Members satisfied with 88% 85% +3 Based on 113 sur- the quality of center vey responses performances Members satisfied with 90% 85% +5 Based on 113 sur- the variety of choices of vey responses types of performances Although this program kept its members satisfied, it did not achieve its target of recruiting new members. This may suggest a need to investigate if other types of performances might attract new members, or if the marketing program is adequate. It might also be worth examining whether the target was realistic. Use 2: Identify Client Groups That Need Attention Programs with different types of clients often find that outcomes vary widely by client characteristics. In human services, client characteristics often include age, gender, race or ethnicity, and income level. But different factors may affect outcome variations in other types of programs. In environmental protection programs, out- come data might need to be broken out by geographical area or pollutant (which part of the stream has problems with which pollutants). The success of all kinds of programs is likely to vary when data are broken out, and nonprofit personnel need information on these variations to make the best use of the data. When outcome information is reported for each breakout group, program results for various groups can be evaluated. Exhibit 5 shows outcomes for multiple client groups, listing the percentage of clients that reported improvement after completing group thera- py, broken out by three demographic characteristics—gender, age, and race or ethnicity.
- 22. 4 Using Outcome Information EXHIBIT 5 Sample Comparison of Multiple Breakout Characteristics Clients That Improved after Completing Group Therapy Considerable Some Little No Number of Improvement Improvement Improvement Improvement Characteristic Clients (%) (%) (%) (%) Gender Female 31 10 19 55 16 Male 43 30 40 21 7 Age Group 21–30 13 23 31 31 15 31–39 28 21 32 36 11 40–49 24 21 29 38 13 50–59 9 22 33 33 11 Race/Ethnicity African American 25 32 20 32 16 Asian 5 0 60 20 20 Hispanic 20 15 40 40 5 White/Caucasian 24 21 29 38 13 Sessions attended 1–2 13 15 8 54 23 3–4 21 24 33 33 10 5+ 40 23 38 30 10 Facility Facility A 49 24 27 35 14 Facility B 25 16 40 36 8 Caseworker Therapist A 19 26 26 42 5 Therapist B 18 11 39 33 17 Therapist C 18 6 17 56 22 Therapist D 19 42 42 11 5 All Clients 74 22 31 35 12
- 23. Detecting Needed Improvements 5 The major finding—outcomes are substantially worse for women than for men— is highlighted. There is little difference among age groups and no clear pattern among racial or ethnic groups. No conclusion should be drawn about the effective- ness of this program for Asians because the number covered (five) is so small. These data indicate that the program’s group therapy service to females needs to be reviewed. Outcome reports can help agency managers and staff identify which, if any, issues need special attention, as illustrated by the exhibit. Any decisions about actions to take depend on the explanations found for these unexpected outcomes. Suggestions for finding explanations are included in the introduction to this guide. Outcome data from Northern Virginia Urban League’s Alexandria Resource Mothers (ARMS) program (Alexandria, VA) for pregnant and parenting teens indicated that Hispanic clients were not attaining certain outcomes at the same rate as non-Hispanic clients. To improve outcomes and better serve these clients, the program hired a Hispanic caseworker and sought to develop more culturally sensitive approaches to encouraging Hispanic clients to obtain employment or participate in educational programs. Use 3: Identify Service Procedures and Policies That Need Improvement Outcome information can be used to help identify service procedures or policies that need improvement and differences among locations and staff members. Man- agers can use this information to improve procedures, facilities, or staff whose out- comes are not as good as others. Such information also can identify apparent successful—or “best”—practices (discussed in use 9). Typical breakouts of outcome data by service characteristics include the following: Particular office or facility; Particular service delivery practice (group versus individual sessions, type of presentation, and venue); Amount of service provided (number and length of sessions); and Individual staff member or team of workers. Exhibit 5 includes data on one outcome indicator broken out by three service characteristics: number of therapy sessions attended, facility, and individual thera- pist. The major findings—substantially worse outcomes for clients that only attend- ed one to two sessions and particularly poor outcomes for clients of one therapist (Therapist C)—are highlighted with circles. There was no clear difference between facilities. Data on service characteristics help identify the patterns; managers must
- 24. 6 Using Outcome Information then find out why some clients only attended one to two sessions, and why the results of Therapist C’s work are so different from those of Therapist D. Data on service characteristics combined with client characteristics may provide a more comprehensive perspective for interpreting outcome data. Therapist C’s poor outcomes may be affected by having had many female clients, or many clients who attended very few sessions. Similarly, additional investigation is needed to determine whether Therapist D should be recognized or rewarded for good performance, or whether his or her practices should be considered “best practices” for adoption by other therapists. Perhaps clients in Therapist D’s caseload had less severe problems than those of the other therapists, for example. Exhibit 6 shows an analysis that examines two characteristics simultaneously. The outcome data show the relation- ship between the outcome (successful employment) and the two characteristics (edu- cation level at entry into the employment program and the length of the program). This examination indicates that clients who had not completed high school (a client characteristic) achieved employment more often if they attended the long training program (a service characteristic). Based on this information, clients who have not completed high school should be encouraged to take the longer training program (a policy change). EXHIBIT 6 Sample Two-Characteristic Breakout For the Outcome “Clients Employed Three Months after Completing Service” Number Education Level at Entry of Clients Short Program Long Program Total Completed high school 100 62% employed 64% employed 63% (of 55 clients) (of 45 clients) (of 100 clients) Did not complete high 180 26% employed 73% employed 48% school (of 95 clients) (of 85 clients) (of 180 clients) Total 280 39% 70% 54% (of 150 clients) (of 130 clients) (of 280 clients) Is action needed? Encourage clients who have not completed high school to attend the long program. Use these figures to help convince clients of the long program’s success with helping clients secure employment.
- 25. Detecting Needed Improvements 7 Thus, outcome reports help nonprofit managers and staff identify policies and procedures that need improvement. Decision about actions must depend on the expla- nations found for unexpected or unusual outcomes, as described in the introduction. Use 4: Identify Possible Improvements in Service Delivery Managers and supervisors are often considered the primary users of outcome information. However, staff members and volunteers providing services directly to clients should find outcome data very useful in identifying where their clients need improvement and in helping improve service delivery. At the very least, the data can help support proposals to change particular program practices or policies. Service workers (which include staff and volunteers who directly provide serv- ices) can use outcome information in the following ways: 1. Adjust services for individual clients or groups of clients while clients are still in service. For example, mental health program clinicians may obtain periodic data on a client’s progress by administering a mental health assessment cover- ing such items as the client’s functioning and mental distress levels. The clini- cian can then make any needed adjustments to the client’s treatment. Staff of Crossway Community’s (Kensington, MD) parenting and life skills training program conduct a structured interview with clients before entry in the program to develop a “Family Evaluation Summary” (FES), which is repeated at 1-, 6-, and 12-month intervals and at pro- gram graduation. Interview questions address such topics as parenting skills, education, employment status, finances, and family relationships. Caseworkers use FES scores to devel- op individual client goals and guide decisions regarding individual client services and referrals. Scores for the incoming class of clients are also are used to tailor course content to the needs of that group of clients. Similarly, service workers can look at aggregate outcome data for all clients receiving the same service to identify any patterns that indicate a need to modify procedures and implement those modifications. Reports prepared for individual service workers should break out the outcomes for cases handled by that worker. 2. Adjust service delivery approach for future clients. Service providers can look at aggregate data on their former clients to identify potential problems and modify services. For example, if many former clients indicated that they had difficulty reaching a certain worker, a change in schedule might be considered to reduce this problem for future clients. Managers and service workers can look at aggregated outcome data on former clients of individual service workers as long as any confidentiality guarantees are preserved. Individual service workers can receive responses
- 26. 8 Using Outcome Information to open-ended questions if the responses have been adequately screened, and wording modified as needed, to prevent identifying who provided the partic- ular comments. 3. Work with other staff to identify service delivery approaches related to par- ticularly successful or unsuccessful outcomes. Service workers might hold service worker–only meetings to review outcome findings, or “How Are We Doing?” meetings with the program manager (described under use 6). These discussions focus on identifying problems and making recommendations to management for changes to program procedures or policies. This process can generate new ideas and identify patterns. Service workers in the home health care program of Jewish Social Service Agency (JSSA) (Rockville, MD) reviewed data on client falls at home and found that a high proportion of elderly clients had suffered falls. Workers concluded that these falls were commonly associ- ated with tripping over furniture or rugs, especially when getting out of bed at night. As a result, program intake procedures were modified to include a safety review as part of the ini- tial home visit. The service worker conducting the visit now looks for trip hazards and makes recommendations to the client (and family members, where applicable) to reduce possible problems. After implementing the safety review, the number of clients falling at home declined. 4. Make a case for additional training, particular equipment, or other changes. Outcome data can provide evidence of a need and subsequently document the improvement after a change is made. 5. Experiment with new procedures. Nonprofits should encourage individual service workers to identify new procedures and try them out systematically. To do this, the service worker might apply the new procedure to every other client and code each client’s record according to which procedure was ap- plied. Data on the outcomes for new and old procedures can provide strong evidence to the service worker about the relative effectiveness of new versus old procedures. This evidence can be presented to other service workers or program managers to promote wider adoption of procedures that result in better outcomes. This procedure is discussed in more detail under use 10. Managers can encourage and facilitate these constructive uses of outcome infor- mation and most effectively work with these front line workers by adopting the fol- lowing practices: Involve service workers in developing the outcome measurement process and selecting the outcome indicators—so the service workers feel comfortable with the outcome information collected. Such involvement can lead to identi- fication of more appropriate outcome indicators, as well as staff commitment to achieving those indicators.
- 27. Detecting Needed Improvements 9 Managers of Crossway Community involved staff of its training program for health care careers in identifying outcome indicators (graduation rates had been used as its indicator). Staff and management chose job placement rates as an additional outcome indicator, which led to the introduction of several program changes to increase the likelihood of clients’ obtaining employment. These changes included providing opportunities for students to meet health services employers early in the program to promote awareness of employment oppor- tunities, initiating a job fair midway through the program, and providing career counseling and assistance in writing resumes. Involve service workers in setting the target values for each year. View the outcome management process as a way to help service workers, not as a way to cast blame if outcomes are not as good as hoped. Share outcome reports with service workers promptly when they become available—and before the results are widely disseminated—so there is an opportunity for service workers to review the information and explain un- expectedly good or poor results. Include outcomes achieved when evaluating personnel, to the extent appro- priate. As outcomes are often affected by many factors outside the influence of staff members, this use should be undertaken cautiously. Use outcome information to demonstrate that service workers and the or- ganization itself are accomplishing their objectives. This can help attract a motivated workforce that wants to stay with an effective organization. If the outcome data are improperly used as a threat, however, it could actually discourage the retention and hiring of quality personnel.
- 29. SECTION II Motivating and Helping Staff and Volunteers Efforts by those on the front line of program delivery are key to achieving good outcomes. Outcome information can help them identify what works for their clients. An ongoing outcome measurement process enables service workers to learn about effective practices and provides evidence to support modifications that may help their clients. Four important uses of outcome information aimed at helping, mot- ivating, and strengthening staff efforts to achieve better service outcomes are discussed in this section. Use 5: Communicate Program Results Outcome reports from the organization’s outcome measurement process are like- ly to interest service workers and motivate them to look for service improvement opportunities in their approach. These reports also make it clear that the organization is serious about outcomes and not merely about the amount of activity and number of clients served. The outcome measurement process should provide feedback on both the condi- tion of clients after receiving service and the quality of service delivery. For exam- ple, clients might indicate problems with the hours of service availability or access to the service locations. If a large proportion of clients reported the problem, staff might consider changing the hours of operation and finding ways to provide trans- portation for clients.
- 30. 12 Using Outcome Information United Community Ministries (UCM) (Alexandria, VA) learned from client comments that the location of its parenting skills class was difficult to reach. The program began using its van to bring clients to class, and also helped arrange car pooling among clients. The information provided to service workers should include the findings from any analyses, including breakouts by client demographic groups, service facilities, and caseworkers (where applicable). Northern Virginia Family Services’ Healthy Families Program (Falls Church, VA) reports out- comes for each of its five local offices against each other as a motivator for the individual offices, as well as to enable the program to track progress against targets. Copies of the outcome reports might be given to all personnel or posted in an office area where they work. JSSA’s home health care program manager posted its outcome data on a bulletin board so all staff could see them. The manager found the staff were interested in the data and proud of the good outcomes, which motivated them to maintain their performance. The UCM employment services program director sometimes posts the number needed to reach a target level for a particular outcome on a whiteboard in the main staff office as a motivator. The director then encourages staff to brainstorm approaches to improve out- comes and reach targets. Use 6: Hold Regular Program Reviews Perhaps no procedure involves staff in seeking service improvements more than regular reviews of progress toward desired outcomes. Such meetings should be held soon after preparing the periodic outcome report. Program managers and service workers review where outcomes have been especial- ly good and where they have fallen short. If results have been very good, the group should discuss why this occurred, and if particular service procedures are considered responsible for the outcomes, whether these procedures should be expanded. These meetings are also an opportunity for recognizing the staff members and volunteers that contributed to successes. Where results have been disappointing, the group should discuss possible reasons and seek solutions. If the group cannot identify likely causes for the short- falls, it might suggest ways to find the causes, such as setting up a small working group. The group should also assess if previous actions recommended by the group actually improved outcomes.
- 31. Motivating and Helping Staff and Volunteers 13 Crossway Community uses a multi-stage approach to review quarterly outcome data with employees. Organization executives initially meet to review outcome data from all programs. They then meet separately with managers of each program (program staff may also be included in these meetings). These meetings focus on reviewing that program’s outcome data and changes that might be needed to improve its outcomes and meet targets. Next, an all- staff meeting is held to report progress (and deficiencies) across all programs. This meeting informs staff about how the organization is doing overall and how their work contributes to the whole. It also is used to generate suggestions for improvement from all employees, not just those associated with particular programs. Exhibit 7 provides suggestions for “How Are We Doing?” meetings. Use 7: Identify Training and Technical Assistance Needs Often the findings on outcome indicators show that service workers need a cer- tain type of training or technical assistance. KCMC (Kansas City, MO) identified several outcome domains (including language, literacy, science, mathematics, creative arts, and emotional development) for its early child develop- ment program. Based on the outcome data collected, KCMC concluded that teachers needed additional training in the areas of mathematics, science, and language. As a result, educa- tion consultants were placed at all sites to provide training to teachers. Feedback on the quality of care a program provides may indicate that training or technical assistance are needed. For example, if a substantial proportion of former clients reported that staff timeliness, helpfulness, or courteousness was not fully satisfactory, this would likely spark attention to these problems. The ARMS program coordinator reviews client satisfaction survey responses to identify con- cerns related to individual caseworkers as well as the overall program. If clients identify any shortcomings (such as a caseworker providing fewer than the expected number of visits), the coordinator raises that issue in her bimonthly meetings with each caseworker. This is to iden- tify any assistance the caseworker may need, as well as to remind him or her of program expectations.
- 32. 14 Using Outcome Information EXHIBIT 7 “How Are We Doing?” Meetings Who should participate? Include staff members and perhaps volunteers, if appropriate. In nonprofits with more than one program, a high level manager could hold such a meeting with some or all program man- agers. When should the meetings be held? Convene them shortly after the latest outcome data for the program become available. Preferably, meetings should coordinate with the data collection schedule, perhaps monthly or quarterly. How should outcome data be used? Provide the latest outcome reports in advance to attendees, with notes that highlight potential problems and successes to help start the discussion. Participants should dis- cuss why outcomes appear unexpectedly poor or unexpectedly good. What kinds of data are most useful? Compare recent outcome data to data from prior time periods, to targets, or among clients with different demographic characteristics. Reviewing data by key client characteristics, locations of offices or facilities, or service delivery practices enables the group to pinpoint what changes could bring improvement. What if there are poor outcomes? Identify reasons and brainstorm ways to improve the program. What if there are good outcomes? Identify “successful practices” or “good practices” that can be adopted elsewhere in the program or would also apply to other programs in the organization. Staff responsible for the successful outcomes can be recognized. What are the next steps? If additional time and study are needed, ask a few group members to develop a plan identi- fying what needs to be done, who is responsible for doing it in what time period, and with what expected results. At future meetings, review the plan to assess progress.
- 33. Motivating and Helping Staff and Volunteers 15 Use 8: Recognize Staff and Volunteers for Good Outcomes When outcome information identifies good performance in achieving outcomes, those who contributed (individuals or the group as a whole) should be recognized or rewarded. Recognition or rewards should be based on sustained high level of out- comes, not outcomes achieved in one reporting period. Organizations also should consider recognizing substantial improvement in outcomes, not just high levels of outcomes. Recognition itself can be viewed as a type of reward and also motivates service workers and other staff. JSSA uses its employee newsletter to recognize good outcomes, including one department’s considerable improvement in customer satisfaction (as indicated in client satisfaction surveys). While monetary rewards are popular in the business community and with many governmental jurisdictions, they are often difficult to implement successfully. It is often tricky to ensure rewards are fair and perceived as such by all involved. For non- profit organizations in particular, monetary rewards may not be worth the effort. However, if tangible rewards—even recognition such as an “employee of the month” designation—are provided, outcome information serves as an unbiased way to deter- mine who should receive the reward or recognition.
- 35. SECTION III Other Internal Uses Although it seems obvious that service workers and program managers can use outcome data to change services delivery, an organization can learn from outcome data in many other important ways. The data can help with design and imple- mentation of new and improved programs, and aid financial planning and client recruitment. Use 9: Identify Successful Practices Identifying service practices associated with successful outcomes is the first step toward expanding their use. This in turn can improve future outcomes. Two approaches help identify particularly successful practices. First, analysis of the outcome data sometimes highlights particular service procedures or identifies service workers associated with especially good outcome levels. In the latter case, the program manager might then identify what those work- ers were doing that would explain their high success rates. In cases where programs are delivered in conjunction with other organizations or facilities in the community, this information can be used to advocate or negotiate for replication of practices or conditions associated with good outcomes.
- 36. 18 Using Outcome Information Big Brothers Big Sisters of Central Maryland (Baltimore, MD) began collecting outcome infor- mation by individual schools where its mentoring program is located. It plans to identify factors that appear associated with better youth outcomes, such as the school providing a quiet room for meetings of mentors and students. The organization plans to use such infor- mation to demonstrate to schools that those conditions should be provided for the mentor- ing program. The organization may consider discontinuing their program in sites that are unwilling to do so, replacing them with those where conditions are more likely to lead to better student outcomes. Second, a meeting where service workers share the differences in their proce- dures might be useful. Those with better outcomes could be asked to make a pres- entation. Keeping these meetings informal should encourage sharing among participants. Information about successful practices obtained from either of the above ap- proaches could be distributed to service workers in a “tip sheet,” an article, or an organization newsletter. A “job shadowing” arrangement could also be established in which the practices of more successful workers are observed occasionally by others. A word of caution: Before deciding that particular practices have been more suc- cessful than others, check for external circumstances or data errors that might have influenced the findings. For example, a new employer in the community might have created a large number of job opportunities for clients completing an employment program. Or, sometimes the clients assigned to a particular worker or facility might by chance have been clients that were particularly “easy” to help, leading to a high success rate that is not a result of the worker or service approach. Use 10: Test Program Changes or New Programs A program that regularly collects outcome information can test the effectiveness of program changes or new service practices before full implementation. This test- ing can encourage creativity and innovation. For example, nonprofit organizations can test the effects of different operating procedures, staffing arrangements, instruc- tional content, levels of service provided to individual clients, and presentation of information to clients. Two types of testing procedures can be used: Replace the old practice with the new one for a period of time. The outcome data can be used to evaluate the effect of the change—whether outcomes improved and, thus, whether the change should be made permanent. To use this procedure, data on outcomes are needed for periods before and after the change.
- 37. Other Internal Uses 19 Implement the new service practice for some clients but not others. This approach is less common, but can provide considerably stronger evidence. To use this approach, clients in both groups must be similar in important charac- teristics that might affect their outcomes. This similarity maximizes the like- lihood that differences in outcomes occur because of the change and not because of client characteristics. If clients are randomly assigned to the new procedure, it is more likely that the two client groups will be similar. To ran- domly assign clients, use a systematic method, such as flipping a coin or assigning every other new client to the new procedure. Exhibit 8 summarizes the steps for this procedure. Note, however, that random assignment may not always be appropriate or feasible. Whatever testing approach is used, it may take some time for the new procedure to show results. Differences in some outcomes, such as knowing about good study habits, might appear soon, while others, such as improving school grades with a new tutoring approach, might take several months, or even a year or more, to be demonstrated. If the program has an ongoing outcome measurement process, testing service delivery options need not be costly, as it requires only a little extra effort. Use 11: Help Planning and Budgeting Feedback is essential when planning and budgeting for programs, and outcome data provide an excellent source. Looking at the outcome information during the planning and budgeting process allows the organization to better allocate funds and set priorities. Nonprofit resources are almost always limited. Knowing the outcomes of various programs will help when decisions are made to expand, contract, or eliminate some. Particularly suc- cessful programs might be expanded, or less successful ones might be given more resources to take corrective actions. Without outcome data, it is diffi- cult to make these decisions. justify requests for and allocation of funds. Outcome data can help make a case for shifting priorities from one program to another and reallocating funds. If a program has particularly good outcomes, then arguably it should have the same or more resources next year. If a program is ineffective, per- haps more resources are necessary to make needed changes, or, perhaps it should be dropped and its resources allocated to programs that have demon- strated their successes. An organization with historical outcome data can use the data to project what is achievable with the funds requested.
- 38. 20 Using Outcome Information EXHIBIT 8 Steps for Comparing Alternative Program Approaches 1. Identify the service practices or approaches to be compared. Typically, one approach is an existing practice and the other a new approach. 2. Choose a method for deciding which incoming clients will be served using which approach. The method should select a representative sample of the participants for each practice with approximately the same proportion of difficult participants in each comparison group. Some form of random assignment is highly preferable. Randomization helps ensure that the comparisons will be valid and greatly increases the strength of the evidence. Simple methods of random assignment include flipping a coin and using a table of ran- dom numbers. If the arrival of participants is random, another method is to assign incoming participants alternatively to each practice. 3. As each participant enters the program, assign the participant to one of the two groups by the procedure you have just identified. 4. Record which participants are assigned to which procedures. 5. Track the outcomes for each participant in each practice over whatever period of time believed necessary to indicate the outcomes of these practices fairly. 6. Tabulate the values on each outcome indicator for each practice. 7. Compare the findings and make adjustments to program practices as appropriate. You may want to drop the practice that shows the poorer outcome. Alternatively, you might decide that you have not received a clear enough picture from the outcome data to make a decision, in which case you should continue the comparison for a longer time. Source: Reprinted from Measuring Program Outcomes: A Practical Approach (1996). Used by permission, United Way of America.
- 39. Other Internal Uses 21 Big Brothers Big Sisters of Central Maryland reports outcome information for youth in its mentoring program by type of program. It recently used data that showed better outcomes for school-based mentoring programs than community-based mentoring to convince its board to allocate more resources to increase the number of school-based programs. develop plans and set targets. When targets for programs have been set and data collected throughout the year, more realistic planning for the next year can be completed. If, for example, outcomes have been poor for a particular class of clients, the organization might want to develop a plan to correct this over a specified period, such as the next year or two. improve budget projections. Organizations know the outcomes achieved with last year’s budget allocation and can cost out the changes needed to improve outcomes from analysis of outcome data. Then, any costs of the needed changes can also be included in the budget projections. provide a basis for long-term strategic planning. If the nonprofit organization undertakes long-range strategic planning, outcome data can provide a start- ing point. Such information identifies the strengths and weaknesses of the organization, such as which services and clients have had particularly good or poor outcomes. The plan should contain future-year targets for key out- come indicators against which annual progress can be measured. Use 12: Motivate Clients Outcome information is not commonly used to motivate clients, but some non- profit organizations have found that sharing information about clients’ outcomes can motivate them to work toward better outcomes. Staff of Crossway Community gave individual clients their scores on the Family Evaluation Summary. Caseworkers believe that providing information on the changes in clients’ lives helps motivate them and encourages feelings of accountability for their own progress. Another caution: Care must be taken not to share information about outcomes of other individual clients that would violate their confidentiality. However, information on clients as a group may be shared, as may information that is not sensitive. The sec- ond example above illustrates a case where clients implicitly consented to information sharing by participating in a program that provides rewards for achievements.
- 40. 22 Using Outcome Information Boys and Girls Club of Annapolis and Anne Arundel County (MD) posts the record of members who satisfactorily complete homework on the wall of a main club room. Points are awarded to youth for attendance and completing their homework. Inexpensive prizes (such as book bags, basic scooters, and batteries) have been given to youth for achieving specified numbers of points. A major incentive for youth is that to participate in a sports league, they need to maintain a “C” average and have logged three homework-hours a week at the club, with home- work completion checked by staff or volunteers.
- 41. SECTION IV Reporting to Others Although outcome data help organizations improve their services, these data also likely provide information of considerable interest to others, including board mem- bers, funders (including foundations, individual donors, corporations, and govern- ments), volunteers, other community members, and former and future clients. The organization may want to create a different report for the external uses outlined in this section, one that is more of a summary and includes explanations of the results uncovered during internal analysis as well as steps taken to address problems or reward success. Use 13: Inform Board Members Although boards in some nonprofits may drive the initiation of outcome meas- urement systems, in other organizations board members are relatively passive re- cipients of outcome information. The following suggestions may help increase the interest and involvement of board members: Start small. Don’t overwhelm board members with all the data for all pro- grams. One approach is to focus on outcomes for one program at a meeting, with a discussion led by the manager of the program. Make reports user-friendly. Add explanations of terms, if necessary. Provide reasons for the results, particularly for any disappointing or good outcomes. Make sure reports are clear and concise.
- 42. 24 Using Outcome Information Focus on how the information has been used and address problems high- lighted by the report. Tell board members not only what the information indicates, but also what is being done to address problems identified. Seek improvement suggestions from the board. Ask how programs with lower than expected outcomes can be improved and how to recognize and reward programs with excellent or greatly improved outcomes. Try reserving time at each meeting for such discussions. Have a few members attend a “How Are We Doing?” meeting. Only one or two members should attend, so the meeting will not be affected by a large board presence. Establish a board committee to review outcomes, if appropriate. Members could be briefed on the process and use of the data. Some could become actively involved in working groups to identify problems and seek improvements. Outcome information can also be provided to the board to support recommenda- tions for resources. Northern Virginia Urban League staff used outcome information to support their request to board members to hire professional staff that could provide more regular and effective client services, rather than relying on volunteers that often met with clients sporadically. Use 14: Inform Current and Potential Funders Responding to funder requirements is, of course, a major use of outcome infor- mation. Nonprofit organizations that receive funds from foundations, government agencies, or the local United Way may be required to submit reports on outcomes. Even if not required, outcome information can be used to answer the question that most funders want addressed: is the program achieving its goals and making a dif- ference? Reports to funders, whether in new applications for funds or to satisfy existing reporting requirements, should include the following outcome-related elements: A brief description of the outcome measurement process; Outcome highlights, including successes and disappointments; Explanations of both successes and disappointments; and Actions the organization is taking or planning to take to address problems.
- 43. Reporting to Others 25 Sometimes, it may be better to report more outcome information than is required. For example, reporting variations among outcomes attained by different client groups may help funders better understand the achievements reported. Such information, though often not required, can considerably enrich understanding of the program results. Another key use of outcome information is to support requests for new funding. Grant and contract proposals often request information on the number of clients served and the type and amount of services provided. However, only a few nonprof- its substantiate the results they achieve or the outcomes they expect with the funds requested. An organization should consider highlighting key outcome information in applications, progress reports, mailings, brochures, advertisements, or newsletters as a part of effective fundraising. This approach can be useful in seeking support from organizations and individuals. Data about successful outcomes indicate to current and potential funders that their funds are making, and will continue to make, a difference. Even data on out- come deficiencies may be used to seek funding needed to correct weaknesses. Data on outcomes can be an invaluable part of proposals by providing new fun- ders with indications of a potential grantee’s past performance, before the funders commit any funds. Use 15: Report to the Community Nonprofits should consider providing reports on outcomes to the community, past and potential volunteers, and past and future service users and clients. Outcome information can increase visibility, support, and good will for the organization throughout the community. A demonstration of accomplishments in specific terms will help set an organization apart from others providing similar services. recruit new clients and customers for programs and services. If previous clients have had demonstrated success, then the service should appeal to potential clients with similar needs. attract volunteers. Outcome information will indicate to volunteers that the time they donate is likely spent in effective programs and is worthwhile. An organization that emphasizes results is likely to attract volunteers interested in making a difference. Big Brothers Big Sisters of Central Maryland plans to use its outcome data to help recruit volunteers for its mentoring programs. Agency officials believe that reporting outcome data will demonstrate their emphasis on results and show that mentors make a difference in the lives of children, both of which will help attract volunteers.
- 44. 26 Using Outcome Information Some organizations use one report to convey outcomes to multiple audiences, including board members, funders and potential funders, and the community in general. The nonprofit Community Shelter Board (CSB) in Columbus, Ohio, distributes funds to part- ner agencies that provide services to prevent homelessness, including prevention services, emergency shelter, and transitional housing. The partner agencies provide annual data on the achievement of program outcomes to CSB, which compiles a report. All agencies report on the proportion of clients that achieved successful outcomes and the recidivism rate for that period, but the definitions may differ for different types of services. For example, success for a homeless prevention program may be the percentage of clients that moved to permanent or transitional housing, while an outreach program may track the number placed in an emer- gency shelter. The report provides data on semi-annual trends in past outcomes; the goals and actual results for the current period, listing whether goals were met; and recommenda- tions for future measures and goals. After each agency verifies the data, the complete report is given to the agencies, CSB board member, and funders, and posted on the CSB web site. CSB also uses the data to allocate funds and establish targets. Volunteers of America (VOA) publishes an annual services report, presenting outcome data on various programs across the United States, that is released at the organization’s na- tional conference and posted on the Volunteers of America web site. The report describes the organization’s programs in relation to key client outcomes, and provides detailed information on the results of VOA programs. For example, promoting self-sufficiency among those experi- encing homelessness and other personal crises is a major focus of the organization, so the report includes detailed information gathered from VOA’s transitional housing programs. The report reveals that 83 percent of the clients leaving these programs secured permanent housing and 87 percent attained financial stability.
- 45. SECTION V Cautions and Limitations This guide has focused on using outcome information to help manage programs. But nonprofits should note a number of cautions before taking major actions based on outcome information. Don’t jump to conclusions based solely on the data. Data do not tell whether the program caused the outcome. Like the score in a baseball game, the out- come data only identify the results. This information is vital but insufficient for assigning praise and blame or deciding what actions to take. If outcomes are poorer than expected, this does not necessarily mean that the program should be cut or discontinued, or the personnel blamed. The pro- gram may need additional resources (or the data may be incorrect). If it is decided that program changes are required, then changes should be imple- mented and given an opportunity to improve outcomes. However, if poor out- comes continue over a sustained period, then changes in programs or personnel may be required. It is always important to check the accuracy of the data to ensure that exceptional outcomes are not the result of data entry or math errors. Also, always look for explanations that may not be related to the program. When there are differences in outcomes among offices, facilities, or serv- ice workers, check whether the clients served by the different offices, facilities, or service workers have similar client characteristics.
- 46. 28 Using Outcome Information Check to see that the right outcome is measured. What looked like the right indicator to measure when the outcomes were originally set up may not pro- duce the best information for making program improvements. The example from Crossway Community (see use 4) illustrates this situation. When its employment training program began tracking job placement rates in addition to graduation rates, staff began to focus on increasing the likelihood of stu- dent employment and took such actions as promoting awareness of employ- ment opportunities, providing assistance in writing resumes and career counseling, and holding job fairs. Assess the impact of sample size and composition. If the outcome data are based on samples of clients, are the findings applicable to all clients? When the percentage of clients for whom outcome data were obtained is very low, considerable caution should be exercised. Consider what is not covered by the measurements. Not all outcomes can be directly measured. Often “proxy” or “surrogate” indicators are used. For example, it is almost impossible to measure the number of cases prevented (whether the cases are of child abuse, crimes, mental health problems, and so on). Instead, the number of cases that were not prevented is counted. This situation is not ideal but is likely the best that can be done. Indicators that measure the percent of improvement in client knowledge or attitudes often focus on initial or intermediate indicators that may or may not lead to the desired end outcome. As a result, they may be weak proxies for end outcomes. For example, increased knowledge about the advantages of not using alcohol or drugs may or may not change actual alcohol or drug use. For other programs, the link is strong. For example, the fact that immunizing infants in a timely matter leads to less illness from preventable diseases is well supported. Thus, organizations must use common sense and research results to assess the strengths of the links between initial or intermediate outcomes and end outcomes.
- 48. The Urban Institute 2100 M Street, NW Washington, DC 20037 Phone: 202.833.7200 Fax: 202.429.0687 E-mail: paffairs@ui.urban.org http://www.urban.org