GPRAMA Implementation After Five Years

Editor's Notes

• #6: Chief operating officer (COO): overall responsibility for improving agency management and performance Performance improvement officer (PIO): assists the agency head and COO with performance management activities Goal leaders: designated for CAP goals, strategic objectives, and APGs Performance Improvement Council (PIC): assists OMB to improve the performance of the federal government and achieve the CAP goals and facilitate the exchange of useful performance improvement practices and work to resolve government-wide or crosscutting performance issues
• #8: GPRAMA includes several provisions related to reporting of performance information. Performance.gov single, government-wide performance website to communicate government-wide and agency performance information Program inventory publicly available list of all federal programs identified by agencies, along with related budget and performance information Performance information quality requirements for agencies to describe how they are ensuring the accuracy and reliability of the data used to measure progress toward APGs and performance goals Major management challenges challenges may include programs or management functions that have greater vulnerability to fraud, waste, abuse, and mismanagement addressed in performance plans
• #10: OMB and Agencies Generally Agreed with GAO’s Prior Recommendations to Improve GPRAMA Implementation, but Most Have Not Yet Been Implemented Since GPRAMA was enacted in January 2011, we have made a total of 69 recommendations to OMB and agencies aimed at improving its implementation. OMB and the agencies have generally agreed with the recommendations we have made thus far, and have implemented some of them. However, of the 69 recommendations we have made, 55 (about 80 percent) have not yet been implemented, while 14 recommendations (about 20 percent) have been implemented. OMB, which has been the focus of most of our recommendations, has implemented just over one-third (14) of the 38 recommendations we have made to it.
• #12: Food Saftey: HHS and USDA vary in the amount of detail they provide on their crosscutting food safety efforts and they do not include several relevant crosscutting efforts in their strategic and performance planning documents. HHS and USDA have mechanisms in place to facilitate interagency coordination on food safety that focus on specific issues, but they do not provide for broad-based, centralized collaboration. Centralized mechanisms needed. NEMT An interagency coordinating council was developed to enhance federal, state, and local coordination activities, and it has taken some actions to address program coordination. Limited leadership – has not convened since 2008.
• #18: Managing for Results: Practices for Effective Agency Strategic Reviews: GAO‑15‑602. Characteristics of strategic reviews: Leadership-driven Held annually Assessment of an agency's progress toward their strategic objectives Identification of strategies for performance improvement, such as strengthening collaboration to better address crosscutting challenges using evidence to identify and implement more effective program designs
• #19: GPRAMA requires data driven reviews for agencies APGs Characteristics of data driven reviews: Regularly scheduled, structured meetings Include key players Leaders and managers review and analyze data on progress toward agency performance goals Used to identify at-risk goals and strategies to improve performance Participants engaged in rigorous and sustained follow-up
• #20: Most agencies reported that relevant contributors from other federal agencies did not participate in their reviews As we previously reported in 2013, failing to include all goal contributors may lead to missed opportunities to have all the relevant parties apply their knowledge of the issues and participate in developing solutions to performance problems As a result, in that 2013 report, we recommended that OMB work with the PIC and other relevant groups to identify and share promising practices to help agencies extend their performance reviews to include, as relevant, representatives from outside organizations that contribute to achieving their agency performance goals. OMB staff said that while agencies have found that at times it is useful to engage external stakeholders in improving program delivery, officials view data-driven reviews as internal agency management meetings and believe it would not always be appropriate to regularly include external representatives
• #21: According to OMB staff, agencies used different approaches for valid and legitimate reasons and a one-size-fits-all approach would not work for all agency inventories. While this may be true, OMB could do more to direct agencies to find common ground on similar programs. One of OMB’s stated purposes for the inventories is to facilitate coordination across programs that contribute to similar outcomes. However, OMB put the plans for updating the inventories on indefinite hold and agencies have not published updated inventories with program-level budget information, in part due to the enactment of the DATA Act. OMB staff told us that they are considering how implementation of DATA Act requirements can be tied to the program inventories.
• #23: Comparing reported use of performance information by federal managers in 2007 and 2013, only 2 agencies experienced a statistically significant improvement, while 4 agencies experienced a statistically significant decline.
• #24: Reviews are designed to shift agency practices from the passive collection and reporting of performance information to a model where agency leaders actively use this data to assess performance, diagnose problems, and decide on next steps to improve performance. Data-driven reviews. GPRAMA and OMB require that in-person reviews of progress of agency priority goals be held at least quarterly, be led by the agency head and/or COO, and be used to assess progress toward each goal and develop strategies to improve performance where necessary.
• #25: Strategic reviews. OMB has directed agencies to conduct strategic reviews, which are leadership-driven, annual reviews of progress on an agency’s strategic objectives (the outcome or impact the agency intends to achieve through its various programs and initiatives).
• #26: Other Practices Have the Potential to Increase the Use of Performance Information Agencies can adopt leading practices to enhance the use of performance information for policy and program decisions aimed at improving results: Aligning agency-wide goals, objectives, and measures Improving the usefulness of performance information Developing agency capacity to use performance information Demonstrating management commitment Communicating performance information frequently and effectively.
• #27: Evidence-based tools that can increase use of performance data include: Program evaluations. Systematic studies of program performance can influence program management and policy decisions if leaders support using evaluations for program improvement, there is a strong body of evidence, and stakeholders are engaged throughout the process.
• #28: Evidence-based tools that can increase use of performance data include Pay for Success. Under PFS, government contracts for specific performance outcomes and requires that a program’s impact be independently evaluated. Contract provisions can also require service providers and stakeholders to regularly review performance data.
• #29: The third area in which we evaluated GPRAMA’s impact was in aligning individual and agency performance with results. GAO’s prior work on performance management has emphasized the importance of practices that create a clear linkage – or “line of sight” – between individual performance and organizational success (GAO-03-488). These practices include: Aligning individual performance expectations with organizational goals, and Making meaningful distinctions in performance. At the organizational level, aligning performance with results involves setting meaningful goals for performance and measuring progress against these goals. The GPRAMA requirements listed on the slide support this alignment, by emphasizing goal setting, accountability for goals, and review of goal progress.
• #30: Goal Leader – For our July 2014 report on the role of the agency priority goal leader, we interviewed 46 goal leaders. Most of them said that the goal leader designation had positive effects, including that it provided accountability for goal achievement. Data-driven reviews – As part of our July 2015 report on data-driven reviews, we surveyed performance improvement officers at the 23 agencies that have APGs. Of the 22 agencies that reported holding regular, in-person data-driven reviews, 21 reported that the reviews have had a positive effect on their agency’s ability to hold goal leaders and other officials accountable for progress toward goals and milestones. Strategic reviews – Similar to data-driven reviews, these reviews can promote accountability for results. In our July 2015 report, in which we identified and illustrated practices that facilitate effective strategic reviews, we reported that accountability for results is one of the key features of the reviews. Leaders should hold strategic objective leaders and other responsible managers accountable for managing progress on strategic objectives.
• #31: Our work has found problems with agencies’ efforts to link performance to results. One of the most well-known of these problems was illustrated in the VA scandal in which veterans were left waiting for or never received care, while schedulers were pressured to use practices to make wait time data appear more favorable than it actually was. The VA’s goal of providing timely care was reflected both in 1) performance contracts for senior leaders, and 2) the agency’s annual budget submissions and performance and accountability reports. On the surface, this is a good thing! But… the wait time data on which goal progress was measured was not reliable. VA staff had used gaming strategies to manipulate it to appear more favorable. This manipulation may have resulted in increased wait times and delays in providing care.
• #32: Performance appraisal systems, including individuals’ performance plans, Goal leader performance plans – we reviewed agency priority goal leaders’ and deputy goal leaders’ performance plans as part of our July 2014 report on the role of the goal leader. Reviewed plans for leaders/ deputies for the 47 APGs in our sample (just about half of all APGs for 2012/2013). 32 goal leader plans and 35 deputy goal leader plans. Very few plans referenced the APGs. Only one goal leader and one deputy goal leader plan linked performance standards to goal outcomes. Going back to key practices associated with line of sight – this is a failure to align individual performance expectations with organizational goals Senior Executive Ratings: Our January 2015 report on SES ratings looked at SES performance plans and ratings distributions among agencies. The 5 agencies we studied all linked performance plans with organizational goals, BUT… We found wide variation in ratings distributions. For example DOD rated ~30 % of its SES employees at the highest rating level; DOJ rated ~74% of its SES at this level. Going back to key practices associated for line of sight – this raises questions about whether agencies are making meaningful distinctions in performance. We made recommendations related to each of these findings (see source reports).
• #33: Agencies Have Long-standing Difficulties in Measuring Performance of Selected Program Types A critical element in an organization’s efforts to manage for results is its ability to set meaningful goals and to measure progress toward these goals. GPRAMA reinforces the need to set meaningful goals by directing agencies to publish a balanced set of performance measures across program areas. However, measuring the performance of different program types—such as grants, regulations, and tax expenditures—is a significant and long-standing government-wide challenge and one we have addressed in our previous work. The table on this slide illustrates some of the examples from our work over the past two years in which we identified agency problems measuring the performance of different program types. As you can see, the related issue areas range from measuring the effectiveness of grants targeting child abuse to food safety issues. More details are in the capping report.
• #34: A performance measurement problem we found at several federal programs was in measuring customer service. Our October 2014 report on customer service examined customer service standards of six federal programs (at five agencies). We compared these programs’ standards to key elements of customer service standard (identified based on our review of GPRAMA and executive orders). We found that three of the six were not effectively measuring customer service. Specifically, these programs did not: - Include targets or goals for performance - Include performance measures For example, we reported that the National Park Service did not have performance goals or measures directly linked to those goals. Because of this, the agency is unable to determine the extent to which the standards are being met agency wide, or strategies to close performance gaps.
• #36: Program inventories have the potential to improve the transparency of performance information, but we identified issues that affect their usefulness: For example, we found that some inventories did not identify a program’s contribution to the agency’s goals. The effectiveness of the inventories could also be improved by their being presented in a more dynamic, web-based format.