Annual Results and Impact Evaluation Workshop for RBF - Day Two - Assessment Tool for Cross-Country Learning on Results-Based Financing
Download as DOCX, PDF0 likes193 views
The document presents an assessment tool for reviewing results-based financing (RBF) programs in health sectors of developing countries, focusing on key implementation challenges and success factors. It aims to facilitate cross-country learning by structuring assessments that identify strengths, weaknesses, and possible solutions while engaging stakeholders at various levels. The tool emphasizes the importance of evaluating real-world implementation rather than just theoretical frameworks to improve program outcomes.
Annual Results and Impact Evaluation Workshop for RBF - Day Two - Assessment Tool for Cross-Country Learning on Results-Based Financing
- 1. 1 Assessment Tool for Cross- Country Learning on Results Based Financing (RBF) A. Introduction RBF is a promising strategy that is increasingly being used by national health programs in developing countries to accelerate progress towards the achievement of the health Millennium Development Goals (MDGs). In order to ensure real time learning while waiting for the formal impact evaluation results to become available, it is important to regularly review ongoing RBF experiences and identify key implementation challenges and factors for success. While the evidence for the importance of particular design elements is usually sub-experimental, it may still be compelling. For example, experience so far suggests a few key features are critical: Paying for results regularly, predictably, and using a simple to understand formula; Verifying and counter verifying results regularly by independent agencies, e.g. monitoring of actual receipt of services and quality of care; Provision of substantial autonomy to health facilities in the use of resources they earn including being able to purchase drugs independently from approved private sector suppliers; Regular supervision and the use of robust tools for quality assessments, i.e., quantitative supervisory checklists Monthly or quarterly performance review meetings at health facility, local government, and higher levels B. Scope of the Assessment Tool Countries may find it useful to visit another country and assess their RBF programs and/or impact evaluations in order to learn from each other’s experience and adapt their program accordingly to ensure better results. The assessment tool is meant to guide and structure such an RBF program assessment. It will thereby allow for a comprehensive assessment of the program, and help identify successes and current challenges of the RBF programs as well as possible solutions. At this point, the assessment tool does not aim to cover the review of the impact evaluation of RBF programs. The assessment tool aims to ensure (i) all areas relevant to the success or failure of a RBF program are covered in the assessment, (ii) the assessing team can use the assessment tool with various stakeholders at all levels of the program, from service delivery to policy-making, (iii) using the tool, the assessing team is able to identify the strengths and weaknesses of a program, and propose ways to adjust it to remedy or prevent issues. Please note that the assessment tool is currently framed to apply mostly to a Performance-Based Financing (PBF) scheme, as PBF is implemented in most countries currently implementing RBF within the HRITF. However it can easily be modified to apply to other types of RBF programs. C. Assessment Tool Questions This assessment tool aims to evaluate the implementation of a RBF program by focusing on the areas that are most critical for achieving the intended results. The following table highlights these core areas. For each area, the assessment tool proposes questions that may be explored with various stakeholders to understand the program, and pinpoint its strengths and weaknesses. Not all questions will be relevant to a given stakeholder, however all core areas can be explored with a variety of respondents, at the health facility level, administrative level, etc.
- 2. 2 It is important to note that for such an assessment to be most useful you want to focus as much as possible on what happens in reality. You are not just interested in how it should (as may be described in the RBF manual) but what happens in reality in the RBF program implementation. This will help focus on possible challenges and can help highlight potential solutions and recommendations that may be needed. Note the questions are there to guide you; you may inquire further on specific aspects when deemed necessary. However, please keep in mind that you are here to understand how the program works and be a listener: use open-ended questions (not yes/no questions), try to understand how things are done and leave your own opinions aside during the interviews. Core area Exploratory Areas The RBF program overall 1) What is the general description of the RBF program, including whether it focuses on the supply or demand side? 2) What has been the timeline of the program? 3) Who are the different actors involved in the RBF program and what are their functions, from the local to national level? 4) Which actions or services do the RBF indicators cover at all levels? How are results paid for (e.g. per service, per target achieved, by weight in overall budget…)? How do quality and quantity of care interact in the payment for results? How have indicators or their tariffs changed since the beginning of the program? Data Quality and Verification 5) What are the data sources for service outputs (quantity)? What is the interaction of RBF data and Health Management Information System (HMIS) in the country? 6) How is the quality of care measured; by whom and what does it focus on? 7) Is there a way client satisfaction is assessed? How is this incorporated? 8) Who carries out the quantity and quality verification, what are the periodicity and sampling procedure? 9) What is the ex-ante mechanism for verification, i.e. the mechanism for verification used prior to making the payment? 10) How is this data on quantity and quality counter-verified, i.e. what is the mechanism that checks that the verification is done correctly (often ex-post, after the payment has been made)? 11) What happens when verification and counter-verification show discrepancies between reported and achieved results? 12) Are there any possible conflicts of interests (e.g., between the providers and the verifiers or between the verifiers and the counter-verifiers)? Results Achieved 13) Taking into account quality of the data, what are the results that have been achieved so far? 14) Which indicators (of quantity and quality) have improved and which have not? 15) What geographical areas are lagging behind or surging ahead? 16) Are there other disparities? 17) What does the trend look like? Is there evidence of plateauing?
- 3. 3 Core area Exploratory Areas RBF Financing and Sustainability 18) What are the sources of financing for RBF? 19) How much is the Government contributing? 20) Which development partners are contributing? How are the relationship and dialogue between the partners (including the Bank) and the Government? 21) What is the level of financing per capita? 22) How does the RBF scheme fit into the referral system? What proportion of funding is going to hospitals? 23) Which indicators absorb the largest percent of the RBF budget? 24) What percentage do RBF payments contribute to health workers’ take- home pay? How motivated is the staff? 25) What are the views of stakeholders in RBF, at all levels including beneficiaries? 26) How sustainable is the RBF financing scheme, financially or otherwise? Capacity Building 27) What has been the effort in Capacity Building at the health facility level and at other levels (initial and ongoing)? 28) To what extent have these efforts worked? 29) How are costs for capacity building distributed? 30) What kind of mechanism for Technical Assistance is in place? Supervision and Monitoring 31) How is the RBF program supervised at all levels? 32) What does supervision entail and how often does it happen? 33) How structured is the supervision? 34) To what extent are the operational data or HMIS data used for program monitoring, supervision and adjustments? Project Management 35) What are the management structures for RBF at the different levels, what is their role and who do they include? 36) How effective are the management structures – Steering Committee, health center committee etc.: how regularly do they meet, what do they focus on, what attention do they give to client satisfaction? 37) How effective is the RBF Implementation Unit (often a Project Implementation Unit- PIU) and how have the key people been trained? 38) Does the project have other support elements (e.g. human resources, Cash-on-Delivery arrangements etc.) and are they moving in tandem? Institutional roles and functions 39) Is there a separation of roles of purchaser, fund holder and verifier? 40) Are there clear contractual relationships between them? Facility Autonomy 41) How is facility autonomy operationalized (e.g. how does the facility use the RBF and possible other resources; what is the decision making process for the use of the funds)? 42) Do facilities have bank accounts and who is the signing authority? 43) To what extent do approval processes or procurement rules support or hinder actions (to be) taken at the facility?
- 4. 4 Core area Exploratory Areas Payment for Results 44) What is the per capita cost of RBF? 45) Are funds transferred directly to facilities for outputs received? 46) What are the time lag quarters between performance and payment? 47) Are there cost containment measures in place? 48) How are payments for results being used? Community Involvement and Beneficiaries 49) What kinds of community management structures are in place at the health service delivery level? 50) How effectively does the facility link with the community management structure? 51) How and how often does the facility check on beneficiaries’ experiences at the facility and satisfaction with the services? 52) How does the facility integrate the views of the community and beneficiaries within its own management?