Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF
1 like370 views
The document outlines key lessons and recommendations from performance-based financing (PBF) implementation in healthcare, focusing on ten critical insights for improving effectiveness. Timely payments, continuous quality improvement, understanding demand-side issues, and political sustainability are among the highlighted lessons. The authors emphasize the importance of data utilization, stakeholder engagement, and adaptability in ensuring the success of PBF schemes.
Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF
- 1. The Science of Delivery: Impact Evaluation Results and Lessons from RBF Dinesh Nair, Benjamin Loevinsohn and Ifelayo Ojo Learning from Implementation
- 2. The Why and How of Top Ten Lessons • Review of ongoing PBF operations suggested some consistent challenges & useful lessons • Focusing on the most important lessons will facilitate learning by other teams • Selection of the Top Ten was done by TTLs and RBF experts involved in a portfolio review of eight ongoing RBF schemes 2
- 3. Lesson #1 “Show me the money!” - Make timely payments • Over 70% of projects have had issues with making payments on time 3
- 4. Outpatient visits in Nigeria declined when payments were interrupted 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Adamawa Nasarawa Ondo Payments interrupted 4
- 5. Lesson 1: “Show me the Money” Make Timely Payments Recommendations Track time required for payments to reach facilities Establish explicit standards for maximum allowable delay (2 or 3 months) Provide resources up-front to health facilities: (i) gains their confidence; (ii) indicates PBF has started; and (iii) provides investment funds 5
- 6. Lesson #2 “Keep moving the goalposts!” Continuous Quality Improvement (CQI) implies Changing the Quality Indicators • Many facilities make rapid improvements in quality and then plateau 6
- 7. 0 20 40 60 80 100 Mwaro Muramvya Kirundo Cibitoke Buja-Rural Kayanza Ngozi Makamba Rutana Bubanza Bururi Gitega Karuzi Muyinga Ruyigi Cankuzo Buja-Mairie Qualityscores by province Average quality score plateaued after 3 quarters in Burundi 7
- 8. Lesson 2: “Keep moving the goalposts!” Continuous Quality Improvement (CQI) implies Changing Quality Indicators Recommendations Quantified quality checklists should emphasize process over structural indicators Revise checklist every 1–2 years to emphasize continuous improvement Include: 1) observation of patient-provider interaction; 2) records reviews (need standard records); 3) Vignettes; 4) Exit interviews, etc. Invest in training supervisors 8
- 9. Lesson #3 “To be discerning, We keep on Learning” • RBF as a tool allows for experimentation, learning and course-correction 9
- 10. What did Zimbabwe do? Quantitative analysis Qualitative analysis Design demand and supply-side RBF (urban pilot) Targeted urban maternal voucher (demand-side) Pay-for-quality RBF Reviewed pricing Improved quality checklists 10
- 11. Operational research studies should be budgeted Carry out process evaluations and use results for implementation dialogue and decision-making. Participate in RBF portfolio reviews – look for new ideas Lesson 3: To be discerning - Keep on Learning 11
- 12. Lesson #4 “Money is NOT the root of all evil, Lack of Money is” – Worry about Financial Sustainability • PBF should not be made to substitute deficiencies in national health financing 12
- 13. PBF sustainability requires wider health system reforms Burundi PBF is combined with FHC Government pays almost half–1.4% of budget allocated Common pool for development partners– two decreased contribution Cumulative deficit of almost US$ 8M Rwanda PBF health reforms were part of larger reforms CBHI reform HR reforms 13
- 14. Where is the PBF money going? 14
- 15. 35% 37% 39% 40% 45% 11% 10% 10% 9% 5% 16% 16% 12% 13% 10% 29% 31% 31% 33% 37% 8% 7% 7% 5% 3% Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Intervention HFs Intervention DMO Hospital Control 1 HFs Control 1 DMO Zambia: Aggregate consumption of PBF funds from Q2, 2012 – Q2, 2013 Where is the PBF money going? 15
- 16. Recommendations Track overall PBF expendituresand understand where most PBF funds are going It is easier to increase tariffs than decrease them. This suggests starting with relatively lowertariffs and increasing them Insist that Government(s) live within an envelope budget. Be careful about increasing expenditure untied to improved performance. Lesson 4: “Money is NOT the root of all evil, Lack of Money is” – Worry about Financial Sustainability 16
- 17. Lesson #5 “Math-Phobes of the World Unite!” – Use your data • Data is vital but under-utilized, despite a lot of effort invested into collecting, verifying and putting payment data on the web 17
- 18. Internet applications with public front- end displaying performance & financial information 18 Burundi Benin Nigeria
- 19. How Zambia uses its data • Quarterly analysis • Extensive analysis by expert o Diff-in-diff analysis of trends using HMIS to compare across the 3 study arms RBF vs. Additional financing Coef 0.904 0.815 0.696 12.944 2.220 -9.316 -2.783 p-value 0.045 0.231 0.229 0.002 0.019 0.735 0.024 RBF vs. Control Coef 1.174 1.954 1.586 7.850 2.243 -39.929 -2.761 p-value 0.005 0.002 0.011 0.055 0.031 0.158 0.011 Attendance outpatient total (calc) Immunised fully <1 year new Antenatal 1st visit before 20 weeks IPT 3rd dose to pregnant woman Postnatal care within 6 days Attendance Family Planning total (Calc) Delivery by skilled personnel 19
- 20. Lesson 5: Math-Phobes of the World Unite! – Use your data Recommendations Create reliable systems for data collection and compilation Ensure consistent flow of PBF data with minimal reporting burden on health workers Make it somebody’s job to analyze data Deploy software to help with data analysis Crowd-source analysis! Attempt to triangulate all available data sources to determine trends in utilization and outcomes 20
- 21. Lesson #6 “Understand your Customers” -Demand side issues may be under-appreciated • There is need to understand root-causes of the problem RBF is trying to solve. 21
- 22. Even with large improvements in utilization since PBF was introduced, absolute coverage levels remain low in Nigeria Barriers to service utilization in two Nigerian LGAs: Transportation Challenges Variable & unpredictable fees for Services and Drugs Social and cultural Barriers Proposed solution: A voucher scheme to improve service uptake Continuous cycle of learning and responding to help households overcome service utilization barriers and improve health outcomes. 22
- 23. Lesson 6: Understand your customers – Demand-side issues may be under- appreciated Recommendations Consider demand-sideissues if coverage levels remain low in spite of PBF (even if PBF has made a big difference). Understand barriers to access – cultural, social and financial barriers through household surveys; focus groups, key informant interviews, etc., Options for addressing demand side include: (i) closer work with community structures; (ii) CCTs; (iii) vouchers; (iv) BCC through facility staff or NGOs 23
- 24. Lesson #7 “Away with Flat-Liners” - Identify poorly performing regions & facilities. Do something about them! • Detailed understanding of predictors of success and failure is required 24
- 25. There are clearly facilities that are NOT making progress i.e. “flat-liners” Non-determinants • Number of staff • Remoteness • Qualification of in- charge • Business planning - 20 40 60 80 100 120 140 Positive deviants • Community engagement • Management capacity of in- charge Institutional Deliveryin Adamawa health facilities, normalized by 100,000 population 25
- 26. Lesson 7: Away with “Flat-Liners” Recommendations Use existing data to identify facilities that have not improved or are not performing well Carry out quantitative and qualitative studies (e.g. Nigeria and Zimbabwe) Consider the kind of technical support needed for the poorly performing facilities and re-examine what is currently available. 26
- 27. Lesson #8 “You can run but you can’t hide” – Worry about the politicaI threats to PBF • Even successful RBF schemes face political pressures that threaten their sustainability 27
- 28. Zambia RBF Pilot RBF PIU not mainstreamed into MOH structure Institutional changes in MOH Key champions moved 28 Watch for winners and losers
- 29. Recommendations Political economy expertise should be engaged to study factors important for scale-up and sustainability of RBF, to figure out what can be done differently. Institutionalizing roles such as purchasing and project implementation within government structures may help improve buy in 29 Lesson 8: Worry about the political threats to PBF
- 30. Lesson #9 “Even the Best Laid Plans can go wrong” – Analyze your Project Design to Ensure Assumptions Remain Valid • Understand key-determinants of the PBF program and the linkages between them to produce results 30
- 31. Benin Bureaucracy makes it difficult for facilities to spend PBF incentives May reduce health worker motivation to perform User-fees easier to spend High fees may decrease healthcare utilization 31 Challenges with health facility autonomy in Benin and nested PBF in Afghanistan made implementation different Afghanistan Unique environment of performance agreement with NGO Ongoing security concerns PBF overlaid on PPA
- 32. Recommendations Periodically review the progression of elements in your theory of change, not just what you are trying to achieve, but how you get there Useful analyses for every project include: What is additional percentage of PBF to health workers' take-home pay? Are they motivated by this amount? Balance of supervision versus autonomy at the health facility level Resources available at the front-line for health facilities 32 Lesson 9: “Even the Best Laid Plans can go wrong” – analyze project design to ensure assumptions remain valid
- 33. Lesson #10 “KISS – Keep Impact Studies Straightforward” - Avoid Impact Evaluation Questions that are TOO Subtle • It is very difficult to assess differential impacts when there is minimal variation between experimental groups 33
- 34. 0 600 1200 1800 2400 1* 2 3 4 1 2 2012 2013 Children immunized Control Treatment The RBF intervention and additional financing group in Benin are looking the same No clear difference is emerging Another control group with no added finances has been included for counterfactual 34 0 2000 4000 6000 8000 1* 2 3 4 1 2 2012 2013 Assisted delivery Control Treatment
- 35. Experimental & Control groups in Nigeria – try hard to be explicit on differences !!!! 35
- 36. Recommendations Assess in advance if experimental variations are likely to make a difference in outcomes. Work with the IE team from the start Ensure counterparts fully understand research questions, and are convinced of utility. 36 Lesson 10: “KISS – Keep Impact Studies Straightforward” Avoid Impact Evaluation Questions that are TOO Subtle
- 37. A ship in the harbor is safe – but that is not what ships are built for.-John A Shedd (1928) Implement, learn and share lessons!