The Transfer Project: Some lessons on making evaluations useful for policy action
Download as PPTX, PDF1 like881 views
The Transfer Project conducted evaluations of cash transfer programs in 10 African countries between 2009-2019 to generate evidence and address criticisms of social protection programs. Key factors that led to research uptake included building trust between evaluators and program implementers, understanding stakeholder needs, embedding impact evaluations into broader learning agendas, and strategically disseminating results through workshops, briefs, and conferences. Examples from Ghana, Zambia, and Malawi illustrated how evidence was used to increase cash transfer values, address concerns about dependency, and influence other development partners and policies. While using evidence takes effort and commitment over time, the Transfer Project model identifies important ingredients for research to impact policy and programs.
![Transfer as Share of Participant Consumption:
LEAP level is very low by international standards
[Actual slide shown at baseline workshop, 2011]
7
15
17
20
22
25
27
29
0
5
10
15
20
25
30
35
Transfer Share](https://image.slidesharecdn.com/angelesgdconferencebonn2019v6reallylatest-191030084100/85/The-Transfer-Project-Some-lessons-on-making-evaluations-useful-for-policy-action-11-320.jpg)
The Transfer Project: Some lessons on making evaluations useful for policy action
- 1. The Transfer Project – Some lessons on making evaluations useful for policy action Gustavo Angeles, University of North Carolina at Chapel Hill (UNC) On behalf of the Transfer Project Evaluation Team 2019 Global Development Conference Bonn, October 23, 2019
- 2. The Transfer Project: Background & Motivation Mid-2000’s, growing momentum for social protection in sub-Saharan Africa Cash transfers (CT) in a few places but skepticism and concerns Latin American CT evaluations known but African programs different: Unconditional; different supply conditions and institutional context Limited administrative capacity So, large evidence gap in Africa at a time of increasing demand for evidence Response: In 2009, Transfer Project created as a collaboration between UNICEF, FAO and UNC to generate the needed evidence.
- 3. The Transfer Project: Background & Motivation Key defining features: Evaluate ongoing national government CT programs Address 3+3 main criticisms of impact evaluations: They don’t answer the questions relevant for policy makers Results not available at needed time Presented in format and language too technical Limited learning component beyond impact Distant from the policy and program processes Limited involvement of local researchers.
- 4. 4 Transfer Project: - Conducted evaluations in 10 countries - This book documents experience across eight countries - Describes evaluation process and results - How was evidence used? - What were key features that led to research uptake? - Chapters written jointly by evaluators and program managers.
- 5. https://transfer.cpc.unc.edu/making-research-count-lessons-on-turning-evidence-into-action-from-the-transfer-project/ Read the blog, if you are convinced then read the book… (350+ pages)
- 6. Key ‘ingredients’ to research uptake • Relationship between evaluator and program implementers • Build trust, same team, same objectives • Close partnership with govt and managers to understand and respond to their concerns • No “arm’s length” evaluation • Understand the stakeholders’ motivation for the evaluation and the political economy of the program • Tailor the evaluation according to the evidence needs of stakeholders • This leads you beyond standard IE of “1 program and 1 outcome”: Added local economy-wide impacts, productive activities; cost; examine “expected” negative behaviors
- 7. Key ‘ingredients’ to research uptake • Managers interested in program design/operations, policy makers can’t wait 2-3 years: Embed impact evaluation into broader learning agenda • Many questions on program design/operations don’t require IE • Make baseline surveys useful: • Targeting analysis • Cash transfer size as share of household consumption • Simulations of impact • Qualitative assessments • Barriers to use of services • Presence of other services
- 8. Key ‘ingredients’ to research uptake • Partnership with national researchers • Understand local context, sensitivities, research questions • Readily available to present results when necessary • Keep it as rigorous as reality permits • Everybody values that; provides credibility • But, balance it with political and operational constraints; RCTs not always feasible. Needs consensus.
- 9. Key ‘ingredients’ to research uptake • Use multiple channels of communication: • 2-day workshops with national & provincial program managers, Ministry and development partners on programmatic implications • Policy briefs • Overall and specialized reports • Newsletters And, Transfer Project Conferences: • 7th in Arusha in April 2019 • 130 delegates in 2019, up from 39 participants in 2010 • Country teams: government, program managers, local academics, and development partners • Exchanges on latest evidence and its use.
- 10. Ex. 1: Ghana LEAP Cash Transfer Program • Initial impact evaluation 2010-2012 • Used baseline data to calculate transfer value as share of consumption of beneficiaries - Extremely low by international standards - Simulations showed no/small positive impact - High administrative cost relative to benefit transferred • Ministry used evidence to successfully triple value of transfer • Operational aspects of the programs became relevant
- 11. Transfer as Share of Participant Consumption: LEAP level is very low by international standards [Actual slide shown at baseline workshop, 2011] 7 15 17 20 22 25 27 29 0 5 10 15 20 25 30 35 Transfer Share
- 12. Endline evaluation showed limited impacts—a key assumption in the log frame did not hold 0 1 2 3 4 5 6 Apr-10 May-10 Jun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 #ofpayments Ghana LEAP Transfers were lumpy and irregular rather than predictable
- 13. Ex. 2: Zambia cash grants IEs • Explicit request from Ministry for IE to examine effects on dependency, hand-outs, laziness • Strong economic/productive angle to IE design • No work decline; increases in production, • Strategic, repeated dissemination of these results • Other ministries, Cabinet, Ministry of Finance • Social protection week • Research team estimated a ‘multiplier’ effect of 1.5 • Powerful, succinct rebuttal to dependency claims
- 15. Zambia: Cash transfer scale-up and evaluation timeline GoZ budget contribution went from $5m to $35m in 2014 and $45m in 2015 0 50000 100000 150000 200000 250000 2003 2005 2007 2009 2011 2013 2015 2017 Households Reached by Cash Transfers in Zambia CGP MCP follow-ups CGP MCP baselines Monze baseline Monze follow-up CGP MCP follow-ups GoZ take over Election
- 16. 16
- 17. Ex. 3: Malawi Social Cash Transfer Programme (SCTP) • Initial IE 2006-07 Mchinji pilot • Led to expansion to 8 districts • Stagnation 2008-2012 • GoM interest in Public Works (MASAF) and FISP • Ministry/UNICEF kept citing Mchinji results, developed a National Social Protection Policy (NSSP), ‘kept pedaling the bicycle’ • SCTP take-off from 2012+ • Donors supported NSSP, more $ commitment from GoM • New IE 2013-2016 as part of expansion
- 18. Malawi - Uptake of evidence from 2013-16 IE • Baseline simulations on transfer value directly led to increase in transfer size in 2015 • Strong positive impacts across the board • Demonstrated Ministry’s ability to implement program, seriousness and professionalism in using evidence • Maintained donor enthusiasm and commitment • Strong positive impacts coincided with ‘negative’ impact results from MASAF public works • World Bank (finally) took note of SCTP, diverted funds from MASAF to SCTP • SCTP scaled up to all districts now • Limited effects on nutrition and morbidity • Need to link beneficiaries with other programs
- 19. Over 95% of eligible households have received their transfers* 95 95 95 95 99 99 97 97 97 95 95 97 0 10 20 30 40 50 60 70 80 90 100 I II III IV V VI VII VIII IX X XI XII PercentageofeligibleHHreceivingtransfer *: According to the transfers payment data. Transfer s Midline Endlin e
- 20. Malawi evolution of real value of transfers • Amounts increased in May 2015 by about 55% • However, about 60% inflation between Aug 2013 and May 2015 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 MalawiKwachas Transfer: 4+ persons Nominal: 2,400 MWK 3,700 MWK Real -37% -54% Baseline (Aug 2013) Midline (Dec 2014) Endline (Nov 2015)
- 21. Summary: Program is operating well, providing regular and predicable transfers, but must protect real value of transfer. Payday in Salima: Photo credit, Jacob de Hoop
- 22. Examples illustrate different paths to evidence uptake • Ghana: Actual IE results were null, but highlighted key operational issues that led to change • Zambia: Focus on addressing critics, strategic and concerted dissemination efforts • Malawi: Patience, long-run perspective; influencing other development partners
- 23. Reflection on whether evidence has been used to improve programs • Yes, but it takes special effort; Transfer Project model identifies some key ingredients: • Implementers must trust evaluator, same team • One-off IE not the best strategy – need broad learning agenda • Understand the political debate • Strategic packaging and dissemination • National research counterparts to present evidence at key moments • Patience…need confluence of other factors, but must have evidence available at the right moment All this requires time, commitment, skills, patience and funding.
- 24. Transfer Project website: https://transfer.cpc.unc.edu/ Facebook: https://www.facebook.com/TransferProject Twitter: @TransferProjct For more information