Using case-based methods to assess scalability and sustainability: Lessons from evaluating innovations of the Supply Chain for Community Case Management (SC4CCM) project in Rwanda, Malawi, and Ethiopia
2 likes536 views
The SC4CCM project aimed to enhance the availability of key products for sick child management in Malawi, Rwanda, and Ethiopia, assessing the scalability and sustainability of innovations through midline and endline evaluations. The project analyzed the effectiveness of interventions and their institutionalization using a mixed-methods approach, including qualitative case studies, surveys, and cross-case synthesis. Findings indicated varying success across countries, with a focus on contextual factors influencing scalability and sustainability of the implemented strategies.
Using case-based methods to assess scalability and sustainability: Lessons from evaluating innovations of the Supply Chain for Community Case Management (SC4CCM) project in Rwanda, Malawi, and Ethiopia
- 2. Background to the SC4CCM project • In 2009, SC4CCM set out to improve community-level availability of key products for sick child management in 3 countries: Malawi, Rwanda, Ethiopia. • Tasked with demonstrating that tested innovations could be scaled and sustained after project end. • Midline evaluations (2012-13) focused on measuring effects; findings kicked off scale-up process. • Endline evaluation (2014) focused on learning about scalability and sustainability of SC4CCM innovations.
- 3. Over-arching Endline Evaluation Questions • To what extent (geographic breadth and institutional depth) have the SC4CCM innovations to support commodity availability at the community level, specific to each country, been scaled up? • To what extent have the program effects of SC4CCM observed at midline been maintained at endline? • To what extent have the country- specific innovations been institutionalized at endline? • What aspects of the SC4CCM design, implementation, and overall project approach contribute to scalability and sustainability of the particular innovation supported in a country?
- 4. Case selection strategy • Compare original with scale-up districts • Learn from extremes: best and worst performing with SCM/PA • Management component focus: choose near and far CHWs; high and low QI • Get multiple perspectives – district, HC, CHW
- 5. Multiple, Mixed methods design Qualitative case studies • Purposively selected cases • In-depth interviews (district, HC, CHWs) • Structured observations of tool use (HC and CHW) • RSP tool documentation review (HC, CHW) • QI team documentation review (district, HC) • Standard field manual Endline surveys • Representative selection of HCs and CHWs • Comparable to midline LIAT • Structured survey interviews • Physical inventory of key products • Assessment of knowledge and competency using RSPs
- 6. Within-case (country) analysis process • Team discussion and analysis notes after each district; feedback from evaluation advisor • One week focused analysis immediately after data collection – whole team, with project management and evaluation advisor participation • Writing of country-specific endline report, integrating LIAT data • Presentation of endline findings to country stakeholders Focused analysis steps: • District-by-district analysis, using notes, original program theory, evaluation questions; ongoing identification of contextual factors • “Deep dives” – RSP procedures: improvements, benefits, challenges; QI teams: functionality, benefits, challenges; PA: perceived improvements, other changes, challenges • Analysis of contextual factors – contributions and barriers; likelihood of affecting scale-up or sustainability; can it be changed? • “Validation” and revision of program theory based on the data
- 7. Cross-case synthesis process • Common themes identified from program theory and country-specific analyses • Codebook developed • Qualitative data coded in atlas.ti – to make the data more manageable, and allow for focused cross- theme analysis to understand relationships and non-linearity • One week focused synthesis analysis with whole team participation Synthesis analysis steps: • Data integration - PA: used qual to explain survey findings; RSP: qual- survey validation; QI: used qual findings as lead • Differences and similarities between countries; full range of benefits and challenges (RSP and QI); explaining PA; context • Scale: compared how it is happening across countries; RSP v. QI; partner implementation • Sustainability: Data visibility and use comparisons and mechanisms; evidence of political commitment • Project inputs and implementation strategy contributions
- 8. Data-based country program theories
- 10. Acknowledgements JSI Research & Training Institute, Inc. (JSI) implemented the Supply Chains for Community Case Management (SC4CCM) project with funding from the Bill & Melinda Gates Foundation. JSI partnered with the Milken Institute School of Public Health at The George Washington University for the endline evaluation. •GWU team: Sangeeta Mookherji, Jillian Dunning, Teemar Fisseha, Anisa Saleh •SC4CCM team: Yasmin Chandani, Sarah Andersson, Alexis Heaton, Megan Noel, Barbara Felling, Savitha Subramaniam, Mildred Shieshia