Looking at implementation: how useful is realist evaluation?
0 likes448 views
The document discusses a workshop on realist evaluation methods in implementation science for global health, particularly focusing on user fee exemption policies and health policy dialogue in Africa. It highlights the objectives and research questions of two studies aimed at understanding the outcomes and mechanisms of these policies and dialogues. The document emphasizes the importance of qualitative methods and a realist lens to analyze the roles of agents and contextual factors in the implementation process.
Looking at implementation: how useful is realist evaluation?
- 1. Looking at implementation: how useful is realist evaluation? METHODS FOR IMPLEMENTATION SCIENCE IN GLOBAL HEALTH April, 20th. 2017 McGill University Workshop co-organised by REALISME Chair and McGill of Global Health Programs Emilie Robert, Ph.D., McGill University, RI-MUHC / ERIT Funded by CIHR
- 2. Context • A realist synthesis on user fee exemption policies in sub-Saharan Africa (S1) – National policies aiming at providing free health services in a context of weak health systems and cost-recovery as the standard – Objective: to improve access to healthcare by removing the main financial barrier • A realist evaluation of WHO support to health policy dialogue for health planning towards universal health coverage in Africa (S2) – Multiplicity of stakeholders in the health planning processes – Low ownership of MoH and unstable participation of stakeholders – Objective: to foster health policy dialogues that are evidence-informed, inclusive, participatory, and led by MoH
- 3. Objectives of the studies (research questions) • S1: What are the outcomes of user fee exemption policies implemented in Africa? Why do they produce such outcomes? What contextual elements come into play? • S2: How, in what contexts, and triggering what mechanisms does WHO support policy dialogue for health planning towards universal health coverage? 1) To highlight the contexts in which WHO can, or cannot, act as a broker and a convener, creating synergy among the actors involved in policy dialogue for health planning; play its role as technical expert, orienting policy dialogue for health planning in accordance with available evidence and equity principles; and support ministries of health in their leadership and stewardship functions. 2) To identify, for each of these mandates, the mechanisms at work and the outcomes produced; 3) To clarify the process chains and the links between them.
- 4. Method • Qualitative methods (however mixed methods may be useful) • Realist lens : – Focus on agents, rather than activities, money or procedures (resources) : agents create change or maintain status quo through their actions, reactions and behaviours. – Intervention as the result of the reasoning and action of agents, especially implementors, faced with new resources in context – Mechanism as the explanatory concept of the causal relation: • between an intervention AND an outcome • between resources AND any outcome in the implementation process – Theory-based evaluation
- 5. Results • S1 : 1st level of analysis (before final theory) Delays and unpredictability in the policy financing at the health facility level (reimbursement or kits / medicine) (C) encourage health staff to adjust the prices of health services (M = COPING). As a consequence, users do not consistently benefit from free healthcare (E). Health staff ajusting prices of health services (C) leads users to protect themselves from potential costs related to seeking care (M = BENEFIT-RISK DECISION-MAKING) and thus constraints their opportunity to benefit from free healthcare (E). Increase in utilization of health services associated with implementation failures (C) deteriorate the initial enthusiasm of health staff (M = DISILLUSION), which in turn contributes to the deterioration of their relationship with users (E). S2 : Initial intervention theory (before data collection & analysis) Evidence-informed, inclusive, participatory, and MoH led policy dialogues would trigger three mechanisms: - Mutual understanding of issues, evidence & information, and courses of action, - Stakeholders’ buy-in of decisions, - Appropriation of policy dialogue process by MoH These should contribute to the formulation of robust and comprehensive health policies, the alignment of stakeholders & a strengthened leadership of MoH.
- 6. Lessons learned for implementation science in global health If you like to solve puzzles, you will like investigating implementation using the realist approach! Strengths • Implementors are agents (individuals / groups / organizations). • Theory building includes explanatory implementation / process models using a realist lens • Mechanisms as the missing pieces of the implementation puzzle. Limits • May be time consuming… • Needs good understanding of the epistemological roots • Needs attention to details • May need to focus on specific processes
- 7. To go further • Pawson, R., & Tilley, N. (1997). Realistic evaluation. London, UK: Sage Publications. • Maxwell, J. A. (2012). What is realism, and why should qualitative researchers care? In J. A. Maxwell (Ed.), A Realist Approach for Qualitative Research (pp.3-13). Thousand Oaks, CA: Sage Publications. • Lacouture, A., Breton, E., Guichard, A., Ridde, V. (2015). The concept of mechanism from a realist approach: a scoping review to facilitate its operationalization in public health program evaluation. Implementation Science, 10(1):153. doi: 10.1186/s13012-015- 0345-7 • Dalkin, S.M., Greenhalgh, J., Jones, D., Cunningham, B., Lhussier, M. (2015). What’s in a mechanism? Development of a key concept in realist evaluation. Implementation Science, 10:45. doi:10.1186/s13012-015-0237-x • Pawson, R. (2003). Nothing as practical as good theory. Evaluation. 9:4(471-490). doi: 10.1177/1356389003094007
- 8. Let’s discuss together! Meet the experts Thank you for your attention