Evaluating complex change across projects and contexts: Methodological lessons from a macro evaluation of DFID's social accountability portfolio
Download as PPTX, PDF2 likes761 views
This document discusses the methodology used to evaluate DFID's social accountability portfolio, which included 361 diverse projects. The evaluation used Qualitative Comparative Analysis (QCA) on 50 cases and narrative analysis on 13 case studies. Combining QCA with narrative analysis provided both numerical evidence from a large number of cases and in-depth qualitative analysis to interpret the QCA findings. Some challenges encountered were data quality issues, difficulty unpacking complex context factors, and limitations due to the project timeline not allowing for full iteration between methods. Overall, mixing QCA and narrative analysis proved useful for understanding what approaches worked best across different contexts.
Evaluating complex change across projects and contexts: Methodological lessons from a macro evaluation of DFID's social accountability portfolio
- 1. P r e s e n t a t i o n a t U K E S a n d C D I , 2 7 a n d 9 A p r i l 2 0 1 6 J e r e m y H o l l a n d a n d F l o r i a n S c h a t z Evaluating complex change across projects and contexts: Methodological lessons from a macro evaluation of DFID's social accountability portfolio
- 2. Background Part of a wider macro evaluation of DFID’s Policy Frame for Empowerment and Accountability (E&A) Objectives: – To understand what works, for whom, in what contexts and why, by conducting cross-case analysis of DFID’s E&A project portfolio since 2011; and – To generate new evidence that informs policy and practice in DFID and other development organisations. DFID’s E&A portfolio: 361 diverse projects (different focus areas, different countries and regions, different budgets and timeframes, different modalities and approaches, etc.)
- 4. QCA and Narrative Analysis QCA (50 cases) Social science research method that applies a systematic comparison to case study research Helps to identify the determinants of outcomes by looking at the similarities and differences of cases in terms of the causal factors and outcomes obtained (Cress and Snow 2000) Situated between qualitative and quantitative research approaches Narrative analysis (13 case studies) In-depth qualitative comparative analysis to interpret and explain the patterns identified through QCA Focus on identifying explanatory models that explain differences between True Positive cases and False Positive cases (i.e. cases with the same configurations of conditions but a different outcome) Based on secondary (project documentation) evidence supplemented by primary (Key Informant Interviews) evidence
- 5. 5 1. Database of projects meeting inclusion/ exclusion criteria (produced 180 SAcc projects) 2. Screen for quality of outcome contribution analysis (produced 50 SAcc projects) 5. Identifying and coding project ‘conditions’ QCA contexts, mechanisms and outcomes (CMOs) 6. Extracting data and scoring conditions QCA binary scoring 4. Literature review and DFID consultations 7. Developing testable hypotheses Hypotheses expressed as CMO configurations 8. QCA of hypothesis configurations 9. Selecting hypotheses and project cases for narrative analysis 10. Narrative analysis of 13 projects Analysis supplemented with key informant interviews Sample included ‘true positive’ and false positive’ cases selected via Hamming distance measure 3. Representativeness analysis Population of 2379 DFID projects screened 10 Step Approach
- 6. Challenges and Lessons 1. The data quality challenge …using QCA: • Despite selecting the projects with the best available data and filling data gaps through Key Informant Interviews, we had 104 out of 1200 data points missing • This required the manual construction of different sub-datasets for each hypothesis and limited our ability to perform more inductive analysis …using narrative analysis: • The quality of data varied considerably in terms of coverage and analytical depth • Key Informant Interviews were effective in deepening our understanding but a tight timeline prevented us from reaching more than 20 stakeholders -> mixed evidence base
- 7. Challenges and Lessons 2. The challenge of unpacking ‘context’ …using QCA: • Despite including context conditions in our hypotheses, we were not able to generate interesting or strong associations involving project contexts • Our context conditions were based on nationally comparable global indices and possibly too broad …using narrative analysis: • At the level of individual case studies, context factors were important but so specific that it was difficult to generalise across cases while also retaining ‘granularity’ • This limited our ability to generate evidence on ‘what works in what contexts’
- 8. Challenges and Lessons 3. The challenge of sequencing and iteration Combining QCA with narrative analysis required sequencing each evaluation step carefully, which resulted in a long timeline: – Finalising hypotheses before data extraction/coding – Finalising QCA before narrative analysis But subsequent steps threw up additional factors, hypotheses and data points that would require revising preceding steps -> Iteration, for which there was insufficient time
- 9. Challenges and Lessons 4. The utility of mixing methods Combining QCA with narrative analysis proved useful. The narrative analysis helped interpret and explain QCA findings, while QCA provided numerical evidence based on a relatively large number of cases to back up our findings. The next slide presents an example finding from the macro evaluation:
- 10. Hypothesis QCA finding Narrative analysis finding Higher-level (at-scale) service delivery (O2) is achieved only when SAcc mechanisms include support for feeding evidence and learning into higher-level discussions (M7) and higher- level legislative and policy change (M1). Hypothesis rejected. The combination of feeding evidence upwards (M1) and directly supporting policy change (M7) is neither necessary nor sufficient for improved service delivery at scale. However, it improves the likelihood of success. In the cluster of (24) cases where both these mechanisms were present, there was evidence of improved higher-level service delivery in 7 (or 29%) of cases. In the cluster of (5) cases where both of these mechanisms are absent there is no case of improved higher-level service delivery. With coverage of 53% and consistency of 29%, the combination of M1 and M7 shows the strongest association with the outcome. No single condition was necessary, sufficient, or strongly associated with the outcome. The narrative analysis confirmed that linking SAcc to higher-level policy advocacy (M1) through upward feeding evidence (M7) could help improve service delivery at scale. The contribution of these mechanisms to improved outcomes was explained by the following factors: • SAcc processes worked better when embedded in policy or programme frameworks • Upward feeding evidence was effective when channelled directly into these policy and programme processes • Upward feeding evidence was effective when combined with citizen participation key decision making processes.
- 11. Thank you For more information: http://www.itad.com/knowledge-and- resources/dfids-macro-evaluations/
Editor's Notes
- #7: (although the majority (63) of these were due to the use of project context indices with missing country profiles) Therefore, the evidence base for our findings is mixed and stronger for some hypotheses than for others