Presentation ukes 2015 v7
Download as PPTX, PDF2 likes337 views
The document outlines the design and challenges of a participatory impact evaluation for an IFAD-funded project in Ghana aimed at improving rural livelihoods through enhanced production and processing of roots and tubers. It highlights the objectives of generating rigorous evidence, facilitating inclusive analysis, and creating a scalable self-evaluation model while addressing complexities in evaluating government programs. The evaluation employs a variety of qualitative and quantitative methods and emphasizes stakeholder involvement in understanding impacts and contributions to poverty reduction.
![Hypothesis:
[Enhanced smallholder R&T production and processing at scale] + [Sustainable and inclusive CC market linkages]
=> Improvement livelihoods and poverty reduction in rural Ghana
Assumption:
· Livelihoods and poverty status in rural Ghana can be improved by commericializing smallholder R&T production and processing businesses combined with the
establishment of competitive market-driven and inclusive CC linkages.
Evaluation/learning question:
· To what extent and for whom does this assumption hold true (or not) and under which conditions? Does it hold true for resource-poor women and youth/young
adults in remote rural areas?What conditions need to be changed to enable women and young adults overcome cultural and socio-economic barriers?
Causal link House-hold
level
Community cluster
level
District level Zonal & national level Sampling approach
I2→I1 HH survey
(with households in the
the supply chain
catchment area)
Review of the 2010 Ghana
Living Standard Survey
report and other relevant
secondary data
Stratified sampling of 30
households from the
community clusters in each
sampled district
O3+O2+O1
→I2
Generic change analysis
(in gender/age-specific focus
groups of community members from
the supply chain catchment area)
Review of RTIMP RIMS
baseline ad other M&E data
Stratefied sampling of
community members from
the community clusters in
each sampled district
O2
O1
I2
I1
O3
Impact claim – Poverty reduction](https://image.slidesharecdn.com/presentationukes2015v7-150531175855-lva1-app6892/85/Presentation-ukes-2015-v7-9-320.jpg)
![Contribution Claim 3 – Enhanced R&T processing
Hypothesis:
[Access to business financing] + [Demonstration of good practices]
=> Development of profitable processing enterprices by R&T supply chain farmers and processors
=> Enhancement and scaling of smallholder R&T processing, contributing to the growth of the R&T supply chains
Assumptions:
· Resource- poor processors and farmers who are well trained in quality management, business planning and marketing apply
for matching grant funding (MEF) to invest in their businesses. PFIs are prepared to provide credit to well- trained resource- poor processors and farmers up to 50%
of their planned investments.
· GPCs can reach and teach resource- poor processors to develop more profitable agri- processing businesses by demonstrating good quality processing and
management practices, including the use of improved technologies and stnadardized equipments. As a result, resource- poor processors apply to the MEF and invest
in new technologies and equipment that help them to produce greater volumes of higher quality at lower cost.
Evaluation/learning question:
· To what extent and for whom do these assumptions hold true (or not)?
· What conditions need to be in place for GPCs to become profitable and attractive businesses particularly for young adults living in remote areas?What supports or
hinders GPCs to better link the supply chain farmers to old and new markets, and how is this influenced by the DSF?
· Reach and added value of the MEF? Effects of the MEF on growth of the funded agro- processing businesses? Avoidance of elite capture?
Causal link Household
level
Community cluster
level
Disrict level Zonal & national level Sampling
approach
M3b→
C3a+C3b→O3
Livelihood analysis
(in gender/age-specific focus
groups with supply chain farmers
and processors)
KIIs with GPCs Review of RTIMP and REP
M&E data and supervision
reports (incl. the 2014
thematic impact studies on
MEF and GPC); the
comparative case study on
matching grant facilities
Stratified sampling of
supply chain farmers,
seed growers and
processors
Stratified sampling of
GPC- and non-GPC-
participants (incl. MEF
beneficiaries)
M3b+M3c+C1a
→C3c
Constituent feedback
(with mixed groups of (non-)
GPC participants and MEF
beneficiaries )
KIIs with BACs and PFI local
branches
C3c
M3b C3b
M3c
C3aM3a
O3](https://image.slidesharecdn.com/presentationukes2015v7-150531175855-lva1-app6892/85/Presentation-ukes-2015-v7-10-320.jpg)
Presentation ukes 2015 v7
- 1. Balancing rigor, inclusiveness and feasibility Learnings from the design of a participatory impact evaluation of the IFAD-funded RTIMP in Ghana. Glowen Kyei-Mensah (PDA) Adinda Van Hemelrijck & Irene Guijt (IFAD & BMGF) UKES May 2015 London
- 2. Impact Evaluation under IFAD9 Attribution of complex government programs? Participation of people? Learning with partners? 80m people out of poverty?
- 3. Improved learning initiative for the piloting of a Participatory Impact Assessment & Learning Approach (PIALA) ✔ Almost ✔ $ 90 K $ 260 K DBRP RTIMP
- 4. Objectives Assessing to what extent impacts occurred (or not) Debating how impacts can be enhanced Explaining why impacts occurred (or not) LEARNING 1. Produce rigorous qualitative and quantitative evidence for global reporting and advocacy 2. Facilitate inclusive analysis and reflection for collaborative learning 3. Generate a scalable model for strengthening IFAD’s self-evaluation system & Purposes
- 5. Design challenges • Serving the different purposes with limited budgets • Understanding the typical attributes of IFAD- funded government programs • Generating solid conversation about “what has worked for whom, under which conditions and why?”
- 6. PIALA Quality Assurance Framework 1. Focus & frame the evaluation 2. Describe & link changes 3. Identify causes 4. Manage quality 5. Synthesise findings 6. Analyse & debate contributions 7. Report & share Rigour Inclusiveness FeasibilityEvaluation Design http://www.participatorymethods.org/authors/adi nda-van-hemelrijck-and-glowen-kyei-mensah
- 7. 1. Focusing & framing the evaluation • Upfront discussion with sponsors about design options (scope & scale) and budgets • Reconstruction of Theory of Change based on desk review and consultations • Design workshop with national stakeholders for identifying – Impact & contribution claims – Core assumptions – Evaluation & learning questions
- 8. M2b: Training & starter pack for commercial seed growers to multiply certified R&T seeds C3a: R&T processors grow and develop into GPCs that are profitable enterprises O3: Enhanced R&T processed volumes of high quality at scale O2: Enhanced R&T productivity and production at scale M2c: Farmer Field Forums (FFF) engage farmers, extension agents and researchers in developing, demonstrating and promoting appropriate R&T production technologies C2a: Resource-poor R&T farmers & seed producers gain access to and adopt improved R&T seed varieties, technologies & inputs to improve crop husbandry, soil fertility and pest management practices C2b: Resource-poor R&T farmers organise and register as FBOs that can access credit and bargain better market prices C1b: Resource-poor R&T processors, farmers & seed producers commercialize and establish effective supply chain linkages C1a: R&T supply chain farmers & processors are capable of developing and implementing viable business and marketing plans C3c: R&T supply chain farmers and processors gain access to business financing and market-linking services M3b: Subsidized upgrading of advanced R&T processors into Good Practice Centres (GPCs) that demonstrate and promote good quality processing & management practices C3b: R&T supply chain processors gain access to and adopt standardized processing technology and good quality management practices O1: R&T supply chain actors effectively solve their supply & demand issues and timely obtain technical support, resulting in sustainable and inclusive CCs linked to old and new markets I2: Improved R&T- based livelihoods for the rural poor in CC catchment areas M2a: R&D for developing bio-agents M1c: Information, Education & Communication (IEC) about CC support services, inputs and technologies M1a: Training of resource-poor farmers and processors involved in the R&T supply chains in business development and marketing M3c: Co-financing of R&T supply chain farmers and processors by matching 40% RTIMP funds with 50% loans from PFIs and 10% self-financing through the Micro-Enterprise Fund (MEF) M3a: Training of artisans to produce and maintain standardized processing equipment for R&T supply chain processors and GPCs I1: Rural poor people in CC catchment areas have increased access to food & income to sustain an active and healthy life M1: District Stakeholder Forums (DSFs) for addressing supply & demand issues and technical support needs of R&T supply chain actors members M1b: Supply Chain Facilitation (SCF) and market linking through the Initiative Fund (IF) EC1 EC3c EC1 EO2 EO1 EC3b MEF GPC DSF FFF Roots & Tubers Improved Marketing Program (30m)
- 9. Hypothesis: [Enhanced smallholder R&T production and processing at scale] + [Sustainable and inclusive CC market linkages] => Improvement livelihoods and poverty reduction in rural Ghana Assumption: · Livelihoods and poverty status in rural Ghana can be improved by commericializing smallholder R&T production and processing businesses combined with the establishment of competitive market-driven and inclusive CC linkages. Evaluation/learning question: · To what extent and for whom does this assumption hold true (or not) and under which conditions? Does it hold true for resource-poor women and youth/young adults in remote rural areas?What conditions need to be changed to enable women and young adults overcome cultural and socio-economic barriers? Causal link House-hold level Community cluster level District level Zonal & national level Sampling approach I2→I1 HH survey (with households in the the supply chain catchment area) Review of the 2010 Ghana Living Standard Survey report and other relevant secondary data Stratified sampling of 30 households from the community clusters in each sampled district O3+O2+O1 →I2 Generic change analysis (in gender/age-specific focus groups of community members from the supply chain catchment area) Review of RTIMP RIMS baseline ad other M&E data Stratefied sampling of community members from the community clusters in each sampled district O2 O1 I2 I1 O3 Impact claim – Poverty reduction
- 10. Contribution Claim 3 – Enhanced R&T processing Hypothesis: [Access to business financing] + [Demonstration of good practices] => Development of profitable processing enterprices by R&T supply chain farmers and processors => Enhancement and scaling of smallholder R&T processing, contributing to the growth of the R&T supply chains Assumptions: · Resource- poor processors and farmers who are well trained in quality management, business planning and marketing apply for matching grant funding (MEF) to invest in their businesses. PFIs are prepared to provide credit to well- trained resource- poor processors and farmers up to 50% of their planned investments. · GPCs can reach and teach resource- poor processors to develop more profitable agri- processing businesses by demonstrating good quality processing and management practices, including the use of improved technologies and stnadardized equipments. As a result, resource- poor processors apply to the MEF and invest in new technologies and equipment that help them to produce greater volumes of higher quality at lower cost. Evaluation/learning question: · To what extent and for whom do these assumptions hold true (or not)? · What conditions need to be in place for GPCs to become profitable and attractive businesses particularly for young adults living in remote areas?What supports or hinders GPCs to better link the supply chain farmers to old and new markets, and how is this influenced by the DSF? · Reach and added value of the MEF? Effects of the MEF on growth of the funded agro- processing businesses? Avoidance of elite capture? Causal link Household level Community cluster level Disrict level Zonal & national level Sampling approach M3b→ C3a+C3b→O3 Livelihood analysis (in gender/age-specific focus groups with supply chain farmers and processors) KIIs with GPCs Review of RTIMP and REP M&E data and supervision reports (incl. the 2014 thematic impact studies on MEF and GPC); the comparative case study on matching grant facilities Stratified sampling of supply chain farmers, seed growers and processors Stratified sampling of GPC- and non-GPC- participants (incl. MEF beneficiaries) M3b+M3c+C1a →C3c Constituent feedback (with mixed groups of (non-) GPC participants and MEF beneficiaries ) KIIs with BACs and PFI local branches C3c M3b C3b M3c C3aM3a O3
- 11. Selection of appropriate methods specific to the links and questions: • HH survey – correlation between changes in “access to food & income” and R&T livelihood changes and investments • PRA-based methods – causes of R&T livelihood changes and investments • SenseMaker lithe – patterns in experiences of “R&T livelihood changes” (400) • Constituent feedback – effects and reach of selected program mechanisms (DSF, FFF, MEF & GPC) 2. Describing & linking changes
- 12. 3 zones 8 regions 4 commodity chains 25 random districts 30 community clusters 150 Parti FGDs (with 1200 ppts, 45% women) 860 HH Surveys Parti Sensemaking (in 23 districts with 640 ppts; national with 100 ppts) 100 KIIs with officials, bankers, researchers, enterprises…
- 13. 6. Analysing & debating contributions • Configurational analysis and integrated QUAL-QUANT synthesis – Systematic collation of data from the different methods at district and aggregated levels (with 0-6 rating of strength of links and evidence for each cluster) – Analysis of patterns in the evidence resulting from different with/without configurations across districts for each cluster • Sensemaking involving stakeholders in a collective analysis and debate of evidence of impact and contributions
- 14. PIALA QAF: 2. Describe & link changes 3. Identify the causes 4. Manage quality Rigour Inclusiveness Feasibility Multi-stage sampling enabling comparative analysis of with/without configurations of program treatment Nested mixed-methods • consistent and equal QUANT & QUAL data collection relevant to the links & questions • complementarity of types of information • Triangulation of different sources & types of data Instant data entry and linking enabling on-site integrated analysis and sensemaking Data quality monitoring and process reflection every evening while doing data collation Field research capacity (nr of teams, time in the field, logistics & mobilization, supervision & quality assurance, time for instant data processing)• Classic HH survey • PRA-based methods • Testing of new tools (SenseMaker & CF) to overcome respondent & researcher bias • District sensemaking workshops • 8 regions, 3 zones • 30 clusters in 25 districts (propotional to CC size) • 30 HHs/ cluster (tot 900) • 40 participants/cluster (tot 1200, 45% women) • Process vs data (R + I) • Participation vs independence (R + I) • Scope vs depth (F + I) • Scale vs voice (R + I) Trade-offs under budget constraints?
- 15. Final notes Value for money? @ baseline mid & end? Replicability? Harvesting the best of all Involving policy- makers, donors & constituents Answering the questions Capturing variability in program treatment Capacity trumps all
Editor's Notes
- #3: IMI -> learning initiative International development desperate to ‘prove impact’ In context where IFAD had committed to 30 impact evaluations under the 9th replenishment “number of people lifted out of poverty” Extractive, for funders, statistical rigour Ignores participation Limited feasibility: how to evaluate complex multi-level and multi-actor systemic change programs with things like self-targeting mechanisms and interferance of many other players and fuders? both the IOE and SSD are struggling with finding appropriate approaches serving both learning and accountability purposes and capturing the complexity of the IFAD-funded projects Part of IFAD’s ‘Improved Learning Initiative’ generates rigorous, contested & debated evidence of project contributions to and explanations of rural poverty impact facilitates meaningful and equal participation of project stakeholders in collecting and analysing the evidence; presents a potentially scalable model for strengthening IFAD’s self-evaluation system Facilitate reflections with stakeholders at field, country and global levels on the quality of the PIALA in terms of: Rigour: consistency and reliability of methods, processes & evidence Utility: accessibility, credibility and value of methods to generate useful insights to influence decisions, processes and relations Feasibility: replicability, manageability and cost-effectiveness of methods and processes
- #4: IMI -> learning initiative International development desperate to ‘prove impact’ In context where IFAD had committed to 30 impact evaluations under the 9th replenishment “number of people lifted out of poverty” Extractive, for funders, statistical rigour Ignores participation Limited feasibility: how to evaluate complex multi-level and multi-actor systemic change programs with things like self-targeting mechanisms and interferance of many other players and fuders? both the IOE and SSD are struggling with finding appropriate approaches serving both learning and accountability purposes and capturing the complexity of the IFAD-funded projects Part of IFAD’s ‘Improved Learning Initiative’ generates rigorous, contested & debated evidence of project contributions to and explanations of rural poverty impact facilitates meaningful and equal participation of project stakeholders in collecting and analysing the evidence; presents a potentially scalable model for strengthening IFAD’s self-evaluation system Facilitate reflections with stakeholders at field, country and global levels on the quality of the PIALA in terms of: Rigour: consistency and reliability of methods, processes & evidence Utility: accessibility, credibility and value of methods to generate useful insights to influence decisions, processes and relations Feasibility: replicability, manageability and cost-effectiveness of methods and processes
- #6: (not performance): to generate solid conversation about critical issues related to “What works how, for whom, under which conditions and why?” indicators of rural poverty (nutrition, food & income, assets) and enablers (capitals, institutions, relations/processes) incl. WEIA & SLA contribution analysis of complex multi-causal interactions, using mixed-methods, recall, cross-validation & triangulation facilitation of group-based causal change mapping and analysis and cross-validation debates
- #7: Add Rainbow Framework visual
- #13: HH survey on food & income, community orgs membership, credit sources/use and training (gender-disaggregated) in 720 HHs (540+180) in 24 villages (18+6) sufficient for comparison with RIMS baseline (900 HHs) Participatory causal change mapping of livelihoods and institutional relationships (gender-specific), and wealth & wellbeing (gender-mixed) with ± 550 participants (390+130) in 8 villages (6+2) sufficient for causal explanation KIIs and FGDs on institutional capacity at communes, districts & province with ± 80 leaders & officials in 8 villages/communes (6+2), ± 15-20 officials in 3 districts, and ± 20 provincial and national officials Multi-stage cluster sampling From 2008 project population (26 communes/villages) From salt, brackish and fresh water agro-ecological zones (3 districts) Stratification according to distance (2km) to inter-communal road Random selection of 18 ‘focus’ and 6 ‘non-focus’ villages, and 30 HHs per village
- #15: Add Rainbow Framework visual
- #16: IMI -> learning initiative International development desperate to ‘prove impact’ In context where IFAD had committed to 30 impact evaluations under the 9th replenishment “number of people lifted out of poverty” Extractive, for funders, statistical rigour Ignores participation Limited feasibility: how to evaluate complex multi-level and multi-actor systemic change programs with things like self-targeting mechanisms and interferance of many other players and fuders? both the IOE and SSD are struggling with finding appropriate approaches serving both learning and accountability purposes and capturing the complexity of the IFAD-funded projects Part of IFAD’s ‘Improved Learning Initiative’ generates rigorous, contested & debated evidence of project contributions to and explanations of rural poverty impact facilitates meaningful and equal participation of project stakeholders in collecting and analysing the evidence; presents a potentially scalable model for strengthening IFAD’s self-evaluation system Facilitate reflections with stakeholders at field, country and global levels on the quality of the PIALA in terms of: Rigour: consistency and reliability of methods, processes & evidence Utility: accessibility, credibility and value of methods to generate useful insights to influence decisions, processes and relations Feasibility: replicability, manageability and cost-effectiveness of methods and processes