Not forgetting the forests for the trees: The art and science of useful impact evaluations
Download as PPTX, PDF0 likes761 views
The document discusses an impact evaluation presentation focusing on the effectiveness of protected areas in preventing deforestation and assessing land use trade-offs using econometric models and GIS data. It explores factors affecting agricultural expansion and deforestation, emphasizing the importance of understanding local contexts in policy evaluations. Key learnings include the necessity of qualitative work prior to data collection and the value of strong collaboration between researchers and implementing agencies.
Not forgetting the forests for the trees: The art and science of useful impact evaluations
- 1. Not forgetting the Forests for the trees - The art and science of useful impact evaluations Presentation at ICRAF Nairobi, 3rd September, 2015 Dr. Jo (Jyotsna) Puri: Head of Evaluation and Deputy Executive Director, Diana Lopez: Evaluation Specialist Stuti Tripathi: Senior Policy and Evidence Uptake Officer
- 2. Who, Why, What and Why – 3ie • 2008 • Largest grant giving organization for impact evaluations • All areas • Raison d’etre
- 3. Deforestation – do protected areas help? Assessing trade-offs
- 6. Variable 1: Protected Areas
- 7. Variable 2: Road building
- 8. Predicting the location of Deforestation
- 9. Predicting Location of Impacts of the two variables • Where is deforestation most likely to occur? Which areas are vulnerable? • Are Protected Areas Effective? • Hypothesis: Plots devoted to the most lucrative use. So plot is cleared if clearing it is more profitable than letting it stay forested. 9
- 10. The Econometric model • The econometric model that we estimate is thus given by • Zi : Plot attributes (Slope, Elevation, Impedance weighted travel time, Soil Dummy, Population density) • Y1i*: Net profit from clearing • Y2i*: Net utility from protecting a plot otherwise0;0* 2 if1 222 * 2 otherwise0;0* 1 if1 1121 * 1 i Y i Y i e i WB i Z i Y i Y i Y i e i YB i Z i Y
- 12. Data – A spatially explicit GIS with many layers • Roads data (1982) • Land Use (1:1m, LDD) • District level census data on population (1990) • Physiographic: Elevation (DEM), Slope • Soil (FAO) • Socio-Economic: Population density • Cost of Travel to the market; Roads (DCW and LDD) • Protected Areas (IUCN) 12
- 13. 13 Roads (1982) and Forests Of North Thailand (1986) NORTH THAILAND Elevat.shp Elevat.shp 1 - 1000 feet 1000 - 3500 feet 3500 - 6000 feet 6000 - 7700 feet Elevat.shp
- 14. Sampling • Grid squares of 100ms, 28 Million data points • Sampled at 5 km (spatial autocorrelation): 6550 sampled pixels. • Impedance weighted cost of travel (Costdistance in Arc/Info)
- 15. NORTH THAILAND Elevat.shp Elevat.shp 1 - 1 000 fe et 100 0 - 3 500 fe et 350 0 - 6 000 fe et 600 0 - 7 700 fe et Elevat.shp
- 16. 16 Cleared Land (Y1 = 1) T- Stats Slope (degrees) -0.088 -10.652 Elevation (ms.) -0.001 -8.095 Population density1990 (people/km2) 0.003 4.532 Log(cost) (1982)** -0.191 -9.729 Soil and Province Dummies Not Shown Protected Area dummy (1986) -0.077 -0.332 Constant 1.295 8.870 Protected Area (Y2 = 1) Equation Slope (Degrees) 0.034 5.297 Elevation (ms.) 0.001 9.058 Population density1990 (people/km2) 0.001 2.297 Log(cost) (1982) 0.192 7.477 Soil and Province Dummies Not Shown Watershed dummy 0.188 3.543 Constant -4.098 -14.010 Log Likelihood -3714.7 No. of observations 4946
- 17. Results • In a static model, protected areas are not very successful in preventing deforestation; Agnostic about Wildlife Sanctuaries. • Roads hasten clearing. • Model useful in predicting vulnerable areas.
- 18. Accuracy of Predictions Actual Predicted Cleared Forested % of Predictions Correct Cleared 872 296 75% Forested 657 3133 83% % Correctly predicted 57% 91% 18
- 19. 19 Long way still....Khao Sanam Phriang Nam Nao Thung Salaeng Suang Predicted Threatened areas Wiang Kosai Mae Yom Chae Son Areas predicted to clear after cost is reduced by 100 units
- 20. Exploring the potential for using GIS/Big data: What affects agricultural expansion?
- 24. What affects Deforestation in the area? • Assessing impact of population and road bulding on the agricultural frontier Forest Reserves. • Uses panel data over 11 years (1986-1996) 24
- 25. Data Problems • Village level data • Unbalanced data set (Selection bias in villages?) • Ambiguous property rights • Groups of crops tracked. No consistent data on crop area. • Short run crop (soybean maize tobacco peanut) • Long run crop (upland rice, tea, tree crops) • Paddy rice • Self-reported data: Confounded zeros and missing values • Data on prices of crops
- 26. The Econometric Model • Log (Land devoted to crop i by village j in year t )= f(Pjt, Tjt , Wjit, Git, Nit, Lit , Ait, t) • Where • Pjt : Population of village j at time t • Tjt : Time taken to travel by most favored mode from village j to the market at time t • Ajt: Proportion of adults in village j at time t • Wjit : Availability of water in village j at time t for crop i • Gjt : Use of credit in village j at time t • Njt : Land productivity in village j at time t • Ljt: Status of property rights in village j at time t • t : time trend
- 27. Ag area Coefficient Intensity of cultivation Year 15.35** 0.0379 Area devoted to Paddy Rice 0.46** 0.01399** Area devoted to Upland Rice 0.21* 0.0011 Area devoted to Soybean -0.19+ 0.0037* Constant -641.17* 63.314** Sigma-u 1054.16 14.0611 Sigma-e 375.89 16.392 0.89 0.4239 Observations 1979 2174 R-square Within 0.042 Groups 622 629 R-square Between 0.054 R-square Overall 0.056 Gaussian Wald Statistic (chi2, 4df) 85.5 20.03 Prob > Chi2 0 0.0005 Results: Agricultural area & Intensity
- 28. Results •Overall, ease in access led to a substitution between upland rice and paddy rice •Upland rice area decreased. •Environmental AND livelihood benefits!
- 29. Conclusions • Magnitudes of policy instruments are clearly important to judge trade-offs. • The ‘where’ and the ‘how’ specially important. • Crop wise exploration is important. • Would have been excellent to have satellite and ground truthed data on • verifiable soil, • crop and • agricultural land data • Boundaries of forest reserves and location.
- 30. Promoting Agricultural Innovation in Sub-Saharan Africa and South Asia: an Impact Evaluation Program Challenges and Learnings ICRAF-Nairobi. September 3rd 2015
- 31. General Features of the Interventions • Target at the farmer association level • Target supply and demand: multiple intervention arm
- 32. Common challenges that can arise • Complex: Difficulties in identifying the outcomes of interest • Measuring the synergies among the different treatment arms • Very likely to have to deal with contamination • Eligibility and placement of geographical areas and farmers is not always clear cut • Women are usually underrepresented in the farmer associations
- 33. Learnings (1) • Doing qualitative work prior to first round of data collection it is key in order to : • Better understand of the the change the intervention can have on the population of interest • Better define the groups on which the intervention should have a differential impact • Better identify the possible sources of contamination and possible channels of spillover effects
- 34. Learnings (2) • Doing a pilot survey is also key to: • Better design of the data collection instruments • Identifying the constraints farmers may have to participate in the program
- 35. Learnings (3) • A good team work between researchers and implementing agencies is key for the project to move and implement successfully.
- 36. Evidence uptake and use
- 37. Generating evidence that is useful Helps to answer a specific policy question Rooted in the local context Opportunity exists for evidence to be used Message gives solutions (that can be drawn from the evidence) Affordable, logistically possible, politically feasible RELEVANT CONTEXTUAL DEMAND DRIVEN CLEAR FEASIBLE
- 38. Evidence uptake in 3ie-funded studies 0 2 4 6 8 10 12 14 16 18 Expand successful progs Close unsuccesful progs Change pol/prog design Inform discussions of pol/prog Inform design of other progs Inform global poldiscussions Improved culture of use of evidence
- 39. Expand successful programmes Wage subsidy programme in South Africa
- 40. 3ie’s ToC for evidence uptake 3ie funds high- quality policy relevant studies Researchers engage with key stakeholders Results in uptake of study findings We get applications - that ask policy relevant question - Comprises of a study team that has demonstrated experience in stakeholder engagement ASSUMPTIONS Researchers have the - Willingness and commitment - Understand how evidence uptake happens - Have the tools and resources to invest in stakeholder engagement Study makes policy relevant recommendations that are - Evidence based - Answer what works and why - Propose feasible solutions given context and cost ASSUMPTIONS ASSUMPTIONS Grant management is about making sure that the assumptions hold!!
- 41. 3ie’s ToC for evidence uptake 3ie funds high- quality policy relevant studies ASSUMPTIONS We get applications - that ask policy relevant question - Hold potential for policy impact - Comprise a study team that has demonstrated experience in stakeholder engagement Institution of preparation phase to facilitate context understanding and stakeholder engagement Increased weightage to policy relevance and impact How is this ensured
- 42. 3ie’s ToC for evidence uptake 3ie encourages researchers to engage with key stakeholders ASSUMPTIONS Researchers have Willingness and commitment - Understand how policy influence happens - Have the tools and resources to invest in stakeholder engagement Developing a plan and setting aside a budget for engagement Encouraging involvement of in country researchers Theory based evaluations– what works and why How is this ensured
- 43. 3ie’s ToC for evidence uptake Results in uptake of study findings Study makes policy relevant recommendations that are - Evidence based - Answer what works and why - Propose feasible solutions given context and cost ASSUMPTIONS Ongoing engagement with 3ie more dynamic model of interaction End of project interviews to learn and adapt processes How is this ensured
- 44. Ensuring quality and achieving relevance to users Use multiple prongs: integrated and regular use of many communication channels: social media, media, listservs, events, meetings, website Early Widely Often Establish champions
- 45. Thank you Jyotsna (Jo) Puri jpuri@3ieimpact.org