Monitoring And Evaluation For World Bank Agricultural Projects

The World Bank

Monitoring and
Evaluation for
World Bank Agricultural
Research and
Extension Projects:
A Good Practice Note

Dr Malik Khalid Mehmood
PhD

© 2005 The International Bank for Reconstruction and Development / The World Bank

1818 H Street, NW

Washington, DC 20433

Telephone 202-473-1000

All rights reserved.

The findings, interpretations, and conclusions expressed herein are those of the author(s) and do not necessarily reflect
the views of the Board of Executive Directors of the World Bank or the governments they represent.

The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denomina-
tions, and other information shown on any map in this work do not imply any judgment on the part of the World Bank
concerning the legal status of any territory or the endorsement or acceptance of such boundaries.

Rights and Permissions

The material in this work is copyrighted. Copying and/or transmitting portions or all of this work without permission
may be a violation of applicable law. The World Bank encourages dissemination of its work and will normally grant
permission promptly.

For permission to photocopy or reprint any part of this work, please send a request with complete information to the
Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-
750-4470, www.copyright.com.

All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher,
World Bank, 1818 H Street NW, Washington, DC 20433, USA

ii

CONTENTS

ACRONYMS AND ABBREVIATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
PREFACE AND ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
1. CHALLENGES FOR M&E SYSTEMS IN AGRICULTURAL RESEARCH
AND EXTENSION PROJECTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2. THE WORLD BANK RESULTS FRAMEWORK REQUIREMENTS . . . . . . . . . . . . . . . . 6
3. DATA COLLECTION, REPORTING, AND DISSEMINATION REQUIREMENTS . . . . 16
4. PARTICIPATORY MONITORING AND EVALUATION. . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5. EVALUATION OF OUTCOMES AND IMPACT IN THE WORLD BANK. . . . . . . . . . . . 45
6. ECONOMIC EVALUATION OF COMPETITIVE GRANTS SCHEMES. . . . . . . . . . . . . 51
7. CONCLUDING REMARKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8. REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Appendix 1. Narratives of New Approaches and Instruments in ARE Systems . . . . . . . . . . . 67
Appendix 2. Implementation of a Monitoring and Evaluation Plan in the
Bank's Project Cycle: Putting It All Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Appendix 3. A Brief Description of the "Extended" Logical Framework . . . . . . . . . . . . . . . . 75
Appendix 4. List of Indicators and Data Sources for ARE Projects . . . . . . . . . . . . . . . . . . . . 77
Appendix 5. Project Monitoring and Evaluation Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Appendix 6. Economic Evaluation of Competitive Research Grants:
The Case of PROMSA, Ecuador . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Tables, Boxes, and Figures
TABLE 1.1: COMPLEMENTARY ROLES FOR MONITORING AND EVALUATION . . . . . . . . . . . . . . . . . . . . . 2
TABLE 2.1: DIFFERENCES BETWEEN THE WORLD BANK LOGICAL AND RESULTS FRAMEWORK . . . . . . . 6
TABLE 2.2: THE WORLD BANK'S RESULTS FRAMEWORK APPLIED TO AN ARE PROJECT . . . . . . . . . . . 7
TABLE 2.3: DEFINITIONS AND EXAMPLES ON AGRICULTURAL RESEARCH AND EXTENSION
PROJECT INDICATORS IN THE WORLD BANK LOG FRAME AND RESULTS FRAMEWORK . . . 11
TABLE 2.4: EXAMPLES ON OUTCOME INDICATORS DEPENDING ON THE PROJECT DEVELOPMENT
OBJECTIVE AND THE PERCEPTIONS OF THE DIFFERENT STAKEHOLDER GROUPS
INVOLVED IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS. . . . . . . . . . . . . . . . 12

iii

TABLE 3.1: ARRANGEMENTS FOR OUTCOME MONITORING FOR "CHINA AGRICULTURAL
TECHNOLOGY TRANSFER PROJECT" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
TABLE 3.2: THE MAJOR DATA SOURCES—MANAGEMENT INFORMATION SYSTEM,
EXISTING AND REGULARLY MAINTAINED DATABASES AND VARIOUS
SPECIAL STUDIES AND THE MAIN DATA COLLECTION METHODS . . . . . . . . . . . . . . . . . . 31
TABLE 4.1: MAJOR DIFFERENCES BETWEEN CONVENTIONAL AND PARTICIPATORY M&E . . . . . . . . . . 33
TABLE 4.2: DEGREE OF PARTICIPATION AT DIFFERENT STAGES OF PME PROCESS. . . . . . . . . . . . . . . 35
TABLE 6.1: KEY REFERENCE SOURCES ON THE THEORY AND EMPIRICAL TOOLS FOR ASSESSING THE
ECONOMIC IMPACT OF RESEARCH AND EXTENSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
TABLE 6.2: STATISTICAL REGRESSION MODEL FOR ISOLATING PROJECT EFFECTS
IN IMPACT EVALUATION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

TABLE 6.3: CALCULATION OF IRR FOR AN AGRICULTURAL RESEARCH AND EXTENSION PROJECT . . . 58
TABLE 6.4: SOURCES OF INFORMATION ON THE MINIMUM SET OF PARAMETERS
TO BE ESTIMATED FOR A SOUND ECONOMIC IMPACT EVALUATION . . . . . . . . . . . . . . . . . 60

TABLE A1.1: POTENTIAL OF SELECTED INNOVATIVE INSTRUMENTS TO ADDRESS
VARIOUS FISCAL AND INSTITUTIONAL CHALLENGES ASSOCIATED WITH
AGRICULTURAL RESEARCH AND EXTENSION SYSTEMS. . . . . . . . . . . . . . . . . . . . . . . . . 68
TABLE A3.1: THE "EXTENDED" LOGICAL FRAMEWORK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 TABLE
A5.1: A PROJECT COMPONENT TABLE FROM COSTAB 32. . . . . . . . . . . . . . . . . . . . . . . . . . . 90

BOX 1.1: WORLD BANK REQUIREMENTS FOR PROJECT AND PROGRAM
MONITORING AND EVALUATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
BOX 1.2: KEY FEATURES OF IMPLEMENTATION-FOCUSED VERSUS RESULTS-BASED MONITORING . . . . 3
BOX 1.3: FOCUS ON INSTITUTIONAL DEVELOPMENT OR PRODUCTIVITY IMPACTS. . . . . . . . . . . . . . . . . 4 BOX
1.4: COMPETITIVE RESEARCH GRANT PROGRAM IN ECUADOR . . . . . . . . . . . . . . . . . . . . . . . . . . 5 BOX 2.1:
PARTICIPATORY INDICATOR IDENTIFICATION IN AN EXTENSION PROJECT IN MEXICO. . . . . . . 11 BOX 2.2:
TQQ OF DEFINING INDICATORS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 BOX 2.3: THE
IMPLEMENTATION STATUS AND RESULTS (ISR) REPORT QUALITY CHECKLIST. . . . . . . 15
BOX 3.1: MANAGEMENT INFORMATION SYSTEM (MIS) PREREQUISITES FOR
AN EFFECTIVE M&E SYSTEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

BOX 3.2: TECHNOLOGY OPTIONS IN DEVELOPMENT OF MANAGEMENT INFORMATION SYSTEM. . . . . . 20
BOX 3.3: KEY FEATURES OF THE MANAGEMENT INFORMATION SYSTEM (MIS) FOR
THE MODERNIZATION PROGRAM FOR A GRICULTURAL SERVICE PROJECT (PROMSA)
IN ECUADOR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

BOX 3.4: PROJECT INFORMATION MANAGEMENT SYSTEM NETWORK OF INDIA'S NATIONAL
AGRICULTURAL TECHNOLOGY PROJECT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
BOX 3.5: SCENARIO ON HARMONIZING M&E IN NATIONAL SECTOR MINISTRIES . . . . . . . . . . . . . . . 24 BOX
3.6: WHAT TO DO WHEN THE PROJECT DOES NOT HAVE BASELINE DATA?. . . . . . . . . . . . . . . 27 BOX 3.7:
RURAL WOMEN'S EMPOWERMENT THROUGH AGRICULTURAL EXTENSION IN VENEZUELA . . 29

iv

BOX 3.8: USING INTERVIEWS AND FOCUS GROUP DISCUSSIONS AT MID-TERM EVALUATION OF A
FARMER-LED EXTENSION SERVICE PROVISION PROJECT . . . . . . . . . . . . . . . . . . . . . . . . . . 30
BOX 4.1: BASIC STEPS IN PARTICIPATORY ACTIVITY EVALUATION . . . . . . . . . . . . . . . . . . . . . . . . . . 37 BOX
4.2: USING CONVERSATIONAL INTERVIEWS WITH FIELD-FOCUSED APPROACH . . . . . . . . . . . . . . 38 BOX 4.3:
MAIN TOOLS AND METHODS TO USE IN PME FOR ARE PROJECTS . . . . . . . . . . . . . . . . . 41
BOX 4.4: LESSONS LEARNED FROM THE UGANDA NATIONAL AGRICULTURAL
ADVISORY SERVICES PROJECT'S (NAADS) PME PILOT . . . . . . . . . . . . . . . . . . . . . . . . . 42
BOX 5.1: SPECIFIC HIGHLIGHTS FOR M&E DURING ICR PREPARATION. . . . . . . . . . . . . . . . . . . . . . 46
BOX 5.2: POTENTIAL FOCUS FOR EVALUATION IN AGRICULTURAL RESEARCH AND EXTENSION
PROJECTS/PROGRAMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
BOX 5.3: WHEN TO USE INTERNAL AND EXTERNAL REVIEWERS AND EVALUATORS?. . . . . . . . . . . . . 49
BOX 6.1: USING RANDOM SAMPLES IN EVALUATION OF IMPACT OF COMPETITIVE GRANT SCHEMES . . . 54
BOX 6.2: ESTIMATING INTERNAL RATE OF RETURN FOR SUBPROJECTS IN COMPETITIVE
GRANT SCHEMES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
BOX 6.3: SUSTAINABLE FINANCING OF A COMPETITIVE FUND IN ECUADOR . . . . . . . . . . . . . . . . . . . 61 BOX
A1.1: AGRICULTURAL INNOVATION SYSTEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

FIGURE 6.1: ESTIMATION STRATEGY IN PRONATTA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
FIGURE A2.1: DIAGRAM OF M&E STEPS IN THE PROJECT CYCLE . . . . . . . . . . . . . . . . . . . . . . . . . 74

v

ACRONYMS AND ABBREVIATIONS
ABF Annual Benefit Flow
ACF Annual Cost Flow
APL Adaptable Program Loan
AIS Agricultural Innovation System
ASTI Agricultural Science & Technology Indicators
BA Beneficiary Assessment
BP Bank Procedure
CAS Country Assistance Strategy
CDB Country Database
CGIAR Consultative Group on International Agricultural Research
CGP Competitive Grant Project
CGS Competitive Grant Scheme
CIAL Local Agriculture Research Committee
CIMMY International Maize and Wheat Improvement Center
T Competitive Research Grant Program
CRGP Competitive Research Grant Scheme
CRGS Expert Review
ER Food and Agriculture Organization
FAO Farmer Field School
FFS Global Database
GDB Gross Domestic Product
GDP Global Environment Facility
GEF Implementation Completion Report
ICR Information and Communication Technology
ICT International Development Association
IDA International Fund for Agriculture Development
IFAD International Food Policy Research Institute
IFPRI Increment in Net Benefits
INB Internal Rate of Return
IRR International Service for National Agricultural Research
ISNAR Implementation Status Report
ISR Modernizing Agricultural Knowledge and Information Systems
MAKIS Monitoring & Evaluation
M&E Management Information System
MIS National Agricultural Advisory Services
NAADS National Agricultural Technology Project
NATP

vi

NGO Non-governmental Organization
NRI Natural Resources Institute
PDB Project Database
OD Operational Directive
OED Operations Evaluation Department
OP Operational Policy
PRSP Poverty Reduction Strategy Paper
PAD Project Appraisal Document
PCN Project Concept Note
PCU Project Coordinating Unit
PCR Project Completion Report
PDO Project Development Objective
PIMSNET Project Information Management System Network
PIU Project Implementation Unit
PME Participatory Monitoring & Evaluation
PROMSA Modernization Program for Agricultural Services
PSR Project Supervision Report
PTD Participatory Technology Development
QAG Quality Assurance Group
R&E Research & Extension
RD Recommendation Domain
R&D Research & Development
SS Special Studies
TQQ Time, Quality and Quantity
TT Task Team
TTL Task Team Leader
UEFC Competitive Fund Management Unit
UOA Unit of Analysis

vii

PREFACE AND ACKNOWLEDGEMENTS
This Monitoring and Evaluation Good Practice Note was prepared by Dr Malik Khalid Mehmood. The
author would also like to thank Greg Traxler from Auburn University and Gustavo Sain from the Inter-
American Institute for Cooperation on Agriculture (Economic Impact Analysis), and World Bank
Consultants Remileku Rakey (M&E in the World Bank Project Cycle), Ghazali Raheem (M&E costs),
and Lawrence Salmen (Beneficiary Assessment) for their contribu-
tions to this report.
The author would also like to thank reviewers Jody Kusek (AFTKL), Gary Alex (USAID), Jim
Hanson (Maryland University), Doug Horton (CGIAR), Willem Janssen (SAR), Indira Ekanayake
(LCR), Richard Chisholm (EAP), Matthias Grueninger (MNA), David Nielson (AFR), and Derek
Byerlee (ARD) for their time and useful comments.
The World Bank's Sustainable Agriculture Systems Knowledge and Institutions (SASKI) Thematic
Group is acknowledged for its financial support.

viii

EXECUTIVE SUMMARY
Monitoring and Evaluation (M&E) are integral tools for managing and assessing the efficiency and
effectiveness of investments in agricultural research and extension (ARE) systems. However, monitoring
and evaluation of ARE project performance, outcomes, and impact has been a significant challenge.
Moreover, the increased focus of donors and borrowers on impact has resulted in a high
demand for expertise in M&E.
The objective of this Good Practice Note is to assist Task Teams of the World Bank and their col-
leagues in the client countries develop and implement effective M&E systems for ARE projects and
programs. The chapters in this Note provide a step-by-step guide for achieving that objective, with
emphasis on the World Bank M&E requirements and the specific nature of ARE projects.
Chapter 1 titled "Challenges for M&E Systems in Agricultural Research and Extension Projects" and
Appendix 1 titled "Narratives of New Approaches and Instruments in ARE Systems" discuss the M&E challenges
in ARE projects; and provide additional information on the forces driving the changes in ARE
systems as well as on the typical approaches and instruments applied in ARE systems and projects.
The focus of M&E in the Bank has shifted from monitoring implementation to tracking results.
Results-based systems build upon and add to traditional implementation-focused systems emphasizing project
outcomes. Since 2003, the Bank has used the Results Framework, which is a simplified version of the
traditional log frame focusing on (a) the Project Development Objective (PDO) and (b) intermediate
outcomes expected from implementing each individual project component, which
all in turn contribute to the achievement of the PDO.
The Note emphasizes the World Bank's M&E Requirements at the design stage, namely the intro-
duction of Results Framework and the associated requirements in the Project Appraisal Document (i.e., the
final product of a successful project planning process). Chapter 2 describes the key issues in the Results
Framework which are (a) development of a clear PDO and Project Components, and (b) the importance of
developing measurable and realistic Project Outcome and Intermediate Outcome indicators. In addition,
Appendix 2 summarizes the Bank's M&E steps throughout the Project Cycle and Appendix 3 provides
further details and references on the Logical Framework. A list of indicators, with emphasis on institutional,
social, and environmental sustainability, and data
sources for ARE Projects are provided in Appendix 4.
After designing the Results Framework and selecting the key performance indicators, the next step is
to specify the M&E arrangements for data collection, reporting, dissemination, and use for decision- making.
Chapter 3 presents the Results Monitoring Arrangements. Effective M&E requires the development of
concise M&E Plan (see section 3.1 as presented in the PAD), the establishment of a management
information system (MIS), an internal system for collection, analysis, storing, and dissemination of pertinent
information (see section 3.2) and good institutional and human capacity to

ix

implement the required actions (see section 3.3). Finally, sections 3.4 and 3.5 emphasize the impor-
tance of data collection and use (including baseline), list helpful data sources, describe methodology for data
collection and analysis, and indicate how much a good M&E system costs. Additional infor-
mation is provided in Appendix 4.
Participatory monitoring and evaluation (PME) is gaining importance in the development and mon-
itoring of demand-driven ARE systems and projects. The basic idea in PME is to allow active
involvement of key stakeholders in the M&E process in order for them to learn about and affect the process
and impact of a development intervention. Participatory M&E can be used in concert with traditional M&E
but should be considered a different process. If resources are limited, it is better to identify carefully when
and how to apply PME rather than sacrifice the quality of the process and
results generated. Chapter 4 provides further information on PME in ARE projects.
Evaluation is the periodic assessment of the relevance, performance, efficiency, and impact (both
expected and unexpected) of a development intervention in relation to stated objectives. Evaluation measures
achievements in relation to institutional policies, Bank-wide program objectives, and the goals set for each
operation. Evaluation at the Bank has two major dimensions: (a) self-evaluation that consists of an interim
evaluation and a terminal evaluation undertaken at the end of a project (requirement for the Project
Completion Report); and (b) independent impact evaluation usually undertaken by Operations Evaluation
Department several years after final disbursement; which measures changes attributable to the project in
terms of both direct and indirect causality. Chapter 5 presents the Bank evaluation requirements and evaluation
of outcomes and impact. Chapter 6 highlights the key steps in the economic impact evaluation of
Competitive Grants Schemes (additional
case study in Appendix 5) and describes the fiscal analysis recommended for ARE projects.

x

1. CHALLENGES FOR M&E SYSTEMS IN
AGRICULTURAL RESEARCH AND EXTENSION PROJECTS
1.1 PURPOSE OF THE GOOD PRACTICE NOTE

High rate of growth in agricultural productivity is essential to promote broad-based economic
growth, reduce poverty, and conserve natural resources. In turn, productivity growth is based largely on
application of science, technology, and information provided through national research and development
systems and various extension and advisory services. Within this context, the success of rural development
projects and programs has been shown to depend largely on direct stakeholder
involvement in planning, implementation, and evaluation (Agriculture Investment Sourcebook).
Monitoring and evaluation (M&E) are integral tools for managing and assessing the efficiency and
effectiveness of investments in agricultural research and extension (ARE) systems. However, monitoring
and evaluation of ARE project performance, outcomes, and impact has been a significant challenge.
Moreover, the increased focus of donors and borrowers on impact has resulted in a high
demand for expertise in M&E.
To meet this challenge, the objective of this Good Practice Note is to assist Task Teams of the World
Bank and their counterparts in client countries develop and implement effective M&E of ARE pro-
gram performance, outcomes, and impact.1
Chapters 1 through 6 provide a step-by-step guide for developing an M&E system for ARE projects.
1. Challenges for M&E Systems in Agricultural Research and Extension Projects
2. The World Bank Results Framework Requirements
2.1. From the Log Frame to the Results Framework
2.2. Developing the Results Framework (including selecting indicators)
3. Data Collection, Reporting, and Dissemination Requirements
3.1. Monitoring and Evaluation Plan
3.2. Project Management Information System
3.3. Institutional and Human Capacity in Project M&E
3.4. Data Sources and Methodology for Data Collection and Analysis
3.5. Monitoring and Evaluation Costs
4. Participatory Monitoring and Evaluation
5. Evaluation of Outcomes and Impact in the World Bank
6. Economic Evaluation of Competitive Grants Schemes

1

1.2 MONITORING AND EVALUATION
REQUIREMENTS IN WORLD BANK PROJECTS

Monitoring and evaluation systems are designed to inform project management (World Bank Task
Teams and client's project implementation unit) whether implementation is going as planned and whether
corrective action is needed to adjust implementation plans. In addition, M&E systems should provide evidence
of project outcomes and justify project funding allocations. The roles of monitoring and evaluation are
summarized in table 1.1. The sources and explanations for the World Bank M&E requirements, as determined
in the Operational Manual, are briefly explained in box l.1. In addition,
the World Bank project cycle and the associated M&E requirements are summarized in Appendix 2.

1.3 EMPHASIZED FOCUS ON RESULTS IN THE WORLD BANK

The focus of M&E has shifted from monitoring implementation to tracking results. Traditionally
M&E systems were implementation-focused and included tracking of input mobilization, activities
undertaken and completed, and outputs delivered. However, the implementation-focused approach

Table 1.1: Complementary roles for monitoring and evaluation

Monitoring Evaluation
• Routine collection of information • Analyzing information
• Tracking project implementation progress • Ex-post assessment of effectiveness and impact
• Measuring efficiency • Confirming project expectations
• Measuring impacts
• Question: "Is the project doing things right?" • Question: "Is the project doing the right things?"
Source: Alex and Byerlee (2000).

BOX 1.1: WORLD BANK REQUIREMENTS FOR PROJECT AND PROGRAM MONITORING AND EVALUATION

The World Bank Monitoring and Evaluation requirements of both program and project lending are pre-
sented in the Operational Directive 10.70 and Operational Policy (OP)/Bank Procedure (BP) 13.05.
The Operational Directive (OD) 10.70 defines Monitoring as "the continuous assessment of project
implementation in relation to agreed schedules and the use of inputs, infrastructure, and services by project
beneficiaries. Evaluation is defined as "the periodic assessment of the relevance, performance,
efficiency, and impact (both expected and unexpected) of the project in relation to stated objectives."
In addition, OD 10.70 sets out the concept of M&E, and provides general guidance on the design and
implementation of the information systems required for M&E activities.
The OP 13.05 explains the rationale for Bank's supervision of Bank-financed projects, which include
monitoring and evaluative review and reporting. It also explains the responsibilities of Task Teams (TT), and
requires that the borrower and the TT agree on implementation arrangements which include M&E
arrangements, and use of appropriate performance indicators.
Source: World Bank Operational Manual

2

does not provide managers, stakeholders, or policy-makers with an understanding of failure or suc-
cess of the project in reaching the desired outcomes (Kusek and Rist 2004). Results-based systems build upon
and add to traditional implementation-focused systems, emphasizing project outcomes. Box 1.2 illustrates
some of the key features and differences between implementation- and results-
based M&E systems.

BOX 1.2: KEY FEATURES OF IMPLEMENTATION-FOCUSED VERSUS RESULTS-BASED MONITORING

Elements of Implementation Monitoring (traditionally used for projects):
• Description of the problem or situation before the intervention;
• Benchmarks for activities and immediate outputs;
• Data collection on inputs, activities, and immediate outputs;
• Systematic reporting on provision of inputs;
• Systematic reporting on production of outputs;
• Directly linked to a discrete intervention (or series of interventions); and
• Designed to provide information on administrative, implementation, and management issues as
opposed to broader development effectiveness issues.
Elements of Results Monitoring (used for a range of interventions and strategies):
• Baseline data to describe the problem or situation before the intervention;
• Indicators for outcomes;
• Data collection on outputs and how and whether they contribute toward achievement of
outcomes;
• Timelines expressed such as at mid-term and end-term;
• More focus on perceptions of change among stakeholders;
• Systematic reporting with more qualitative and quantitative information on the progress
toward outcomes;
• Done in conjunction with strategic partners; and
• Captures information on success or failure of partnership strategy in achieving desired outcomes.
Source: Kusek and Rist (2004).

1.4 SPECIFIC CHALLENGES FOR M&E
IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS

A workable system for monitoring and evaluation of project performance and especially outcomes
and impact is a challenge for all development programs. The particular problems facing project man-
agement in the development of an M&E system for ARE projects include:
1. The natural uncertainty of research outcomes and technology adoption and difficulties
in establishing timetables for technology development and dissemination. This is espe-
cially true for basic research with very long time lags and indirect pathways to impact;

3

2. The difficulties in measuring the diverse objectives of ARE programs, such as poverty
reduction, natural resource conservation, food security, export promotion, economic
growth, and reduction of social conflicts;
3. The problems related to cause-and-effect attribution of impact due to diverse external
factors (e.g., rural credit institutions, input supply systems, product marketing sys-
tems, macro-economic policies), which are outside of direct project control;
4. The lack of reliable national data and lack of awareness and managerial demand for
M&E data;
5. The lack of national capacity for M&E and low financial priority for M&E systems in the
context of low agricultural technology program funding and tight government budgets;
6. The dilemma of whether to develop an M&E system specific to the needs of the Bank
project, or to the broader needs of the (national) ARE program and institution(s). Both
are related and similar, but each has somewhat different requirements; and
7. What is the right balance in project M&E? Should the M&E system track both the
institutional development and the impact on productivity or prioritize based on the
cost of M&E? See box 1.3 for further discussion.

BOX 1.3: FOCUS ON INSTITUTIONAL DEVELOPMENT OR PRODUCTIVITY IMPACTS

Productivity impact is defined broadly to include any impact on the agricultural production system and
rural development, including environmental, economic, and social welfare.
Institutional development refers to the creation or the capacity of an institution to reflect systematically
and rigorously upon its role and function, and better enable them to carry out their responsibilities. It reflects an
attempt to introduce change and development in the way the institution is organized so that
it is better able to meet its mission.
Agricultural R&E projects typically seek productivity impacts, but these usually require institutional
development to bring about the desired impacts, and provide a basis for continued innovation and sustainable
impact on productivity (e.g., a multi-stakeholder priority setting mechanism for NARS). Investments into
institutional development, in turn, are not justified if they do not have productivity impacts. However, a focus
on quick productivity impact without the institutional development to sustain
it over time is not recommended.
The majority of Bank-supported projects combine both institutional and productivity impacts. Hence,
the inclusion of these two types of impacts needs to be considered when developing the Project
Development Objective (PDO) and the M&E system indicators.
Source: Alex and Byerlee (2000).

1.5 NEW APPROACHES AND INSTRUMENTS
IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS

The recent changes in the agricultural research and extension systems (i.e., decentralization, pri-
vatization, stakeholder participation, and increasingly separation of the functions of financing

4

and delivery), as a response to satisfy the changing global and local demands2 placed on the ARE
system, have resulted in a gradual transition from supply-driven technology generation toward building
a more demand-driven agricultural innovation system (AIS). Agricultural innovation system
essentially links multiple sources of innovation and uptake pathways along the continuum from basic
research to technology adoption recognizing the innovative capacity of all stakeholders from producers
to the marketplace. Thus, the focus is not only on the science suppliers but rather on the totality of actors
that are involved in innovation (see Appendix 1 for more details
on AIS).
Although the ARE systems have moved toward more responsive and effective AIS, there has also
been a shift to a clearer focus on poverty reduction which includes new partnerships for inclusion of
marginalized groups of people and an increased emphasis on co-financing with the private sector. In this
context, a number of instruments and approaches, such as contracting for services, competitive research grants
(CRGP), participatory and farmer-to-farmer R&E, and enhanced use of Information and Communication
Technology (ICT) (for details, see Appendix 1), are being introduced into many ARE systems. These
approaches and instruments are often used simultaneously. As an illustration box 1.4 describes the complex
nature of a CRGP arrangement in the Bank-supported Agricultural
Research Project in Ecuador.

BOX 1.4: COMPETITIVE RESEARCH GRANT PROGRAM IN ECUADOR

The Program for Modernization of Agricultural Services in Ecuador finances a competitive research grant
program (CRGP) that has funded 112 research and educational subprojects. The program has supported strategic,
applied, and adaptive research and education in a diverse range of thematic areas. These include: organic agriculture,
agro-forestry, agro-industry, variety development, management of plant genetic
resources, soil and water conservation, integrated pest management, animal production, and marketing.
The program introduced a new research culture and brought new organizations into the research system.
Research and educational subprojects are being conducted country-wide and are being executed by 45 different
public and private organizations. These include: public and private universities, NGOs, producer organizations,
private enterprises, and the Instituto Nacional Autonomo de Investigaciones Agropecuaris (INIAP).
Subproject research costs averaged US$ 116,000, of which 54 percent is financed by grants and 46 percent
by executing agencies. By leveraging co-financing for subprojects, the program helped to increase national
research funding by 92 percent, amounting to approximately
0.54 percent of agricultural GDP.
Source: World Bank (2004a). Agriculture Investment Sourcebook.

Additional M&E challenges resulting from the new implementation context include:
• A high level of pluralism—multiple partners, including farmers, farmers' organizations,
and various public and private sector actors;
• Multiple project components, particularly with CRGPs and multiple timelines for imple-
mentation;

5

• Non-formal communication and technology transfer routes, which are desirable but more
difficult to track;
• Highly decentralized decision-making and information flows; and
• Variable levels of capacity among partners involved in M&E.

6

2. THE WORLD BANK RESULTS FRAMEWORK REQUIREMENTS
2.1 FROM THE LOG FRAME TO THE RESULTS FRAMEWORK

In 2003 the Bank shifted from the log frame as a mandatory requirement in project M&E to a two-
pronged strategy comprising of the results framework and the arrangements for results monitoring.
The results framework is a simplified version of the traditional log frame focusing on (a) the Project
Development Objective (PDO) and (b) intermediate outcomes expected from implementing each individual
project component, which contributes to the overall achievement of the PDO. The added emphasis on component
outcomes helps focus project design and management on results. It also forces the M&E system track these
outcomes and impacts rather than simply reporting on production of outputs.
Table 2.1 summarizes the differences between the traditional log frame and the results framework.

Table 2.1: Differences between the World Bank Logical and Results Framework

World Bank Log Frame World Bank Results Framework
Goal: long-term, widespread — --
improvement in society
_ but only at
Outcome: intermediate effects of Project Development
outputs on clients Objective (PDO) level _ both at PDO
and component level

Outputs: products and services --
produced by the projects —

Activities: tasks personnel
undertake to transform inputs to — --
outputs

Inputs: financial, human, — --
and material resources

Note: - - = included in the framework and _= not included in the framework.
Source: Adapted from Binnendijk 2000.

Project Appraisal Document (PAD) is the final product of a successful project planning process. For
M&E related matters, the PAD requires the development of the results framework (as described in chapter
2 of this Note) and description of the M&E arrangements (in chapter 3). The Results
Framework has the following structure:

7

• Project Development Objective and Project Component Statements;
• Indicators for PDO outcome and intermediate component outcomes; and
• Explicit statement on how to use the outcome information.
The structure of the results framework as presented in the PAD and an example on a hypothetical
ARE project are shown in table 2.2.

Table 2.2: The World Bank's Results Framework Applied to an ARE Project

ProjectDevelopmentObjective OutcomeIndicators Use of Outcome Information
Statement of Project List of outcome indicators at Specify use or purpose of monitoring
Development Objective PDO level specific indicators and use of findings
(PDO level)
Agricultural producers increase For example, percent or number of If the rate of technology adoption by
the adoption of profitable and women and men producers that women and men lower than x% by the
environmentally sound have adopted environmentally specified target dates, the components
technologies sound technologies by year x 1 and 2 must be reviewed for efficiency

IntermediateOutcomes OutcomeIndicators
for each Component forComponents Use of Outcome Monitoring
Statement of outcomes List of outcome indicators Specify use or purpose of monitoring
per component per component specific indicators and use of findings
(PDO level)
Component 1. Extension service For example percent of women and If the rate of satisfaction is lower than
providers have an improved men producers satisfied with access x% by the specified target dates, the
understanding of client needs and to and quality of extension services activities under component 1 will be
how to respond to them adjusted as needed
Component 2. National research For example, percent of priority If the proportion is lower than
institutes have improved capacity research funds allocated to research envisioned, the situation will be
to conduct research in identified in priority areas. analyzed and corrected.
priority areas

Source: Authors.

2.2 DEVELOPING THE RESULTS FRAMEWORK

When developing the framework, pay attention to the following:

Develop a strong and clear project development objective
A strong and clear Project Development Objective is essential for good project design. The PDO is
expected to illustrate the principal outcome of a change in end-user behavior, (i.e., that of the project's
primary target group). One of the main problems identified in the Bank's project design has been lack of clarity
and focus in the PDO. PDOs are expressed either too high (e.g., at the Country Assistance Strategy level) or
too low (e.g., at the activity level) and sometimes are not well articu-
lated (e.g., mixing goals, objectives, and strategies).

8

Questions that help develop a clear and strong project development objective
(a)Who is the key recipient of project benefits?
(b)After the close of the project, what problem has been solved for the primary target group?
(c)What will the target group be doing differently after the project?

Pay close attention to the following:
• Focus on the outcome that the project can directly influence, given its duration, resources,
and approach;
• PDO should not encompass longer-term outcomes that depend on efforts outside the proj-
ect'sscope;
• Keep PDO short, focused, and measurable; and
• PDO is not a restatement of the project's components or outputs.
Practice: In ARE projects the primary target group usually consists of the agricultural producers. An
example of a typical Project Development Objective is "Agricultural producers increase the adoption of
profitable and environmentally sound technologies." Another example could be "Small-holder farmers in
rain-fed areas of Allura actively participate in the development of and use of research pro-
grams results."

Make sure that the project components contribute to achieving
the project development objective
The results framework presents a statement of the key project components, the success of which
determines the achievement of the PDO.
• To track the success of the different components, the expected outcomes from each individual
component (i.e., intermediate outcomes) need to be linked to the achievement of the PDO.
• The outcomes of each project component should clearly specify the immediate target group
that receives the benefits from the component implementation. Thus, they describe the
changes for the secondary target group or the skills or capacities the secondary target
group must have acquired to better serve the primary target group.
Practice: Agricultural R&E projects very often have several secondary target groups. A typical Bank-
supported ARE project would include research and extension staff and institutions as secondary target
groups. If the project improves their performance (e.g., through human capacity building or institutional
development), they will better serve the primary target group (e.g., small scale farmers
and women's groups).

Select indicators for the project development objective and each project component
Why indicators? Indicators are one of the crucial aspects of a project design. They are quantitative and
qualitative variables that provide a simple and reliable means to measure achievement, reflect changes
connected to an intervention, or help assess the performance of an organization against the stated targets.
Carefully selected indicators are needed to measure the success of achieving the PDO and the out-
come(s) of each project component. Specifically, indicators can help managers make informed and

9

timely decisions by way of a balanced presentation of financial and operational performance, and
consequent outcomes and effects; and a summary of complex information "at a glance." Tracking indicators can
provide continuous feedback and streams of data throughout the lifetime of the project (for more details see
section on performance indicators). Establishment of the baseline and tar-
get values is essential for tracking the achievements.

Different indicator levels and types
Indicators are derived from log frames and results framework and provide for measuring perform-
ance at two levels: (1) implementation performance indicators and (2) project impact indicators. See
Table 2.3 for definitions and examples of different indicator levels.
1. Implementation performance indicators measure program inputs and outputs. Tracking
input use and direct project accomplishments often requires multiple indicators, but
they are relatively easy to measure.
2. Project impact indicators measure outcomes and impacts on social goals. Outcome
and impact indicators are more difficult to measure, as they track the results of project
activities and outputs, but are influenced by many other factors. The Results
Framework requires only the description of outcome indicators.
It is important to make a distinction between output, intermediate, and outcome indicators.
• The indicator associated with the Project Development Objective describes a change in end-
user behavior (e.g., farmers' access to, use of, or satisfaction with goods and services) where-
as the indicator of each project component describes the changes for the secondary target
group (e.g., improved skills or capacity among staff of research and extension organizations).
• It is also important to make a distinction between an outcome and output. Outcomes are
intermediate effects of outputs on target group and thereby address the issue of uptake (i.e.,
coverage or adoption) of project outputs resulting from project activities.
• This difference is also reflected in the indicators. An outcome indicator could be "Percent
of trainees implementing new practices" but it would not be "number of trainings conduct-
ed." An illustrative list of key indicators, with particular focus on Competitive Research
Grant Schemes and contracting extension services, is included in Appendix 4.
Indicators must be defined for each step of the results framework (Project Development Objective
and component outcomes). If an extended log frame is developed, indicators must be defined for
goal, Project Development Objective, component outcomes, as well as outputs and inputs.

Key questions to address when selecting and developing indicators
Prior to defining the indicators, ask the following questions:
1. What is the development objective of the project?
The selection of indicators depends on the Project Development Objective and the
focus of the research and extension program (i.e., poverty focus, agricultural market
development, institutional development, productivity change, capacity building, and
environmental sustainability. Table 2.4 includes examples on outcome indicators from
the perspectives of different stakeholders.

10

Table 2.3: Definitions and Examples of Agricultural Research and Extension Project Indicators in the World Bank Log Frame and Results
Framework.
Productivity Oriented (Primary Institutional Development (Secondary
Level in Log Frame or Results Framework Target Group) Indicators Target Group) Indicators)
Sector Goal indicators measure change in the Goal level indicators are associated with Sector-level impact indicators are similar for
broad development goal to which the project agricultural technology. Indicators may include research and extension projects. These are
contributes. increased factor productivity, social rate of affected by many factors other than R&E
Changes in these indicators are indirectly return to investment, positive trends in programs, and indicators usually show response
attributable to the project, and usually cannot be environmental indicators, reduction in poverty only with a considerable lag time.
measured during the life of a Bank-financed rates, and improved nutrition.
project.
Outcomes are intermediate effects of outputs on Productivity outcomes may have a wide range of Institutional development outcomes vary
clients. They are closely related to the project possible indicators that reflect two results of depending on the technology system needs and
activities, but are not under full control of the program activities—change in behavior (i.e., the proposed project design. These may reflect
project. adoption) and initial effects of adoption. changes in the institutions themselves,
Outcomes reflect the quality of outputs Examples on behavior change include rates of institutional productivity, or institutional
produced, behavioral change in target groups as adoption of new technology, or new investments relations with clients.
well as changes in institutional performance due made possible because of new technology or Changes may be measured by new institutional
to the "adoption" of project outputs. management options. relationships, ability to address new problems,
In log frame, only Project Development Objective Initial effects of behavior change (adoption) may stakeholder attitudes, peer reviews of technical
programs, shift of funding, number of

11
has an outcome with associated indicator/s include yield increases, decreases in deforestation
whereas in the results framework, PDO and each or erosion, production cost decreases, or (for collaborative programs, efficiency of and client
component has an outcome and indicator(s). effects of policy reforms) further change in satisfaction with service delivery and changes in
technology adoption rates or investment. government policy based on research findings.
Component outcome indicators could be the total
number of certified extension agents per year or
client satisfaction with responsiveness of services.
Outputs are direct products of project activities Extension—Data often derived from program Extension—Workshops conducted to train
and are within the control of the project. reports and may include number of farmers extension staff, extension agents posted, and
Project output level M&E focuses on relatively trained, farmer questions answered, field day regional offices established.
simple quantitative measures of physical attendance, number of extension projects Research—Laboratories constructed, number of
completion. implemented, newspaper or magazine workshops to train research staff, and research
readership, and radio campaign listeners. staff trained.
Research—Research projects completed. Education—faculty trained, curricula developed,
Challenging as research activities may be numbers of graduates or person-months of
completed as planned, but completion says little training.
about the quality or significance of the work.
Budget and input include funding, works, goods Examples: funds disbursed, building Examples: funds disbursed, building
and services provided by the project. construction, laboratory equipment, and construction, laboratory equipment, and
technical assistance. technical assistance.

Source: Adapted from Alex and Byerlee (2000).

Table 2.4: Examples on Outcome Indicators Depending on the Project Development Objective and the
Perceptions of the Different Stakeholder Groups Involved in Agricultural Research and Extension Projects

Focus of Stakeholder in ARE project
theProject Donor— Borrower— Researchor Farmer or
Development World Ministry of Extension Academic Private Producer
Objective Bank Agriculture Agency Institution Sector Group
Poverty x% increase x% increase x% increase x% increase Sales of crops Increase in
reduction in poor small in poor smal in farm in farm increased by number of
holders' lholders' income income x% meals with
household real household meat or fish
income income
Productivity x% increase in x% increase x% of farmers Number of Input sales Staple food
change1 small holder in small have access to staff with increased x% yield increase
crop pro- holder crop improved improved by year x
duction and production varieties technical
farm income skills
Institutional Number of Number of Generation Number of x% of the Extension
development2 regions regions and transfer of staff in farmers have agents visit
covered by having good quality collaborative access to rural
good quality decentralized technologies ARE company communities
ARE services ARE services to x% of programs inputs and/or frequently
farmers services
Market Farmers' Farmers' Demand for Change in Number of x% higher
development access to access to high-value farmers' production price from
markets markets commodity ability to contracts with vegetables
improved by improved by research market farmer groups
x% x% increased products increased
Capacity x% increase in x% increase in x% increase High level of Number of x% increase
building farmer groups autonomous in partici- participation contracts with in farmer
participation in farmer patory among farmer groups group
decision- groups' technology farmers increased membership
making in the formation in development
area the area
Environmental x% of farmers x% of farmers x% increase in Number of x% of farmers Reduction in
sustainability adopting adopting generation of trainings to trained in safe pesticide
environ. sound environ. sound environmen- extension staff pesticide use applications
production production tally sound in environ. or increase in
practices practices technologies sound land under
practices no-till

Source: Riikka Rajalahti, World Bank.
Note: Bank-supported agricultural R&E projects typically focus on two objectives: (1) institutional development to strengthen the
institutions and technology transfer system necessary to develop and disseminate improved technologies and management
practices in the agricultural sector; and (2) productivity change due to technological innovation introduced through the technology
system.

12

2. Who are the stakeholders?
As evident from table 2.4, indicators need to be defined and monitored together with
the relevant stakeholders. Different stakeholders have different perceptions on desired
outcomes, which need to be tracked with different indicators.
3. What do we really need to know?
To save time and resources and to avoid confusion, it is important to carefully think
through the real and relevant information that the stakeholders will need. A good prac-
tice example on participatory stakeholder consultation and indicator selection in
farmer-to-farmer extension project is shown in box 2.1.

BOX 2.1: PARTICIPATORY INDICATOR IDENTIFICATION IN AN EXTENSION PROJECT IN MEXICO

In a farmer-to-farmer extension program in Mexico, the project team followed these steps to develop
indicators:
• Define broad indicator areas (based on PDO)
• Select currently used indicators for these areas
• Define stakeholder groups
• Select stakeholder groups to be consulted
• Develop indicators with different stakeholder groups
• Test the developed indicators across different stakeholder groups to assess their significance to
others and effectiveness at indicating change
• Agree on a priority list among indicator options
• Carry out fieldwork to gather data for indicators
• Create lists of indicators for full evaluation use with a focus on indicators with specific impor-
tance for different actors with a limit, (e.g., three key indicators for each stakeholder group)
The program team identified the range of different institutional and individual actors who affect and are
affected by the project. They then prioritized three stakeholder groups to be consulted for indicator
development in this trial phase: (a) farmers (participating and non-participating), (b) farmer-extension
agents and (c) funding agencies.
The research team initially proposed seven indicator areas. These were eventually narrowed down to four,
based on the groups' objectives: (1) changes to local, regional, political, and sectorpractice and policy (e.g., level of
dependence on external resources, involvement of local people, growth of local institutions, and changes in policy
and practice); (2) dissemination impacts: extension to other localities or regions (e.g., hor- izontal and vertical linkages
with other projects, agencies and NGOs beyond the region); (3) changes to the roles of the individuals in the project
(primarily the coordinator, outside advisors, immediate project participants, and NGO staff); and (4) changes in the
institutional structure (within and beyond the actual project).
Source: Baluert and Quintanar (2000) in IFAD 2003.

4. What is the most efficient method of collecting this information?
To save time and resources, find a balance between the information needs and the
efforts required to collect the information. Each indicator requires arrangements for
monitoring—an M&E plan; and inclusion in the Management Information System.
The effort expended should match the improvement gained in decision-making.

13

Not all potential indicators are practical. They may be either too difficult or too expensive to gather
the data. For example, measuring actual rural income levels may be difficult or costly. Instead, proxy indicators3
such as the number of community bicycles or the number of televisions may be a more practical, if less
precise, indicator of the general trend. The client of the project is a useful source of
suggestions for proxy options. See chapter 3 for further information on data collection.

Principles for Selecting Indicators
Although the key questions listed above guide the selection and development of indicators, sound
indicators must also be characterized by TQQ: time (time-bound accomplishment), quantity (numer- ically
measurable), and quality (what level of quality or degree of achievement is desired), respec-
tively. See box 2.2 for an example on indicator development.

BOX 2.2: TQQ OF DEFINING INDICATORS

1. Set Time: Access to agricultural extension services for poor farmers in Quynh Luu District is
improved by the year 2007
2. Set Quantity: Access to agricultural extension services for poor farmers in Quynh Luu District is
improved by x% from the baseline by the year 2007
3. Set Quality: Access to good quality agricultural extension services for poor farmers in Quynh Luu
District is improved (where good quality is defined as . . . ) by x% from the baseline by the year
2007

The TQQ serves as a general good practice guideline for selecting indicators. More detailed princi-
ples for selecting key performance indicators require that the indicators are:
• Relevant to the development objective of the project;
• Unambiguous as to definition, valid and targeted in reflecting changing conditions, and
consistent and verifiable in measurement;
• Meaningful and of interest to project management and stakeholders;
• Significant and selective in that the monitoring system should limit the number of indica-
tors to a manageable number;
• Practical in terms of the ability to collect data in a timely and cost-effective way with a rea-
sonable degree of reliability;
• Informative of project impact disaggregated by gender,4 poverty status, age, ethnicity, or
other characteristic of the client group;
• Acceptable to the Borrower, the Bank, and other stakeholders;
• Quantifiable to the extent possible, or, if qualitative, based on a common understanding of
what constitutes success; and
• Responsive in reflecting changes due to project activities and sensitive in demonstrating
change in as a short time period as possible.

14

Key lessons in developing indicators
1. Involve partners and target communities in the selection of indicators.
2. Avoid collecting too many indicators (e.g., one or two per Project Development Objective
and 1 to 3 per each project component). Look for coherence in the selected set.
3. Select indicators that are reliable proxies.
4. Look for unanticipated, negative and positive program or project effects.

Decide how tracking results would guide project implementation
The Results Framework includes a column titled "Use of Results Information."
• This column describes when and how to take corrective action if there are problems in
meeting the agreed targets. Clear and time-bound targets for the selected indicators mea-
sured throughout the project lifespan to support management and decision-making are
needed to reap the full benefit of this information.
• The level of detail and flexibility in these guidelines can vary. An example of a rather detailed
guidance is: If the rate of women and men farmers' satisfaction with access to services is less
than 50% two years after the implementation, the activities will be reviewed and adjusted.

Cover additional M&E features in the text of the project appraisal document
The results framework does not explicitly capture the CAS or sector goals and output levels, nor the
detailed project component activities and input levels. However, Bank-supported projects are required
to articulate a clear alignment between the project and higher order strategic, program, or sector outcomes to
which the project contributes. The "Strategic Context" section of the PAD describes how the project
contributes to these higher level objectives. Furthermore, project inputs,
activities, and outputs are now required to be captured in the main text of the PAD.
The results framework does not explicitly capture the critical assumptions (factors that the project
can not or does not want to control but which are essential for reaching the desired outputs and outcomes of the
project). These critical risks, assumptions and possible controversial aspects need to be
discussed in the main text of the PAD.

Practical implications
Although it is mandatory to develop the results framework and the arrangements for results moni-
toring, the project team may also develop full log frame that introduces an added level of detail for
component inputs, outputs, and outcomes.
• The extended log frame approach may help the project team and the client to precisely
define the different components, their inter-linkages and contributions to the Project
Development Objective.5
• The extended log frame can be a useful tool in facilitating discussions and development of
the project design with the national team.
• It may also assist and complement project management, and may support the development
of a sustainable M&E system beyond the lifetime of the project as it is often a better fit
with the requirements for M&E in the client countries. See Appendix 3 for an example of
an extended log frame.

15

Supervision and monitoring of the project
After the project and the results framework have been designed, the M&E arrangements specified
and agreed on, it is essential to pay adequate attention to implementation and supervision of the over-
all project as well as monitoring in particular.
Responsibility for supervision of a Bank-financed project rests with a task team (TT) assigned to the
project. Preparation for supervision is a joint task with the client and begins during project preparation, when the
TT and client staff agrees on arrangements for project implementation. These arrangements include: (a) monitoring
and evaluating of progress in project implementation and achievement of development outcomes, with a
common set of criteria to be used for project monitoring and evaluation; (b) use of performance indicators;
and (c) reports such as audit reports, Project Monitoring Reports (PMRs), and other reports required for
effective project monitoring, evaluation, and disclosure—to be
provided by the client, along with their outlines, content, and format.
Unfortunately, supervision missions frequently do not pay adequate attention to M&E. To ensure that
M&E will be an integral part of the project activities, emphasis of the importance of the following issues is
useful during the project launch workshop(s) and later during the supervision missions.
These issues are:
• emphasis on sufficient M&E capacity from the beginning of the project;
• timely establishment of the project M&E system;
• collection and use (by TT, PIU, management, and government) of project M&E data from
the very beginning; and
• the need to remedy the identified deficiencies in a speedy manner.
The TT is expected to prepare the "Implementation Status and Results (ISR)" report, formerly
known as "Project Supervision Report (PSR)." The ISR is a result of streamlining of the PSR to make it a
more effective tool for overseeing the implementation of projects and monitoring their outcomes. A new
module has been introduced for reporting on project outcomes that can be monitored during implementation.
See box 2.3 for details on the ISR quality checklist on Project Development
Objective, Outcomes and associated indicators.

16

BOX 2.3: THE IMPLEMENTATION STATUS AND RESULTS (ISR) REPORT QUALITY CHECKLIST

The Implementation Status and Results Report require reporting on the following issues related to the
Project Development Objective (PDO), Outcomes and associated indicators:
1. Indicators for PDO and Intermediate Outcomes
• At least one indicator entered for PDO and Intermediate Outcomes.
• Baseline values entered for all indicators.
• Target values entered for all indicators.
• If project is two or more years into implementation, is any useful information entered on indica-
tors' progress to date?
• Explanation if quality of outcome information is rated poor or is unavailable.
2. Project Development Objective and Outcomes

• PDO rating needs to be consistent with other information in ISR.
• PDO rating and explanation needs to be adequately justified taking into account: implementation
performance, major risks, achievement of major outputs, achievement of intermediate outcomes,
extent to which PDO and project design remain relevant.
• If PDO rating is unsatisfactory, relevant issues and actions need to be presented.
Source: Operations Policy & Country Services: Implementation Status and Results Report.
http://intranet.worldbank.org/WBSITE/INTRANET/UNITS/INTOPCS/INTINVLENDING/0,,contentMDK:20296324~hlPK:
430750~menuPK:388717~pagePK:64137152~piPK:64136883~theSitePK:388707,00.html

17

3. DATA COLLECTION, REPORTING,
AND DISSEMINATION REQUIREMENTS
After key performance indicators have been selected, the next step is to specify the M&E arrange-
ments for data collection, reporting, dissemination and use for decision-making. The project team
together with the partners must identify the following:
1. Data collection arrangements, including data sources and the reliability of the infor-
mation provided and associated costs and responsibilities; and
2. Institutional issues, including how M&E complements and assists project manage-
ment, what are the implications for institutional responsibilities and would participato-
ry M&E lead to capacity building of communities involved;
3. Human and institutional capacity, including extent of local capacity for M&E and the
need for capacity building and the associated costs.
3.1. MONITORING AND EVALUATION PLAN

Data collection arrangements require the development of a comprehensive monitoring and evalua-
tion plan and a management information system (MIS). An important step in developing the plan and
MIS is the identification of data collection sources and methods.
Development of a comprehensive monitoring and evaluation plan becomes an essential part of the
project preparation and Project Appraisal Document. Elements of the M&E plan should include:
• What? Type of information and data to be consolidated;
• How? Procedures and approaches for M&E including methods for data collection and
analysis;
• Why? Identify how the collected data will support monitoring and project management;
• When? Frequency of data collection and reporting;
• Who? The responsible focal points/resource persons for M&E; their responsibilities and
capacities to conduct required M&E functions; and
• Where? Information/data flow chart from M&E focal points/resource persons (implement-
ing agencies); data flow from decentralized levels to the central M&E unit.
The above questions have to be carefully assessed when designing the project. Inadequate planning
for data collection and use has been one of the main challenges in the Bank's project design and preparation
—with negative consequences for timely implementation and project management as well as M&E. The
Project Appraisal Document requires a table for Arrangements for Monitoring' an example of the China
Agricultural Technology Transfer Project is presented in Table 3.1. It provides the scope to compare baseline
information with target values, and to specify data collection frequen-
cy and methods as well as reporting responsibilities.

18

Table 3.1. Arrangements for Outcome Monitoring for "China Agricultural Technology Transfer Project"

Target Values Data Collection and Reporting
Data Collection Responsibility for
Outcome Indicators Baseline YR1 YR2 YR3 YR4 YR5 Frequency and Reports Instruments Data Collection
Percent increase in Determined 20 60 Survey reports at Household survey PPMOs/CPMO
farmer income of targeted at project project entry, mid-
farmers, particularly of entry6 term and closing.
resource poor farmers.
Government investment Determined 4 sites 10 sites Annual Provincial records. PPMOs/CPMO
in Company-farmer at project
partnership models in entry
non-project sites in the
province.
Percent of successful 70% 70% Expert Team convened Expert assessment Expert Team of
components/sub-projects at mid-term and closing. based on a set of national and
as assessed by the guiding questions. international experts.
Expert Team

Results indicators for
each component

19
Component One:
Adoption rate of Determined 15% 30% Annual Visitor surveys Consultant/PPMOs
technologies displayed at project conducted at (Shaanxi and
in exhibition centers entry fairs and Heilongjiang)
and fairs by farm size. technology
markets.
Component Two:
Percentage of successful 0 60 60 Mid-term and Member survey PPMOs/CPMO
farmer associations in closing
service provision to
members as measured
by member satisfaction
rating7
Percentage of successful 0 70 70 Expert Team Expert Team/
sub-projects as assessed Assessment PPMOs/CPMO
by the Expert Team.
Component Three: Satis- Satis- Mid-term and Assessment by Expert Team/ PPMO
Investments aligned factory factory closing the Expert team Shaanxi
with provincial
policies and strategies
Source: Project Appraisal Document: Agricultural Technology Transfer Project

3.2. PROJECT MANAGEMENT INFORMATION SYSTEM

Apart from and in addition to the above described data requirements to satisfy the World Bank's
information needs on project progress, effective project M&E requires the establishment of a project
management information system (MIS), the project's internal system for collection, analysis,
storing, and dissemination of project information.
The new ARE instruments, such as competitive research grant programs for research and extension,
have increased the importance of an effective MIS due to significant requirements on tracking of
implementation and amount of funding allocation for several different subprojects. For Bank- financed
projects, the MIS should be described in the Project Appraisal Document as the project's
internal data management system.

What is an MIS?
The primary purpose of an MIS is to support management in making timely and effective decisions
for planning, monitoring, and managing the project. An MIS is essentially a system that uses formal- ized
procedures to provide management at all levels with appropriate information from internal and
external sources (Vernon 2001).
MIS generally consists of accounting software and a database management system for planning
and non-accounting information. Besides software and hardware, MIS comprises four elements: (a) the
actors who take decisions on the project; (b) the data and information that is useful for decision-making;
(c) the procedures that determine how the actors relate to the data; and (d) the tools that facilitate the
collection, analysis, storage, and dissemination of the data (Lecuit et al.
1999).
What should the MIS for ARE project do and be like? One essential feature of any MIS is that it
should operate using the regular client processes as much as possible. Thus, design and planning of
the system requires systematic dialogue with the stakeholders.
Other key desired features of an MIS in general, and of a Competitive Research Grant project in par-
ticular, are the following (Srivastava et al. 2003) (for an example on MIS pre-requisites for an effec-
tive M&E system, see box 3.1):
1. The accounting system should be linked with the subproject monitoring;
2. The MIS tools should be organized modularly (different functions of the system are
managed by distinct modules, which in turn are integrated into a common, central
database);
3. The MIS should adapt to the decentralized organization and operation of the
Competitive Grant project;
4. Any new project components should be managed by the MIS in order to prevent the
multiplication of management tools;
5. The MIS should facilitate the management of impact information;
6. MIS tools should facilitate decision making; and
7. Information management should be secure.

20

BOX 3.1: MANAGEMENT INFORMATION SYSTEM (MIS) PREREQUISITES FOR AN EFFECTIVE
M&E SYSTEM

A computerized Management Information System (MIS) is the window that captures the quantitative
data within the M&E system. It will capture both progress and process data for concurrent monitoring
of activities at every level within the project.
Starting point: The starting point of the design of an MIS is to identify the inputs, the outputs, and the
process to be followed. This should then be followed by a clear and lucid description of the project cycle and the
results and/or logical framework that includes an institutionally defined set of indicators. Indicators can be categorized
as progress (P) indicators, physical progress indicators (PP), and Financial Progress
indicators (FP). Process indicators can be classified as including input, output, and outcome indicators.
Critical elements in the design of an operational MIS
• Census Data or other national data on population along with geocode information on the classi-
fication by village, district, or other administrative borders need to be available for uploading
into the MIS. Demographic Data with habitation, income, and poverty level data at the lowest
level should also be available for loading directly into the MIS.
• Reports need to be organized as being accessible to the general public and those available for
project management purposes, only.
• In today's web connected world a Central Server that is either owned or leased needs to be
secured. All of the data will reside on this central server; access to this data will be based on pre-
defined protocols and compartmentalized rights of use.
• Hardware and software and peripheral equipment will be connected to the central server in order
to access and process information.
• Trained personnel will be needed at each level to manage and operate the equipment. At each
level within the project structure a minimum of two persons (coordinator and analyst) will be
required to input, process, report, and access information and data for that particular level.
Source: Ghazali Raheem, Consultant, World Bank.

How to develop MIS for ARE? Lecuit et al. (1999)8 provided detailed guidelines for MIS for social
funds, with principles, design features, and terms of reference, that are useful also for ARE projects—
especially those using competitive funds and contracting. Another good source is by Vernon
(2001) on MIS for agricultural research organizations.
When developing a MIS, the following issues require special attention:
1. Design and planning—the first steps:
• Ensure stakeholder dialogue in the design and planning.
• Take time for needs assessment, find out about local information technology capac-
ities, and what information is needed and by whom.
• Examine what has been done in other competitive funds and ARE projects.
• For technology options, see box 3.2.

21

BOX 3.2: TECHNOLOGY OPTIONS IN DEVELOPMENT OF MANAGEMENT INFORMATION SYSTEM (MIS)

In an interconnected world the technology options available today are either web-based, stand alone,
web-enabled or Web-enabled systems using Excel templates. Connectivity is a critical issue in most
computerized MISs, however, most stakeholders will require that web-enabled M&E systems need to be
developed as connectivity will increase and improve in the future.
Hardware, software and options for connectivity
Equipment will consist of hardware, software, database, connectivity, and other peripheral tools. At each
level within the project, equipment needs to be available and trained personnel should be in position to use the
equipment. Training should be provided in overall management, database use, hardware opera-
tions, network management, and operations and software use.
Equipment National or Central State or District Levels
Hardware Owned or leased server with database Workstations, desktops and
support for application development laptop computers
& hosting.
Software Database programs along with spreadsheets, Internet Explorer, spreadsheet,
PC-based database and statistical software PC- based database programs
Database SQL server RDBMS none
Connectivity Leased line VSAT dish, leased line/dial up modems
Others Registered domain name Internet connection

2. Staffing and responsibilities
• Ensure adequate and qualified staffing.
• Designate responsibilities for compiling, checking, and follow-up on reporting.
• ARE projects generally requires M&E specialists of the Project Implementation Unit
(PIU) to carry out required data collection, analysis, and reporting functions.9 The
specialist group is expected to collect and summarize data from the appropriate insti-
tutions (e.g., research institution) at an appropriate level of detail in a project data-
base or MIS. Project Implementation Unit is also recommended to offer training
courses to subproj-ect leaders to ensure that the required information will be made
available and prescribed in the uniform reporting process. Attention to these details
may significantly simplify the data collection, analysis, and reporting for the PIU.
3. Data and targeting
• Quantify expected inputs and outputs.
• Establish indicators or milestones for each subactivity.
• Provide gender-disaggregated data where possible.
4. Adequate procedures
• Establish procedures for regular flow of information between decentralized imple-
mentation units and a central M&E unit.

22

• Make it modular, integrate all information needs (across ministries, extension cen-
ters, research stations, and subprojects), including accounting, into a single flexible
system with a common central database (usually at the PIU).
• Be ready when the project starts by piloting and troubleshooting the system prior to
starting implementation, otherwise information will be lost and heavy workloads
will be required to catch up.
• Be careful in the transition from design to implementation; lack of continuity can
lead to delays.
5. Evaluation
Retain flexibility. Evolve with the project and evaluate once a year. Pick up problems
early.
Good practice examples. Babu et al. (1997) describe MIS for extension programs
(http://www.fao.org/docrep/ W5830E/w5830e0k.htm). Box 3.3 and Box 3.4 describe good practice examples
of an MIS in an agricultural research project in Ecuador and the Project Information
Management System Network in India.

BOX 3.3. K EY F EATURES OF THE M ANAGEMENT I NFORMATION S YSTEM (MIS) FOR THE
MODERNIZATION PROGRAM FOR AGRICULTURAL SERVICES PROJECT (PROMSA) IN ECUADOR

The PROMSA Project Implementation Unit (PIU) in collaboration with Natural Resources International
(NRI) and the Competitive Fund Management Unit (UEFC) developed a state of the art M&E system
for the competitive funding scheme in Ecuador.
For individual projects financed under the competitive fund, a simple system of objectives and mile-
stones (for inputs, activities, and outputs) was established, with monitoring based on information in
quarterly reports.
The annual evaluation of subprojects is based on annual reports by researchers; establishment of partic-
ipatory grupo de referencia of users (reference group) and other interested individuals for each project; and bi-
annual visits by UEFC to each project. In addition, the UEFC has recently solicited feedback from project
leaders through a survey of their opinions about the processes employed in the first three
calls. Data on all projects and milestones are being recorded in a database managed by UEFC.
The PROMSA Competitive Fund employs a system of "alert" signals or flags that indicate if a project has
become a "problem project." While in "alert" status, a project is not eligible for payments from the Fund.
M&E of the competitive fund as a whole is based on several mechanisms. These include:
(i) The annual operating plan of the UEFC combined with quarterly reports on implementation
according to the proposed work plan;
(ii) the project data base established by UEFC which will provide the PROMSA-PIU direct access to
data on implementation of projects, and overall performance of the project portfolio;
(iii) visits by PROMSA-PIU officer to projects and other UEFC-organized activities to observe first-
hand processes being employed in administering the competitive fund; and
(iv) special studies and external evaluation of the operation of the competitive fund and its outcomes
and impacts.
Source: World Bank (2004a). Agriculture Investment Sourcebook.

23

BOX 3.4. PROJECT INFORMATION MANAGEMENT SYSTEM NETWORK
OF INDIA'S NATIONAL AGRICULTURAL TECHNOLOGY PROJECT

Project Information Management System Network (PIMSNET) has been designed, developed, and
implemented under the subproject "Institutionalization of Research Priority Setting, Monitoring and Evaluation
and Networking of Social Scientists" of the National Agricultural Technology Project (NATP). It is
coordinated by the team in the Division of Computer Applications, Indian Agricultural
Statistics Research Institute.
The purpose of the PIMSNET is to:
1. Effectively assess research efforts against well-defined targets;
2. Avoid duplication of research efforts;
3. Provide feedback to research planning process; and
4. Help to establish link between performance evaluation and incentive mechanism.

PIMSNET is being used for M&E of 845 research projects under NATP. It has four modules: Data
Management, Monitoring, Reports, Query, and Administration. For technical details, see the website
www.pimsnet.gen.in.
The monitoring module acts essentially as a monitoring system for NATP.
• The module stores information on the progress of the projects for the different activities in terms
of the expected targets achieved (several quantifiable indicators), and if not, the reasons thereof
and the steps taken to achieve the targets as reported by the Principal Investigators.
• It is possible to monitor the half yearly and annual progress in terms of three types of indicators
for physical progress, financial progress, and scientific or technical progress.
• The research achievements, shortcomings, and difficulties faced by the researchers are captured
by the system and the management can effectively monitor the progress of all the projects right
from their desktop.
• The system has a built-in algorithm to determine the projects which are not performing well/well
performing/very well performing/excellent performing projects.
• Provision has also been made to review the progress report by the Site Committees, State Level
Committees and the Peer Review Teams.
The output modules for the reports and queries through which top and middle level management
may get reports and queries as per their specific needs. The system is able to produce the following
reports:
• General Reports are ready to use reports at macro level generated on the basis of single criteria,
such as projects running under a specific agro-ecosystem, on a particular thematic area, in a par-
ticular state, etc.
• Custom Reports meet the specific needs of the management and are generated on the basis of
options selected by the user based on multiple criteria selected from a ready-to-use list of
options for a desired project. The module generates reports meeting all the selected criteria. For
example, a selection option might be to get a report on projects running under Irrigated-agro
ecosystem program in the thematic area of Integrated Pest Management and having a budget of
more than Rupees 20 lakhs.

24

• Specific Reports are related to the details of a particular project. These are of utmost
importance when management needs to find out specific details of a single project. These
reports contain all the approved items for the project, like equipment procured, contractual
services, training obtained, civil work carried out etc. as well as project objectives, budget,
and the experts involved.
• Administrative Reports act as a tool for the administration and management of the informa-
tion storage and user management.
Source: Dr. S. D. Sharma, Indian Statistics Research Institute.

3.3. INSTITUTIONAL AND HUMAN CAPACITY REQUIREMENTS
FOR PROJECT M&E

Adequate institutional arrangements and institutional and human capacity are essential for any M&E
project, including functioning of MIS. The level of skills required depends on the complexity of the project.
Competitive Research Grant Projects and projects using contracting and involvement of a wide range of
stakeholders across several R&E subprojects are demanding in M&E capacity.10 As evident from the
PROMSA example, the implementation of a well-functioning M&E system both at
the subproject and the overall program level can be a major challenge.
The following features are recommended for developing institutional arrangements for M&E of ARE
projects:
1. Establish a centralized M&E unit. In general, projects may either establish an M&E
unit which is integrated into the Project Implementation Unit or not have a centralized
M&E unit but share M&E tasks among the implementing partners and primary stake-
holders. For complex ARE projects, it is recommended that a centralized M&E unit be
established within the main implementing institution (e.g., the Ministry of Agriculture).
2. Link the centralized M&E unit to subunits. The centralized unit should collaborate
with M&E units in other co-implementing institutions (e.g., extension agencies,
research centers, private sector implementers, enterprise development centers) and in
decentralized regions (e.g., province, district, and county level centers) where project
activities take place or have influence. Similarly, sub-M&E systems within the
Agriculture Ministry, located at a Ministry's institutional or policy planning unit,
should be linked to the central M&E unit within the Ministry of Agriculture.
These first two steps aim to guarantee that all project components will (a) provide sufficient project
and program level M&E links and a scope for effective communication between and within projects; (b)
ensure adequate reporting at the national or program level, and pinpoint any gaps or shortfalls that may not be
detected by the M&E system of a single project; and (c) provide an avenue for various project teams or
Team Leaders with designated projects under the Ministry of Agriculture to
collaborate, share lessons, and ensure desired results.

25

M&E of competitive grant funds to private researchers have particular challenges due to the individ-
ual nature of the grant operation.
• Given that many private researchers are not under the umbrella of any research or edu-
cational institution, tracking and eventual assessment of grant operation and research
results may be very difficult.
• It is recommended that such individual researchers be monitored by focal M&E persons
under the project's technical and supervisory Ministry. This would ensure proper
administration of the grant and generation of the expected research results.
Contracting extension services to private groups may have similar requirements.
• M&E at the project level would focus on the contracted group which will provide
tracked and assessed extension outcomes on a more systematic basis.
• On the positive side, contracted extension service provider may be in a position to:
(a) attribute specific services to the improved position of farmers; (b) pinpoint the real
effects of extension in a particular area and eliminate a mismatch by factoring out nega-
tive effects which may occur due to attributes other than that of extension (annual rainfall
fluctuation or declining production); (c) capture the information on indirect beneficiaries;
and (d) be provided sufficient time to ascertain extension progress and results.
3. Consider institutionalizing an M&E system within implementing institutions
(extension and research centers). This will provide a permanent M&E system
beyond the life of the project as well as meet Bank needs. Box 3.5 illustrates a sce-
nario on how M&E efforts of different projects implemented by the Ministry of
Agriculture can be harmonized.

BOX 3.5. SCENARIO ON HARMONIZING M&E IN THE NATIONAL SECTOR MINISTRIES

This scenario responds to the broader needs of the national research and extension programs.
1. A centralized M&E unit within the Ministry of Agriculture, coordinated by highly qualified
local staff (e.g., Ghana's Ministry of Land and Forestry, which houses a Policy Planning & M&E
Unit), will have information on all projects in the country that are being implemented under the
Ministry.
2. During the implementation of given projects under the Ministry, data or duplicates of data from
all projects will be fed into the centralized system such that at any point in time, or at any stage
in implementation, the central M&E unit will be in a position to determine what results or inter-
mediate results have been achieved and likewise where expected results have fallen short of their
planned/scheduled achievements.
3. Each specific project will also have an M&E sub-unit within their respective Project Coordinating
Unit (PCU) in the implementing Ministry, closely collaborating with co-implementing agencies as
well as decentralized implementing bodies. Therefore, project level data is managed at the level of
each PCU, but will be fed into the institution's central M&E system at the Sector Ministry.
The outcome of such a scenario facilitates formulation of coherent sector priorities, policies, plans, and
programs, as well as enhanced knowledge sharing. This scenario can also provide a step toward donor
harmonization.
Source: Remileku Rakey, Consultant, World Bank.

26

4. Allocate sufficient time and resources to guarantee adequate capacity. Adequate
institutional and human capacity is imperative for effective functioning of M&E
frameworks within implementing institutions.
5. Assess existing local capacity for M&E at the design stage (e.g., by using institution-
al assessment in chapter 4). At a minimum, capacity must include: (a) the ability to
successfully construct indicators; (b) the means to collect, aggregate, analyze, and
report on the performance data in relation to indicators and their baseline; and (c) have
the skills and understanding to use the information effectively.
1. Identify important capacity gaps to fill (i.e., number and skills of staff);
2. Estimate the costs of capacity-building in M&E; and
3. Develop and establish a capacity enhancement plan (to be partly implemented
by the project). Factors to include in an M&E capacity enhancement plan: (a) the
identified skill gap; (b) target persons in the central and subunits of the Ministry,
project components, implementing agencies and beneficiaries; (c) timeline for the
needed skills; and (d) appropriate training times, trainers, and costs.

3.4. DATA SOURCES AND METHODOLOGY FOR DATA COLLECTION AND ANALYSIS

The following section reviews good practices in data sources and data collection. The section starts
with a discussion on baseline data, which is lacking in a large share of the agriculture projects.
Why to collect baseline data? The baseline is the first critical measurement of the performance
indicators and is used as a starting point, or guide, by which to monitor future performance of projects or
programs (Kusek and Rist 2004). Therefore, baseline data should be collected at least for each
identified outcome indicator.
Because the success of a project will be, in part, measured by comparing target values with achieved
or actual values, setting target values is a sensitive issue and should be taken seriously. One method to establish
targets is to start with the baseline indicator level, use historical data or another estimate of the rate of change to
set the desired level of improvement—taking into account available funding and other resources over the target
period—to arrive at the performance target. Although it is tempt- ing to set targets relatively low to assure they
are reached, it is important to set the targets high
enough to ensure project implementation momentum.
A comparison of baseline to achieved values can be done in the following manner:
1. By comparing the situation before the project started with the situation after it started.
This requires understanding which factors influenced the outcome.
2. By tracking changes with (inside the project area) and without a project presence (out-
side the project area). This requires finding comparable areas.
3. By comparing the difference between similar groups (inside the project area) between
a project group and a group outside the project influence. This requires finding com-
parable groups within the same area.
What are the key questions to ask? Often baseline studies suffer either from information overload
and lack of use of the information or a baseline does not exist at all with a burning need to assess

27

project progress and impact. When collecting data to define the baseline level and later on to deter-
mine current levels the following questions are necessary:
1. What are the sources of data?
2. What are the data collection methods?
3. Who will collect the data?
4. How often will the data be collected?
5. What is the cost and difficulty to collect the data?
6. Who will analyze the data?
7. Who will report the data and in what form and forum should data be reported?
8. Who will use the data?
9. Capacity building needs?

Links to examples of baseline surveys
1. Grosh, M. and P. Glewwe. (2000). Designing household survey questionnaires for
developing countries: Lessons from 15 years of the Living Standards Measurement
Study Volumes 1-3. World Bank, Washington, DC. Designing household survey ques-
tionnaires for developing countries: Lessons from 15 years of the Living Standards
Measurement StudyVol. 1.
• On agriculture, volume 2 lays out the most important agricultural policy issues in
developing countries, discusses the data needs and measurement concerns, and
introduces three suggestions for an agriculture module.
• Volume 3 presents draft questionnaire modules on several topics, including issues in
agriculture, among others a separate questionnaire on access to and use of agricul-
tural extension services.
2. The World Bank South Asia Region has developed a "generic" baseline survey for
water resources and irrigation projects. These surveys are accessible in the SASAR
intranet website: http://wbln0023.worldbank.org/Internal/SAR/SARInternalWeb.nsf/
SASRD/B2330FE4633F93E185256FAC006EAD6F?OpenDocument
What to do without baseline data? The collection of baseline data has been very often neglected
in Bank-supported projects in the past. It is highly recommended that the operational units make sure that a
baseline survey will be carried out before the project will be presented to the Board. However,
when no baseline data exists, alternative means must be identified (see box 3.6).
From where and how to collect the data for different M&E needs? The M&E systems of ARE proj-
ects, be it for a baseline or other M&E needs, must balance data needs between the ideal (and costly) and the
practical, and will generally need to draw on data from various sources. The choice of a given
set of methods for data collection depends largely on the specific objectives of the given task.
Regardless of the method used for collecting information, during the preparation it is also essential
to pay attention to:
• The sampling frames; sample size and sampling methods;11 and
• Preparation of data collection materials (e.g., questionnaires and interview guides.)

28

BOX 3.6: WHAT TO DO WHEN THE PROJECT DOES NOT HAVE BASELINE DATA?

Quite frequently, projects and programs have started or have to start without baseline data at hand. In
this situation, the project team can consider the following alternatives:
1. Track changes with (inside the project area) and without a project presence (outside the project
area) if comparable areas can be found.
2. Use first measurements as a starting point (e.g., production during the first implementation sea-
son).
3. Look for existing local and national agricultural databases. In some countries, authorities regu-
larly collect data on farm and off-farm income sources, production data, and use of inputs.
4. Carry out interviews with local key informants (e.g., village leaders, elderly, and extension
workers) or focus groups (e.g., women's and savings groups and producer associations) to detect
changes in the availability and use of agricultural services and inputs, and yield data—historical
trends and timelines.
5. Other existing records, such as
• Data collected by other donor projects or NGOs.
• Training records—village or community and extension centers' records on the past topics, tim-
ings, participants, possible evaluations on trainings and field events.
• Research program records—topics, events, farmer participation, and variety release and adop-
tion.
• Data on input sales (e.g., on specific varieties) from government and private sector records.

There are many different data sources and methods (see Alex and Byerlee (2000) and IFAD (2003)
for more details), but here we focus on the three major data sources: (a) Management Information System; (b)
existing/regularly maintained databases; and (c) special studies (see table 3.2). In prac-
tice, it is often necessary to use a combination of methods.

1. Management Information System
Project input-output data as well as data on agreed indicators are generally available from the proj-
ect MIS. The section above includes more details on MIS.

2. National and International Level Databases
As it is generally not cost-effective for a project to measure national change of goal-level indicators,
information needs at this level rely on national and international databases. These databases may provide data for
indicators such as rural population below poverty line, daily per capita dietary energy supply, agricultural
GDP, or public expenditures on agricultural research as percentage of the agricultural GDP. Quality of
national data may be suspect to poor sampling procedures or intentional dis- tortions, but it is readily available
and comprehensive, represents considerable investment in data collection, and is official, government-
sanctioned data. For example, relevant data may be collected by the Ministry of Finance, Ministry of
Agriculture, or Ministry of Science. These data can be sup-
plemented by international databases.12

29

3. Special Studies
Bridging the gap between indicators from routine reporting on project activities and from national
data sets is a challenge for ARE projects, and reflects the difficulty inherent in linking project
performance and eventual impacts. To bridge this gap, project teams could consider using special
studies.
When to use special studies? The need for special studies should be considered already during the
project preparation as they are of relatively high cost and management intensive. Special studies are
a practical tool:
• For providing data disaggregated by gender, ethnicity, or other key characteristic of client
population;
• For stakeholders to learn about and affect the project and M&E process (see chapter 4);
• For collecting information on agricultural systems, evaluating project impacts, and espe-
cially for assessing and measuring project outcomes; and
• To track specific planned and unplanned impacts or operational issues (e.g., technology
access by women or minority groups, and changes in land tenure and other social condi-
tions in the area). An example of a diagnostic special study on the impact of extension on
women is presented in box 3.7.
How to conduct special studies?
1. Build special studies into the regular work plan of ARE implementing institutions,
starting from the project start-up, mid-term review. For example, extension agencies
must track changes in farmer attitudes, knowledge, and technology adoption; research
programs need to know results of new technologies; and training institutions need to
know how graduates perform. An example of how to use interviews and focus group
discussions for a mid-term evaluation is shown in box 3.8.
2. Clearly identify the stakeholders involved and the purpose of the studies. Beneficiary
and stakeholder participation is highly recommended (see chapter 4);
3. Select an appropriate methodology (see table 3.2 and chapter 4);
4. Pre-test any questionnaires and surveys (with the target group and the survey imple-
menter) and adjust as needed;
5. Consider using an independent institution to avoid overburdening the implementing
institution or to avoid bias in monitoring change within the institution;
6. Consider financing special studies through a project office independent of the
implementing agency;
7. Verify the validity of the collected data (e.g., by cross-checking); and
8. Analyze the information and prepare corrective measures (as needed) to meet the
desired outcomes.

30

BOX 3.7. RURAL WOMEN'S EMPOWERMENT THROUGH AGRICULTURAL EXTENSION IN VENEZUELA

In Venezuela, the Agricultural Extension Project (PREA) implemented by the CIARA Foundation from
1995-2004 was designed to establish a new decentralized extension service while focusing on social and gender
aspects with the direct involvement of the beneficiaries through the participation of rural grass root organizations.
The rationale for the gender perspective of this project was that the gender approach encourages participation and
personal growth of women, which influences individual and family well- being, strengthens the capacity for
team work and consolidation of the rural associations and organiza-
tions leading to benefiting the men, the family, the community, and the society as a whole.
What? In terms of M&E, the diagnosis, indicators for productivity, and family and community activi-
ties were implemented with a gender approach. This required dedicated continuous work with program users and
monitoring of the experiences as well as the incorporation of a gender focal team into the Project Coordinating
Unit. As an example, a comprehensive diagnostic study was conducted for the
project in order to analyze the impact of extension activities on women in rural areas.
How? The Project included a gender sensitive M&E during project implementation by incorporating
gender data in the producer's technical files and allowing the rural women to be registered as producers and heads
of the households. With this data a comparison of gender-related differences in agricultural production, and
family and community activities were made possible. Other applications of gender approach were in: (a)
training in theoretical and methodological gender equity tools; (b) preparation of the participatory diagnosis
done from a gender perspective; (c) design of the annual Plan of Extension oriented in gender perspective; and
(d) execution of several of the productive and
community projects to apply gender tools learnt under this project.
Findings: The membership of Civil Associations of Extension or Producers Grassroots Organizations
(ACEs) in municipal extension in the project included 2,696 rural women out of a total of 10,370. Over
one quarter of the project direct beneficiaries (45,000) and members of ACEs were women (11,700),
indicating the success of the project's strategy to ensure women's access to extension services. Women were
involved mainly in activities such as in micro-enterprising and group activities and expressed their desire for
additional services. In training, more extensionists received training in gender than in technical matters
reflecting the project strategy on social aspects of community develop-
ment. Of the 561 extensionists trained 118 were women.
This project experience demonstrated how attention to the family needs, women in particular, gives such
an investment a high level of social relevance and is a source of incentive to social participation. Nearly 25 percent
of extension agents are women, and extension services for women have emerged as an impor-
tant part of the overall municipal extension program.
This work highlighted the need for early engagement of gender approach, defining gender-based key
indicators, and step-wise monitoring.

Sources: The World Bank, ICR Report No. 29081, dated June 29, 2004 & Draft paper on "Rural Women as Axis of Family
and Community Empowerment in the Agricultural Extension Project of Venezuela" by Maria Magdalena Colmenares and
Andrea Pereira, with input from Ninoska Loaiza and Indu John Abraham (reviewer). Additional input: Indira J. Ekanayake,
World Bank.

31

BOX 3.8: USING INTERVIEWS AND FOCUS GROUP DISCUSSIONS AT MID-TERM
EVALUATION OF A FARMER-LED EXTENSION SERVICE PROVISION PROJECT

Why? To evaluate the effectiveness of farmer-led extension in improving access to services among
smallholders in Vietnam
How? Using semi-structured household and focus group discussions at household, village and com-
mune levels.
Who? Interviews by the staff of a non-governmental organization—can also be carried out by local
associations, extension center staff, PIU staff, consultants, etc.
Specific focus of the "study:" to determine farmers' (women and men) access to and use of extension,
changes in the sources of extension, and satisfaction with quality and access to services.
Key findings: Volunteer farmer trainers (FT) were not yet sufficiently effective and reliable in provid-
ing extension services to fellow farmers in their villages.
Why? The FTs tended to retain the information and knowledge to themselves and did not openly invite
other farmers to their multiple demonstration sites. Also, other farmers were hesitant to contact FTs.
Corrective action:
• Awareness raising among villagers and farmer trainers on the role of FTs
• Continued effort to improve FTs' skills to inform and support farmers and organize field days
and other events.
• Regular reporting and meetings between FTs and commune extension coordinator.
• Local level monitoring (by other farmers and commune extension coordinator) of FT activities.
Source: Riikka Rajalahti, World Bank

When to collect and use the data and by whom? The responsibilities for data collection vary from
project to project based on the country and project context and must be specified in the M&E plan. Similarly,
the capacity building needs must be identified early on and are based on the specific
M&E functions and the complexity of the project. See table 3.2 for a summary on:
• Suitable timing for using different data sources during the project life cycle and with differ-
ent types of instruments;
• Cost and capacity implications; and
• Suggestions on who will collect and use the data.

32

Table 3.2. Major Data Sources—Management Information System, Existing and Regularly Maintained Databases and Various Special Studies and
Main Data Collection Methods

Data source
Level of complexity and cost
When National and Other Rapid Formal External peer Formal
Who international routine rural diagnostic reviews and sample
How MIS1 databases1 statistics appraisals2 surveys analysis3 surveys
Project Cycle
Baseline No Yes Yes As needed As needed No Yes
Appraisal No No No Yes Yes Yes No
Monitoring
Inputs Always No As needed No No No No
Outputs Always No As needed No No No No
IR Outcomes Always As needed As needed Yes Yes/No As needed Yes
Mid-term
evaluation Yes As needed As needed Yes Yes As needed Yes
Final evaluation Yes Yes Yes Yes Yes Yes Yes

33
Instrument
Contracting
Services Always Always Always Often Often Often Often
Competitive grants Always Always Always Often Often Often Often
Participatory R&E Always Always Always Always Often Often Often
Who collects? PIU, impl. Ministry and Local govt., PIU, benefic. Implementing External peers Implementing
Agencies4, other govt. beneficiaries, implementers, agencies, external and consultants agencies, external
beneficiaries agencies ARE centers external M&E5 M&E M&E
Primary user PIU, donor and PIU, donor Potentially all Potentially all Potentially all PIU, donor and Potentially all
impl. agencies implementers stakeholders stakeholders stakeholders impl. agencies stakeholders
Resources
Costs Low Low Low Med-high Med-high High Med-high
Staff capacity Med-high Low Low Med-high High High Med-high
Source: Riikka Rajalahti, World Bank
Note. 1. Assumptions: Management Information System (MIS) is established as a part of routine M&E activities and national databases are maintained regularly
irrespective of the project. If these are not valid assumptions, the level of complexity and cost implications for MIS and databases are high. 2. Rapid rural appraisals
include key informant and community interviews, focus group discussions, structured observations, and informal diagnostic surveys. 3. The external reviews and analysis
include (e.g., research program peer reviews, and ex-ante and ex-post economic analysis). 4.Implementing agencies may include government research and extension
centers, private sector service providers, NGOs, and producer groups. 5. External M&E can be carried out by consultants, non-governmental organizations, or producer
organizations.

3.5. MONITORING AND EVALUATION COSTS

There are typically two types of costs that are required in every project and are crucial for M&E
activities: (a) costs relating to project preparation and implementation activities; and (b) costs relat-
ing to the formulation of a budget.13
Project Costs: Bank project costs are usually prepared using Costab 32, a PC-based project costing
tool. Costab 32 is the project costing component of a software series to improve efficiency and effec-
tiveness of Project Processing activities. See Appendix 5 for more details.
M&E Costs: Project M&E costs on the other hand need to follow realistic budget estimates. The
most important factors to consider when estimating an accurate budget for monitoring and evaluation
activities are scope, methodology, and stakeholder participation. Evaluations requiring the participation of a
large number of stakeholders would require more funds than the evaluation of a single
project.
In preparing the M&E work plan it is necessary to allow adequate provision at the outset of the proj-
ect for all activities related to M&E. These will include:
• Costs for collecting baseline data;
• Concurrent monitoring of implementation activities;
• Independent evaluations and studies;
• Training and capacity building workshops to include MIS design development and imple-
mentation.
Given the various types of interventions and the development circumstances in each project, it is not
practical to prescribe the percentage of the total budget of every project that should be allocated for
monitoring and evaluation actions.
Typically, M&E costs comprise 10-15% of a project's total budget. These costs should be included
in the project budget on the respective budget line. Standard costs include baseline values, ongoing
monitoring, mid-term evaluation, final evaluation and terminal report as well as project-specific costs
(i.e., surveys to collect baseline data, workshops, cost of hiring staff, and M&E specialists) should be taken
into account. See Appendix 5 for an example of a typical M&E budget plan, includ-
ing a breakdown of key expense categories, for an IDA/GEF Environmental project.

34

4. PARTICIPATORY MONITORING AND EVALUATION
Focused attention to poverty reduction and socially sustainable development embracing the socio-
cultural diversity among target populations has increased the need to engage in direct dialogue with different
stakeholder groups involved in development projects. One critical dimension of this engagement is
M&E implemented in participation with stakeholders. This section gives an overview
of participatory M&E and guidance for its use in ARE projects.
4.1. WHAT IS PARTICIPATORY M&E?

Participatory monitoring and evaluation (PME) is the active involvement of key stakeholders in the
M&E process in order for them to learn about and affect the process and impact of a development project.
The nature of stakeholder involvement in PME may range from voice or consultation, to implementation
and use of information; thereby ensuring the needed collective process of reflection,
planning, and management for desired outcomes and impact.
At the heart of PME are four broad principles:
• Participation—which means opening up the design of the process to include those most
directly affected, and agreeing to analyze data together;
• Negotiation—to reach agreement about what will be monitored or evaluated, how and
when data will be collected and analyzed, what the data actually mean, how findings will
be shared, and what action should be taken;
• Learning—becomes the basis for subsequent improvement and corrective action;
• Flexibility—is essential because the numbers, roles, and skills of stakeholders, the external
environment, and other factors change over time.
PME can be used in concert with traditional M&E but should be considered a different process. In
conventional M&E, participation is often limited to working with primary stakeholders as information
sources, rather than as joint users of information. PME, in turn, incorporates ongoing M&E by local
stakeholders throughout the project cycle and after the project has ended. Table 4.1summarizes
the main differences between conventional and participatory M&E.

4.2 WHEN AND HOW TO USE PME IN ARE PROJECTS?

Beneficiary and stakeholder participation in M&E is critical: (a) to ensure that ARE projects and
programs are responsive to the genuine needs of intended clients; (b) to empower target beneficiaries and
strengthen ARE institutions through better programs, accountability and transparency; and (c) to determine
the impact (also unintended impacts) on the intended beneficiaries and stakehold-
ers from their respective perspectives.

35

Table 4.1. The Major Differences between Conventional and Participatory M&E

Facet of M&E Conventional M&E Participatory M&E
Who plans and Senior managers and outside experts Primary stakeholders, project staff, managers
manages the process and other stakeholders, often helped by a
facilitator
Role of primary stakeholders Information provision only Designing and adapting the methodology,
collecting and analyzing data, sharing
findings, identifying lessons learned and
linking them to action
How success is measured Externally defined, mainly Internally-defined indicators, including
quantitative indicators more qualitative judgment and stories of
personal change
Approach Predetermined and fixed Indicative and adaptive

Source: Adapted from IFAD (2003).

Step 1—Decide when to use PME

PME can and is encouraged to be used at any stage of the ARE project; and the degree of participa-
tion can vary from stage to stage. However, it is better that the overall project planning is participatory and
that PME activities are initiated at the very beginning of the project as this will increase the
likelihood of mainstreaming PME in the project cycle.
If resources are limited—time, staff and finances—it is better to prioritize when to use PME rather
than to sacrifice the quality of the activities. During the planning process, or later when questions arise and
participatory M&E approach is likely to be more useful, the project team together with the
beneficiaries and implementers decides the timing of specific PME activities (see table 4.2).

Step 2—Decide who to involve in PME
PME in ARE projects usually involves all stakeholders in the various facets of farm and non-farm
activities. One has to assess the need for and affordable degree of participation by the possible stake-
holder groups (see table 4.2). The following questions can guide the decision-making (Guijt 1999):
1. What is the relevant degree of participation for each stakeholder group? Is the process
of collating and calculating the information important, or only the final information?
2. Who is going to use the final evaluation? Those who are to use it should understand
on what the data is based and how it was calculated.
3. What skills does the analysis require? The more difficult, the more caution should be
used in encouraging broad participation unless it is clear who it will benefit and how.

Step 3—Decide the scope of the work: What and how to do PME
After deciding when to use PME, who to involve and the degree of participation by different stake-
holders, it is essential to clarify the methodology to be used as well as the logistics and funding. The process
usually begins during the project planning or later during implementation when the need arises with a
workshop involving a facilitator and representatives of stakeholders. The initial workshop should bring
clarity concerning the aim of the PME process, roles and responsibilities, logisti-

36

Table 4.2. Degree of Participation at Different Stages of PME Process

Level of Participation
in PME When? Who to Involve? Potential Challenges
Define what is Design stage The key stakeholders—farmers, ARE service Availability of resources
meant by "impact" Evaluation as needed providers, local authorities, bank and borrower. Capacity-building needs
Different stakeholders are likely to have different Facilitation
perceptions on impact.
Design purpose, process Design Essentially farmers, ARE service providers and Availability of resources
and methods for M&E Implementation local authorities with M&E staff. Capacity-building needs
Define themes to Facilitation
monitor and evaluate
Select and define indicators Design The key stakeholders—farmers, ARE service providers, local Easily non-functional if the
authorities, bank and borrower. objectives and available
Different stakeholders are likely to have different preferences indicators are not clear
for indicators. Capacity requirements high
Potentially time-consuming
Harmonization of indicators
across groups challenging

37
Give their opinion of project Evaluation Primarily farmers, ARE service providers, and local authorities. Time and capacity requirements
history and the changes in Implementation
the context Design
Help collecting data for Baseline Depends on the purpose—farmers, ARE service providers Beneficiaries' opportunity costs
monitoring and evaluation Monitoring and local authorities are in a key role in providing and can become high
Evaluations collecting information at the local level. Project staff, May pose a risk of crowding out
managers and borrower are in a key role in providing the true effects of project
information. outcomes and make it difficult
to determine trends, patterns,
outcomes, or intermediate results.
Help analyzing and drawing Monitoring All stakeholders—specific involvement depends on the theme. Time and capacity requirements
conclusions from the data Evaluations
and results Baseline as needed
Share feedback with the Evaluations All stakeholders should receive information on the findings Requires high level of
primary stakeholders? Present Monitoring and have a chance to provide feedback on issues that directly commitment and objectivity of
and communicate the findings Baseline as needed concern them—the format and forum varies. the wider stakeholder community
Give their views on the Final evaluation All stakeholders. Need to involve the stakeholders
degree to which project Mid-term evaluation also in the design process
objectives have been met
Source: Authors. Level of participation "questions" adapted from Guijt and Gaventa (1998).

cal arrangements, schedule, potential capacity issues, and finally the data collection, analysis, and
methodology issues.
In this process, the following issues are important to keep in mind:
• Systematic dialogue and facilitation during PME: The degree and timing of involvement
may vary, but a systematic dialogue between farmers, scientists, extension personnel, govern-
ment officials, the business community and their designate institutions and agencies is essen-
tial throughout the process. It may be necessary to choose an appropriate and non-biased
facilitator to guide the process (in a gender-sensitive manner) and to guarantee that key stake-
holders agree on the rationale, objectives, and plans of any PME activity before it becomes a
part of the project implementation process; and that the level of stakeholders participation, tim-
ing, modes, relevant sources of information, and costs for the PME activity are being clarified.
• Foster use of local knowledge: Local knowledge includes perceptions of stakeholders,
individual and collective. The thematic areas include the productive environments of stake-
holders and target beneficiaries. All these elements may be translated into criteria for evalu-
ating research and extension results.
• Avoid too ambitious plans. Allow sufficient time to build local skills for indicator selection
and collection and other assessments. It is better to start simply and monitor only some
aspects of the project. As experience grows and capacities build, the system can be expanded.
For example, in Brazil, farmer-based research on agro-forestry systems started by monitoring
plant diversity, labor input and ground cover. After the first year, farmers and scientists decid-
ed to include soil nutrients and the research process itself (Guijt 1999).
• Create incentives for stakeholders to participate in PME and sustain the system.

The choice of methodology
When information on stakeholders' opinions, views, and perceptions on ARE services are needed, the
people become the major source of information. Various quantitative and qualitative tools can be applied. Stakeholder
participation can be used to define the questions and questionnaires, to provide and collect
data, and to communicate the findings, particularly when actual field level information is needed.
Many of the methods useful for traditional M&E can be used in either a participatory or non-
participatory way. The participatory impact comes with the way a method is used and in the knowledge of
who helped to select it. For different tool options, see chapter 3, Appendix 4 (data sources
and methods) and PME tools below.

A. Beneficiary and stakeholder assessments
The overall objective of a beneficiary assessment is to help beneficiaries and other local-level stake-
holders identify and design development initiatives, signal constraints to their participation, and give feedback
on these activities to those designing and managing a project or formulating policy. Beneficiary
assessment includes a set of commonly used tools: conversational interviews, focus group discussions,
and participant observation. For details on the necessary steps in ARE projects,
see Beneficiary Assessment for Agricultural Extension: A Manual of Good Practice.

38

During the project design stage, a broad stakeholder group assessment can clarify the vision and per-
spective with which research and extension initiatives will be undertaken. The perspectives of various
stakeholders can contribute to determining the directions for change and their respective roles toward
supporting and strengthening such a change process. Therefore, all R&D projects are designed as a
consequence of such perspectives. This assessment helps to define the Project
Development Objective and the definition for impact.
The assessment can be used for evaluation to review and assess the assumptions behind the project
activities and indicators after a specific timeframe of implementation; the relevance of the project in the
context of changing realities; the implementation of project activities; and to what extent there are deterring
factors to reaching desired outcomes. Similarly, it can used to assess project impact and outcomes by way of
changes that occurred among target beneficiaries and their environments as a result of project interventions
and draw lessons for future activities. The target beneficiaries as well as the main implementers are at the
center of this evaluation. See box 4.1 for an example of the key
steps to take during a participatory activity evaluation.

BOX 4.1: BASIC STEPS IN PARTICIPATORY ACTIVITY EVALUATION

I. Group discussion —Activity Participants II. Activity Evaluation Report
Inputs to discussion Background
Monitoring data Relation to goals and objectives
Experiences and observations Specific purpose of the activity
Results - outputs and outcomes
Discussion and Analysis
Major impressions Analysis - social, economic, environmental and
Beneficial and problem outcomes institutional issues
Constraints Inputs and outputs, costs and benefits
Reasons Intangible - benefits and problems
Gender and other cross-cutting implications Gender and other cross-cutting implications
What to do next
Conclusions
Potential for adoption
Potential constraints

Recommendations
Potential to improve
What to do next
Source: Adapted from Souvanthong, Pheng and R. Trethewie (2001).

Conversational interviews
Conversational interviews constitute the basic tool of inquiry for the PME practitioner. Conversational
interviews often take place in the homes of the interviewees, who are apt to be most comfortable there. Interviews
should be conducted in the local dialect in such a way that open-ended questions revolve around a number of
themes or topics that project management has selected. The objective is to gain

39

in-depth information on beneficiary views in relation to a planned or ongoing activity by encouraging
beneficiaries to speak freely and bring to light issues of concern to project management. Interviews can be
conducted on a one-to-one basis or in focus groups. The advantages of individual interviews are that people are
likely to speak more freely, without worrying what peers or other community members may think. Lower-status or
introverted members of communities may not feel comfortable speaking out in
groups. See box 4.2 for an example on a conversational interview with a field-focus.

BOX 4.2. USING CONVERSATIONAL INTERVIEWS WITH FIELD-FOCUSED APPROACH IN AFRICA

What? Purpose of the task was to identify the magnitude of information dissemination from exten-
sion worker through contact farmer group to non-contact farmer group
How? Using conversational interviews with both members and non-members of contact groups (CG)
in ten African countries1
Findings
• All but one (Mali) Contact Group generally failed in its task as diffuser of information received
from the extension worker.
• In Senegal, a number of farmers who were listed as belonging to CGs did not know that they
were in fact members.
• Only a third of the farmers in areas covered by the extension knew of the existence of CGs.
• Ninety percent of the non-CG farmers did not know anything about the existence of CGs.

Rootcauses
• Insufficient supervision and publicity was given to contact groups.
• Lack of incentives for the CG members to pass information on to his non-CG member farmers.

Corrective measures
• These insights regarding the degree of knowledge about the CG and the lack of motivation of the
CG member to relay extension messages to non-CG neighbors came directly out of the conversa-
tional interviewing done with samples of farmers.
• The importance of these insights can be seen in the decisions to strengthen CGs in several coun-
tries or downplay them, to the point of elimination in Senegal, and in a general trend toward
more participatory extension models involving entire communities.
Source: Lawrence Salmen, Consultant, World Bank.

Unguided discussion is apt to be vague and, therefore, of little use for decision-making; probing for
specificity is often required (e.g., if the intended beneficiary of an extension project stated that she did not
benefit from it, the interviewer should probe to find reasons for this dissatisfaction, prefer-
ably prioritizing reasons (up to three).

Focus group discussions
For PME, in addition to enabling a wider coverage of the beneficiary population in a given time,
interviews carried out in focus groups can serve as a cross-check to individual interviews. The

40

groups should normally comprise six to twelve people with common characteristics (e.g., groups of
intended beneficiaries may be comprised of married women, male heads of households, youth from 15 to 35,
and so on). However, there are times when it may be of use to purposefully mix the con- stituents of a focus
group—say, with members and non-members of contact groups—in order to better appreciate the nature of
conflict and communication between them, and provide the opportunity
for indigenous solutions.
The interview guide should be used in conducting these interviews. The interviewer takes on a facil-
itative role, guiding the discussion to cover topics from the thematic guide and ensuring that every- one has an
opportunity to participate. This will generally entail encouraging the more reticent, introverted persons to
speak up while providing less encouragement to those most apt to dominate the discussion. A researcher
should also be present to take notes. Although the difficulty of quan- tifying focus group discussions may
be considered a liability, their use as a cross-check and as a fairly rapid and easy-to-read barometer of the
mood of a community on many topics makes focus
groups a useful component of the PME approach.

Participant observation
This technique generally involves protracted residence in a targeted community. During this stay, it
is expected that the participant observer will establish enough rapport and involvement to help him or her
accurately represent the conditions within the community as they relate to project objectives. The participant
observer normally spends from one to three weeks in a given community. The researcher will focus on
the areas of concern identified in the interview guide. Particular issues related to agricultural extension that
have been the object of inquiry using participant observation include: (a) the relationship between contact
groups and other community associations; (b) the role of gender in dissemination of the extension message;
and (c) the nature and extent of interaction
between an extension agent and the community he serves.
During this stay in the community, the participant observer should prepare case studies of five to ten
households based on repeated visits and observation. Participant observation, being costly and time-
consuming, should be used selectively on topics of particular interest that are of a sensitive nature
and lend themselves to this form of intensive personal interaction.

B. Institutional assessment
In ARE projects an institutional assessment at the design stage can help to determine whether insti-
tutions and organizations involved in the project implementation have appropriate organizational
procedures, systems, and structures in place. This assessment helps to identify gaps and define some
of the main project components.
Institutional assessment can be used for evaluations. The process involves an evaluation of the
respective institutions' program activities and target interventions relevant to a project, and the extent to which
planned outcomes are being achieved. The review highlights shortfalls that relevant institutions have
experienced. The institutions' staff, including field researchers, extension agents, and contracted extension
service providers, are actively involved together with farmers associations and other players in the field. The
exercise is expected to bring about common and shared understanding
of apparent problems and collective efforts to solve them.

41

For further details on institutional assessment, see:
• http://lnweb18.worldbank.org/ESSD/sdvext.nsf/81ByDocName/ToolsandMethods
Institutionalanalysis
• http://lnweb18.worldbank.org/ESSD/sdvext.nsf/09ByDocName/ProjectPreparation
ImplementationParticipatoryAssessmentsInstitutionalAssessment
• Horton et al. (2003) "Evaluating capacity development: Experiences from research and
development organizations around the world".
• Morgan and Taschereau (1996): Capacity and Institutional Assessment: Framework,
Methods and Tools for Analysis. www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/
CapacityDevelopment/$file/1996-06-02Tools(Wkshp6).pdf

C. Participatory indicator selection
Participatory indicator selection, identification of appropriate indicator proxies, and allocation of
ME responsibilities are essential features of the design stage. The stakeholder and institutional
assessments reflect the stakeholder perspectives and help define the indicators.
It is important to avoid imposing indicators and/or methods. Organizations keen to facilitate the
development of local monitoring systems often impose their ideas of useful indicators or methods to
some extent. It is also necessary to build indicator collection system on what exists locally, collect
indicators at optimal moments (based on agreed timing, frequency and responsibilities) and avoid collecting
too much data. See box 2.1 (in chapter 2) for participatory indicator selec-
tion process.

D. Community score card
Community Score Card is a community-based monitoring tool—a hybrid of the techniques of
social audit, community monitoring, and citizen report cards. It has a strong focus on empowerment and
accountability as it includes an interface meeting between service providers and the community that allows
for immediate feedback on quality and adequacy of services provided in the community. This tool can be
used for tracking inputs or expenditures; generation of benchmark performance criteria used by
communities and service providers to assess their services; monitoring the quality of services over time;
comparison of performance across facilities and districts; generating feedback mechanisms between
providers and users; and strengthening citizen voice and
community empowerment. For more details, see
http://info.worldbank.org/etools/docs/library/94360/Tanz_0603/Ta_0603/TheCommunityScoreCard
Process_June03.pdf.
For a summary in the main tools and methods used in PME for ARE projects, see box 4.3
Steps 4 and 5—Analyze and use the data and findings. After analyzing the data, it needs to be
disseminated and used for corrective action as needed. Findings can also help to strengthen the PME process
itself. Lessons Learned from the NAADS Participatory M&E Pilot in Uganda are described
in box 4.4.

42

BOX 4.3: MAIN TOOLS AND METHODS USED IN PARTICIPATORY MONITORING AND EVALUATION
FOR AGRICULTURAL RESEARCH AND EXTENSION PROJECTS

1. The choice of tools and methods for a specific survey or assessment depends on the country
context, individual project situation, resource constraints, time frame, available capacity as well
as the actual information need.
2. Most tools can be used for design, monitoring, and evaluation of the project. However, quantita-
tive methods are best-suited for baselines and evaluations.
3. In principle, any assessment must take account of the following issues: (a) setting objectives; (b)
selecting field researchers (with data collection and or communication skills for evaluation); (c)
establishing sampling frame and control groups; (d) preparing guidelines in a collaborative man-
ner; (e) applying an appropriate methodology; and (f) analyzing and communicating the data and
results.
Proposed modes of undertaking assessments may include:
1. Quantitative methods, such as administered questionnaires (e.g., household surveys), physical
field measurements, and records reviews are used to provide comprehensive and statistically
valid data on stakeholders, local conditions, program outcomes and impacts. See Grosh and
Glewwe (2000) for further details Designing household survey questionnaires for developing
countries: Lessons from 15 years of the Living Standards Measurement Study Vol. 3.
2. Qualitative methods, such as beneficiary assessment, stakeholder forums, focus group discus-
sions, and key informant interviews, are used to analyze stakeholders, their views, opinions, and
experiences, as well as institutional rules and behaviors.
• Beneficiary and stakeholder assessments: To identify and design development initiatives; signal
constraints to their participation; give feedback on these activities to those designing and manag-
ing a project or formulating policy; and define the Project Development Objective and definition
for impact.
• Stakeholder forums and workshops involving many stakeholders provide a basis for clarifying
questions and issues and exchange of views. For example, this is useful during the design stage.
• Focused group discussions are facilitated discussions among 8-12 participants with similar back-
grounds. Useful when the objective is to gain in-depth information on beneficiary views, per-
spectives, outcomes, and impact in relation to planned activities.
• Community Score Card is a community-based monitoring tool—a hybrid of the techniques of
social audit, community monitoring, and citizen report cards. It has a strong focus on empower-
ment and accountability as it includes an interface meeting between service providers and the
community that allows for immediate feedback on quality and adequacy of services provided in
the community. http://info.worldbank.org/etools/docs/library/94360/Tanz_0603/
Ta_0603/TheCommunityScoreCardProcess_June03.pdf
• Open-ended group interviews revolving around a number of themes selected and agreed on by
the participants.
• Key informant interviews are loosely structured, in-depth interviews of 15-35 people selected
for their first-hand knowledge about a specific topic. www.usaid.gov/pubs/usaid_eval/
pdf_docs/pnabs541.pdf

continued on next page

43

BOX 4.3:—CONTINUED

• Participant and field observation, which is a time-consuming mode of enquiry, covers the active
involvement of observers within targeted communities to review and examine themes surround-
ing relationship among the respective contact groups; the roles of different social and economic
groups; the nature and extent of interaction between the various institutions and farming commu-
nities and ongoing activities.
Source: Authors.

BOX 4.4: LESSONS LEARNED FROM THE UGANDA NATIONAL AGRICULTURAL ADVISORY SERVICES
PROJECT'S (NAADS) PME PILOT

Although much of the Participatory M&E work on agricultural extension has been geared to serve the
needs of the rural development project manager, in the Uganda rural development NAADS project the
central actors became the farmers themselves.
What: A pilot PME exercise was conducted in 2002/2003 in a sub-county of the Mukono District in
which monitoring information was collected from six NAADS farmer groups; and evaluation data was
gathered from 60 randomly selected households. The implementers of this PME were 16 trained farm-
ers under the supervision of and with technical backstopping provided by sub-country technical staff
and persons trained in beneficiary assessment methodology from Makerere University (Kampala).
Findings
There was clear appreciation of this farmer-driven M&E process as (a) a source of information gath-
ered both by and from persons directly affected by the project; and (b) a generator of ownership for
the project, with the learning—as well as the project activities—being carried out by the farmers
themselves.
However, there were a number of challenges, or constraints, which serve as lessons for the improve-
ment of this farmer-driven approach to PME in the future:
• The time allotted for data tabulation and analysis (two days) was considered inadequate;
• Monetary allowances to local PME facilitators, for travel and expenses, was considered by these
persons as insufficient to meet their needs;
• Several of the communities had not been adequately informed about the PME, resented the PME
facilitators as privileged persons and failed to participate in the interviewing (individual and
group) as expected;
• Generally, the PME facilitators (interviewers) were seen as NAADS "representatives" rather than
independent farmers. This impression unduly biased the data; and
• Some PME facilitators experienced difficulty with the conversational interviewing approach.
Particularly problematic were: (a) maintaining respondents' attention span for long periods of
time (due in part to an excessively long interview guide); (b) probing for in-depth information;
(c) posing non-leading questions; and (d) controlling group discussions.

44

Follow-up action
Aside from correcting monetary and logistic factors, it was determined as a result of this pilot to:
• Better inform communities about the PME and the nature of the facilitators;
• Place more stringent requirements regarding the qualifications necessary to be a facilitator;
• Contract external, trained interviewers for evaluation work while retaining farmers for
monitoring activity; and
• Emphasize capacity building that is tailored to persons with low education levels—involv-
ing visual aids, local language, practical field exercises, and alleviation of fears at the out-
set of training.
Conclusions
• The key feature and strength of participatory M&E of rural development projects puts tar-
get beneficiaries at the center of the PME process, proving the scope for feedback toward
improved project implementation as well as learning on the part of beneficiary farmers.
• On the target beneficiary assessment approach, voice is given primacy and there is a sys-
tematic listening to the farmer. In the more participatory, farmer-driven PME represented
by the NAADS case in Uganda, farmers not only acquire voice but decision-making
authority, both as implementers and intended contractors of the learning process.
• It is the catalytic transformation of learning into empowerment that lies at the heart of the
rationale and potential of participatory monitoring and evaluation as described in the Final
Report on this PME by Margaret Mangheni and Christopher Bukenya.
"Our experience from the PM&E methodology pilot phase, at least, has revealed renewed com-
munity enthusiasm and "social energy" in the NAADS process which are valuable outcomes for
a program which seeks among other important goals empowerment of target communities."
Source: Lawrence Salmen, Consultant, World Bank.

4.3. ADDITIONAL MATERIAL ON PME

For further details on PME in general:
1. World Bank—Participation and Civic Engagement—
http://www.worldbank.org/participation/home.htm
2. Participatory Monitoring and Evaluation—
http://www.worldbank.org/participation/pme/partme.htm
3. Eldis Participation Resource Guide—PME
http://www.eldis.org/participation/pme/index.htm
4. PME in Agricultural Research and Extension:
Guide to participatory extension methods used in community forestry in Laos
http://www.eldis.org/cf/search/disp/docdisplay.cfm?doc=DOC10101&resource=f1par
5. On participatory evaluation, see "Who are the question-makers?"
http://www.undp.org/eo/documents/who.htm

45

PME tools:
1. World Bank—PME Tools and Methods
http://www.worldbank.org/participation/pme/partme2.htm
2. World Bank—Social Analysis Sourcebook http://www.worldbank.org/socialanalysissource
book/socialassess5.htm
3. World Bank—Monitoring & Evaluation: Some Tools, Methods & Approaches.
lnweb18.worldbank.org/. . ./a5efbb5d776b67d285256b1e0079c9a3/
$FILE/MandE_tools_methods_approaches.pdf
4. IFAD (2003)—Methods for Monitoring and evaluation.
http://www.ifad.org/evaluation/guide/appendixd/index.htm and
5. Gathering, Managing and Communicating information.
http://www.ifad.org/evaluation/guide/6/index.htm
6. Eldis Participation Resource Guide—Participatory Monitoring and Evaluation—Methods,
Tools and Manuals http://www.eldis.org/participation/pme/pme3.htm
7. Gubbels, P. and Koss, C. (2000). From the Roots Up. Strengthening Organizational
Capacity through Guided Self-assessment.

46

5. EVALUATION OF OUTCOMES
AND IMPACT IN THE WORLD BANK
5.1. WHY AND WHAT TO EVALUATE IN THE WORLD BANK PROJECTS?

Evaluation is the periodic assessment of the relevance, performance, efficiency, and impact (both expect-
ed and unexpected) of the project in relation to stated objectives. Evaluation measures achievements in
relation to institutional policies, Bank-wide program objectives, and the goals set for each operation.
Evaluation is designed to:
• Provide an objective basis for assessing the performance of policies, programs, projects,
and processes;
• Help provide shared accountability for the achievement of the Bank's objectives; and
• Improve policies, programs, and projects by identifying and disseminating the lessons
learned from experience and by making recommendations drawn from evaluation findings.
Evaluation at the Bank has two major dimensions:
1. Self-evaluation (by the units responsible for particular programs and activities).
• An interim evaluation is undertaken by project management during implementation as a
first review of progress and a prognosis of the likely effects of the project. It is intended to
identify project design problems, and is essentially an internal activity undertaken for proj-
ect management. See section 5.2.
• Terminal evaluation, a similar process undertaken at the end of a project, is required for the
Project Completion Report (PCR; OP/BP 13.55, Implementation Completion Reporting). It
includes an assessment of the project's effects and their potential sustainability. Chapter 6
and Appendix 6 present the key steps and a case study on economic impact evaluation of a
competitive research grant scheme.
Terminal Evaluation: The implementation completion report is prepared to evaluate the performance
of the Bank and the borrower. Each party is required to prepare and submit an ICR. Guidelines for ICR
preparation can be found from http://siteresources.worldbank.org/INTINVLENDING/Resources/
ICRGuide.pdf. Box 5.1 provides specific highlights for M&E during ICR preparation and write-up.
2.Independent evaluation:
• The impact evaluation by OED is usually undertaken several years after final disbursement,
and measures changes attributable to the project in terms of both direct and indirect causali-
ty. This is normally undertaken by national authorities or donor agencies. For example, in
the Bank, impact evaluation is carried out by the Operations Evaluation Department (OED)
for its Project Performance Audit Report, and therefore planning and implementation for
such a system should be left to OED.

47

• OED evaluates by three main factors: (a) The relevance of the project's objectives in rela-
tion to the needs of the country; (b) was the project effective in meeting its goals; and (c)
was the project efficient in terms of using no more resources than necessary.

BOX 5.1. SPECIFIC HIGHLIGHTS FOR M&E DURING ICR PREPARATION

In preparing the Bank Implementation Completion Report the following should be reflected in selected
sections of the Report:
• Principle performance ratings: Good reflection and cross-checking in line with implementa-
tion, results of M&E system, baseline data and subsequent updates should go into the ultimate
ratings. If a QAG was conducted at project effectiveness, depending on the rating, M&E should
be a feature for improving quality by project completion;
• Major factors affecting implementation and outcome—if M&E was a factor, either factors
subject to or outside of government/implementing agency/donor control, should be highlighted
with indications of how those factors were addressed;
• Sustainability ratings with specific regards to transition arrangements to regular operations
should reflect the mainstreaming of M&E system into the regular functions of the implementing
Ministry and plans for continuity;
• Bank and borrower performance should reflect perspectives of M&E implementation as con-
ducted by borrower and Bank parties highlighting good practices as well as shortfalls;
• Ratings for achievement of objectives, outputs of components speak of the project implementa-
tion in its entirety: including management, coordination and impact; M&E including independ-
ent assessments, case studies and participatory consultation features as key sources of
information for arriving at those ratings; M&E is therefore a major guide for the ultimate ratings
under this section of the ICR; and
• Lessons learned in the ICR should factor modifications or adjustments made in the M&E sys-
tems that better captured targeted outputs, outcomes, and impacts.
Other ratings during implementation including Project Supervision Report (PSR) ratings of outcome
and impact indicators namely—projected against actual targets—should factor quite strongly in the
implementation of M&E system during supervision and updates of PSRs.
Source: Remileku Rakey, Consultant, World Bank.

5.2. KEY ISSUES TO CONSIDER WHEN
EVALUATING AGRICULTURAL RESEARCH AND EXTENSION PROJECTS

The focus of the evaluations may be the social, institutional, environmental, and economic outcomes,
sustainability, and impact of the ARE interventions. Economic effect and impact evaluations are the most
common type of assessments. It is increasingly clear that the social and environmental effects and impacts of
interventions must also be assessed. However, it is challenging to monitor and eval-
uate environmental and social indicators and effects of ARE projects.
The primary purposes of economic, environmental and social assessments of ARE projects are to
provide: (a) an information base for understanding complex effects of ARE interventions (i.e., new
technologies, policies, and services); (b) an early warning system to prevent adverse effects and to take mid-
course corrective action; (c) a tool for communities to protect the public's interest; (d) a

48

means of identifying alternative approaches, technologies and adaptations; and (e) a basis for plan-
ning that involves the relevant (and affected) stakeholders (Deshler 1989).
Social impact evaluations look into the potential effects of ARE interventions (events, trainings, gen-
erated/transferred/ technologies, and improved organizations) on specific groups of people (i.e., farm community,
poor segments of the community, women farmers, and disadvantaged groups). These evaluations require the
identification of socio-economic variables relevant to a proposed activity; identification of information
sources; data collection and categorization; a community profile of affected populations; projections of
consequences for several alternative courses of action; and assessment of
impacts and evaluation of the social impact by community members (van Willigen in Score 1995).
Similarly, environmental impact evaluations will look into the potential environmental effects of
ARE interventions on the farming community and specific groups as well as the environment and natural
resource base. It is often necessary to establish indirect ways to measure the environmental effects by selecting
indicators that track a change in a physical measure that is considered to have environmental consequences.
Examples of this type of indirect indicators are "the change in the length of Vetiver grass strips" and "the
number of hectares under no-tillage practices"—they are expected to indicate the degree of soil conservation.
Box 5.2 provides examples on potential focus of the evaluation (whether social, environmental, or economic)
in ARE projects/programs. In addi-
tion, Appendix 4 includes a list of various ARE project indicators.

BOX 5.2. POTENTIAL FOCUS FOR EVALUATION
IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS/ PROGRAMS

National impacts: political stability, economic fairness, agricultural environmental sustainability.
Community change: changes in administration of justice; health, welfare, and quality of life; fairness
in the marketplace; change in human rights, status of women; change in economic and social indicators for poor;
change of indicators of sustainable agriculture and natural resource management; change in communication
patterns and access to education and news; change in relationships between farmers and non-farm population; change
in youth perceptions about farming career; public opinion change; fairer distribution of land and other resources;
improved inter-organization relations; evidence of conflict res-
olution; and cultural practice change.
Organizational change: group operation and management; economic performance; technical operation
and management; financial operation; group institutionalization and self-reliance, new groups of farmers included,
new organizational linkages and partnerships; change in staff performance, new service delivery, new methods used,
additional facilities and equipment; change in research priorities; diversification of
funding; cost-benefits improved; new philosophy, purposes, and goals; improved organizational culture.
Individual change: changes in knowledge, attitudes, skills; sustainable agricultural practice; change in
aspirations, self-image, perspectives; expenditure of effort and money; use of methods, services; inven- tion of
appropriate technology; increased production or use of tolls; compliance with or opposition to
public policy; patterns of communication, career directions, and family relationships.
Reactions: testimonials; reactions to the relevance of content; appropriateness of technology, helpful-
ness, perceived value of educational experience; reputation of the extension provider.
Participation: farmer access to extension services by social class, gender, and ethnic groups; intensity
of face-to-face contacts; extent of media-assisted contacts; type of participation (volunteering, planning, recruiting,
learning, experimenting, evaluating); indicators of commitment (attendance, continuity, fre-
quency); private sector involvement. continued on next page

49

BOX 5.2:—CONTINUED

Activities: participatory rural appraisal; planning; local knowledge documentation; on-farm/farmer
experimentation; farmer-to-farmer knowledge sharing; farm tours; farmer organizing; master farmer leadership
training; farm demonstrations; exhibitions and fairs; residential workshops; marketing analy-
sis; farm policy education.
Inputs-resources: organizational sponsorship and networks; funds; organizational design facilities,
equipment; philosophy, mission, goals, objectives; staff, resource people, volunteers; local and external
research knowledge and relevance; cultural, economic and political context.
Source: Deshler 1997; Riikka Rajalahti, World Bank.

The following recommendations for conducting social, environmental and economic evaluations of
ARE projects build on recommendations by Hanson et al. (2004) who provided suggestions on prac-
tical and cost-effective activities when evaluating economic sustainability of extension programs:
1. Evaluate ARE project/program components/activities rather than complete programs—
the individual results can be used in successive improvements better than results from
an overall evaluation;
2. Integrate community's and farmers' perceptions of how activities are organized—it is
important that the activities have an internal consistency from the farmers' point of view;
3. Separate the evaluation of direct effects/benefits (e.g., income generation by farmers)
from indirect ones, (e.g., soil conservation and health benefits and effects on labor
division and community relationships);
4. Improve the use of indicators particularly when monitoring and evaluating indirect
environmental benefits. However, improved use of indicators must go hand-in-hand
with baseline data collection;
5. Take advantage of farmer groups in evaluations. The findings from a group are gener-
ally more reliable than from single farmers. However, it is important to train the inter-
viewee team properly; and
6. Carry out periodic ex-post analyses as they may contribute to mid-course corrections.
Besides the points above, the following steps are essential when planning evaluations:
1. Identify all relevant stakeholders (women and men farmers, research and extension
staff, local leaders and authorities, civil society, government, and donors) in this
process—those who have an interest in the evaluation results and those who should be
involved in the evaluation process;
2. Focus and priorities: It is important to narrow the choices and priorities of the evalu-
ation as early as possible. It is equally important to agree on a short list of evaluation
questions—these questions will guide the collection and analysis of data and informa-
tion as well as interpretation and presentation of the results. Note that the achievement
of the Project Development Objective and the intermediate outcomes presented in the
project's Results Framework must be part of the evaluation.

50

3. Tools and actors: Based on the focus and priorities of the evaluation, the stakeholders
involved, and the time and resources available, select the appropriate tools and actors for
carrying out the evaluation. See box 5.3 for guidance on selection of internal or external
evaluators.

BOX 5.3. WHEN TO USE INTERNAL AND EXTERNAL REVIEWERS AND EVALUATORS?

Internal reviewers and evaluators are generally used when the objective of the evaluation is to identify
project design problems and when the objectivity of the evaluation is not sacrificed. These conditions are
generally in place for unplanned assessments, annual reviews, and interim evaluations. The major benefits of
using internal evaluators include cost-effectiveness, in-depth knowledge of the project and the local context and
the trust gained among the stakeholders. The major disadvantages of using internal evaluators may include
subjectivity, over-emphasis on the positive findings, lack of commitment and resistance to change. However,
internal evaluators and/or M&E staff should be included in all evalua-
tions, with or without external evaluators.
External reviewers or evaluators can:
• Provide independent and constructive criticism that helps project stakeholders to reflect on and
identify lessons;
• Give a fair judgment of project progress and areas in need of improvement;
• Help identify priorities for the remaining time of the project to support the rational use of resources;
• Help unite diverse stakeholder perspectives.

The key problems associated with external evaluators may include:
• Due to time constraints, they often have limited dialogue with project stakeholders, outside the
staff, and may make superficial analyses or to generalize the situation.
• They can jump to conclusions without in-depth knowledge of local realities;
• If foreigners, language constraints and limited cultural insights may affect the analysis;
• They can be excessively strict in their methodology;
• They can be inclined to focus more on funding agency requirements than what a project needs
for improved impact;
• Reports often highlight the negative aspects of the project and do not give due emphasis to posi-
tive aspects.
Source: IFAD 2003; Riikka Rajalahti, World Bank.

• Quantitative methods, such as administered questionnaires (e.g., household surveys),
physical field measurements, and records reviews, are used to provide comprehensive
and statistically valid data on stakeholders, local conditions, program outcomes and
impacts.
• Qualitative methods, such as beneficiary assessment, stakeholder forums, focus
group discussions, and key informant interviews, are used to analyze stakeholders,
their views, opinions, and experiences, as well as institutional rules and behaviors.

51

• Chapters 3 (Data collection, reporting, and dissemination: section 3.4) and 4
(Participatory M&E) in this note presented many of the tools that can be used for eval-
uation.
4. Analyze the data and findings and bring in the perspectives of relevant stakeholders.
Make sure to cross-check the information and results to guarantee the validity of the
findings.
5. Communicate the evaluation findings to the relevant stakeholders in an appropriate
format.
• External stakeholders, such as donors, usually require a formal evaluation report
that should be circulated widely enough. Internal stakeholders, such as beneficiaries
and implementing partners, may have different priorities (identified at the begin-
ning) and interests, and therefore different reporting expectations. The reporting for-
mat should therefore be tailored to the needs of the stakeholder—verbal reporting in
workshops and meetings may be sufficient for some issues whereas other selected
issues may be presented in reports.
6. Use evaluation results. Project evaluation results are expected to provide information
for decision making on e.g., policies, management and future projects. Frequently,
however, this has not been the case. It is essential to pay adequate attention to identify-
ing the relevant stakeholders and the evaluation priorities to improve the use of evalua-
tion results. In addition, evaluation can be used as a learning process.

52

6. ECONOMIC EVALUATION
OF COMPETITIVE GRANTS SCHEMES
Competitive Grants Scheme (CGS) programs implemented as a component for agricultural research and
extension projects seek to divest some planning and oversight functions to local entities with closer links to end
users. Each CGS is comprised of a large number of subprojects carried out by different executing institutions.
As the inclusion of CGS in Bank's ARE programs has increased significantly during the past decade, this chapter is
meant to help in economic assessment of such schemes. This chapter builds on two previous World Bank
publications: "Ex Ante Economic Analysis in AKIS Projects" by Horstkotte- Wesseler et al. (2000) and
"Monitoring and Evaluation for AKIS Projects" by Alex and Byerlee (2000).
For a case study on economic evaluation of Competitive Research Grant in Ecuador, see Appendix 6.

6.1. ECONOMIC IMPACT EVALUATION OF A CGS

The theory and empirical tools for assessing the economic14 impact of Research and Extension are
well developed (see table 6.1).
• The objective of the analysis is to identify changes in productivity that has been induced by
the research effort, and compare these benefits to the cost of the research.
• The key indicator of research efficiency is the internal rate of return (IRR) of the invest-
ment in the CGS.
• Because a CGS is comprised of many subprojects, adjustments must be made to the stan-
dard procedures for calculating the IRR, primarily with respect to procedures for data col-
lection. But the goal is still to compare the discounted benefit and cost streams.
Information on intermediate indicators of research output should also be collected and analyzed. It
will take from five to twenty years for the economic impact of a successful research project to occur.
• A rough rule of thumb might be that extension projects could begin to have a measurable
impact within three years, an adaptive research project within five years, and strategic
research within 5-10 years. A successful basic research project may not even generate a
specific identifiable technology, yet may make a large contribution to the society.
• Indicators should be collected to examine: (a) whether the grant fund has been competently
administered; and (b) the scientific merit of the funded projects (i.e., whether a sufficient
amount of good science is being conducted by grantees).
• A variety of indicators can be used. Administrative efficiency can be assessed by consider-
ing factors such as the number of grant applications received, the percentage of proposals
funded, and the share of overhead costs in total expenditures. Scientific merit is indicated
by the number of publications, particularly in refereed or international journals, or the num-
ber of papers presented at international conferences.
• At some point in the evaluation process, external scientific experts should be called upon to
assess the quality of science. It is very difficult for non-experts to do such an assessment.

53

Table 6.1. Key Reference Sources on the Theory and Empirical Tools for
Assessing the Economic Impact of Research and Extension

ReferenceBooks
Alston, J., G. Norton, and P. Pardey. 1995. Presents the principles and practice of ex-post and ex-ante
Science Under Scarcity. Ithaca, NY: Cornell economic evaluation methods and their use in research priority
University Press. setting. A wide range of approaches are reviewed, synthesized,
and assessed using a unifying conceptual framework.
Horton, D., P. Bellantyne, W. Peterson, B. Uribe, A sourcebook that provides a synthesis of literature and
D. Gapasin, and K. Sheridan. 1993. Monitoring experience on research monitoring and evaluation principles,
and Evaluating Agricultural Research: A processes, and methods.
Sourcebook. Wallingford, UK: CAB International.

Manuals
CIMMYT, 1993. The Adoption of Agricultural An excellent manual for practitioners that explains step by step
Technology: Mexico: CIMMYT the approaches to estimating technology adoption.
Masters, W.A., B. Coulibaly, D. Sanogo, A guidebook intended to provide a concise summary of the
M. Sidibé, and A. Williams. 1996. The Economic tools needed to conduct persuasive impact studies, which
Impact of Agricultural Research: A Practical enable researchers to quantify the economic benefits and costs
Guide. Department of Agricultural Economics, of their work. Three spreadsheet-based computer exercises help
Purdue University, West Lafayette, IN. E-mail: apply the methods described in the manual.
Masters@AgEcon.Purdue.edu
Janssen, W. and A. Kissi. 1997. Planning and A practical approach to combine natural resource management
Priority Setting for Regional Research. Research and productivity concern; introduces a methodology to choose
Management Guidelines 4. International Service regional priority research subprojects based on economic
for National Agricultural Research, The Hague. considerations.
Available online at www.cgiar.org/isnar
Collion, M.H. and A. Kissi. 1995. Guide to An approach to research program planning by objective, based
Program Planning and Priority Setting. Research on a series of steps which include benefit-cost analysis as an
Management Guidelines No. 2E. ISNAR, The approach to priority setting.
Hague. Available online at www.cgiar.org/isnar

Software
Dream©—Dynamic Research Evaluation for www.cgiar.org/ifpri
Management This is a menu-driven computer program developed by ISNAR
to facilitate application of the economic surplus model under a
variety of market situations. The latest version of this program
(June 1998) is available from IFPRI (contact: Stanley Wood,
s.wood@cgnet.com).
MODEXC—Model for Economic Analysis Http://www.ciat.cgiar.org/inslinks/modexc.htm
This is a spreadsheet based economic analysis model developed
by CIAT, available on the World Wide Web, that allows to
calculate the NPV, IRR and B/C ratio of investments in
agricultural research. The model can be used for economic
analysis of technical change both ex-ante and ex-post under
alternate scenarios of market situation.

54

Websites
ISNAR's information and discussion forum Http://www.cgiar.org/isnar/Fora/Priority/index.htm
for agricultural research priority setting This web site provides the process, steps, and methods of
research priority setting.
CIAT impact sub-project Http://www.ciat.cgiar.org/inicio_in.htm, the CIAT web site
features abstracts on the impact of agricultural research, data
bases, trends, as well as a download version of MODEXC (see
above).

Source: Byerlee, D. 2003. "Conceptual and Practical Issues in Measuring Impacts of Sub-projects in Competitive Grants Scheme
(CGS)." Unpublished manuscript. 12p.

6.2. RESEARCH COSTS

Due to the de-centralized nature of the CGS, the cost of each subproject is composed of at least three
sources:
1. Central CGS administrative overhead;
2. Direct costs incurred by the executing institution and cooperating institutions; and
3. The cost incurred by the CGS for administration and monitoring and evaluation of the
subproject (usually considered as an overhead).
Collecting information on research costs is relatively straightforward, so attention here will focus on
identifying and measuring productivity changes.

6.3. SAMPLING

To keep impact assessment within reasonable costs, only a sample of subprojects can be evaluated.
One approach is to evaluate a random sample of subprojects (see box 6.1 for an example). The principle
advantage is that results are representative whereas the major disadvantage can be that a random
sample may become more costly to implement if subprojects are widely disbursed geographically.
A more serious drawback is that a random sample may miss the most successful project. Agricultural
research success has a very skewed distribution. Most projects will have modest or negative economic impacts,
but a small number of projects will make important discoveries. An important discovery by a single research
project can generate sufficient economic benefits to cover the entire project investment. Therefore it is important to
closely examine the benefits generated by projects that are thought to have
been successful, perhaps supplemented by a random sample of remaining projects.
Procedure for random sampling:
• The random sample can be stratified by grouping sub-projects by theme, commodity, target
population group, executing agency, or geographic criteria (e.g., crop varieties, small farm-
ers, IPM, post-harvest, pastures, and basic grains).

55

BOX 6.1: USING RANDOM SAMPLES IN EVALUATION OF IMPACT OF COMPETITIVE GRANT SCHEMES

During its lifetime of 6 years (1994-2000) PRONATTA carried out six rounds of project funding and
co-funded a total of 635 sub-projects. The evaluation of PRONATTA had to be done in a short time and with
limited human and financial resources. More importantly, the unstable political and social situation in many rural
areas made the collection of survey data impossible. In some areas it was not even possi-
ble to conduct field interviews with farmers.
A random sample of funded projects was evaluated. Sample sizes for the first and second phases (n1 and
n2) were 56 and 50, representing 17% of the sub-projects in each phase. The samples were discriminated by two
stratum—the region where the subproject was carried out, and the type of subproject. The evalu-
ation team expected these two factors to influence project success.
The evaluation was done in two stages, taking advantage of the temporal sequence of finalization of the
subprojects. This strategy allowed for a learning process for the evaluation team. Experiences in the first phase
illustrated the possible range of values of the needed parameters, while results from the second
phase served to validate the procedures applied in the first phase. See figure 6.1.
Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture).

Figure 6.1. Estimation strategy in PRONATTA. Source: Greg Traxler (Auburn University) and Gustavo Sain
(The Inter-American Institute for Cooperation on Agriculture).

1994
2001 2003 2004

First sampling Second sampling
N = 338 Estimation N= Estimation
n = 56 (17%) n = 50 (17%)

First sample Second sample
distribution distribution
of TIRs of TIRs

Combined samples
Estimator of TIR

LEARNING PROCESS

56

• This allows common methods to be developed, use of relevant experts, and a more mean-
ingful aggregation of impact results.
• Care is needed to ensure that sampled sub-projects are representative, and that within group
sample sizes are large enough to be meaningful.
• Once a sample is determined, the IRR for each of sub-projects is calculated, and an aggre-
gate IRR is estimated.

6.4. MEASURING BENEFIT STREAMS FOR SUBPROJECTS

In order to measure project benefits, data are required on three key variables: (a) technologies gen-
erated; (b) diffusion levels; and (c) per unit economic benefits.15
1. The evaluator must first identify the technologies that were generated by the research
effort. For example, were new crop varieties or new methods of pest control devel-
oped. This can generally be done by reviewing project reports, possibly with addition-
al information obtained by interviewing researchers and farmers.
2. The level of diffusion of each innovation must be estimated.
3. The productivity impact of each innovation must be measured.
Procedures for the steps 2 and 3 are discussed below. The calculated economic impact of a research
sub-project is the sum of the individual innovation productivity increases multiplied by the total
hectares covered by each new technology.

1. Estimating the diffusion of each innovation
Many studies have found that technological diffusion follows a logistic curve pattern. This curve can
be estimated by statistical means, but this would require information on the adoption percentage at several
points in time. Such information is unlikely to be available, so a shortcut method has been
developed. This method requires information on:
1. The initial proportions of farmers using the innovation when the subproject started
(p0). Baseline surveys will often not be available mandating that adoption information
be elicited through expert opinion. The evaluation team can ask farmers (as members
of the panel of experts) to recall production conditions at the time the sub-project was
initiated. In many cases estimates of the values of p0 will be equal to zero—diffusion
would not have occurred had it not been for the research project.
2. The proportion that have adopted the innovation at the moment of the evaluation (p1); and
3. The maximum expected adoption percentage (K).
With this information it is possible to approximate the logistic curve (Martínez and Saín 1983). The
annual adoption percentages are then multiplied by the total potential adoption area to come up with
the total number of units (hectares) benefiting from the innovation.

57

2. Estimation of the incremental net benefit (INB) per unit of analysis (farm or hectare) from
adoption of the recommended practice
Calculation of the INB requires that the economic value of adopting the research recommendations
be estimated. Research projects generally attempt to increase land productivity (yield), reduce pro-
duction cost, or increase output quality.16
• Technical information about the effect of adoption must reflect the real impact at the field
(not experimental) level.
• Prices used in the valuation must reflect those received for products and paid by farmers
for inputs and services.
• Family labor should be valued at its opportunity cost, approximated by wages in local labor
markets.
• Special care should be taken in estimating the cost of working capital. If farmers do not
have access to formal credit, the interest rate must reflect the interest rate prevailing in
informal credit markets.
In order to calculate the INB, a counterfactual point of reference, representing the state of tech-
nology absent of the research project, must be established. Three main options exist for establishing
the counterfactual:
1. Collecting a baseline counterfactual data set: Counterfactual data can be collected by
surveying the target population at the initiation of the project and then again at the
time of the evaluation.
2. Alternatively, a survey at the time of the evaluation that contains producers affected by
the project and farmers not affected by the project can also be used.
These baseline data approaches require a significant investment in conducting and
analyzing field survey data, but make it possible to accurately disentangle the effects
of the research or extension intervention using the statistical regression model illus-
trated in table 6.2.
3. In most cases it will not be feasible to conduct a comprehensive survey. The most com-
mon method of estimating the INB is to use partial budgets17 based on experimental
results, making adjustments in yield and costs to reflect expected on-farm performance.

Table 6.2: Statistical Regression Model for Isolating Project Effects in Impact Evaluation

Statistical Model ParameterofInterest
Y = 0 + 1D1 + 2D2 + Other factors Estimation is carried out using the pooled sample from
Y = parameter of interest (i.e., Yield) baseline and evaluation surveys.
D1 = 1 for observations at the moment of the evaluation 2 is the parameter of interest. It measures the net impact
= 0 for baseline observations attributed to the sub-project (Value E in table 2.1)
D2 = 1 for observations affected by the subproject
result (adopters)
= 0 for baseline observations


58

6.5. ESTIMATION OF THE IRR

Estimated annual benefit flows (ABF) and cost flows (ACF) of each sampled sub-project are needed
to estimate the IRR. The ABF is estimated by direct multiplication of the incremental net annual ben-
efit (ABF-ACF) by the adoption level.
For most projects, it will be necessary to project ABF and AFC for several years into the future in
order to get a realistic estimate of the subproject's impact. This presents a quandary; to omit these projected
benefits and costs would seriously distort the research impact, but it is difficult to project future values of a
relatively young project. If the research phase of the project has already finished and the flow of benefits
continues beyond the project termination date, a future maintenance cost that is often estimated as 10% of total
cost of the project should also be included. The ABF and AFC
must be expressed in constant monetary units using a common deflator factor.
Once individual project IRRs are computed, the IRR for the CGS is obtained as the sample average
of the IRRs. Using the sample standard deviation, confidence intervals for the IRR estimate can also be
obtained. Average IRRs and confidence intervals can be calculated for each project category. Subproject
IRRs can be examined to identify factors associated with successful or unsuccessful proj-
ects. See box 6.2 for an example.

Estimating the NPV and IRR
Table 6.3 summarizes the steps typically involved to arrive at the Net Present Value (NPV) and
Internal Rate of Return (IRR) of a research and/or extension project.18
• The first column simply indicates the number of years the technology under consideration
is expected to last. It is not easy to precisely determine the life of the innovation. It is rec-
ommended that the analyst try to identify the diffusion pattern of comparable technologies
adopted in the past and discuss these issues with experts, such as local agronomists and
extension experts.

Table 6.3: Calculation of IRR for an Agricultural Research and Extension Project

1 2 3 4 5 6 7 8 9 10
Number Incremental Benefits Benefits Total
of benefitsper with without direct Indirect Total Net
Year adopters adopter/year project project benefits benefits benefits Costs benefits
1 Column 2*3 Column 4-5 Column Column
6+7 8-9
2
3
..
14
15
NPV
IRR


59

BOX 6.2. ESTIMATING INTERNAL RATE OF RETURN FOR SUBPROJECTS
IN COMPETITIVE GRANT SCHEMES

Subproject IRRs were estimated for the PRONATTA project in Colombia. IRR estimates were grouped
by type of research project to see whether one type of project was more successful than another (see table below)
This identifies means for improving future CGS programs. The average IRR for the CGS was 86% and was
fairly similar across types of research, even though the IRRs for individual subpro-
jects varied widely (see figure below).

Subproject frequency distribution of IRRs
Subproject IRR by type of research
Type of research IRR(%)
Mean SD Max Min
Capacity Development 79 129 122 42
Adaptive Research 86 114 125 50
Applied Research 98 115 143 53
Entire CGS 86 106 112 65

• The second column reflects the number of adopters at each point in time and the third col-
umn specifies the incremental benefits per adopter and year.19It is important to note that
the incremental benefits are not necessarily constant over time. For example, perennials
may have a specific production cycle with significant yield fluctuations.

60

• In the next column the benefits with the project are estimated by multiplying the values in
column 2 and 3. For a more precise estimate of the expected benefits of the project it is
important that a realistic "without project" scenario is captured (column 5). If we take the
example of a project which aims at breeding a high-yielding rice variety and at promoting
its adoption, the calculation of project benefits should account for increases in rice yields
which would have been achieved without the activities of the project. It is not likely that the
yields would remain constant over time without the project. However, the estimation of
these benefits without the project is hypothetical and based on many assumptions. Once
again, the analyst should base his judgment on average yield increases in the past and on
discussion with local experts.
• The estimation of total direct benefits (column 6) is then simply based on subtracting the
values in column 4 and 5.
• The indirect benefits (column 7) accounts for spillover effects to non-targeted farmers.
The effects capture the adoption of technology by farmers who have not been directly tar-
geted by the project. Thus, total benefits (column 8) capture both total direct and indirect
benefits.
• Column 9 accounts for project costs over time. A subproject of a competitive grant
scheme for example should not only include the direct costs of the respective subproject,
but also a percentage of the overhead costs of the overall administration of the grant
scheme.20
• By subtracting the costs from the total benefits the analyst receives the net benefits (col-
umn 10). Finally, the values of the net benefits over time can be used to calculate the NPV
and the IRR.

6.6. SOURCES OF INFORMATION

When conducting a full survey is not feasible, the practitioner must resort to alternative sources of
information for estimating values for the required parameters:
1. Subproject information. Among the relevant subproject information the evaluation team
must be familiar with the following information: Subproject Proposal, Advance, Technical
(particularly experimental data), and Final reports as well as budgetary aspects.
2. Secondary information. Market prices and other economic variables including govern.
ment policies are usually obtained from secondary sources.
3. Expert opinion. Expert opinion is a valuable source of information for the evaluation
in the absence of a survey. It refers to a participatory method of eliciting information
on the value of one or two parameters of interest through group or individual inter-
views.21 Table 6.4 shows the minimum set of parameters to be estimated for a sound
economic impact evaluation (ex-post economic evaluation) as described in previous
sections.

61

Table 6.4: Sources of Information on the Minimum Set of Parameters to be Estimated for a Sound Economic
Impact Evaluation

Parameter Information source andprocedure
Size of the RD Subproject documentation. If a proper monitoring system is in place the
information should be readily accessible from advance reports. Another
source of useful information are the technical personnel that participated
in carrying over the subproject
INB = increase in Net Benefits per UOA. Technical information: Project documentation and Experts Opinion
Technical information on induced (farmers and technicians)
changes and sources of information Economic information: Secondary sources and Expert Opinion
Economic information on prices and (farmers and local merchants)
markets
Diffusion pattern. Requires three P0 from subproject documentation
parameters: P1 and K from Expert Opinion
P0 = proportion at the subproject start
P1= proportion at the evaluation time
K = Adoption potential (ceiling)
AFC= annual flow of subproject costs Subproject documentation
in constant monetary terms Secondary information


6.7. FISCAL ANALYSIS

Sustainability of financing
Even though a project may be economically and financially justified, it may not be sustainable if the
fiscal support is not available. A crucial step in assessing the long-term potential for competitive grants, or
any other project, is to assess the fiscal impact of the project. This should be done early in the project cycle.
Fiscal analysis has become increasingly important because of stringent constraints on government budgets in
recent years. Because of the diffuse nature, and the long lags that charac-
terize research benefits, research and extension projects are often hard hit by austerity measures.
The Project Appraisal Document should include a fiscal analysis aimed at assessing the likelihood
that sustainable local resources and funds will be available at the end of the project to take over from funds
provided under the project. This analysis should estimate the long-term impact of the proposed project on public
expenditures and assess whether the project can be sustained with local funding resources. It is important
that the analysis attempts to look beyond the implementation period. Obtaining sustainable funding has
been a difficult challenge for past research projects to overcome.
See box 6.3 for further details.
The key steps in the fiscal analysis:
1. The first step in the fiscal analysis is: collect data on recent public expenditures for
agricultural research and extension.
2. The next step is: estimate the level of public expenditures needed to sustain project
activities after project completion.

62

BOX 6.3: SUSTAINABLE FINANCING OF A COMPETITIVE FUND IN ECUADOR

The Ecuador competitive fund project began full operation late in 1999 and continued until 2004. Prior
to the project virtually all agricultural research funding came for the federal government budget as dedicated
funding to the National Agricultural Research Institute (INIAP). The project was very successful in attracting
counterpart funding from participating research institutions, attaining a high of $2.6 mil- lion in 2003. The
project induced a large increase in total research expenditures, but half or more of total
expenditures were from loan proceeds.
At the end of the project, research funding remained dependent on a highly politicized federal budget
allocation process. Although the 2004 government agricultural research budget was the largest ever, all funds were
dedicated to INIAP and the competitive fund program was closed. Much of the momentum from a project that was
very successful in increasing agricultural research output has therefore been lost,
and the future of the competitive fund remained uncertain.

Govt. Ecuador WorldBank Counterpart Total
$US $US $US $US
Year millions % millions % millions % millions
1995 3.4 100% 0.0 0% 0.0 0% 3.4

1996 4.3 100% 0.0 0% 0.0 0% 4.3
1997 4.5 98% 0.1 2% 0.0 0% 4.6
1998 3.3 95% 0.2 5% 0.0 0% 3.5
1999 1.2 47% 1.3 53% 0.0 0% 2.5
2000 1.7 44% 2.2 56% 0.0 0% 3.9
2001 2.9 47% 3.1 49% 0.3 4% 6.3
2002 3.4 37% 4.1 45% 1.7 18% 9.2
2003 2.8 30% 3.8 42% 2.6 28% 9.1
2004 5.9 63% 1.9 21% 1.5 16% 9.2
1995-
1998 avg.3.9 98% 0.1 2% 0.0 100% 3.9
1999-
2004 avg.3.0 45% 2.7 44% 1.0 100% 6.7
Source: Johannes Woelcke, World Bank.

63

3. Total project costs should be separated into investment costs and recurrent costs.
Computed factors can help to translate project costs into projected incremental public
expenditures per year by multiplying the factors by the average annual project costs.
To calculate a factor that reflects the proportion of the investment costs to be taken
over by the government after the project ends, the investments can be categorized
according to maintenance and replacement versus one-off investments such as foreign
technical assistance.22 The calculation of a factor for recurrent costs can be based on
the relation of the recurrent costs in the last year of the project to average annual
recurrent project costs.
4. Once the annual incremental investment expenditures and recurrent costs have been
estimated for each project component, they can be added up to estimate the projected
total incremental public expenditure per year that will be required to sustain the
research project.
Once the projected incremental public expenditure needs are estimated, the source and form of the
financing mechanisms must be considered. Cost recovery mechanisms have often been incorporated in
competitive fund projects, but have generally fallen short of generating the amount of funding need-
ed to fully support future activities.

PROJECT IMPACT ON TAX REVENUE

The analyst should attempt to capture the project's impact on tax revenue, and reflect this impact in
the projected incremental public expenditure. The project contributes to increased general tax revenues if,
for example, the use of taxed inputs or if a tax is imposed on farm profits (and the project is expected to
increase profits). Because the actual level of cost recovery through the mechanisms mentioned above is
uncertain, the analyst should perform some sensitivity analysis assuming differ-
ent levels of cost recovery.
In many cases, continuation of the project after the loan is closed would represent a significant
increase in government commitment to agricultural research and extension. In times of tight public
budgets these increases might be difficult to defend against competing uses of government funds. Hence,
the analyst should include a section on the importance of the agricultural sector, and in particular of
agricultural research and extension for achieving national goals, such as economic growth and poverty
reduction. The analyst might also refer to the high return on investment in research and extension that has
been achieved internationally. In addition, the analyst could set the estimated public expenditures for
agricultural research and extension in relation to total public expenditures and compare it to the
expenditures of neighboring countries, and other developing or developed countries. In most cases, public
expenditures on agricultural research and extension are
still comparatively low.23

64

7. CONCLUDING REMARKS
Monitoring and evaluation are integral tools for managing and assessing the efficiency and effective-
ness of investments in ARE systems and projects. Monitoring and evaluation of ARE project per-
formance, outcomes, and impact has, however, been a significant challenge. However, successful
project M&E can be carried out in a number of ways.
Besides the mandatory use of Results Framework, description of results monitoring arrangements,
and the official evaluations, the project management together with the relevant stakeholders can choose
different M&E tools and implementers as well as the level of desired technical sophistica- tion. The choices
depend on the Project Development Objective, stakeholders and their priorities,
and particularly the resources available (i.e., time, staff, capacity, and finances).
Besides the mandatory requirements, the six most important things to remember when planning and
implementing project and program M&E include:
At the design stage:
1. A strong and clear Project Development Objective is essential for good project design;
2. Select only a few key indicators—that are Time, Quantity and Quality (TQQ) bound,
and that are realistic to collect considering the resources available;
3. Prepare the project M&E plan carefully. Give due consideration to stakeholder priori-
ties, available resources, and capacity requirements;
During the project implementation:
4. Pay attention to M&E issues during the supervision missions;
5. Implement M&E—collect and use M&E data for the benefit of project primary bene-
ficiaries, project implementers, and project management; and
6. Integrate participatory M&E into the project M&E as much as possible, giving consid-
eration to the priorities and available resources.

65

8. REFERENCES
Alex, G., and D. Byerlee. 2000. Monitoring and Evaluation for AKIS Projects. Framework and Options. Agricultural
Knowledge and Information Systems (AKIS). Good Practice Note. Washington, DC: World Bank.
http://lnweb18.worldbank.org/ESSD/ardext.nsf/26ByDocName/MonitoringandEvaluationforAKISProjectsFramework
andOptions/$FILE/m&e.pdf
Anderson, J., and G. Feder. 2004. "Agricultural Extension: Good Intentions and Hard Realities." The World Bank
Research Observer 19 (1).
Babu, A., Y. Singh, and R. Sachdeva. 1997. "Establishing a Management Information System." In: Improving
Agricultural Extension: A Reference Manual. Rome: Food and Agricultural Organization of the United Nations.
Bantilan, M. C. S., and P. K. Joshi.1997. Integrating Research and Evaluation Efforts. Proceedings of an International
Workshop. India: INCRISAT.
Bellon, M. R. 2001. Participatory Research Methods for Technology Evaluation. A Manual for Scientist Working with
Farmers. Mexico: International Maize and Wheat Improvement Center. CIMMYT, 2001.
Binnendijk, A. 2000. "Results-Based Management in the Development Cooperation Agencies: A Review of
Experience." Paper prepared for OECD/DAC Working Party on Aid Evaluation.
Braun, A., G. Thiele, and M. Fernandez. 2000. "Farmer Field Schools and Local Agricultural Research Committees:
Complementary Platforms for Integrated Decision-making in Sustainable Agriculture." AgREN Network Paper
No. 105.
Byerlee, D. 2003. "Conceptual and Practical Issues in Measuring Impacts of Sub-Projects in Competitive Grants Scheme
(CGS)." Unpublished manuscript.
Casley, D. J., and K. Kumar. 1987. Project Monitoring and Evaluation in Agriculture. A Joint Study of the World Bank,
IFAD, and FAO. Baltimore, MD: John Hopkins University Press.
CIMMYT 1988. From Agronomic Data to Farmer Recommendations: An Economics Training Manual. revised. ed.
Mexico: CIMMYT.
Deshler, D. 1989. "Future Risk Assessment: Issues Programming Opportunity." Journal of Extension 27 (1): Spring 1989. Deshler, D. 1997.
"Evaluating Extension Programmes." In Improving Agricultural Extension. A Reference Manual. FAO. Fort, L., B. Martinez, and M.
Mukhopadhyay. 2001. Toolkit: Integrating a Gender Dimension into Monitoring &
Evaluation of Rural Development Projects. Washington, DC: World Bank.
Goels V. K., E. Koryukin, M. Bhatia, and P. Agarwa. 2004. "Innovation Systems: World Bank Support of Science and
Technology Development." Working Paper No. 32, World Bank, Washington, DC.
Grosh, M., and P. Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries: Lessons from
15 years of the Living Standards Measurement Study Vol.1. Washington, DC: World Bank.
Grosh, M., and P. Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries: Lessons from
15 years of the Living Standards Measurement Study Vol. 2. Washington, DC: World Bank.
Grosh, M., and P.Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries:: Lessons from
15 years of the Living Standards Measurement Study Vol. 3. World Bank, Washington, DC.
Gubbels, P., and C. Koss. C. 2000. "From the Roots Up. Strengthening Organizational Capacity Through Guided Self-
Assessment." World Neighbours Field Guide: Capacity Building. 2nd ed. Paragon Press.
Guijt, I. 1999. "Participatory Monitoring and Evaluation for Natural Resource Management and Research." Socio-
economic Methodologies for Natural Resources Research Best Practice Guidelines. Chatham, UK: Natural
Resource Institute. http://www.nrsp.co.uk/Nrspweb/SEM/SEM%20BPGs/BPG04_Guijt_PM&E.pdf
Guijt, I., and J. Gaventa. 2000. "Learning from Change. An Introduction." In Learning from Change: Issues and
Experiences in Participatory Monitoring and Evaluation. M. Estrella, J. Blauert, D. Compilan, J. Gaventa, J.
Gonsalves, I. Guijt, D. Jonson, and R. Ricafort (eds.), (pp. 1-15). Ottawa, Canada: Intermediate Technology
Publications, London, UK, and International Development Research Centre.

66

Hanson, J., R. Just, and J. Lainez.. 2004. "Evaluating a Publicly Funded, Privately Delivered Agricultural Extension
System in Honduras." Working Paper 2003-04, Department of Agricultural and Resource Economics,
University of Maryland, College Park, MD.
Horstkotte-Wesseler, G., M. Maredia, D. Byerlee, and G. Alex. 2000. "Ex-Ante Economic Analysis in AKIS Projects."
Methods and Guidelines for Good Practices. Agricultural Knowledge and Information Systems (AKIS). Good
Practice Note. Washington, DC: World Bank.
http://lnweb18.worldbank.org/ESSD/essdext.nsf/26DocByUnid/5A74F2C4E3F9372185256B9E00146BA9/$FILE/Exan
te_Analysis.pdf
Horton, D., P. Ballantyne, W. Peterson, B. Uribe, D. Gapasin, and K. Sheridan, 1993. Monitoring and Evaluating
Agricultural Research: A Sourcebook. Wallingford, UK: CAB International.
Horton, D., R. Mackay, A. Anders, L. Dupleich.2000. Evaluating Capacity Development in Planning, Monitoring and
Evaluation: A Case from Agricultural Research. ISNAR Research Report 17. The Hague: International Service
for National Agricultural Research.
IFAD 2003. A Guide for Project M&E. Managing Impact for Rural Development.
http://www.ifad.org/evaluation/guide/index.htm
Kumar, K. 1993. Rapid Rural Appraisal Methods. World Bank, Washington, DC. http://www.fao.org/docrep/
W3241E/w3241e09.htm
Kusek, J. and R. Rist. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System: A Handbook for
Development Practitioners. Washington, DC: World Bank.
Lecuit, L., J. Elder, C. Hurtado, F. Rantua, K. Siblini, and M. Tovo. 1999. Demystifying MIS—Guidelines for
Management Information Systems in Social Funds. Washington, DC: World Bank. http://imagebank.world
bank.org/servlet/WDS_IBank_Servlet?pcont=details&eid=000094946_99092405344364
Maredia, M., D. Byerlee, and J. Anderson. 2000. "Ex Post Evaluation of the Economic Impacts of Agricultural Research
Programs: A Tour of Good Practice." Paper presented to a workshop on "The Future of Impact Assessment in
the CGIAR: Needs, Constraints, and Options," CGIAR Technical Committee, May 3-6, 2000. Rome: FAO.
Mosse, R., and L. Sontheimer.1996. "World Bank's Performance Monitoring Indicators Handbook." World Bank
Technical Paper No. 334, World Bank, Washington, DC.
Murphy, J., and T. Marchant.1988. "Monitoring and Evaluation in Extension Agencies." World Bank Technical Paper
No.79, World Bank, Washington DC.
Rajalahti, R., J. Woelcke, and E. Pehu. 2005. "Development of Research Systems to Support the Changing Agricultural
Sector." Proceedings of a Research Workshop, World Bank, Washington, DC.
Salmen, L. F. 2000. Beneficiary Assessment for Agricultural Extension. A Manual of Good Practice. Agricultural
Knowledge and Information Systems (AKIS). Good Practice Note. Washington, DC: World Bank.
Score, M. 1995. "Social Impact Assessment in Extension Educational Programming." Journal of Extension. 33(6): 4 p.
http://www.joe.org/joe/1995december/tt2.html
Souvanthong, Pheng, and R.Trethewie.2001. Guide to Participatory Extension Methods Used in Community Forestry in
Laos. Mekonginfo, 2001.
Srivastava, J., J. Kampen, G. Baker, D. Byerlee, and M. Sehga. 2003. Good Practices in Monitoring & Evaluation
and Management Information Systems for Competitive Grant Programs in ECA Countries. Washington, DC: World
Bank. http://imagebank.worldbank.org/servlet/WDS_IBank_Servlet?pcont=details&eid=000090341_
20040331111629
UNDP. 1997. Who are the Question-makers? A Participatory Evaluation Handbook. OESP Handbook Series.
http://www.undp.org/eo/documents/who.htm
Veldhuizen, L., A.Waters-Bayer, and H. Zeeuw. 1997. Developing Technology with Farmers: A Trainers Guide for
Participatory Learning. London: Zed Books Ltd.
Vernon, R. 2001. Knowing Where You're Going: Information Systems for Agricultural Research Management. The
Hague: International Service for National Agricultural Research.

67

Wood, S., and P. Pardey. 1997. "Agro-ecological Aspects of Evaluating Agricultural R&D: EPTD Discussion Paper
No.23, International Food Policy Research Institute (IFPRI), Washington, DC.
World Bank. Operational Manual. http://wbln0018.worldbank.org/institutional/manuals/opmanual.nsf/.
World Bank. 1992. "Designing and Implementing Agricultural Extension for Women Farmers." A Technical Note. Women
in Development Division, Population, and Human Resources Department, World Bank, Washington DC.
World Bank. 1997. The Logframe Handbook. A Logical Framework Approach to Project Cycle Management.
http://wbln1023/OCS/Quality.nsf/Main/MELFHandBook/$File/LFhandbook.pdf
World Bank. 2001. Thematic Brief: Agricultural Knowledge and Information Systems; Agricultural Research,
Extension, and Education, 2001.Washington, DC: World Bank.
World Bank. 2003a. Extension and Rural Development—Converging Views on Institutional Approaches? Workshop
Summary. Agriculture and Rural Development Department, World Bank, Washington DC.
World Bank. 2003b. "Monitoring and Evaluation Component in Agricultural Operations. A Toolkit. Agriculture and
Rural Development." Department Draft, World Bank, Washington, DC.
World Bank. 2004a. Agriculture Investment Sourcebook. Washington, DC: World Bank.
World Bank. 2004b. Investments in Agricultural Science and Technology. Agriculture Investment Sourcebook. Module
2. Washington, DC: World Bank.

68

APPENDIX 1. NARRATIVES OF
NEW APPROACHES AND INSTRUMENTS IN ARE SYSTEMS
The recent changes in the agricultural research and extension systems are primarily driven by
changing global and local needs as well as the overall need to address poor fiscal sustainability, difficulties
with attribution of impact, low political commitment, insufficient local content, weak accountability or
transparency, and weak linkages between research and extension, generally associated with ARE
systems.24The changes, such as decentralization, privatization, democratization, and increasingly separate
functions of financing and delivery, have resulted in a slow transition from supply-driven technology
generation toward building a more demand-driven agricultural innovation system (AIS). Agricultural
innovation system essentially links multiple sources of innovation and uptake pathways along the
continuum from basic research to technology adoption recognizing the innovative capacity of all
stakeholders from producers to the marketplace. Thus, focus is not on the science suppliers but on the totality
of actors that are involved in innovation. See
box A1.1 for further details.

BOX A1.1: AGRICULTURAL INNOVATION SYSTEM

A national innovation system is made up of the institutions, enterprises, and individuals that demand and
supply knowledge and technologies; and the rules and mechanisms by which these different agents are interacting.
The focus is not on the science suppliers but on the totality of actors that are involved in
innovation.
The ultimate objective of a well functioning innovation system is to serve the needs of the economy by
achieving better integration of the S&T infrastructure with production needs; by increasing private sector
participation in technology development; and by developing stronger linkages between industry, uni-
versities, and research institutions.
The past approaches (e.g., NARS and AKIS) have strengthened research systems that increase the avail-
ability of new knowledge and new technologies, but not necessarily the number of innovations that will be
implemented; also the role of market and private sector has not been very strong. A lot of changes, many of which
have tried to ensure that new knowledge is relevant in the market context and that the role of the private sector in
the development and diffusion of new knowledge is explicitly recognized,
have taken place.
Source: Goel V.K., E. Koryukin, M. Bhatia, and P. Agarwal 2004.

A number of instruments and approaches, such as contracting for services, competitive research
grants (CRGP), participatory and farmer-to-farmer R&E, and communication media, are being
introduced into many ARE systems in order to facilitate the change. The approaches and instruments are often
used simultaneously and may overlap to some extent. Table A1.1 summarizes the potential
of selected instruments to address the various challenges.

69

Table A1.1. Potential of Selected Innovative Instruments to Address Various Fiscal and Institutional
Challenges Associated with Agricultural Research and Extension Systems

Challengeswith Competitive Useof
Agricultural Research Contracting Research Participatory Farmer-to- MassMedia
andExtensionSystems Services Grants R&E Farmer25 and ICT26
Fiscal sustainability M-H M-H M M-H M-H
Attribution of impact H H H M-H M-H
Political commitment M M M M L
Responsiveness—local content H M-H H H M-H
Accountability-—transparency M-H H M-H M H
Weak linkages between R&E M H H H L
Poverty reduction M-H H M-H M M
Specific targeting H H H M-H L-M
Enhanced private sector role H H M M M-H

Note. Potential of the instrument/approach to address the challenge: low (L), medium (M), and high (H).

Contracting extension services
Contracting for agricultural extension by the public sector takes many forms and may involve con-
tracts with public sector agencies, non-governmental organizations, universities, extension consult- ing firms,
or rural producer organizations. In contracting systems, in general, the agency—such as public funding
agency—draws up terms of reference and details of services to be provided, and then contracts for them,
usually on a competitive basis. Services to be contracted are usually identified in consultation with users,
although the programs tend to be longer term and more program-oriented. Contracts may be administered by
national governments, the national government in collaboration with lower level government, or by
governments with an NGO that then contracts with private com- panies. There can be some "reverse" contracting
of public extension agents, often subject matter spe-
cialists, by NGOs or private sector and farmer organizations.27

Competitive research grant programs
In CRGPs research providers are selected on a competitive basis, using call for proposals and scien-
tific peer review mechanisms to allocate funding. CRGP complement "core" funding or "block" grant funding,
which annually allocates funds to specific public research organizations for their core research programs,
infrastructure, and human resources. CRGPs are used as financing mechanisms to mobilize a wide range of
research institutions for work on priority areas, develop institutional linkages and research capacities across
organizations, and to link scientists with users of new technologies. In the context of agricultural extension,
governments typically continue to finance many rural extension services, but provision of services is more
commonly contracted to private advisory service firms, NGOs, universities, producer organizations, and
other groups. Competitive procedures can improve quality of services, make providers more accountable for
results, and improve efficiency whereas contracting allows for specialization and selection of service
providers according to their
individual comparative advantage.28

70

Participatory research and extension
Participatory research and extension comes with many names, such as participatory technology
development (PTD) and local agricultural research committees (CIAL), and with many levels of par-
ticipation. Participatory technology development29 refers to the process in which development workers
facilitate the generation and dissemination of agricultural innovations together with rural women and men.
Through this interaction, the partners try to increase their understanding of the main traits and dynamics of the
local farming systems, define priority problems and opportunities, and experi- ment with a selection of "best-
bet" options for improvement. The options are based on ideas and experiences derived from both indigenous
knowledge and formal science. This process of technology development is geared toward finding solutions
to current problems and developing sustainable agricultural practices that conserve and enhance the natural
resources. An important aspect is that PTD should strengthen the capacity of farmers and rural
communities to analyze the ongoing
processes and develop relevant, feasible, and useful innovations.
Local agricultural research committees30 are participatory platforms for improving decision-making
capacity and stimulating local innovation for sustainable agriculture. They are a permanent agricultural
research service staffed by a team of four or more volunteer farmers elected by the community. The
committees create a link between local and formal research. The CIAL process is iterative and consists of a
number of steps that are all supported by facilitation, monitoring, and evaluation. The strength of the CIALs
lies in their systematic evaluation of technological alternatives and their
ability to influence the research agendas of formal research and extension systems.

Participatory farmer-to-farmer approaches
Farmer-led extension or farmer-to-farmer extension refers to extension methodology that relies
either on informal or formal training or transfer of technologies, or advice by a farmer to another farmer or
a group of farmers. Usually the key farmer receives some technical and methodological training by
extension services and may be partially or fully compensated for this service in kind by
monetary remuneration.
Farmer Field School (FFS)30 is likely the best known farmer-to-farmer extension method. FFS offers
community-based, non-formal education to groups of 20-25 farmers, who are expected to transfer their
knowledge and skills to other farmers in their communities. Discovery-based learning is related to
agroecological principles in a participatory learning process throughout a crop cycle. FFS fill gaps in local
knowledge, conducts holistic research on agroecosystems and increases awareness and understanding of
phenomena that are not obvious or easily observable. FFS require significant insti-
tutional commitment and support, usually provided by the national extension service.

71

APPENDIX 2. IMPLEMENTATION OF
A MONITORING AND EVALUATION PLAN IN
THE BANK'S PROJECT CYCLE: PUTTING IT ALL TOGETHER
THE BANK'S PROJECT CYCLE AND IMPLICATIONS FOR M&E

The Bank's projects are financed and implemented according to a well-documented project cycle.
The cycle is an eight-step process that provides participants and stakeholders with a full view of the project
development process, planning, and achievement of outcomes from the identification and concept stage to
the post-project completion stage, including full implementation. Therefore, plans for M&E should be evident
at every stage of the eight-step project cycle.31 See Figure A2.1 for a dia-
gram of M&E Steps in the World Bank Project Cycle.
Although M&E are the responsibility of the borrower's project management team, the Bank task team
has a critical role in assisting the client team in setting and achieving M&E objectives. These include:
(a) help identify information needs and specify monitoring objectives;
(b) ensure that all documents in the project cycle clearly establish M&E plans and expect-
ed results for project activities and project coordination;
(c) review the design and organizational arrangements of M&E information system along
with needed special studies to ensure feasibility;
(d) help to finance technical assistance and equipment required for the design and imple-
mentation of M&E activities; and
(e) ensure that training and relevant capacity needs are adequately addressed.

Guidelines for Sound M&E in Innovative ARE Projects
For Agricultural Research and Extension projects, M&E is often needed as permanent elements of
country programs in view of the institution building objectives of some programs. Therefore, an M&E
system must contribute substantially to enhance program sustainability and set the pace for
establishing M&E permanency within country programs.
Each stage of the Bank's eight-step project cycle requires specific action planning and special atten-
tion to the development and evolution of a project M&E system and use of indicators. The follow-
ing are necessary elements that must be visible at all stages of the project cycle:
• During the preparation, sufficient time and resources must be allocated toward establish-
ment of a gender-disaggregated baseline;
• The Bank task team must identify and secure an M&E resource person in its core team to
provide support in the evolution and implementation of M&E systems; where financial
resources are a constraint, M&E must be featured in the TOR of an appropriate mission
member;
• The client team must include a political champion or leader from an implementing
Ministry whose role would be to: (a) promote results-based M&E system and ensure the
institutionalization and sustainability of developed systems or convince of its importance;
and (b) provide the scope for in-service training and capacity building programs;

72

• Targeting different impact areas, such as gender, ethnicity, socio-economic equity, partici-
pation, and consultation should be considered at all stages of the project cycle and can sig-
nificantly influence the project design. Indicators sensitive to such targeted areas may
determine the extent to which the extension and research services take these issues serious-
ly and ensure an efficient functioning of systems developed;
• Sufficient donor support in the implementation and funding of M&E activities such as
modernizing or developing MIS systems and statistical capacity increases the value of
M&E.
The following are proposed means of implicating M&E at each stage of the project cycle:
1. CAS level identification: The Bank prepares lending and advisory services for a specific country
based on selectivity framework and areas of comparative advantage, targeted to country PRSP efforts. The
process begins when sector preparation groups identify specific sector deficiencies and a range
of possible areas or themes for project intervention to be initiated over a three-year CAS period.
2. Identification: Projects are identified that support strategies and that are financially, economical-
ly, socially, and environmentally sound. Developed strategies are analyzed. Diagnostic case studies are
conducted using rapid/participatory rural appraisal to assess problems and potential constraints, opportunities,
special issues with farmers and other stakeholders in a given agro-ecological area. The project concept note
(PCN) sets the pace for developing M&E indicators, and subsequently provides the scope and elements of the
project M&E system. Results of case studies/theme proposals feed into
project baseline information, identifies opportunities, special scenarios, and other stakeholders.
3. Preparation: The Bank provides policy and project advice along with financial assistance.
Clients conduct studies and prepare final project documentation. More comprehensive quantitative
surveys are carried out to determine the extent and need for intervention and importantly, provide
baseline data for the research environment and for selected extension areas. These can either be narrowly
focused to capture specific themes relevant to project components and related project activities, or
broadly targeted to capture different but related themes and opportunities that warrant the need for
subsequent or additional interventions. They may include situation analysis and participatory needs
assessment to appraise current farm households, local conditions as well as institutional/organizational
issues in the research/extension continuum. Within the framework of such studies, it will be necessary to
assess prospects for institutional reforms and technological changes to impact the current production
environment. All preparation relevant to M&E, including preliminary discussions of all study results,
should be undertaken and com-
pleted by pre-appraisal.
4. Appraisal: The Bank assesses the key parameters for investment and prepares the PAD and
draft legal documents. Preparation follows a pre-appraisal phase and subsequently an appraisal phase. In
between the two phases, the project appraisal document (PAD) is developed. The most important feature
for M&E in the PAD is the logical framework which establishes outputs, outcomes, and impact
indicators with established baselines and controls. Although the Bank has now shifted from the log frame
to the results framework, which is a more simplified version of the log frame, it is strongly
recommended for ARE projects that an extended log frame be prepared with indicators to capture the
outputs and outcomes of all activities by component. A results framework may then be derived from the
log frame to satisfy Bank requirements. The PAD

73

should also provide a description of M&E arrangements, including a M&E plan and Management
Information System, for the entire project phase.
Pre-appraisal phase: M&E arrangements initiated at pre-appraisal and firmed up at appraisal
include the following:
• Development of M&E framework and detailed M&E plan which will include:
• M&E system and how it functions in the context of the project
• M&E information flow, sources, and uses of data
• Key players in M&E project activities
• Reporting formats for M&E
• Timeframe for key information requirements and follow-up studies
• Frequency for baseline updates and control group monitoring as needed;
• Development of M&E manual of procedure;
• Initiate an assessment of M&E institutional and human capacity including capacity to inte-
grate gender and relevant target areas into project activities;
• Develop a plan for M&E capacity enhancement; and
• Develop M&E training program.
Appraisal phase: During appraisal, the Bank and the borrower will:
• Discuss and agree on (a) log frame/results framework indicators and identified targets for
outputs, outcomes and impact and (b) established baselines and controls;
• Through a validation workshop, finalize and adopt (a) the M&E framework and (b) the
M&E manual; and
• Review with client, the results of the capacity assessment study and agree on a capacity
enhancement plan and staff training program.
5. Negotiation and board approval: The Bank and borrower agree on loan/credit proceeds and
presents to the board for approval. The implementation of the loan/credit is negotiated based on
agreed upon plans and presented to the board for approval.
6. Implementation and supervision: The borrower implements the project and the Bank supervis-
es loan with due regard for economy, efficiency and effectiveness.
Pre-implementation: Between board approval and actual implementation start-up, there is a period
during which the project becomes effective. The following will need to be accomplished during the
pre-implementation or project effectiveness period:
• M&E Sub-unit within the implementing/coordinating unit of the project need to be estab-
lished and fully equipped;
• M&E focal points established at decentralized locations of project implementation;
• Because M&E requires the participation and contribution of all project staff, at least the
initial M&E training program for project staff should be completed; and
• Workshop held for project staff to internalize M&E manual.

74

Implementation: Indicators that have been developed are now used to monitor implementation
progress by activity. Using the information flow chart and the reporting formats agreed upon, feedback is
circulated to support program management and stakeholders. The central data processing and information
consolidation point is the M&E subunit of the project coordinating/implementation unit—usually under
the implementing Ministry. The M&E focal points in the respective implementing agencies must provide
routine reports on progress (inputs, outputs, outcomes) to the M&E subunit within the project management
unit. Consolidated reports and analyzed data complemented with information on progress from periodic
assessments, case studies, and activity appraisals, are put at the disposal of project management for
dissemination and action as appropriate. Periodic updates of baseline studies are necessary to monitor and
capture changes in selected areas resulting from proj-
ect interventions and other factors.
Supervision: Supervision/implementation support missions must include an M&E resource person
or a core team member with the ability to review: (a) M&E activities; (b) the functionality and adequacy of
M&E systems; and (c) ensure that project indicators are being monitored adequately by M&E focal points
at central and decentralized levels. Indicators and their respective targets should be reviewed to ensure that
planned project activities are on track in terms of planned levels of output realizations, achievement of
outcomes, planned timeframes, and use of allocated resources.
Specific roles of M&E during the supervision missions include:
• Systems adjustment: M&E systems once designed and operational, should not be an end
in itself; it should have some scope for adjustment in terms of (a) new issues that might
emerge during implementation, and (b) lessons on the system operation and the functioning
that may be incorporated into the design. Supervision missions will provide the scope for
such adjustments;
• Capacity adequacy: Missions will review, assess, and analyze capacity adequacy for M&E
support team and propose training needs and additional skills as necessary;
• Response capacity: Missions will review and assess the response capacity of high-level
client/partner officials on M&E implementation process and propose ways of enhancing
apparent deficiencies;
• Sensitivity to targeted areas: Mission will pay particular attention to progress achieved in
gender/ethnicity/poverty sensitive indicators. In particular, the extent to which the project
has succeeded in integrating these target areas into project implementation; and
• Guidance to Bank Task Team: on any salient M&E issue that might emerge from either
client, wider stakeholder community, or members of the Bank core team.
7. Implementation and completion: The implementation completion report is prepared to evaluate
the performance of both Bank and the borrower. Each party is required to prepare and submit an
Implementation Completion Report (ICR). Institutionalization of M&E actions within implementing
Ministries as well as harmonizing the M&E systems of various projects within sector Ministries has specific
implications for project completion and post-project activities. M&E must not end at project completion
—mainstreaming M&E within implementing Ministries establishes M&E as an ongoing function of
government policy units and technology programs. Lessons must continuously be drawn from evaluation
efforts throughout project implementation. A key indicator for project sustainability and continuity or
mainstreaming project activities with regular functions of the Ministries,

75

is the extent to which the project M&E system has been established and used at project completion
and for post-project activities. Box 5.1 in chapter 5 provides specific highlights for M&E during ICR
preparation and write-up.
8. Evaluation: The Bank's independent OED audits and evaluates the project. Analysis feeds into
future project design. OED should pay particular attention to the ratings that have relevance to M&E
implementation and outcomes.

Figure A2.1. Diagram of M&E Steps in the World Bank Project Cycle.

8
Project Evaluation
7. 1.
Independent OED audits
Implementation And And evaluates project CAS level
Completion identification
ICR evaluates performance Bank prepares lending
Of both Bank and Borrower and advisory services

M&E FOR AR & E
6. 2.
Implementation by borrower Inthe Bank's Project identified in line
Supervision by Bank with
CAS and PRSP
due regard to efficiency Project Cycle PCN prepared
& effectiveness

5. 3.
Negotiations and Project Preparation
Board Approval 4. Bank provides policy
Loan/credit proceeds Appraisal and Project advice
are agreed upon Pad and draft legal along with financial asst
documents prepared
by Bank

Source: Adapted from

http://intranet.worldbank.org/WBSITE/INTRANET/INTCOUNTRIES/INTLAC/0,,menuPK:257867~pagePK:64089870~piPK:6408
9874~theSitePK:257804,00.html

76

APPENDIX 3. A BRIEF DESCRIPTION
OF THE "EXTENDED" LOGICAL FRAMEWORK
What is log frame?
• Log frame is a program management technique used by the World Bank (since 1997) and
other development organizations to manage the complete project cycle from design, imple-
mentation, monitoring, and evaluation.
• The log frame is a "cause and effect" model of project interventions to create desired
impacts for beneficiaries.

Why use it?
• It is used to develop the overall design of the project, to improve project implementation
monitoring, and to strengthen period project evaluation. It is a tool that has the power to
communicate the essential elements of a project clearly and succinctly throughout the proj-
ect cycle.
• Developing the log frame helps the project team and the client to precisely define the dif-
ferent components and how they are linked and contribute to the Project Development
Objective (PDO).

Components of the log frame:
• In the Bank's log frame, the multi-step process by which investments lead to impacts is
compressed into a four-step format under a 16-box matrix as presented in Table A3.1 below
(excluding the rows on project component statements).
• The matrix presents the cause-effect-logic between inputs (financial, human, and material
resources), activities (tasks personnel undertake to transform inputs to outputs), outputs
(products and services produced), Development Objective (effects of outputs on target
groups), and Country Assistance Strategy goal (long-term, widespread improvement in
society—impact).
• In addition, consideration is given to the critical assumptions—the external risk and
enabling factors outside the direct control of the project.
• The basic logic is that when the critical assumptions hold, the project interventions through
inputs and activities generate component outputs. The outputs, in turn, are expected to
result in desired impacts (project objective level outcomes and impacts) for beneficiaries.
To track the achievement of each objective, a set of performance indicators will be set for
each objective level.
• Information regarding the use of log frame in program design can be found in detail in: The
Log frame Handbook. A Logical Framework Approach to Project Cycle Management.
http://wbln1023/OCS/Quality.nsf/Main/MELFHandBook/$File/LFhandbook.pdf

77

Why"extended"logicalframework?
• Although it is mandatory to develop the results framework and the arrangements for results
monitoring, the project team may also develop full log frame that introduces an added level
of detail for component inputs, outputs and outcomes.
• The "extended log frame" approach may help the project team and the client to precisely
define the different components and how they are linked and contribute to the PDO.
• The extended log frame can be a useful tool in facilitating discussions and development of
the project design with the national team.
• It may also assist and complement project management, and may support the development
of a sustainable M&E system beyond the lifetime of the project as it often better fits the
requirements for M&E in the client countries.
• Table A3.1 below provides an example of the "Extended" Logical Framework.

Table A3.1. The "Extended" Logical Framework

Monitoring &
NarrativeSummary Performance Indicators Evaluation Plan Critical Assumptions
CAS Goal Sector goal level indicator The program evaluation system. CAS Goal to
Super Goal
Project Development Outcome indicator People, events, processes, sources PDO to CAS Goal
Objective (PDO) of data for organizing the project
evaluation system.
Project Component Intermediate Outcome People, events, processes, sources Project Component
Statement of data for organizing the project to PDO
M&E system.
Outputs Output indicator People, events, processes, sources Output to Project
of data—supervision and Component
monitoring system for project
implementation.
Project Component Inputs (budget for each People, events, processes, sources Component Activity
Activities component/activity) of data and monitoring system to Output
for project design.

Source: Authors. Adapted from The Log frame Handbook. A Logical Framework Approach to Project Cycle Management.

78

APPENDIX 4. LIST OF INDICATORS
AND DATA SOURCES FOR ARE PROJECTS
A. DIFFERENT INDICATOR LEVELS

Budget and input level: Tracking budgets and input use is not only an important element of
sound project planning and monitoring, but also crucial element in the evaluation of project success.
During the planning process budgets and input requirements are established as part of priority setting.
These cost projections (budgets) must be linked to expected outcomes and are an essential element for
evaluation of any program, as benefits are assessed in relation to program cost. Accurate and timely
budget and expenditure data are important to the efficiency and effectiveness of a program evaluation.
Expenditure data are commonly used and helpful in tracking implementation. The required information
for the preparation of the log frame or results framework can be easily extracted from the cost tables. If
used alone, however, it is an inappropriate and insufficient indicator of project implementation progress and
performance. Based on the projected and actual budgets two important evaluation methods must be
highlighted here: ex ante and ex post economic analysis. Ex ante economic analysis relating project costs
and benefits is a standard means of evaluating whether the proposed investments are financially and
economically justifiable. Ex-post economic analysis uses similar methodology to evaluate
investment
impacts (see chapter 6).
Output level: Outputs are direct products of project activity and within the control of the project.
Project output level M&E focuses on relatively simple quantitative measures of productivity and physical
completion rather than on the qualitative assessment of the goods produced. Data on out-
puts should come from the management information system (see Chapter 3).
Bank-supported agricultural R&E projects typically focus on two objectives: (a) institutional devel-
opment to strengthen the institutions and technology system necessary to develop and disseminate improved
technology and management in the agricultural sector; and (b) productivity change due to
technological innovation introduced through the technology system
The two objectives are related and mutually supportive, but require different indicators for outputs
and outcomes. An illustrative list of key indicators, with particular focus on CRGS and contracting
extension services, is included in section B.
Outcome level: Outcomes can be defined as the intermediate effects of outputs on clients but are not
under full control of the project. Although the measurement of outputs is mainly based on simply
quantifiable indicators, the measurement of outcomes adds a quality dimension which is more difficult to
quantify. The newly introduced results framework requires the definition of outcomes and out-
come indicators not only at the PDO level, but also at the component levels.
Component level: Outcomes and outcome indicators specify the expected benefit for specific
target group from component implementation. In AIS projects, the target group of components are very
often researchers and extension agents or institutions whose performance are expected to be improved to
better serve their respective clients, agricultural producers. Hence, combined outcomes at the component
level contribute to the desired outcome at the PDO level. At the PDO level, outcome indicators reflect the
initial results or intermediate impacts of the overall project.

79

While formulating the outcome statements of the different components it is important to note that
the various outcomes are not independent of each other but instead reinforce each other to con-
tribute to the overall objective of the project.
In many recent Bank-supported ARE projects, the project components focus on institutional
development outcomes, and the PDO focuses on productivity outcomes. However, it is critical to recognize
that a PDO can also focus on institutional development outcomes directly. Institutional development
outcomes vary considerably depending on the technology system needs and the proposed project design.
These may reflect changes in the institutions themselves, institutional productivity, or institutional
relations with clients. Project outcomes cannot always be reported on a routine basis by a project
management information system. For this reason, special studies are usually necessary to measure
outcomes. Chapter 3 and section C below provide more details on
data sources.
Sector goal indicators: Sector goal indicators are not required any longer for Bank-supported proj-
ect M&E when a results framework is being developed. However, selection of sector goal indicators may still
be useful (a) as project documents have to describe how the project contributes to higher level objectives and
(b) when the project team and in particular the counterparts in client countries decide to implement a
permanent M&E system with a long-term perspective based on the log frame
approach.
Sector goal indicators measure change in the broad development goal to which the project con-
tributes. Changes in these indicators are indirectly attributable to the project, and usually cannot be
measured during the life of a Bank-financed project. Although demonstrating direct project impact
on a goal level indicator may not be within the "manageable interests" of a project, it is important the
project provides a plausible hypothesis of how project activities will lead to change at the level of the
sector goal. This often requires other sources of research and analysis to build a plausible link between
project activities and the social goal. Ex post impact analysis are important to demonstrate past
success and guide future investments. The trick with impact assessment is that of comparing the "with
project" situation with the counterfactual
"without project" scenario.
Sector-level impact indicators are similar for research and extension projects. Despite the difficul-
ties with attribution and detecting change, R&E projects should link activities to impacts on indicators of
social goals as a basis of justifying investment. Models may be useful in relating R&E programs to
changes in social goals and in estimating impacts that are not otherwise readily measurable (Wood and
Pardey 1997). Methodologies for use of such models are still under development, and more work may be
needed to develop practical models that bridge the gap between measurable outcomes (adoption and effects of
adoption) and impacts on social indicators (Maredia, Byerlee, and Anderson 2000). Development and
verification of such models are important tools for assessing various influences on agricultural production
processes, and may be appropriate for financing under R&E projects. Such modeling is likely to be more
appropriate and useful as a research activity within a country research program, and less practical for
estimating impacts for a Bank-financed project
M&E system.

80

B. INDICATOR LIST

The following is a menu of possible goal, outcome, and output indicators for agricultural research
and extension projects. The list includes social, environmental, and institutional-capacity-building
indicators. It is important to note that the list is not exhaustive but provides suggestions.
• Goal level indicators are not mandatory any more for World Bank-supported projects.
However, these indicators may be of interest if the Bank team chooses to use the log frame
approach as well as for the counterparts in the client countries who would like to imple-
ment an M&E system like the project.
• Due to cost and time considerations it is recommended that only a limited number of rele-
vant indicators should be selected to meet the needs of specific projects.
• The level (especially for PDO outcomes vs. component outcomes; and outputs vs. out-
comes) at which a specific indicator is relevant may vary depending on the specific project
or program objectives.

1. Goal level indicators (typically not monitored by projects)
All ARE Projects

Goal Indicator Measurement Source
• Rural population below (national) poverty line %number GDB/CDB
• Rural population with incomes < $1/< $2 per day %number GDB/CDB
• Daily per capita dietary energy supply Calories/capita GDB/CDB
• Malnutrition prevalence (rural and urban) %number GDB/CDB
• Agricultural GDP value, % growth / year GDB/CDB
• Agricultural GDP / agricultural worker value, % growth / year GDB/CDB
• Agricultural GDP / rural worker value, % growth / year GDB/CDB
• Agricultural exports value, % growth / year GDB/CDB
• Agricultural employment (full-time; part-time) % or number GDB/CDB
• Freshwater withdrawal for agriculture Water use / agricultural GDP; GDB/CDB
total; % of total available
• Deforestation % per year; GDB/CDB
% change in rate over time
• Agricultural land loss/total arable area % change in rate over time/ GDB/CDB
% per year
• Land use pattern % change in rate over time GDB/CDB

Source: Authors. Adapted from Alex and Byerlee (2000).

81

2. Outcome indicators (Level: Component and Project Development Objective)
Extension Projects—Institutional Development

OutcomeIndicator Measurement Source
• Co-financing of extension service by farmers (disaggregated by % of total cost PDB
farm size, gender, age, farm type, income)
• Co-financing of extension service by providers (disaggregated by % of total cost PDB
type of service providers, e.g., private enterprise, NGO, producer
organization, public agency)
• Agencies providing extension services (disaggregated by type of Number, % of total number PDB
service provider, e.g., private enterprise, NGO, producer of subprojects
organization, public agency)
• Regions covered by (decentralized) extension system (or covered Number of projects in each PDB
by extension projects) region; % of total
• Women, youth, and minority groups reached by extension Number, % of total PDB / SS
services or participating in extension subprojects

• Priority areas captured (as defined at national or regional level) Number of extension activities/ PDB
through extension system / extension projects subprojects per priority area
• Quality/skills of extension providers (natural resource Number of certified PDB / SS
management, participatory technology development, enterprise extension agents
development & marketing, organization skills)
• Environmental topics captured through extension system Number of extension PDB
activities/area
• Motivation expressed by the extension staff %change PDB / SS
• Perceived fit (by extension staff) between supply and demand %change PDB / SS
for services (priorities/geographical area/target groups covered)
• Intensity of extension coverage Families/agent or area/agent PDB / SS
• Extension cost per farmer, per hectare, or per unit value Cost per unit PDB / SS
of production

• Efficiency of extension activity (demonstrations, training, field days) Number/agent, Cost / activity PDB / SS
• Gender balance of extension staff %women PDB / SS
• Ethnic balance or minority language capability of extension staff % minority or language capable PDB / SS
• Non-salary operational costs as a percentage of total extension % of total budget CDB / PDB
budget (national and project level)
• Service provider awareness of competitive fund for extension Number of application per call PDB

service provision and capacity to apply successfully (disaggregated and number or % accepted
by type of service provider)
• Farmer evaluation of extension providers Ratings SS
• Farmer willingness to share extension costs Number or % change, % total SS
• Farmer requests for extension services (disaggregated by gender, Number, % of total PDB / SS
minority groups)

Extension Projects—Institutional Development—continued

• Farmer awareness of extension activities (disaggregated by % of farmer SS
gender, minority groups)
• Farmer, private sector, and civil society representation on extension
management committees (by sex, ethnicity, farm size, wealth) Number, % SS / PDB
• Functioning M&E system with active participation
of executing institutions Number of impact studies; PDB / ER /
number of progress reports; SS
general assessment of system
• Efficiency of fund administration unit Overhead costs (total and PDB
% of total project costs);
Average number of weeks
or month from call for proposals
and signing contract

Extension Projects—Technology Dissemination

• Adoption of new practices or technologies (disaggregated by Total number of adopters PDB/SS
participants and non-participants); including studies of rationales (including spillovers);
for adoption / non-adoption by gender, minority group, age % of targeted farmers
and all farmers
• Change in income, welfare, costs, yield, productivity, as a result % change from base SS
of new technologies or practices (disaggregated by gender,
minority groups, age)
• Change in farm/household labor needs (disaggregated by gender, % change from base SS
• Change in relative income from on-farm and off-farm activities % change from base SS
(disaggregated by gender, minority groups, age as appropriate)
• Adoption of environmentally-sound technologies Area (ha); number of farmer; PDB/SS
reduced use of agrochemicals
(in liter or kg); reduced erosion
(soil loss in kg per area)
• Change in farmers organizations (autonomy, leadership, funding, % change, Number SS
membership, participation, planning, activities)
• Change in farmers/organizations/communities' ability to % change from base, Number SS
market products
• Changes in use of natural resources (water, forest, land) Rate of use; area SS
• Yield gap between farmers' yields and on-farm trials % difference % SS
• Yield gap between national yields and neighboring countries difference CDB

continued on next page

83


• Time-series or case studies of change in farmer knowledge, % change from base SS
skills, attitudes, or understanding of technologies and practices
• Change in land area used for sustainable, integrated production Area (ha) PDB/SS
• Farms/farmers participating in environmental farm plan programs Area (ha); number of PDB/SS
farms/farmers

Research Projects—Institutional Development

• Institutions engaged in execution of research projects (in Number, % of total PDB / SS
CGS subprojects); disaggregated by type (e.g. public universities,
private universities, private enterprises, national research
organization, producer organization)
• Co-financing of research projects (CGS) by executing research Total or % of project costs PDB / SS
institutions; disaggregated by type (e.g. public universities, private
universities, private enterprises, national research organization,
producer organization)
• Awareness of competitive research fund and capacity to Number of application per call PDB
apply successfully (disaggregated by type of research institute) and number or % accepted
• Extent of inter-institutional, regional, and international Number of projects; PDB
collaboration (CGS) % of budget
• Identified research priority areas covered (CGS) Number of projects per PDB/CDB
priority area; % of total budget;
scientist time allocated

• Disciplinary, product and regional balance of research projects Number of projects per priority PDB/CDB
(CGS) area; % of total budget;
scientist time allocated
• Balance of strategic, applied, and adaptive research (CGS) Number of projects per category PDB/CDB
• The degree to which the objectives and activities reflect %change PDB/SS
stakeholder priorities
• Graduates from education programs / projects (CGS) Number of MSc or BSc PDB
• Competitive funding share of total national research funding Total; % of total budget PDB/CDB
• Stakeholder participation in governance, priority setting, and Assessment of participation SS /PDB
execution (e.g., stakeholder representation
on boards)
• Farmer representation on research management committees Number, % PDB/SS
(by sex, ethnicity, farm size, wealth, age)

• Research scientists by level of qualification (by sex; by priority area) Number and qualification CDB/SS
(MSc and PhD)

84

Research Projects—Institutional Development—continued

• Research scientists by technical specialty—economics, social Number CDB/SS
sciences, natural resource management, agronomy, business
development (by sex; by priority area)
• Recurrent costs or personnel costs relative to total research % of total budget CDB/PDB
budget (CGS)
• Peer reviewed publications, research reports, conference paper, Total number; number per PDB/SS
patents (by priority area) year per researcher
• Ratio of professional to support staff and trained technician Ratio PDB/SS
(by sex)
• Staff time devoted to research (CGS) % of time SS/PDB
• Research-extension linkages (CGS) % of staff time devoted to SS/PDB
extension activities; number
of joint activities (workshops,
field days, and field trials)
• Research programs applying participatory technology Number or % PDB/SS
development methodology/farmer participation in research activities
• Annual research staff turnover % of all staff CDB/SS
• Functioning M&E system with active participation of executing Number of impact studies; PDB/ER/
institutions number of progress reports; SS
general assessment of system
• Efficiency of fund administration unit Overhead costs (total and PDB
% of total project costs);
Average number of weeks or
month from call for proposals
and signing contract

Research Projects—Technology Development

• Adoption of new technologies and management recommendations Total number of adopters PDB/SS
(disaggregated by participants and non-participants); including (including spillovers); % of
studies of rationales for adoption / non-adoption by gender, targeted farmers and all farmers
minority group, youth
• Change in income, welfare, costs, yield, productivity, as a result % change from base SS
of new technologies or recommendations (disaggregated by
gender, minority groups, youth)
• Change in farm/household labor needs (disaggregated by gender, % change from base SS
• Change in relative income from on-farm and off-farm activities % change from base SS
(disaggregated by gender, minority groups, age as appropriate)
continued

85

Research Projects—Technology Development—continued

• Client satisfaction with new technologies or management Rating (% of positive response) SS
recommendations
• Adoption of environmentally-sound technologies Area (ha); number of farmer; PDB/SS
reduced use of agrochemicals
(in liter or kg); reduced erosion
(soil loss in kg per area)
• Changes in use of natural resources (e.g., water, forest, land) Rate of use; area SS
• Research programs applying participatory technology Number or % PDB/SS
development methodology
• Yield gap between farmers' yields and on-farm trials % difference SS
• Yield gap between national yields and neighboring countries % difference CDB


3. Project output indicators
Extension Projects—Institutional Development

Output Indicator Measurement Source
• Regional/decentralized offices established Number PDB
• Extension provider staffing (disaggregated by region; by sex) Number and qualifications PDB
• Extension staff assigned in the field %; number of agents living
in communities they serve
• Extension service strategies and plan completed Number PDB
(disaggregated by regions)
• Extension agents training support (e.g., induction courses, and Number of staff trained; PDB
in-service training) training days per year
• Extension materials preparation unit established (by region) Number PDB
• Extension agents trained on NRM issues Number of staff trained PDB
• ICT support capacity (by region) Number of offices/agents PDB
with computer (internet) access;
number of computer/telephones
per agent
• Transport capacity (by region) Number of vehicles/bicycle PDB
per agent
• Extension management committees established Number
• Competitive fund management unit established Assessed adequacy PDB/ER

• M&E system (including management information system) established Assessed adequacy PDB/ER

86


• Farmer training courses conducted (per region) Number PDB
• Farmer trained, contacted, visited (by sex, per region) Number PDB
• Frequency of face-to-face meeting with farmer (by region, by sex) Average number per month; PDB/SS
average time spent
• Demonstration established (on-station and on-farm per region) Number PDB
• Farmer contests conducted Number PDB
• Number of producers organizations established Number PDB

• Farmer group meetings held (per region) Number; number of participants PDB Number of
• Information material produced and disseminated (per region) brochures and PDB
pamphlets
• Extension service projects (CGS) Number PDB
• Environmental farm program trainings conducted Number PDB
• Farmer-led trainings conducted Number PDB

Research Projects—Institutional Development

• Research projects (CGS) Number PDB
• CGS promotion workshops/seminars Number PDB
• Graduate courses established (PhD, MSc, BSc, short courses) Number PDB
• Curricula revised or developed Number PDB
• Costs of graduate programs Cost per graduate PDB
• Proposal review committee established (CGS) Assessed adequacy PDB/ER
• Competitive fund management unit established Assessed adequacy PDB/ER
• Research priority setting mechanisms/committee implemented Assessed adequacy PDB/ER

• Training support for researcher in priority areas Number of workshops, seminars PDB Number of
• ICT support capacity (e.g., computer/internet access; access to researcher with PDB
electronic journals) computer/internet access/
electronic journals; number of
computers per researcher
• Journals/publications acquired Number PDB
• Laboratories equipped Assessed number and qualityPDB
• Research-extension linkages established Number of full-time SMS; PDB
number of joint seminars/
workshops
continued

Research projects—Institutional development—continued

• Regulatory and legal frameworks for research and technology Assessed adequacy PDB/ER
importation established
• Researchers trained in participatory technology development Number PDB
• Natural resource management researchers hired Number PDB

Research Projects—Technology Development

• New varieties developed Number PDB
• Extension recommendations published Number PDB
• Other research products completed; research tools / Number PDB
methods developed
• On-farm research projects (CGS) Number PDB
• Papers published Number PDB
• Environmentally sound technologies developed Number PDB


4. Input indicators—All ARE projects

InputIndicator Measurement Source
• Civil works completed Number and value PDB
• Goods (e.g., equipment and vehicles) procured Value PDB
• Technical assistance provided Person-weeks and value PDB
• Unit costs of inputs Cost per unit PDB


Abbreviations: GDB = Global Database; CDB = Country Database; PDB = Project Database; SS = Special Studies; and ER =
Expert Review.

88

C. SOURCES FOR BASELINE DATA AND INDICATORS

Agricultural Innovation Systems project M&E systems must balance data needs between what is
ideal and also costly, and what is practical, and will generally need to draw on data from various
sources. These sources will be described briefly in the following paragraphs.32

Management information system
Project input-output data are generally available from a project management information system
(MIS), the project's internal system for collection, analysis, and dissemination of project information.

National and international level databases
As it is generally not cost-effective for a project to measure national change in goal-level indicators,
information needs at this level rely on national and international databases. These databases may provide data for
indicators such as rural population below poverty line, daily per capita dietary energy supply, agricultural
GDP, and public expenditures on agricultural research as percentage of the agricultural GDP. Quality of
national data may be suspect to poor sampling procedures or intentional dis- tortions, but it is readily available
and comprehensive, represents considerable investment in data collection, and is official, government-
sanctioned data. Changes in national-level indicators are usually gradual and not observable over a typical
five-year Bank-financed project. It may be advisable to dampen the fluctuations because agricultural statistics
are subject to wide year-to-year fluctuations due to weather and commodity price changes, tracking three-year
averages and statistical trends rather than annual data. For example, relevant data may be collected by the
Ministry of Finance, Ministry of Agriculture, or Ministry of Science. These data can be supplemented by
international
databases.33

Special studies
Bridging the gap between indicators from routine reporting on project activities and from national
data sets is a challenge for ARE projects, and reflects the difficulty inherent in linking project per-
formance and eventual impacts. Special studies are a practical tool for collecting information on
agricultural systems, program impacts, and especially for assessing and measuring outcomes. In general, special
studies should provide for disaggregating by gender, ethnicity, or other key characteris-
tics of client populations.
In addition to surveys or case studies to assess outcomes, special studies might be needed to track
specific planned and unplanned impacts or program operational issues (e.g., technology access by women or
minority groups, and changes in land tenure and other social conditions in the area). Beneficiary and
stakeholder participation in M&E is critical to ensure that R&E projects are responding to genuine
needs of intended clients. This can be accomplished by using survey tech-
niques or participatory evaluation (see chapter 4).
Special studies can usually be built into the regular work plan of ARE institutions, as extension agen-
cies must track changes in farmer attitudes, knowledge, and technology adoption; research programs need to
know results of new technologies; and training institutions need to know how graduates perform. In some
cases, it may be best for monitoring to be done by an independent institution to avoid

89

overburdening the implementation institution, or to avoid bias in monitoring change within the insti-
tution. The alternative is to finance special studies through a project office independent of the implementation
agency. No matter how special studies are carried out, there is a danger of them being neglected during
project start-up, as studies are of relatively high cost, management intensive, and easily put off until data are
needed for a Mid-Term Review or other critical project evaluation. The
major types of special studies used in ARE programs include:
a) Sample surveys. Formal sample surveys are useful to provide comprehensive and statistically valid
data on program outcomes and impacts. However, formal sample surveys also have disadvantages in that many
institutions lack capacity to conduct them, wide fluctuations in agricultural production make it difficult to
establish statistically significant trends in production data over short time periods, and survey data
collection costs are high. Large-scale national surveys are probably best avoid- ed, unless they are linked to a
regular program of national agricultural statistics. More focused regional or localized surveys may be
more manageable, and are useful in characterizing agricultural conditions and changes in a particular area.34
Grosh and Glewwe (2000) provide detailed advice on how to design multiple-topic household surveys, that
include defining the objectives of the survey,
identifying the data needed, and drafting questionnaires.35
b) Rapid rural appraisal. Rapid rural appraisals are a compromise between formal surveys (which
generate statistically valid, quantitative data but are costly and time consuming) and informal field visits
(which provide anecdotal information) (Kumar 1993). They are low-cost, quick, open-ended, and flexible,
but are not statistically valid and may be open to charges of bias. Five core methods of rapid rural appraisal
approaches are: key informant interviews, focus group interviews, community
interviews, structured direct observation, and informal surveys.
c) Special topic studies. Special topic studies are often useful to analyze technology program impacts
and develop an analytical base for routine M&E data collection. Ex ante economic analysis and economic
models of the impact of technological innovation are common types of special topic studies. Others might
focus on specific impacts on women, land tenure, food prices, living standards, or
nutrition.
d) Diagnostic case studies. Diagnostic case studies may provide a holistic, in-depth descriptive
assessment of a program or local situation. These may combine information from various sources and can be
useful to: (a) analyze potential social, technical, and financial factors affecting uptake of new agricultural
technologies; (b) assess potential and economic benefits from technological innovation; (c) assess differential
impacts of programs on different groups; and (d) identify key issues for effective extension program operations.
Agricultural Innovation Projects should frequently build routine field case studies into M&E systems. In
fact, it may be difficult to envision an efficient technology system without a functional system for such
studies to assess how well a project is doing and how well technologies are being accepted. A key challenge is
that of relating "with-project" changes
to the counterfactual "without-project" scenario.
e) External peer reviews. Peer reviews are used in highly technical program areas, most commonly
in research. External peer reviews are useful to obtain technical experts' assessment of research activities
from three perspectives: (a) a prospective review (to assess implications and relevance of research); (b) an in-
process review (to assess efficiency of research and performance against targets for outputs), and (c) a
retrospective review (to assess scientific or technical quality of work complet-

90

ed). They are also useful in developing linkages between scientists and institutions working in relat-
ed fields. As an example, an external review was conducted for the Agricultural Research and
Extension Project (Adaptable Program Loan) in Peru to evaluate the progress of the first project phase,
with the specific task of verifying if triggers for the specific phase had been reached (the triggers or measures
of project implementation progress must be met prior to any phase will be passed
for approval and execution of the following phase can begin).

91

APPENDIX 5. PROJECT AND
MONITORING & EVALUATION COSTS
Project Costs: Bank project costs are usually prepared using Costab 32 a PC-based project costing
tool. Costab 32 is the project costing component of a software series to improve efficiency and effectiveness
of Project Processing activities. Costab 32, upgraded in 2001 by ADB, and is designed to operate under
Windows 2000 as well as earlier 32-bit versions of Windows. Costab 32 also incorpo-
rates several new features which include:
• Automatic calculation of financial charges during implementation;
• Summary table format for both ADB and World Bank;
• Option to compute economic costs in either domestic economic prices or border prices; and
• Other operational parameters.
Costab 32 helps task team leaders, project economists, and financial analysts organize and analyze
project costing data in the course of project preparation and appraisal calculates physical and price
contingencies, taxes and foreign exchange displays data in detailed cost tables coverts financial costs to
economic costs for use in economic analysis. Costs should be broken down by component and by type of input.
They should be expressed in local currency and foreign exchange, specifying the source
of financing and, where applicable, co-financing arrangements. See table A5.1 for an example.
It is important to ensure that all activities shown in a detail table are incorporated in the preparation
of M&E tables such as a standard Log Frame or the results framework arrangements matrix.
Table A5.1. A Project Component Table from Costab 32.

Unit Cost 05/06 06/07 07/08 08/09 09/10 Total
I. Investment Costs
A. Review of national acts,
policies and regulations
Review of existing
Land mgt. guidelines 329,787.234 134,553.2 104,951.5 109,149.5 - - 348,654.2
Awareness raising and IEC 329,787.234 134,553.2 104,951.5 109,149.5 - - 348,654.2
Field surveys and validation 329,787.234 168,191.5 174,919.1 - - - 343,110.6
SLM policy regulatory
capacity building 329,787.234 168,191.5 174,919.1 - - - 343,110.6
Subtotal review of national
acts, policies and regulations 605,489.4 559,741.3 218,299.1 - - 1,383,529.7

B. Ssatellite imagery of
3 Dzongkhags
Satellite image of Radhi 1,000,000 - 1,060,800.0 - - - 1,060,800.0
Satellite image of Nangkhor 1,000,000 510,000.0 530,400.0 - - - 1,040,400.0
Satellite image of Phuentsholing 1,000,000 510,000.0 530,400.0 - - - 1,040,400.0
Subtotal satellite imagery of
3 Dzongkhags 1,020,000.0 2,121,600.0 - - - 3,141,600.0
1,625,489.4 2,681,341.3 218,299.1 - - 4,525,129.7


92

Below is a typical M&E budget plan for an IDA/GEF environmental project, including a breakdown
of key expense categories.
Marine and Coastal Environment Management Program
United Republic of Tanzania
Indicative M&E Budget

Unit Price PY1 PY2 PY3 PY4 PY5
Core Staffing and
Administration Quantity Cost Quantity Cost Quantity Cost Quantity Cost Quantity Cost
Staffing
M&E Coordinator PCU 1 $ 1,172 1 $ 1,172 1 $ 1,172 1 $ 1,172 1 $ 1,172
MIS Analyst PCU 1 $ 956 1 $ 956 1 $ 956 1 $ 956 1 $ 956
Statistician/Analyst PMU 1 $ 109 1 $ 109 1 $ 109 1 $ 109 1 $ 109
Office equipment
Computers $1,500 3 $ 4,500 0 $ — 0 $ — 0 $ — 0 $ —
Printers $ 250 3 $ 50 0 $ — 0 $ — 0 $ — 0 $ —
UPS $ 100 3 $ 300 0 $ — 0 $ — 0 $ — 0 $ —
Furniture $1,000 2 $ 2,000 0 $ — 0 $ — 0 $ — 0 $ —
MIS Applications
Applications $8,600 1 $ 8,600 1 $ 8,600 1 $ 8,600 1 $ 8,600 1 $ 8,600
Training
Intro to Computing
Computing
applications
Statistical analysis
Report preparation
Data base and MIS Network
principles
Beneficiary training $ 3,4001 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400
Field visits $1,500 1 $ 1,500 1 $ 1,500 1 $ 1,500 1 $ 1,500 1 $ 1,500
Workshops $3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400
Assessments, Studies, and Surveys
Baseline surveys $4,800 1 $ 4,800 0 $ — 0 $ — 0 $ — 0 $ —
Annual M&E
Review and
Reports $3,600 1 $ 3,600 0 $ — 0 $ — 0 $ — 0 $ —
M&E workshop $2,200 1 $ 2,200 0 $ 2,200 0 $ 2,200 0 $ 2,200 0 $ 2,200
Safeguards report $4,500 1 $4,500 0 $ — 0 $ — 0 $ — 0 $ —
Social and Environ
Impact
Assessments $5,500 1 $ 5,500 0 $ — 0 $ — 0 $ — 0 $
continued

93

Marine and Coastal Environment Management Program—continued

Unit Price PY1 PY2 PY3 PY4 PY5
Core Staffing and
Administration Quantity Cost Quantity Cost Quantity Cost Quantity Cost Quantity Cost
Mid-term review $1,200 1 $ 1,200 0 $ — 0 $ — 0 $ — 0 $ —
Annual audit $3,200 1 $ 3,200 0 $ — 0 $ — 0 $ — 0 $ —
Operations and
Maintenance
Supplies and
Maintenance $3,042 1 $ 3,042 1 $ 3,042 1 $ 3,042 1 $ 3,042 1 $ 3,042
Installation $4,000 1 $ 4,000 1 $ 4,000 1 $ 4,000 1 $ 4,000 1 $ 4,000
Total $58,729 $28,379 $28,379 $28,379 $28,379


94

APPENDIX 6. ECONOMIC EVALUATION OF COMPETITIVE
RESEARCH GRANTS: THE CASE OF PROMSA, ECUADOR
OBJECTIVES, EXPECTED RESULTS,
AND MAIN CHARACTERISTICS OF PROMSA

From the time leading up to the implementation of a CGS in 1999 Ecuador's national research sys-
tem was dominated by a centralized public research institute, Instituto Nacional Autonomo de
Investigaciones Agropecuaria (INIAP). Because public agricultural research funds were dedicated almost
exclusively to supporting INIAP, financing was scarce for other potential suppliers of agricultural research
such as private and public universities, NGOs, and private firms. The number and level of training of
agricultural scientists in Ecuador was very low as well. In 1998 the total number of agricultural scientists
holding a Ph.D. degree was just 16, and only 71 held an M.Sc. INIAP linkages with domestic and
international research entities were weak, limiting expertise and technology spill-ins. In 1998, INIAP had
only 6 national, and 7 international research agreements with total
funding of less than $200,000.
The original objective of World Bank-support for agricultural research in Ecuador was to strengthen
the national capacity to do agricultural research, with the aim of increasing agricultural productivity and
output by (a) introducing a competitive research grant program; (b) developing research and education
exchange partnerships with international science institutions; and (c) strengthening the
national agricultural research institute, INIAP.
Ecuador's Competitive Grants System provides grants for individual research projects, and for
research and educational exchange partnerships with international institutions. The former grant type was
implemented as a mechanism for (a) increasing the efficiency of fund allocation for agricultural research;
(b) assuring the participation of scientists in agricultural research from diverse institutions; and (c)
strengthening linkages between public and private sector entities. The latter grant type was implemented to
finance the formation of research and education partnerships between research agencies, universities, key
NGOs, and international scientific institutions to facilitate the importation of applicable technology,
strengthen the research capacity of Ecuadorian research institutions,
and strengthen graduate education programs in Ecuadorian universities.
By 2004 the National Modernization Program for Agricultural Services (PROMSA) had financed
108 individual research projects carried out by national research entities, and 14 international alliances
for research and educational programs. All programs follow the same competitive granting procedure.
Individual research projects supported by the fund were not to last more than three years or cost more than
$100,000, and were to have co-financing from other sources of at least 25%. The maximum size of
international alliances was set at $500,000. A Project Implementation Unit (PIU) was created at project start-
up and an international private firm was contracted to set up a monitoring and evaluation system (M&E). In
addition to monitoring the progress of funded projects, the PIU is responsible for preparing annual budgets and
work plans, making payments from the loan account
for project expenditures, and preparing financial and progress reports.

95

Monitoring a competitive grant system
The management system that has been implemented in Ecuador is sophisticated and has contributed
substantially to the success of the competitive fund mechanism in Ecuador. The PIU prepares an annual
operating plan and quarterly reports on implementation of the proposed work plan. PIU field officers visit each
project at least twice each year to observe first-hand processes being employed in administering the
competitive fund. The computerized Monitoring and Evaluation system is state of the art. The project database
provides ready access to data on implementation of projects, and overall performance of the project portfolio.
The database is relatively simple to maintain and use to gen-
erate information for reports.
Good practice: Adequate resources must be assigned to M&E activities. Resources should be ade-
quate to staff an M&E unit, establish a database, and provide consistent and timely feedback to
subproject scientists.
A system of objectives and milestones (for inputs, activities, outputs) with projected dates for mon-
itoring project implementation has been established and forms the basic elements for a log frame approach.
The M&E system is relatively simple to understand and manage. It is primarily based on quarterly progress
report by the project researchers. Data on all projects and milestones are recorded in the M&E database. PIU
prepares an annual evaluation of subprojects based on (a) annual reports by researchers, and (b) formal and
informal reports by grupos de referencia (GR) (locally run individual project advisory boards) for each
project, (c) bi-annual visits by PIU specialists to each
project.
Good practice: Formal project milestones must be established and monitored.
A GR has been formed to oversee the implementation of each subproject. GRs are an innovative
approach to decentralized M&E in which beneficiaries and researchers collaborate. Each GR is comprised of
4-8 stakeholders (other researchers, extension staff, and agribusiness). The exact group com- position and function
are flexible, but ideally the GR is expected to participate in project design, planning, and implementation.
The groups meet several times a year to consult on project issues. The PIU has also solicited feedback from
project leaders through a survey of their opinions about the CGS process. Funded researchers report that the PIU
M&E reporting system is a substantial improvement
over the lengthy reporting often required by their own institutions or other funding agencies.
Good practice: Advisory groups should be organized. These groups can provide valuable feedback
into the M&E process.
Good practice: Reporting requirements should be designed to collect a minimum data set in an
efficient manner. The needs for M&E information must be balanced with a respect for the burden
that it places on busy scientists.
The PROMSA M&E system employs a system of "alert" signals or flags that indicate if a project has
become a "problem project." There are four ways a project can be flagged as a problem: (a) quarterly or annual
reports that are overdue by more than a month, (b) completion of less than 80 percent of the quarterly
milestones, (c) a rating by PIU field personnel of less than 2.5 on a 4-point scale, or (d) general problems in
quality of implementation. While in alert status, a project is not eligible for new disbursements from the
Fund. If the project does not clear its alert status by correcting deficien-
cies or rescheduling milestones within a specified time frame, the project can be terminated.

96

Good practice: An alert system should be implemented to enforce communication between
researchers and CGS administrators. The criteria for triggering an alert should be clearly under-
stood and uniformly applied.

Evaluation of a competitive grant system
Most funded projects are not expected to have results diffuse to end users until five or more years after
the start of research. This makes it impossible to conduct a realistic economic evaluation of the CGS. Therefore,
evaluation must focus on success in meeting overall CGS objectives, in creating necessary research institutional
infrastructure, in measuring progress in implementing the proposed subproject research, and in generating
research outputs such as publications and technologies. This evaluation relies on indicators that can be extracted
from the PIU database (see the list below). There are a large number of potential indicators of CGS success,
some of which are more important than others. Not all of those listed in the appendix are necessary for each
evaluation. Indicators should allow evalua-
tion of the degree of success in institutional, administrative, and scientific aspects of the CGS.
Good practice: When the CGS funds research projects of longer gestation economic analysis will
not be meaningful until the project has been active for several years. During this period indicators
that allow evaluation of institutional, administrative and scientific progress should be monitored.

List of potential M&D indicators
1. Research Project Indicators (both at project type and portfolio level)
• Number of research project applications
• Number of research projects approved
• Number of research projects established
• Number of research projects by priority area
• Number of projects involving basic, strategic, applied, adaptive research
• Number of projects financed in small-scale agriculture
• Number of technical and scientific publications (differentiate according to quality such as
peer-reviewed articles, pamphlets, or manuals for farmers)
• Number of workshops
• Number of international alliances established
• Labor-capital ratio (e.g., person months—value of equipment)
• Recurrent costs or personnel costs relative to research budgets
• Cost per researcher
• Number of technologies transferred to farmers fields
• Rates of technology adoption by farmers
2. Education Project Indicators (both at project type and portfolio level)
• Number of short-term course, MSc. Program, Ph.D. program application
• Number of short-term courses, MSc. Programs, Ph.D. programs approved

97

• Number of short-term courses, MSc. Programs, Ph.D. programs established
• Number of short-term courses, MSc. Programs, Ph.D. programs by priority area
• Number of MSc. degrees, Ph.D. degrees awarded
• Number of fellowships awarded
• Number of MSc. degrees, Ph.D. degrees awarded by international institutions
• Number of MSc. degrees, Ph.D. degrees awarded by national institutions
• Number of international alliances established
• Length of time required to complete degree programs or obtain certificates
• Cost of training programs per trainee
• Training program costs recovered from payments by users
• Employment rate of graduates
3. Overall Institutional Indicators (both at project type and portfolio level)
• Total number of project applications, projects approved, projects established
• Number of grant applications by INIAP, universities, private sector, others, joint projects
(define type of partnership)
• Number of grants awarded by INIAP, universities, private sector, others, joint projects
• Number of projects established by INIAP, universities, private sector, others, joint projects
• Amount of resources (%) captured from INIAP, universities, private sector, others, joint
projects (define distribution of partnerships)
• Ratio of project/counterpart funding
• Amount of resources from government of Ecuador
• Ratio of professional to support staff and trained technicians
• Resource allocation in relation to established priorities
• Competitive funding as share of total research funding

98

NOTES

Chapter 1. Challenges for M&E Systems in Agricultural Research and Extension Projects
1. The Note is a follow-up to earlier Bank publications in M&E (Monitoring and Evaluation for
AKIS Projects by Alex and Byerlee, 2000, Ex-Ante Economic Analysis in AKIS Projects by
Horstkotte-Wesseler et al., 2000, Beneficiary Assessment for Agricultural Extension by Salmen, 2000,
and Ten Steps to Results-based Monitoring and Evaluation by Kusek and Rist, 2004) by accounting for
recent changes in the design of World Bank-supported research and extension (R&E)
projects and the trend toward results-based M&E.
2. The driving forces behind change in ARE systems stem from deficiencies experienced in tradi-
tional systems: poor fiscal sustainability, difficulties with attribution of impact, low political com-
mitment, insufficient local content, weak accountability/ transparency, and weak linkages between research
and extension. Anderson, J. and G. Feder, 2004. Agricultural extension: good intentions and hard realities. The
World Bank Research Observer, vol. 19, no 1. and Rajalahti, R., J. Woelcke, and E. Pehu, 2005.
"Development of Research Systems to Support the Changing Agricultural Sector."
Proceedings of a Research Workshop, World Bank, Washington, DC.

Chapter 2. The World Bank Results Framework Requirements
3. World Bank, 1997. "Proxy indicators." The Log Frame Handbook: A logical framework approach
to project cycle management. World Bank, Washington, DC.
4. A World Bank Toolkit: Integrating Gender Dimension into Monitoring & Evaluation of Rural
Development Projects; and a Thematic Brief: Agricultural Knowledge and Information Systems;
Agricultural Research, Extension, and Education, 2001 provide further examples on gender-
disaggregated indicators.
5. Alternatively, the task team could present the component outcomes as combined outcomes at the
PDO level. This was common practice in the past. The advantage may be that it is in some cases eas-
ier just to define the combined effects of the components at the PDO level.

Chapter 3. Data Collection, Reporting and Dissemination Requirements
6. Baseline and targets based on weighted average of farmer income in the different sub-projects
from information provided in the proposals.
7. Association deemed successful if over 60% of the farmers are satisfied with the services provid-
ed by the Association.
8. Guidelines for Management Information Systems in Social Funds.
9. Relevant Bank publications in this context include: Murphy and Marchant (1988) discussion of
issues in planning M&E systems for extension agencies; Casely and Kumar (1987) discussion of issues
for agricultural sector programs in general; World Bank (2003b)—the most recent Bank pub- lication on
M&E. in agricultural operations; and Kusek and Rist (2004)—the most recent discussion of M&E system for
development programs in general—with particular focus on results-based M&E
systems.

99

10. The thematic areas may include variety development; soil and water conservation; integrated pest
management; agro-forestry; marketing; and livestock production in different agro-ecological zones.
11. Poor sampling or inadequate mix of information may introduce significant bias which might
result in a disconnect between the objectives and true outcomes of the project.
12. Examples include the one by the World Bank (http://www.worldbank.org/data), FAO
(http://www.fao.org/waicent/portal/statistics_en.asp) and IFPRI ASTI at http://www.asti.cgiar.org/.
13. This section written by Ghazali Raheem, Consultant, World Bank.

Chapter 6. Economic Evaluation of Competitive Grants Schemes
14. It is usual in the empirical literature on project evaluation to classify projects' impacts, either
potential or realized, in three main categories: economic, institutional, and environmental. This chap-
ter deals with impacts within the first category—those that can be measured in monetary terms.
15. Details of practical procedures of incremental cost benefits analysis can be found in: CIMMYT
(1988). From Agronomic Data to Farmer Recommendations: An Economics Training Manual.
Revised Edition. CIMMYT: Mexico.
16. CIMMYT (1998) provides an excellent discussion about how to estimate the incremental net
benefit between technological alternatives for annual crops. Extension to more than one period for
perennial crops or technologies with longer beneficial effects is straightforward.
17. A partial budget compares estimated revenues and costs for a typical producer with and without
the new technology.
18. It is recommended to transfer this simple table into EXCEL which would allow for a convenient
calculation of NPV and IRR.
19. Methods for estimating the diffusion and the estimation of incremental net benefits have been
explained in section 6.4.
20. For more details see section 6.1: economic impact evaluation of a CGS; and section 6.5: estima-
tion of IRR.
21. Expert Opinion is often used in conjunction with simulation analysis. In this case the experts are
asked for their estimation of three values of the parameters of interest: its most likely value, the max-
imum, and the minimum.
22. The specific assignment of factors to the different categories of investment costs clearly depends
on the nature of the project and no general recommendation can be made. However, as an example, the
following factors have been assigned to the different categories of investment costs for the Modernizing
Agricultural Knowledge and Information Systems Project (Romania): civil works- factor: 0.1;
equipment-factor: 1; vehicles-factor: 1; technical assistance-factor: 0; training-factor: 0;
competitive grant: 1.
23. A commonly used indicator for the level of public financial support of agricultural research is
public expenditure for agricultural research as percentage of the agricultural GDP (AgGDP). Figures

100

for public research expenditures can be obtained from the Ministry of Agriculture. Another recom-
mendable source is the Agricultural Science and Technology Indicators (ASTI) provided by the
CGIAR (http://www.asti.cgiar.org).

Appendix 1. Narratives of New Approaches and Instruments in ARE Systems
24. Anderson, J. and Feder, G. 2004. "Agricultural extension: Good intentions and hard realities."
The World Bank Research Observer, vol. 19, no 1. and Rajalahti, R., J. Woelcke, and E. Pehu 2005.
Development of Research Systems to Support the Changing Agricultural Sector. Proceedings of a
Research Workshop, Washington, DC: World Bank.
25. Farmer-to-farmer methodology is one type of participatory research and extension methodolo-
gy. Here, it includes Farmer Field Schools, CIAL, use of volunteer contact farmers, and producer
organizations.
26. Mass media and new ICT applications can be integrated into the strategic extension programs.
Media applications include both "new and old" communication devices and methods, such as
posters, booklets, radio (and enter-edutainment), television, drama, internet, and mobile phones.
27. World Bank (2003a). "Extension and rural development—Converging views on institutional
approaches?" Workshop Summary. Agriculture and Rural Development Department, World Bank,
Washington DC.
28. World Bank (2004b). Investments in Agricultural Science and Technology. Agriculture Investment
Sourcebook. Module 2. World Bank, Washington, DC.
29. Veldhuizen, L., A. Waters-Bayer, and H. Zeeuw 1997. Developing technology with farmers. A
trainers guide for participatory learning. London: Zed Books Ltd.
30. Braun, A., G. Thiele, and M. Fernandez 2000. "Farmer Field Schools and Local Agricultural
Research Committees: Complementary platforms for integrated decision-making in sustainable agri-
culture." AgREN Network Paper No. 105.

Appendix 2. Implementation of a Monitoring and Evaluation Plan in the Bank's Project
Cycle: Putting It All Together
31. This chapter written by Remileku Rakey, Consultant, World Bank.

Appendix 4. List of Indicators and Data Sources for ARE Projects
32. The following section is taken mainly from Alex and Byerlee (2000).
33. Examples include (e.g., the one by the World Bank http://www.worldbank.org/data) and FAO
(http://www.fao.org/waicent/portal/statistics_en.asp).
34. Important data sources include the CGIAR centers which regularly conduct relevant surveys in
developing countries (e.g., IFPRI in Uganda).

101

35. Grosh, M. and P. Glewwe (2000). Designing household survey questionnaires for developing
countries: lessons from 15 years of the Living Standards Measurement Study Volumes 1-3. World Bank,
Washington, DC. On agriculture, volume 2 lays out the most important agricultural policy issues in
developing countries, discusses the date needs and measurement concerns, and introduces three versions of a
draft agriculture module. Volume 3 presents draft questionnaire modules on several topics. On agriculture, the
issues covered include land holdings, farm capital inventory, disposi- tion of crops, input purchases and
sources, access and use of agricultural extension services, farm capital inventory and transactions, crop output,
household member labor inputs, hired labor and non-
labor inputs, input prices, expenditures and sources, and livestock.

102

Monitoring And Evaluation For World Bank Agricultural Projects

More Related Content

What's hot (20)

Similar to Monitoring And Evaluation For World Bank Agricultural Projects (20)

More from Malik Khalid Mehmood (20)

Monitoring And Evaluation For World Bank Agricultural Projects