Unicef guide 4 monev

http://www.unicef.org/reseval/index.html

A UNICEF Guide for Monitoring and
Evaluation
Making a Difference?
Section I - Introduction
Section II - Organization of Monitoring and Evaluation
Section III - Strengthening Monitoring
Section IV - Strenghening Evaluation
References and Bibliography
Appendices
UNICEF Home | Information | Participation | Organization | Activities

Section I - Introduction
1. Scope
This manual covers monitoring and evaluation policies and procedure. Section II presents the
organization, of monitoring and evaluation in UNICEF: the roles of UNICEF staff in country,
regional and headquarters offices; their responsibility to strengthen national capacity to monitor
and evaluate; and the place of monitoring and evaluation in UNICEF's programming cycle. The
next section is on monitoring: first, of services, programmes and projects; second, of the
situation of women and children in the country and third, of global goals. The final section, on
evaluation, follows the stages of an evaluation: planning, managing, conducting and using the
results of evaluations.
A distinction can be made in analysis, and sometimes in practice, between the monitoring and
evaluation of
a) what the government is doing in a particular field, e.g. an MCH service, primary health care,
an area development programme, and what system of monitoring exists for these
programmes, and
b) what an outside co-operating agency such as UNICEF is doing in a particular field, e.g.
provision of technical assistance, financial and material inputs; the agency wrongly, but
frequently, calls this "its" programme.

The UNICEF Executive Board, like the governing bodies of other agencies, is interested in b)
above. However, they cannot cover effectiveness, efficiency, impact, sustainability, etc., without
a) as a pre-condition. The a) should be done by country authorities, though they may well have
outside support for institutional strengthening, and even their operations. In the worst case,
when outsiders have to attempt a), they must have, as a minimum, the co-operation of the
authorities.
Most of what follows is related to a), for two reasons. Firstly, strengthening country capacity is a
UNICEF objective, so that monitoring and evaluation become regular activities. Secondly, it is
more difficult to provide and support a) than to perform b). Reports emanating from a) may
serve b) with some additions. If a separate report is needed for b) less separate discussion is
necessary about how to prepare it because it will be based on a) and follow the same
guidelines.
Government officials responsible for implementing activities in which UNICEF co-operates and
UNICEF field staff are generally responsible for monitoring, planning evaluations and using
evaluation results.
This manual explains monitoring and evaluation processes and emphasizes practical
suggestions. The examples used are from health services in UNICEF programming.
The manual sets out the ideal for which UNICEF is aiming. Field offices need to adjust these
recommendations according to their situation and in light of the UNICEF goal to strengthen
national capacities. For example, a less than perfect evaluation in which local people are fully
involved may be more useful than an elegant evaluation done by outsiders.

2. Definitions
Monitoring is the periodic oversight of the implementation of an activity which seeks to
establish the extent to which input deliveries, work schedules, other required actions and
targeted outputs are proceeding according to plan, so that timely action can be taken to correct
deficiencies detected. "Monitoring" is also useful for the systematic checking on a condition or
set of conditions, such as following the situation of women and children.
Evaluation is a process which attempts to determine as systematically and objectively as
possible the relevance, effectiveness, efficiency and impact of activities in the light of specified
objectives. It is a learning and action-oriented management tool and organizational process for
improving both current activities and future planning, programming and decision-making.

3. Importance of Monitoring and Evaluation
To evaluate means "to ascertain the value or worth of," according to its Latin root. Knowing what
difference programmes are making motivates workers and their supporters to renewed effort.
Although evaluations may be retrospective, they are essentially forward looking with regard to
their purpose. Evaluation applies the lessons of experience to decisions about current and
future programmes. Good evaluation presents alternatives for decision-makers to consider.
Evaluation can be an excellent learning tool as well as a means to improve programme
performance and demonstrate accountability.

Too often evaluation is perceived as threatening; it should be constructive. For example, an
evaluation can be tapped for developing human resources and improving management and
planning capabilities. Evaluation results can be used in advocacy and fundraising efforts to
obtain greater support from governments, private organizations, and the general public.
The relationship between monitoring and evaluation
Both monitoring and evaluation are management tools. In the case of monitoring, information for
tracking progress according to previously agreed on plans and schedules is routinely gathered.
Discrepancies between actual and planned implementation are identified and corrective actions
taken. When findings are used to monitor the development results (effects, impacts) it is
sometimes referred to as ongoing evaluation.
Evaluation is more episodic than monitoring. It is facilitated by monitoring but utilizes additional
sources of information. Many such sources are identified during project reviews when there is a
need to understand why inputs did not lead to planned outputs. Evaluation focuses on specific
questions related to effectiveness and impact in order to influence future programmes or
services.
Impact assessment is often difficult because causality is difficult to determine, in addition to
being costly and time-consuming. However, managers need to know the effects of project
activities on the intended beneficiaries during implementation. Community monitoring
programmes can record impacts locally and use results to modify project activities. Impacts may
be assessed informally, through conversations with beneficiaries, women's groups, village
elders. This allows managers to adjust strategies, if necessary, during implementation, rather
than continue less than effective activities.
As a practical matter, UNICEF recommends that monitoring focus on project progress. While a
few outcome indicators might be monitored, such as nutritional status of project beneficiaries,
formal assessment of overall impact and cost effectiveness is best handled by conducting
evaluations (CF/EXD-IC-1986-8).
In April 1984, the UNICEF Executive Board requested that monitoring and evaluation be
strengthened in the field, regions and headquarters and that the quantity and quality of
monitoring and evaluation be improved. Two to five percent of UNICEF country programme
funds should be allocated for evaluative activities (CF/PD/PRO/1986-001, p.3).

The complementarity between monitoring and evaluation is seen in the following table:
Table I-1: Complementarity between monitoring and evaluation
Item
frequency
main action

Monitoring
periodic, regular
keeping track / oversight
improve efficiency
basic purpose
adjust work plan
inputs, outputs, process outcomes, work
focus
plans
routine or sentinel systems, field
information
observation, progress reports, rapid
sources
assessments
programme managers
community workers
undertaken by
community (beneficiaries) supervisors,
funders

reporting to

programme managers
community workers
community (beneficiaries)
supervisors, funders

Evaluation
episodic
assessment
improve effectiveness, impact,
future programming
effectiveness, relevance, impact, costeffectiveness
same, plus surveys, studies
programme managers
supervisors,
funders
external evaluators
programme managers
supervisors, funders
policy-makers
beneficiaries

The objectives of monitoring and evaluation in UNICEF are:
a. to improve management of programmes, projects and supporting activities and to ensure
optimum use of funds and other resources;
b. to learn from experience so as to improve the relevance, methods and outcomes of cooperative programmes;
c. to strengthen the capacity of co-operating government agencies, non-governmental
organizations (NGOs) and local communities to monitor and evaluate;
d. to meet the requirements of donors to see whether their resources are being used
effectively, efficiently and for agreed upon objectives; and
e. to provide information to enhance advocacy for policies, programmes and resources,. that
improve the condition of women and children.
Achieving the first purpose - improving management requires better monitoring and evaluation
throughout the programming cycle and prompt supply of information to decision-makers. The
second purpose - to learn from experience, develop and refine intervention policies - will be
achieved only if procedures are set up to disseminate findings to decision-makers and to use
them for corrective action. The third - strengthening national capacity - requires working with
responsible officials and programme staff, and often involves supporting institutional
strengthening. Meeting donor requirements, the fourth purpose, relates to fundraising and often
depends on occasional external evaluations carried out by teams of specialists. Finally,
advocacy for improved policies and programmes and mobilization of greater personal

commitment and financial support require credible information about progress, problems and
potential derived from monitoring and evaluation.

4. New Directions for Monitoring and Evaluation in UNICEF
UNICEF's proposed strategy for the 1990s emphasizes the significance of monitoring the
condition of children. Development goals proposed by UNICEF, WHO and others for the Fourth
United Nations Development Decade are based on the conviction that the survival, development
and protection of children should be one of the means and ends of a strategy of development
with a human face, one that aims to overcome poverty and enhance human capabilities. The
specific goals proposed are measurable targets. For example, a major goal is to reduce the
infant and under-five mortality rates by one-third (or to 50 and 70 per 1000 live births,
respectively) by the year 2000. The list of goals is found in Appendix F.
This goal-oriented strategy calls for implementing a system to track progress. The monitoring
systems which are developed can be tapped for advocacy, social mobilization and leveraging
resources from others. While developing five-year national development plans and joint country
programmes with UNICEF, policy-makers in each country must identify goals and measures for
determining progress toward each objective, which they deem appropriate to their country.
Some baseline data should be established for each goal and systems set up to collect data to
monitor progress at certain milestones.
New directions in evaluation were identified at the first meeting of UNICEF's field staff
"evaluation focal points" in Florence in June 1989. The following five points were raised:
a. Evaluation as co-operation - UNICEF is no longer the small supply organization of its
origins. Its primary focus shifted from providing relief to promoting development. In the
1980s, emphasis shifted more to co-operating with governments to set targets specific goals
to measure improvements in the situation of children and quantifiable objectives for
programmes and projects. Monitoring and evaluation, ways of assessing progress, grew in
importance as UNICEF adopted this co-operative planning approach.
b. Evaluation and money - UNICEF's annual budget increased from $200 million in 1980 to
more than 600 million in 1990. As funding increases, so will the need for accountability
provided by objective evaluations. Moreover, UNICEF's resources are small compared to its
mandate, and rigorous cost analysis enables policy-makers to extend benefits by identifying
the most efficient means to achieve objectives.
c. Evaluation and expertise - UNICEF will need to increase its technical and evaluation
expertise to correspond with these changes. This entails expanding UNICEF's own
evaluation capability, in-house, and that of officials responsible for activities in which
UNICEF co-operates. In addition, UNICEF will continue to call on consultants, universities
and research institutions to undertake evaluations of programmes in which it is co-operating.
d. Evaluation for action - All evaluation must be oriented toward action. Feedback of findings to
the communities involved is as important as feedback to the managers and policy-makers
who can implement recommendations. These communities are not only beneficiaries but
also key actors who carry out programme activities.

e. Evaluation is communication - Like all communication, evaluation works best within a
relationship based on mutual respect and trust. Honest communication should be
supportive, rather than threatening, and it should also be constructive. Evaluations should
be useful and credible to those who can implement recommendations.

Section II - Organization of Monitoring and
Evaluation
1 - Scope
This section covers (a) monitoring and evaluation - the responsibility of government officials,
UNICEF and other donors, (b) how evaluation and monitoring are organized in UNICEF and
what is being done to. strengthen capacity to carry out these responsibilities, (c) UNICEF's
responsibility to strengthen national capacity for monitoring and evaluation, and (d) how
monitoring and evaluation fit into UNICEF's programming cycle. The approach taken is to
describe a fully developed model of evaluation or monitoring. This may not be suitable for some
country situations, but it is easier for the user to simplify the model than to have to make it more
sophisticated.
Both government officials and UNICEF staff are responsible for monitoring and evaluation of
programmes and projects supported by UNICEF. Strengthening government capacity to
conduct development programmes, including the ability to monitor and evaluate them, is one of
the objectives of UNICEF co-operation. The ideal model described here, however, is not always
possible to attain in practice. In some cases, UNICEF staff may have to play a more active role
in monitoring and evaluation activities compared with that of their government counterparts.
Responsibilities
Monitoring progress and evaluating results are key management functions to improve
programme performance of government officials, from the local to the national level, who are
responsible for implementing programmes.
Government officials who supervise programme managers or allocate public resources use
evaluations to ensure accountability, strengthen programmes, decide among activities to fund,
and provide information for policy-making.
Donor agencies are responsible to their contributors for evaluating the effectiveness of the use
of their resources, consistent with their mandates. Donors also use evaluations for
accountability, improving performance, and funding decisions. UNICEF staff must often oversee
how donor resources are used and present reports to the donor agencies. While such
evaluations are no longer a responsibility shared with the country, they can be soundly based
only if they have evaluations of the country activity to build on, and these must inevitably be
made by the country or at least in co-operation with it.
Close communication and negotiation are necessary while planning and carrying out monitoring
and evaluation processes. Deciding who will carry out each of the many tasks involved in
monitoring and evaluation is an important component of programme and project planning.

2. Roles and Responsibilities in UNICEF
In general, the responsibility for monitoring and evaluation in UNICEF lies with the country
offices. Support for monitoring is given by the UNICEF Programme Division in New York and
the Evaluation Office provides assistance for evaluation activities. Regional Offices have an
important responsibility in monitoring country evaluation plans and providing support for
monitoring. They also provide training and technical assistance.
Responsibilities of the Country Representative and Country/ Area Office
The country Representative should ensure that sufficient UNICEF staff resources and time are
dedicated to the tasks involved in monitoring and evaluation at the different stages of the
programming cycle. Also, the Representative should see that adequate communication occurs
with government officials and that they participate in decision-making throughout the process.
Such government participation is especially vital in the programme preparation stage so the
services and activities supported by UNICEF are "owned" by the officials responsible for
implementing them. The country office can provide training and technical assistance to increase
local planning, monitoring and evaluation capacities.
A. Tasks during the situation analysis stage
1. identification of baseline data or means to collect it;
2. aggregation and disaggregation of data;
3. analysis of trends and prospects; and
4. analysis of needs and identification of constraints and possibilities for action.
B. Tasks during the programme preparation stage
The plan for the activity or service in which UNICEF will co-operate should include:
1. objectives that are as specific as possible;
2. indicators and criteria to measure success or progress;
3. monitoring and evaluation activities;
4. budgetary provisions for- monitoring and evaluation;
5. examination of the national capacity for monitoring and evaluating programmes;
6. measures to strengthen country capacity to monitor and evaluate the programme, e.g.
training, technical back-up and institutional strengthening; and
7. opportunity to share, review and use results.
All these provisions have to be adapted to the country's political, administrative and budgetary
context, and some may need discussion with senior officials.
C. Tasks during the programme implementation stage
Monitoring by UNICEF field staff should include:
1. ensuring that data collection is proceeding according to the monitoring plan;
2. reviewing regular progress reports with managers, comparing progress to what was planned;
3. making field visits to project sites;
4. monitoring UNICEF inputs;
5. identifying additional training, technical assistance and other resources that may be needed;
6. obtaining agreement for making annual and mid-course corrections, if needed;
7. changing monitoring procedures, if necessary;
8. identifying additional studies and evaluations needed as the result of programme review; and
9. providing feedback to concerned parties.

D. Tasks during the programme evaluation stage
1. proposing evaluations, when necessary;
2. seeking technical help from the regional or headquarters office, when needed;
3. discussing the evaluation with appropriate ministries, departments, sections and other
partners;
4. reaching agreement on the audience and purposes of the evaluation;
5. obtaining agreement on the Terms of Reference;
6. agreeing on the selection of a team;
7. briefing the evaluation team jointly with country officials;
8. following the progress of the evaluation, and giving support to the team;
9. participating in discussions of the recommendations; and
10. promoting the implementation of the recommendations and use of evaluation results in
present and future programming.

Responsibilities of the Evaluation Focal Point in the country office
Approximately 70 UNICEF field offices have designated a staff member as an "evaluation focal
point." Usually a programme or project officer is appointed to this position but sometimes
another staff person, from the Representative to junior programme officer, is designated. The
chief functions of the focal points are:
a. helping design, update, implement and monitor their office's plans to promote and support
evaluations and share evaluation results;
b. helping link evaluation with planning, that is, incorporating lessons learned from evaluations
into country programme- and project-level planning;
c. proposing topics for training in evaluation to meet needs of government officials and
UNICEF staff,
d. advocating improved monitoring and evaluation, especially with regard to activities in which
UNICEF is co-operating;
e. keeping informed of technical issues in evaluation;
f.

consulting with other programme officers to identify topics for special research; and

g.

maintaining a regular dialogue with the Evaluation Office in New York in relation to these
issues.

Responsibilities of the Regional Director and the Regional Office
The six UNICEF regional offices have planning and evaluation officers. Their responsibility is to
strengthen UNICEF and national capacity by:
a. providing technical support to the field offices;
b. planning training activities;
c. co-operating with Evaluation Office and Programme Division at NYHQ;

d. monitoring evaluation plans in the region; and
d. ensuring that field offices monitor the projects and programmes in which they are cooperating.
Responsibilities of Evaluation Office at Headquarters
The Evaluation Office became an independent unit in 1987 to improve the quality and policyrelevance of evaluation of UNICEF-supported activities. The office reports to the Deputy
Executive Director for Programmes or the UNICEF Executive Director, depending on the task.
Currently, this office has seven professional and six support staff members. The functions of the
Headquarters Evaluation Office are:
a. assisting in setting policies and guidelines for evaluation;
b. building organizational capacity to address evaluation needs;
c. providing technical support and staff training to field offices to improve field capacity in
planning and supporting evaluations;
d. establishing an institutional memory of field evaluation reports;
e. conducting "thematic" evaluations to assess selected sectors or activities in several country
programmes;
f.

responding to donor requests for evaluations;

g. initiating research on selected topics; and
h. providing technical information on evaluations, monitoring, research and studies to country
offices.
The Evaluation Office supports various information exchanges, for example, producing the
UNICEF Evaluation Newsletter (three times a year) and distributing publications on evaluation.
The office also reviews country evaluation plans, assesses evaluations done in the field,
participates in reviews and evaluations in the field, and provides training to field staff. Training
activities include meetings, seminars, and workshops at the global, regional and national levels.
A copy of all evaluations of activities in which UNICEF co-operates should be sent to the
Evaluation Office. This enables the office to disseminate evaluations that may be helpful to
others and provide feedback to country offices.

Initiation of evaluations
The majority of evaluations are initiated at country level, often as the result of advocacy by
country offices. Evaluations are conducted by UNICEF offices for general resources and
supplementary-funded projects alike, according to the workplan. The Headquarters Evaluation
Office, with geographical sections, or donors, initiate approximately 20 per cent of the
equations. These deal with special themes or programmes requested' by donors or the UNICEF
Executive Board and usually concern programme impact, cost and policy issues.

When donors request an evaluation, UNICEF will undertake the evaluation if the recipient
government gives permission, the request is made in a timely fashion and the donor bears the
cost.

Quantity of evaluations supported by UNICEF
The number of evaluations in which UNICEF is involved has increased due to the higher priority
given evaluation. In 1989, 379 evaluations were completed, compared with 129 in 1987 and 218
in 1988. (In 1989, 40 per cent were of immunization programmes and 11 per cent of water and
sanitation projects.)
However, countries infrequently complete the number of evaluations planned for the year.
Nearly 40 per cent of the evaluations scheduled for 1989 were postponed or canceled; but over
half of those completed had not been anticipated in the previous year's annual report.

Quality of evaluations supported by UNICEF
The Evaluation Office in New York assesses every evaluation received from country and
regional offices on the quality of the methodology, findings, recommendations, and presentation
(Figure II-1: UNICEF Evaluation Report Review Sheet). The strengths and weaknesses of
recent evaluations have been identified, as follows:
a. 'Very good" evaluations had a clear purpose, linked with project objectives, used
methodologies judged to be possible models for others, recommendations were clearly
linked with significant findings, and reports usually were well written. Many, however, lacked
an executive summary, limiting their use by policy-makers.
b. 'Poor' evaluations did not relate to project objectives or describe the evaluation methodology
employed, or used methods, such as sampling, inappropriately. Sometimes quantitative
data were not interpreted correctly; other times, qualitative data were omitted. Analysis of
the findings was lacking or poor. For example, some did not indicate progress compared to
a standard or identify reasons for success or failure. Recommendations were weak and not
based on findings. Often, such evaluations were overly philosophical" or subjective and not
sufficiently empirical.

Figure II-1: UNICEF Evaluation Report Review Sheet
Criteria/options for Score

Score

Project objectives listed
Evaluation purposes stated?

1
1

Lessons learnt
- Not specified but some implications
- Clearly specified

1
2

1
1

Recommendations
- Specified?
- Action oriented with timeframe
specified?
- Responsibilities distributed and
spelled out?

1
1
1

Methodology
- Specified?
- Correct

Criteria/Options for Score

Score

Findings
- Some findings, partially meeeting the
objectives
- Many findings, meeting the
objectives
- Many useful findings not only for the
project but also for other projects

1
2
3

Presentation
- Summary?
- Well structured/written?

Analysis
- Little analysis
-Some analysis
- Very good analysis

1
2
3

Total Score
Rating Score 00-05
Score 06-11
Score 12-17

1
1

*
**
***

3. A Co-operative Approach
A valuable lesson that has come out of recent evaluation experience is that evaluations will not
be used unless those who can use the results are involved during the planning and
implementation stages of the evaluation (Patton, 1978). An evaluation should answer questions
and provide information needed for decision-making. Successful evaluations, those that make a
difference, almost always are co-operative efforts. Project managers and other government
officials, funders and other supporters, and beneficiaries should actively participate in planning,
conducting and using the results of evaluations.
The government officials involved include district and local personnel as well as ministry and
department-level officials in the capital. Both service deliverers and civil administrators should
be consulted at the local, provincial and national levels. Usually a hierarchy of officials is
involved in delivering the service or carrying out a project and they should be consulted during
monitoring and evaluation. In addition, views of Civil administrators, such as local authorities,
the district commissioner, the governor, and the political party hierarchy are important. They
may be able to provide not only perspective and another view of the activity, but also valuable
supporting services.
It may not be possible or practical to involve officials from all levels at each of the stages of
design, implementation/data collection and review of findings and recommendations. However,
maximum possible inputs should be sought and particular attention should be placed on
reviewing the recommendations with all concerned parties.
A supportive relationship is threatened if evaluations are seen as negative, adversarial or
threatening. UNICEF staff should orient government officials to view evaluation positively - as a
way to develop management abilities, improve programmes, and help with advocacy.
Participation in evaluations can assist managers to reflect on strategy, design better
programmes, and build national capacity to conduct evaluations.
Government officials should be responsible for key stages of planning monitoring and
evaluations. Specific roles depend on the particular evaluation plan. If evaluations are imposed
from the outside and officials ' involved in the programme have no say in how they are designed
or carried out, institution-building is limited and the evaluation results often ignored.
Several UNICEF offices have tackled this problem by setting up a steering committee of
government and donor representatives to participate in planning and supervising evaluations of

UNICEF-assisted programmes. The committee can make suggestions about who should be on
the evaluation team, meet with the team as it initiates its work, and participate in discussions on
findings and draft recommendations. The committee may also help in disseminating and using
evaluation results.
It is important to include government officials on the evaluation team itself. It helps facilitate
acceptance of findings and promote a sense of ownership. can be done even for external
evaluations, for example by inviting officials experienced in planning or evaluation from another
department. It may only be possible to obtain participation of busy officials at key stages of an
evaluation or programme review.

Co-operating with donors and other supporters
Programmes are often supported by multiple donors, each with different reporting requirements.
This constitutes a burden to managers. Consequently, donor co-ordination in monitoring and
evaluation is highly desirable, in as much as it can conserve human and financial resources.
Multiple donor participation in evaluation can help bring about a coordinated approach to
implementing evaluation results and lead to wider donor collaboration on programming (OECD,
1988; White, 1986).
Additional partners in evaluation may include international and national non-governmental
organizations (NGOs) involved in the same programme or sector in the country and national
organizations such as universities, research institutes, consulting firms, and non-profit
organizations. Representatives of these groups usually understand the cultural, social and
political context; speak the local language (s); do not require airfare; can implement the lessons
learned in other country programmes; and may provide technical assistance or become
programme managers in the future.

Co-operating with beneficiaries
Involving beneficiaries in evaluation is essential, though often neglected. A repeated finding of
evaluations of development assistance is that programmes in which beneficiaries do not have a
voice are likely to fail (OECD, 1988). Beneficiaries should be involved in project selection,
design, implementation and evaluation if their participation is to be effective and their voices
heard.
In every case, evaluators should interview beneficiaries for their opinion on the need for the
service/project, its design, objectives, and possible improvements. Beneficiaries can be involved
further through:
a. discussion during community meetings or with women's groups;
b. beneficiaries and local leaders meeting with evaluators before they design a questionnaire;
c. selection of local persons to conduct interviews or administer questionnaires for the
evaluators;
d. community health workers completing surveys or registers with health data (e.g. children
vaccinated and with faltering growth on growth charts) and providing this information to
Ministry of Health staff; and

e. participatory evaluation.

Participatory evaluation goes further. Not only does it encourage beneficiaries to voice their
views or gather information; it also entails assisting community members to analyse data
themselves and plan actions to improve their situation (Rugh, 1986; Feuerstein, 1986). UNICEF
has assisted such community self-help efforts, for example in the Iringa Nutrition Support
Programme (discussed in Section III, subsection 4).
Supporting local communities to design, implement and evaluate their own activities has many
advantages. Self-help efforts may be more sustainable and have greater impact than others,
because the people themselves are convinced of their value and assume decision-making
responsibility. Training community leaders in evaluation can strengthen their ability to plan and
carry out development activities (Feuerstein, 1986).

4. Strengthening National Capacity for Monitoring and Evaluation
A major UNICEF staff responsibility is to identify the national stakeholders (interested parties) in
a service/programme and help strengthen their capacity for arranging for and using monitoring
and evaluation. This can be done in a variety of ways:
a. advocating that programmes and projects which UNICEF supports be designed with specific
targets and work plans against which progress can be measured;
b. providing technical assistance and equipment for monitoring and evaluation;
c. providing funds for training, workshops, and observation trips locally or within the region;
d. encouraging the participation of beneficiaries in planning, implementing and evaluating;
e. encouraging collaboration between donors and government and
f.

requiring that plans for monitoring and evaluation be part of the country programme
recommendations, plans of operations and plans of action.

Monitoring and evaluation does not depend on highly trained outside experts. Officials and
community groups frequently engage in informal monitoring and evaluation. As noted below,
many elements of the UNICEF programming cycle are evaluative in character. Strengthening
existing skills and activities may be all that is required for an effective evaluation component.
"Demystifying" the evaluation process is likely to enhance its acceptability.
Training programmes for national staff may be designed to cover programme management
(including project development, monitoring and evaluation) or sector-specific topics (such as the
evaluation of water projects or immunization programmes). Training may be provided formally,
through workshops and courses, or informally, through working together. For example,
circulating copies of a well-done monitoring plan or evaluation report of a similar activity gives
managers a concrete idea of what is expected. Formal training followed by on-site technical
assistance is generally more effective than providing formal training alone. Technical assistance
in developing such training activities can be obtained from the Regional Office or Evaluation
Office in Headquarters.

5. Monitoring and Evaluation in Relation to UNICEF's Programming
Cycle
In a broad sense, monitoring is watching over and evaluation is taking stock of a programme,
project or condition. These occur throughout the UNICEF programming cycle, situation analysis,
planning, implementation and evaluation (Figure 11-2: UNICEF's Country Programming Cycle).
UNICEF's programming cycle should be timed to fit in with that of each country in order to be
supportive of national development programming.

Figure II-2: UNICEF's Country Programming Cycle
______________________________________________________________________
Information Collection and Situation Analysis
Review of national policy and past programme performance
Review of Research
Situation studies
Household and special surveys
Census data, etc

Programme Planning
and Project Formulation
Problem identification
Strategy formulation
Objective setting
Constraints analysis
Scheduling, Budgeting

Programme and Project Implementation
Ongoing evaluation, Programme audits, Monitoring, Operational Research,
Standard Input Monitoring System,
Programme Information Management System

Programme and
Project Evaluation
Retrospective evaluations

(Information collection and situation analysis)
_____________________________________________________________________________________________
Source: UNICEF, "Overview of Evaluative Activities in UNICEF" E/ICEF/1984/L.3, 22 February 1984, p. 13.

Evaluation and timing
UNICEF supports a variety of evaluation activities (CF/PD/PRO/1986-001; Glossary, Appendix
A; Sohm, 1978). In each case, the evaluation examines changes and their significance in
relation to one or more of the following issues: relevance, effectiveness, efficiency, impact and
sustainability.
a. Prospective appraisals, sometimes called ex ante evaluations, are carried out before a
programme or project is started. An appraisal is the critical assessment of the relevance,
feasibility (political, technical, financial and institutional), and socio-economic profitability of
an activity before a decision is made to undertake that activity, or approve funding for it.
Baseline surveys, field studies and situation analyses may provide elements of appraisal.
b. Ongoing evaluation is the analysis during the implementation phase of the continuing
relevance, outputs, effectiveness and impact of an activity. It is closely linked to monitoring
and is used to alert managers to problems so that corrective action can be taken. Informal
observation usually plays an important role. Mid-term evaluations are also carried out during
implementation. The results of either may lead to mid-course corrections.
c. Retrospective evaluations, also called ex-post evaluations, analyse relevance, effectiveness
and impact after the completion of an activity. Lessons learned can be used as input for
future planning and funding decisions.

Evaluation focus
Evaluations focusing on organizational or management issues, called process evaluations, are
often done during implementation. Those focusing on effects on beneficiaries, called impact
evaluations, are often scheduled at the end of the programme period.
A distinction is sometimes made between formative and summative evaluations.
Formative evaluations involve the systematic collection of information to aid decision-making
during the planning or implementation stages of a programme. 'They are generally process
oriented. Formative evaluation is also sometimes referred to as context evaluation, needs
assessment or diagnostic research. They usually involve staff directly responsible for the activity
and may also involve external evaluators to bring new approaches or perspectives.
Summative evaluations are usually carried out as a programme is ending or after completion of
a programme in order to "sum up" the achievements, impact and lessons learned. They are
useful for planning follow-up activities or related future programmes. Evaluators generally
include individuals not directly associated with the programme.

Monitoring and timing
Surveying and analysing the situation of women and children at the beginning of a new cycle
provides valuable data for programme planners by identifying where greater assistance is
needed. Situation monitoring also provides a way to identify a correlation between the impact of
previous services and improvement of conditions of women and children.

Monitoring of services or programmes during implementation reveals problems needing
attention and contributes to evaluation by tracking progress and collecting data to measure it.
Similarly, monitoring helps in formulation of new annual plans of action by detailing what has or
has not been accomplished by previous ones.

Demystifying evaluation
Field staff are often engaged in evaluative activities, throughout the programme and project
cycles, and not just when involved in formal, retrospective evaluations in the final stage of the
programming cycle. Often the words, "review," "analysis," "assessment," and even "report"
designate some aspect of evaluation. Field staff and government officials have an oftenunrecognized wealth of evaluation experience to tap. A more systematic or formal approach at
specific times will however strengthen evaluation capacity.

Evaluative elements in the UNICEF Programming Cycle
The roles monitoring and evaluation assume throughout UNICEF's 5-year programming cycle
are described below. Sections of the manual noted in parentheses contain more information.
a. Reviews of past performance and preparation of plans of action for the following years
Annual reviews, conducted jointly by UNICEF with the ministries carrying out services or
programmes in which UNICEF co-operates, ascertain progress, identify problems and possible
solutions in order to reconfirm or revise plans for the coming year. Usually in the third or fourth
year of a five-year co-operative programme, the annual review is expanded to a more thorough
Mid-term Review (UNICEF Policy and Procedure Manual, Book D, Chapter 3, Section 4;
Chapter 4, Section 11). Project monitoring documents should be a prominent source of
information for these evaluative reviews.
b. annual report
Whenever feasible, the results of the annual reviews should be incorporated in the annual report
sent by the country office to UNICEF headquarters and to the ministries concerned. The overall
thrust of the annual report should be evaluative and analytical. 'Ibis report should summarize
which programme or project goals have been accomplished, analyse reasons for success or
failure, and indicate where changed approaches will be taken in the coming year (UNICEF
Policy and Procedure Manual, Book D, Chapter 4, Section 12). The reports should include a
section on evaluation, with annexes listing evaluations and studies (if any) done during the year
and those projected for the next year. The form to be included in the Annual Report is found in
Appendix G.
The following points should be covered in the text of the Annual Report:
•
•
•
•
•
•
•

schedule of evaluations for the coming year;
purposes and intended uses of evaluation results;
type of evaluation and administrative responsibility proposed for various parts of each
evaluation;
indicators and data sources to be used (if possible to determine this far in advance);
cost estimate for each evaluation;
feedback and reporting process; and
actions proposed to develop national capacities in monitoring and evaluation.

c. Special studies and research
Research, especially action-oriented research, and. special studies address specific programme
or project issues. They can contribute both to programme preparation through prospective
analyses of needs, constraints, and opportunities and to evaluations by providing background
data. A special study may assess an area of potential new co-operation between UNICEF and a
ministry or analyse a problem cutting across and affecting several projects. 'These studies may
be undertaken throughout the programme cycle. Often special studies are partially funded by
UNICEF and usually they are prepared by national researchers.
Studies are especially useful when they present both facts and reasons behind them especially from the viewpoint of beneficiaries. While not a substitute for evaluation, research
and studies play an important role by providing detailed data and analysis. Problems of
questionnaire design, sample size and time frame often preclude incorporating the same degree
of detail in an evaluation, but the studies will enrich the evaluation. When evaluations are well
planned, studies can be commissioned to contribute most effectively to the evaluation.
d. Situation analysis
The UNICEF country programming cycle begins with a comprehensive assessment of the
condition of women and children in the country. The resulting document, called a Situation
Analysis, is designed to provide a basis for UNICEF's country programme plan. The findings
should also feed into the National Development Plan. Preparing the Situation Analysis entails
analysing data from various sources, including previous evaluations. The Situation Analysis
attempts to assess economic, social, political and other factors in order to analyse needs and
identify opportunities and constraints. It looks at reasons for poverty and deprivation in the
population, examines why existing programmes and services may not be filling needs (UNICEF
Policy and Procedure Manual, Book D, Chapter 3, Section 3).
Updated at least every five years, the Situation Analysis can be used for policy advocacy,
programme planning, social mobilization and fundraising efforts. The Situation Analysis should
also examine the social and health information systems of the country, their strengths and
constraints, the quality and timeliness of information generated and national resources devoted
to these systems.
Procedures to store and retrieve major findings and recommendations of evaluations and
monitoring reports provide institutional memory and assist in updating the Situation Analysis. A
number of countries are developing mechanisms for this kind of data bank and regular updating.
e. Programme strategy and programme preview meetings
These are usually held by country and UNICEF officials in the year before the next country
programme plan is submitted to the UNICEF Executive Board. Decisions taken during these
meetings reflect national and UNICEF priorities, country conditions and lessons learned from
past cooperation.
i. Programme strategy.- based on the situation analysis and the results of the mid-term review, a
position paper is presented which outlines the framework of the next country programme,
indicates the overall programme direction and describes the hierarchy of objectives.
ii. The programme preview meeting discusses the country and sectoral plans of operations. It is
based on the strategy agreed upon earlier.

Strategy formulation and programme planning will be greatly strengthened if good evaluation
results are available. Retrospective and ongoing evaluations of services or programmes being
considered should be consulted. Evaluations of similar programmes conducted by other
organizations might be helpful. Appraisals, including feasibility analyses of several proposed
courses of action and special studies, can provide invaluable information at this stage.
f. Plan of operations (Planops)
The basic document of the programming cycle, the Plan of Operations, covers the entire country
programme, usually for the upcoming 5-year period. An overview of the effort in the country (the
Master Plan of Operations) is followed by separate plans of operations for each sector or areabased programme/project. 'These must include the objectives of each programme/project,
measures of progress, key activities and who is responsible for each, time-frames, mechanisms
for monitoring and evaluation, schedule of evaluations, and budgets (UNICEF Policy and
Procedure Manual, Book D, Chapter 3, Sections 5-9).
When the plans of operations include specific objectives for each programme/project and
describe who is responsible for collecting data to measure progress at quarterly, semi-annual or
annual intervals, the basis for effective monitoring and evaluation throughout the five-year
period is laid. A more detailed monitoring and evaluation plan should be included in the annual
plans of action.
Figure 11-3 indicates the relationship between the country or area (part of country or region)
programme goals and the programmes, projects, sub-projects and activities relating to the
reduction of mortality by activities in the field of health. 'These relationships are equally
applicable to other fields.
g. Project plans of action
An annual Plan of Action for each project details the problem addressed, project objectives and
annual targets, time frame, schedule of operations (who will do what when, where, why, how
and with what resources), assumptions, monitoring and evaluation, management and budget
(UNICEF Policy and Procedure Manual, Book D, Chapter 3, Sections 5-9).
To facilitate the monitoring and evaluation process outcome-impact objectives must be specific
and reflect the expected result not the process for obtaining the result. Expected results should
be formulated to reflect a hierarchy of levels of objectives: input and output. Vague, general
objectives, such as "to increase the participation of women in community organizations," make
monitoring and evaluation difficult. Specific objectives, for example "that 10 per cent of women
with children under five not presently taking part in existing community organizations become
participants," make it possible to measure the results of the project. A good example of crisp,
measurable objectives is found in the Thailand Plan of Action for Improving Household Security.
The project's objectives are clearly stated: to train and assist 450 rural women entrepreneurs in
30 villages (input); to develop 15 new small-or micro-enterprises (output).
By identifying the specific activities needed to accomplish each objective in the 12-month
period, when they are to occur and who is responsible for each, the groundwork for monitoring
is laid. (the annual Plans of Action should also detail approaches, sources of data, timing, how
analysis is to be done, and how the information will be used).
h. Country programme recommendation (CPR)
The Country Programme Recommendation, an analytical summary of the three-five year
UNICEF country programme plan, including financial information, is the document sent to the

Executive Board for funding approval at its annual session. The 24-page summary includes
highlights of the condition of children, an analysis of past performance during the previous and
current periods (listing studies and evaluations consulted in an annex), and major features of
each programme for the new period. The CPR should outline plans for monitoring and
evaluation to satisfy the Board requirement for adequate control and measurement of progress
toward meeting objectives (UNICEF Policy and Procedure Manual, Book D, Chapter 3, Section
9).
i. Audits
Audits investigate the degree of programme/project compliance with laws, regulations,
guidelines and agreed-on plans and budgets. Audits may be performed at any time during the
programming cycle. A programme audit overlaps somewhat with an evaluation since it
examines accomplishment of proposed work plans and objectives, but an audit does not
normally review the relevance on impact of the project. UNICEF audits are conducted by the
Office of Internal Audit by headquarters and regional auditors and sometimes by external
auditors. 'They examine carefully monitoring documents. 'They focus on oversight of UNICEFsupplied resources and result in internal documents available to UNICEF Headquarters staff,
regional and country Representatives, where appropriate problems are discussed with
government officials.

6. References and Bibliography
a. Feuerstein, Marie-Therese, Partners in Evaluation: Evaluating Development and Community
Programmes With Participants, London, Macmillan, 1986, 194 pp. Designed for those who
work with community groups, with illustrations, a glossary and list of references.
b. Patton, Michael Quinn, Utilization-Focused Evaluation, Beverly Hills, Ca., Sage Publications,
1978.
c. Organization for Economic Co-operation and Development (OECD), Development
Assistance Committee, Expert Group on Aid Evaluation, Evaluation in Developing
Countries: A Step in a Dialogue, Paris, OECD, 1988. Helpful to understand consensus
among donors about evaluation. Glossary.
d. Rugh, Jim, Self-Evaluation: Ideas for Participatory Evaluation of Rural Community
Development Projects, Oklahoma City, Oklahoma, World Neighbors, 1986.
e. Sohm, Earl D., "Glossary of Evaluation Terms," Geneva, Switzerland: Joint Inspection Unit
of the United Nations, November 1978.
f.

UNICEF, Policy and Procedure Manual Programme Operations. Book D, Chapters 1-4, July
3, 1989. Refer to this for greater detail about UNICEF's programming cycle.

g. UNICEF, "Evaluation for Action: First Meeting of the UNICEF Evaluation Focal Points,"
Florence Italy, 4-9 June 1989.
h. UNICEF, Strategies for Children in the 1990s, A UNICEF Policy Review, May 1989.
i.

UNICEF, "Framework for Improvement of the Monitoring and Evaluation, of Child Survival
and Development Activities," CF/EXD-IC-1989-8, 5 March 1986.

j.

UNICEF, "Overview of Evaluative Activities in UNICEF," Policy Review, February 22, 1984.

k. White, Louise G., Methods and Procedures in Aid Evaluation: A Compendium of Donor
Practice and Experience, Paris, OECD, 1986.

Section III - Strengthening Monitoring
1 - Scope
This section covers monitoring of services/programmes or projects (subsections 24) and
monitoring of the situation of women and children (subsection 5). An overview of key terms and
concepts follows the Summary of Action. The section on programme and project monitoring
explains: how to plan a monitoring system to be carried out by those implementing a
programme or project; how UNICEF field staff monitor progress during implementation, and
some examples of community monitoring work. The part on monitoring the condition of children
explains how to determine what to monitor, identify sources of data and analyse data, as part of
a situation analysis. These are country inputs to the monitoring of global goals.

2. Summary of Action
During programme and project planning, government officials and UNICEF staff should ensure
that each programme and project has specific objectives and time frames and that adequate
provision is made for monitoring progress. The monitoring plan should specify who will collect
what information, for what purpose, in what form, when and how it will be used, and ensure that
it should strengthen existing monitoring systems. Counter the tendency to collect too much data
by carefully selecting a few indicators that focus on process and outputs. Those responsible for
carrying out this plan should ensure that it is feasible and adequate resources are available.
Where possible, opportunities for coordinating data collection for monitoring systems should be
sought among different programmes and projects. It is sometimes possible to integrate the
monitoring systems rather then establish parallel data collection systems. Data collected may be
of use to several programmes or projects, especially if monitoring systems are designed to
maximize the utility of the information gathered.
During implementation, government programme managers are responsible for regular reporting
on project progress and resource expenditures, as indicated in the plan. 'This usually takes
place during the annual review. It may take the form of a written progress report that reviews
what has been accomplished compared to what was proposed, identifies constraints, and
suggests
necessary
modifications.
Often, monitoring systems are not planned in sufficient detail in advance. It is obviously
preferable to incorporate a monitoring plan from the beginning of implementation. However, the
development of monitoring systems for new projects is often evolutionary - they tend to be
introduced as need arises and possibilities open up. Other government officials (e.g., planning
officials, those who supervise programme managers) may conduct supervisory field visits,
review progress reports and help identify improvements. Specific responsibilities for monitoring
UNICEF inputs am described in UNICEF Policy and Procedure Manual, Book D, Chapter 4,
Sections 7-9.

UNICEF staff are responsible for following this process and working to strengthen the capacity
of government officials to manage programmes. UNICEF staff can facilitate this process by
working closely with government officials to review monitoring systems and use the information
for project management. 'This also provides the basis for monitoring implementation of UNICEFassisted services and projects. Staff will make field visits, review (and sometimes prepare)
regular progress reports and provide information and suggestions to programme managers.
Being informed about progress, constraints and opportunities requires communication with:
a. those implementing the activity (government managers and staff at the local as well as other
levels);
b. beneficiaries of the programme and other members of local communities;
c. other local government officials responsible for development efforts; and
d. representatives of other development organizations engaged in similar work in the country.
At times, appropriate action includes suggesting revision of work plans or identifying training or
technical assistance needs and resources, usually included in the plan for the following year.

Use of monitoring results for evaluation and planning
UNICEF field staff and government officials are responsible for incorporating monitoring results
into annual and mid-term reviews, evaluations, and for preparing annual Plans of Action and
other steps of future programming. Monitoring results also help determine what evaluations and
studies need to be conducted.
UNICEF country offices should advocate for and work with government officials to develop and
implement a system for monitoring the situation of women and children and progress towards
the global goals for the 1990s. This information should feed into the preparation of the Situation
Analysis and future programming and be used to advocate for improved services and policies.

Overview of Monitoring
Definition. As already noted, monitoring is the periodic oversight of the implementation of an
activity which seeks to establish the extent to which input deliveries, work schedules targeted
outputs, and other required actions are proceeding according to plan, so that where deficiencies
are detected, timely corrective action can be taken.
Monitoring also applies to the systematic checking on a condition or set of conditions, such as
following the situation of women and children.
In practice, monitoring covers a wide range of activities. UNICEF field staff, for example, in
response to a recent request for information about their monitoring systems, submitted materials
reflecting three usages of the term: (a) reporting programme/project progress; (b) assessing the
condition of women and children; and (c) collecting data for use in the oversight process (Table
IIII-1). Monitoring requires data collection, but data collection is not synonymous with it.
Monitoring also implies analysis and use of the data. Expanding on each of the UNICEF field
usages of monitoring:

(a) Project and programme progress reports are periodic reviews and updates of work plans,
often done by project managers on a seniisemi-annualannual basis. Many funders request
regular reports to oversee use of their funds and to identify additional resources needed.
(b) Keeping watch over changes in the status of women and children in a country and progress
with regard to goals for the 1990s, requires selection of indicators to measure the impact of
health and social programmes and other influences. The available national statistics are
often out of date. Support may be given to government statistics offices to help them
improve the system. UNICEF places particular emphasis on assisting countries in this type
of monitoring. Section TV discusses this element in greater detail.
(c) Data can be collected continuously or periodically by those implementing an activity. Some
data are collected monthly by project personnel and then totaled by managers for quarterly
and, later, annual reports.

Table III-1: Selected UNICEF Field Office Monitoring Tools
Country Sector User/level Language Period
Special Features
A. Reporting Programme/Project Progress
project
Tables: workplan and project assessment (comparing targets
Bangladesh all
English
6 mo
mgr
with achievements).
project
Bolivia
all
Spanish 1-4 mo
Table: links financial status with project progress.
mgr
project
Table: targets, progress to date and expected for year.
Burma
all
English
3 mo
mgr
Outputs compared to schedule.
project
Computerized form (Lotus spreadsheet & Gant chart) links
China
all
English
1 mo
mgr
financial status with activities.
UNICEF
Links objectives, results to call forwards. Identifies reasons
Madagascar all
French
6 mo
staff
for failure, corrective action.
nutriti local to
English
Tanzania
3 mo
Tracks impacts at community level, uses Lotus 1-2-3.
on
national
Swahili
B. Status of Women and Children
English +
Burkina Faso na provincial
6 mo
Child Monitoring Bulletin.
French
Dominican
Chart with targets/ progress compared to economic policy
na
national
Spanish
na
Rep.
changes.
Jamaica
na
national
English
na
Forms for many levels.
C. Data Collection Instruments
Dominican
project
Ed.
Spanish
na
Forms to obtain data from individual students, programmes.
Rep.
mgr
state/natio
Systems and users manuals for computerized (D-base III)
India
WES
English
1 mo
nal
tracking of drilling rigs.
local/
Indonesia
EPI
project
English
na
Training forms: Local Area Monitoring.
mgr
Healt
Forms for village nutritional surveillance using graphics for
Mauritania
CHW
French
na
h
non-literate people.
wome
project
User and installer manuals for computerized (Lotus and DNepal
English
1 mo
n
mgr
base) programme.
credit

Sudan

WES

project
mgr

English

1 mo

Compares inputs (supplies) with outputs.

Key Terms
Monitoring can focus on:
a. Projects normally consist of a set of activities undertaken to achieve specific objectives
within a given budget and time period.
b. Programmes are organized sets of projects or services concerned with a particular sector
(e.g. agriculture or education) or geographic region (e.g. a multi-sector programme in one
area) or a particular donor (e.g. UNICEF co-operation in a country programme).
c. Services are based on a permanent structure, and are or have the goal of becoming
national in coverage e.g. health services, whereas programmes are usually limited in time or
area. Sustainable activities usually need to become part of an ongoing service. "Services"
are the vocabulary of governments, "programmes" that of external co-operation agencies.
d. Processes are organizational operations of a continuous and supporting nature (e.g.,
personnel procedures, administrative support for projects, distribution systems, information
systems, management operations).
e. Conditions are particular characteristics or states of being of persons or things (e.g.,
disease, nutritional status, literacy, income level).
In addition, monitoring may focus on different aspects of a service, programme or project:
a. Inputs are resources provided for an activity, and include cash, supplies, personnel,
equipment and training.
b. Processes transform inputs into outputs.
c. Outputs are the specific products, goods or services, that an activity is expected to deliver
as a result of receiving the inputs.
d. A service is effective if it "works", i.e. it delivers outputs in accordance with its objectives.
e. A service is efficient or cost-effective if effectiveness is achieved at the lowest practicable
cost.
When monitoring the status of women and children one may also focus on:
f. Outcomes, which generally refer to peoples' responses to a programme and how they are
doing things differently as a result of it.
g. Impacts, which are the effects of the project/programme on the people and their
surroundings. These may be economic, social, organizational, health, environmental, technical
or other intended or unintended results of the project or programme. Impacts are long-term
effects while outcomes are shorter-term effects relating to objectives.

Output measures refer to outputs as far as possible and cover availability and accessibility of
services; outcome measures refer to utilization; and impact measures to the condition of
children and women in the target area. For example:
Output: number of homes which have received ORS;
Outcome: number of mothers correctly using ORS;
Impact: reduction in deaths due to diarrhoeal dehydration.
An indicator is a measure which is used to demonstrate the change or result of an activity,
project or programme. Where direct measurement is not feasible, indirect or proxy indicators
may be used. Indicators can be categorized by type, such as output indicators or impact
indicators. Targets are often set in terms of indicators to be achieved at specific times.
Indicators should be:
a. measurable quantifiable (percentage, ratio, number)
b. factual mean the same to everyone.
c. valid measure what they claim to measured.
d. verifiable can be checked.
e. sensitive reflect changes in the situation
Indicators should generally relate to objectives, structure and implementation of the
project/programme and therefore are usually best selected locally.
Figure III-1 illustrates input and output indicators relating to communication about oral
rehydration therapy.
Figure III-1: Examples of communication indicators
____________________________________________________________________________
1. Final outcome indicators
Target audience: Mothers
• ORS/ORT use rate
• continued feeding use rate
• % of target audience correctly preparing ORS and/or recommended home fluids
• % of target audience seeking treatment outside the home when necessary
Target audience: Health workers
• % of cases correctly given advice on home treatment or prevention
2. Intermediate outcome indicators
Exposure
• % of target audience who heard, read or saw communication messages and materials
Knowledge
• % of target audience who know about
• correct preparation and use of ORT/ORS/home fluid

•
•
Trial
•

what foods to give
when to seek treatment
% of target audience who have tried correct home treatment at least once

3. Input indicators
Examples:
• Number of health workers who received training in interpersonal communication
• Number of printed materials (of a given type) produced
• Number of mass media programmes or spots produced
• Quantity of broadcasts (number and frequency)
• Quantity of educational talks presented in a given setting with a given audience
_____________________________________________________________________
Source: Communication, A Guide for Managers of National Diarrhoeal Disease Control
Programmes, W.H.O., 1987, p. 53.

Functions of monitoring
a. Monitoring is a management tool that contributes to effective and efficient implementation.
Managers and donor representatives, responsible for monitoring, should be open to
modifying original plans during implementation, if such action seems warranted.
b. Monitoring is an evaluative activity. By comparing what has been accomplished to targets
set up in advance for a given time period, monitoring identifies shortcomings before it is too
late. It also provides elements of analysis as to why progress fell short of expectations,
identifying constraints and pointing the way toward measures to overcome them.
c. Monitoring may be used to adjust specific objectives and revise the workplan during the
current period and for the next reporting period. This does not mean that objectives should
be changed lightly. But when monitoring signals that something is off course, a careful
review of the situation should be undertaken to assess if a modification of objectives is
merited. Adopting a flexible planning style is usually more successful than following original
plans rigidly because unexpected opportunities, as well as obstacles, often arise. In the
many cases where the objective is a change of behavior, it is impossible to accurately
predict people's reaction to interventions; monitoring and subsequent adjustment of activities
are essential.

Organization of monitoring
Monitoring can be conducted from the local to the global level.
As a management tool, monitoring should be organized at each level of management. For
example, monitoring a hospital-based, specialized Maternal and Child Health (MCH) service
would be designed to serve hospital and unit management. Most services in which UNICEF
collaborates extend to the local level - literacy, primary health care, nutrition, water usage,
sanitation, women's income-generating activities and local level managers need the support of
monitoring.
The monitoring system has to fit the management system. In order to accomplish this, UNICEF
advocates:

a. decentralized management, i.e., management decisions taken as close as possible to the
activity. Management from a distance does not motivate personnel, can only react slowly to
changing needs and situations, and does not encourage an economical use of resources.
b. management by objectives, i.e., managers should not merely perform, and oversee the
performance of specified activities, but should be most interested in their outcomes and
impact;
c. development of community participation in certain activities into community responsibility for
these efforts. Ideally, consultation between local government workers and community
members should lead to the community assuming greater responsibility for an effort e.g.
through a management committee in a health centre.
In general, the level that records information should be able to use it. Designing data collection
systems with this principle in mind helps improve chances that the data will be collected
carefully and put to use. Normally, data collected at one level is also passed upward through a
hierarchy of administrative or supervisory levels. Bottlenecks in the process mean that data
quality and frequency of reporting decrease with time, and that regional and national data may
be outdated by the time they are aggregated (sentinel sites are designated to get round this
problem).

4. Monitoring Services, Programmes and Projects
Elements of monitoring
Routine reporting, field visits, detailed activity plans and timetables provide a monitoring
framework. Baseline data describe the problem or situation before the service, programme or
project begins (or begins another phase) and identify a point from which to measure progress.
Often it is not desirable to delay the start of an activity while comprehensive baseline data are
being gathered. In that case data should be gathered in association with the early phase of the
activity, before it has had time to have an impact.
At the beginning of implementation, or immediately prior to it, managers responsible for carrying
out a project programme should review the PlanOps and Plan of Action and confirm or refine
objectives, activities, time frame, resources, baseline data, and monitoring plan. Sometimes
these items have not been identified adequately during the planning period. Personnel or
assumptions made during planning may have changed by the time the work is ready to begin.
Adjustments should be made at this time. For example, if baseline data concerning indicators
selected for monitoring are not available, plans for the initial phase can be revised to include
rapid methods to gather such baseline data, if the indicator is important for monitoring and can
be collected at reasonable cost and effort. A plan of action should be prepared even if baseline
data are deficient Experience has indicated that delays in preparing the plan of action result in
irregular contacts with the project, weak co-ordination, infrequent contact between government
and UNICEF officials and poor implementation.
It is important to link monitoring to normal programming activities. Government officials and
UNICEF conduct annual reviews of their projects during the period July-October of each year.
This review should normally produce the following:
a. an annual report on progress during the past 12 months;

b. a list of problems identified during implementation and for which studies/evaluations need to
be conducted to study causes/ reasons; and
c. a revised plan of action for the following year.
As an alternative to c), managers responsible for a project may prepare during the last quarter
of each year a project plan of action for the following year with a workplan. The annual review
may necessitate revising some elements of the planops (objectives, activities, time frame,
resources, etc.).

Planning a Monitoring System
A. What should be monitored? Select indicators for monitoring. An indicator answers the
question: what is going to show that we have accomplished our objective? To decide on an
indicator, recall the objective of an activity and the targeted audience: what is the expected
result of the effort and who is the intended user of the service? Although many indicators
can usually be established for each objective, it is important to select one or a limited
number of key indicators that will best demonstrate you accomplished your purpose. There
is no standard list. Good indicators limit and focus data collection. The choice of items to be
monitored is influenced by the country context. Narrow the number of indicators to be
followed. Some indicators may not be feasible to track in the time period available or with
limited resources. When a national data collection system is collecting some reasonably
useful data it may be cost effective to make use of the already collected indicators even less
than ideal - rather than establish new monitoring systems to collect the optimal indicators.
Illustrative indicators for selected sectors of UNICEF co-operation are noted in Appendix C.
Ask managers and decision-makers which data will be most useful to them. The basic
principles are: (1) keep the information requirements to a bare minimum; (2) collect the
information that will be most helpful to those who will use it. The most common problem is
that too much data is being collected, of poor quality, that is not being analysed or used,
expending resources that could be better used. A first step in designing a monitoring plan is
to identify who needs what information, for what purpose, how frequently, and in what form.
The national information system should be looked to where possible to provide necessary
monitoring information. Especially in countries with severe resource constraints, this may be
one of the only regular sources of data.
B. How? Select methods to track indicators and report on progress. Many methods may be
used: observation, interviews, routine reporting, sentinel sites (see evaluation methods, in '
Section IV, subsection 7). Piggyback on existing data collection systems whenever possible.
Both formal and informal, quantitative and qualitative methods can be used to check
periodically on certain programme aspects. Decide how information will be recorded
systematically and reported clearly.
Consider the time and skills of those who will collect the data. For example, the number of
items to be monitored in a community health programme depends on the capacity of typical
health workers, whether they are effectively remunerated, and how much time they are
expected to spend on health work. Also, consider which data collection methods are most
credible to policy-makers, planners and managers. Ask each group to comment on the
monitoring plan and make suggestions to improve it. Pretest new monitoring instruments.

C. Who should be involved when? Organize reporting and feedback processes, clearly
identifying who will collect information on indicators, when, and who will receive it. Reporting
project progress is at the core of most monitoring systems. Most data collection and analysis
will be done by the project staff at various levels. The plan should also identify when the
project manager should gather together this data and prepare regular progress reports, what
should be in these reports, who they are to be distributed to and when. Generally, project
managers are responsible for completing written progress reports on a regular basis
quarterly or twice a year - that include financial and programme information. The monitoring
plan can include a progress report form for managers to complete.
The monitoring plan should also identify who will be involved in reviewing progress and
provide feedback to the managers on a regular basis. The manager may be asked to discuss
progress reports with field staff implementing the activities, beneficiaries, superiors and/or
funders, to provide them with information about what has happened and seek their
suggestions about improving the services.
D. What resources are needed and available? Human and financial costs of gathering,
reporting; and reviewing data should be identified. Needed funds and time should be set
aside for this work and, if necessary, the monitoring plan should be adjusted within the
context of possibilities.
E. Consultation and training. The monitoring programme should be discussed with a
representative group from each level before it is put into effect. This involves a discussion of
how the level would use it for management purposes. Subsequently, training should be
provided to those who will be using the monitoring/management systems. UNICEF can help
meet the costs of introducing or improving the monitoring system.

F. Preparing the workplan. Designing a monitoring programme does not necessarily mean that
it will be implemented. Many beautifully designed plans never got implemented. To ensure
its use, a workplan should be prepared for each year listing the main activities to be carried
out, their output, timing and parties involved. It should stress who would use the results and
how. Reviewing the workplan during informal and annual reviews will keep the monitoring
programme alive and facilitate periodic improvements.
UNICEF field staff monitors progress by making field visits, reviewing progress reports, tracking
inputs and providing information to responsible officials.

A. Monitoring through field visits
Field visits can provide valuable qualitative and quantitative information that cannot be obtained
from written reports or conversations with project managers or officials in the capital. Field visits
can be used to monitor project processes, results, participation and to obtain a better
understanding of its setting. In particular, field staff should be alert for the unexpected and seek
out unforseen benefits and difficulties, new opportunities and unanticipated costs. They might
meet with some who criticize the project or ask follow up questions to those who express
opposing views.
Field visits should be planned. They should be based on the following:

•
•
•
•

elements from the workplan for the period;
issues identified during previous visits which require follow-up;
enough flexibility to take note of unforeseen developments or unexpected observations;
and
the visits should be conducted jointly with government staff concerned with the services.

Field visits should be used for observation and interviewing a broad range of individuals:
1. beneficiaries and other members of local communities. If possible, talk informally with
beneficiaries, without project staff present, to obtain insight into the impact of the project on
their lives and behavior. Meeting with small groups of local or opinion leaders can provide
information about access to the service. Sometimes interviewing women separately from
men encourages them to speak.
2. staff on-site, whose hands-on experience and detailed knowledge of prevailing conditions
should be tapped by monitors.
3. representatives of governments and non-governmental organizations.
a. Local government officials have information about the community, access to resources, and
insights about the strengths and weaknesses of the services.
b. Representatives of non-government organizations and other donors addressing similar
problems in the locale. Local NGOs, church groups, researchers, overseas volunteers and
others familiar with the community may provide valuable insights into effectiveness of the
service.
c. In the capital, representatives of donors funding similar programmes, or projects in the same
locations, can be asked to share lessons learned.
Persons should be notified in advance of the purposes of the visit; sites visited should be typical
rather than the most successful or accessible; visitors should examine records on site. Findings
should be discussed with staff and beneficiaries at the site as well as government officials and
UNICEF staff in the capital. Trip reports should always be written in as much as they comprise
an important part of monitoring documentation. An example of a trip report form can be found in
Appendix
E.
UNICEF procedures for planning, conducting and following-up on field observation visits are
described in UNICEF Policy and Procedure Manual, Book D, Chapter 4, Section 10.

B. Monitoring through progress reports
What reports should contain. A variety of regular reporting forms have been developed. UNICEF
has a decentralized system for monitoring project and programme progress. Several country
offices have developed progress report forms of their own that are applicable to all sectors
(Table III-1). The major components of these progress report forms are fairly standard and
include:
1. a list of specific objectives and activities that were to be accomplished during the quarter (or
six month period);

2. the degree to which each was completed, with numbers;
3. identification of constraints or obstacles that explain why certain objectives/activities were
not completed as planned (if applicable);
4. identification of additional resources, training, etc., needed to help overcome these
constraints;
5. updated list of objectives/activities planned for the next period;
6. financial accounting: what was budgeted and spent during the report period and any
proposed changes for the next period; and
7. items to be followed up during the next visit.
Each UNICEF office should develop with government planning officials a system for monitoring
the services that it supports. A quarterly (or biannual) progress report form, containing the
above information may be useful. This form applies to services in all sectors. Sophisticated
computer programmes, such as Timeline, can be used. But a one-page typed chart reproduced
on a copier can be used as effectively. Progress reports do not have to be lengthy documents.
For example, the UCI quarterly report form is only two pages. So is the Half Yearly Project
Assessment Table used by the Bangladesh UNICEF office. Under certain circumstances, oral
progress reports may be appropriate.
Ideally, progress reports should be prepared by government officials responsible for
implementing the services. On-the-job training can be provided when UNICEF staff and service
managers prepare them together. Good quarterly progress reports strengthen managers'
capacity to manage programmes as well as to obtain funding from the government and donors.
Field offices should be encouraged to conduct reviews an informal one in the period March-May
and an annual review during the period July-October. This allows for updating the workplan and
making course correction.
UNICEF field staff can help programme managers overcome COMMON MISTAKES made by
those preparing progress reports:
1. writing (or saying) too much, especially too much description about what project personnel
are doing, or the situation - and not clearly identifying what has/has not been accomplished
in regard to the planned activities, outputs and objectives for the quarter;
2. presenting only general or subjective opinions rather than quantifiable data;
3. omitting an explanation of setbacks and reasons for not accomplishing what was planned;
4. not identifying ways to deal with constraints or new opportunities;
5. not proposing revised objectives and activities for the upcoming period (usually the following
year) in light of the above; and
6. not consulting staff or beneficiaries while preparing the report

C. Tracking inputs to monitor progress

UNICEF has a centralized system to monitor its inputs, using "cash and supply call-forwards",
that is facilitated with the use of the software package of the Global Field Support System
(GFSS). This recently revised system for financial and supply monitoring and control is
described elsewhere (UNICEF Policy and Procedure Manual, Book D, Chapter 4, Sections 7-9).
Verification that assistance has arrived, however, does not tell much about whether or how it
has been used. Therefore, field staff should confirm that inputs are being used as planned (for
example, by making on-site inspections of warehouses receiving and distributing UNICEF
supplies) and that the supply is adequate in light of project needs.
This system of tracking inputs can be used to monitor project and programme progress.
Financial tracking is linked with specific programme activities, enabling those using these
records to pinpoint areas requiring special attention. If call-forwards are falling behind what is
expected, this can be a clue to potentially significant problems. One programme officer even
learned that a project director resigned when a quarterly financial report showed no funds
expended for two months in that fine item. If a project is spending resources faster or more
slowly than planned, an alert staff person will ask for explanations. A review may show that the
project is progressing normally but financial plans need to be revised. When UNICEF is the
principal funder of an activity, it is possible to compare cost effectiveness of different inputs and
delivery systems.
Service managers and UNICEF staff should also be concerned about tracking inputs from other
sources. Governments are increasingly concerned with financial sustainability and are
requesting cost analyses which require complete data on income and expenses; they are
particularly concerned with the burden on their recurrent-cost budget. Thus, keeping records of
all inputs is critical, including cash and in-kind contributions of various governmental
departments, other donors, and local communities.

C. Using monitoring findings and providing feedback
Monitoring findings can be used to make managerial decisions at each level. In addition to direct
use by the level collecting the data, feedback should be given to personnel at each level
regarding the results of the data collected and the analysis made at higher levels. The levels
include:
•
•
•
•

the level of collection (the service unit and community,
including beneficiaries);
the district/province level (supervisor of service unit); and
the central level (administrator).

Where findings are not used at the level of collection, data reporting can become irregular and
of poor quality. Every effort should be made to show to data collectors how the data can be
useful at their level of operation.
Monitoring results, including field trip reports and progress reports, should be summarized and
discussed during the Annual Review and incorporated in the Annual Report. Suggested
revisions should be incorporated in the Plan of Action for the coming year. In addition, the
results of the review will identify new evaluations and studies to be conducted and revise/update
the evaluation plan.

Monitoring social mobilization
Social mobilization can be defined as a process for engaging a large number of people in action
to achieve common social goals.
It aims at:
•
•
•
•
•

getting political commitment from key decision-makers to fully support the
implementation of service delivery systems;
motivating service providers so that they perform with attention to the user's needs;
forming alliances with partners with good potential for contribution to the programme, as
promoters and/or service providers;
empowering communities to demand and obtain satisfaction of their needs; and
positively influencing individuals' behavior and social norms for the adoption of
appropriate practices.

For more information see Chapter 5 of the Policy and Procedures Manual dealing specifically
with Social Mobilization and including an extensive list of process indicators for achieving
sustainability in social mobilization.
Monitoring community participation. All UNICEF-assisted activities seek to strengthen the
capacity of people to solve their own problems and to empower them. Community participation
in activities is a must for them to be successful and sustainable. This implies that the community
fully understands the need for a given activity, participates in the design of the system, and
exercises responsibility during implementation and evaluation. The participatory development
process facilitates outreach to the poorest members of the community, making it possible for
them to organize and begin to take control of their own lives.
The process of dialogue between service personnel and community participants can be
monitored, for example, and the responses of village groups to activities developed in cooperation with them. It is important to know if the poorest groups have been included in this
dialogue.
Table III-2 gives an illustrative ranking scale for process indicators of community participation.

Table III-2: Ranking scale for process indicators of
community participation
Ranks
Indicator

1. Narrow
Imposed from outside with
medical, professional point
of view (CHL, village health
Needs assessment worker, health post staff);
or: Latrine builidng
programme imposed on
community.

Leadership

Organization

Resource
mobilization

Management

2. Restricted

3. Mean

4.Open

5. Wide

Medical point of
view dominated an
"educational"
approach. Comunity
interests are also
considered.

CHL is active
representative of
community
views and
assesses the
needs.

WHC is actively
representing
community views
and assesses the
needs.

Community members
in general are
involved in needs
assessments.

WHC
WHC fully reprsents
functioning
variety of interests in
Active WHC,
under the
community and
leadership of an taking initiative.
controls CHL
independent
activities.
CHL.
WHC imposed
WHC imposed by
WHC actively
Existing community
by health
WHC imposed by health
health services, but
cooperating with
organizations have
services, but
developed some
other community
been involved in
services and inactive
became fully
activities.
organizations.
creating WHC.
active.
Community
Small amount of resources
Considerable amount
Fees for services.
fund-raising
Community fundraised by community. No
of resources raised by
WHC has no control periodically, byt raisng periodically
fees for services. WHC does
fees or otherwise.
over utilization of no involvement and WHC controls
not decide on any resource
WHC allocates the
money collected.
in control of
utilization of funds.
allocation.
money collected.
expenditure.
CHL manages
independently with WHC selfInduced by health services.
WHC self-managed CHL responsible to
some involvement managed without
CHL only supervised by
and involved in
WHC and actively
control of CHL's
of WHC.
health staff.
supervision of CHL. supervied by WHC.
Supervision only by activities.
health staff.
One-sided, i.e. wealthy
minority; imposing ward
chairman; health staff
assumes leadership; or: no
heterogeneous WHC.

WHC not
functioning, but
CHL works
independently of
socal interest
groups.

WHC - Ward health committee CHL = Community health learder Source: World Health Forum, Vol. 10, 1989, P. 471

Community-based monitoring refers to different ways of involving the community in collecting
and using data. Community members collecting information for use by health officials differs
from true community monitoring, in which members of the community collect data and use it
themselves. Several UNICEF-related experiences demonstrate how health planning officials
can work together with communities through various community-based monitoring systems.
The joint WHO/UNICEF Nutrition Support Programme (JNSP) in Iringa, Tanzania provides an
example of the power of community monitoring to motivate grassroots action. Under the villagebased monitoring system, regular reports are prepared by village health committees on nutrition
status and causes of death of children under-five. The committees were established following
an introductory film and talk and weighing and immunizing all children in 168 villages in 1984
(this provided baseline data). Village Health Workers and Child Care Attendants were selected,
trained and are now largely financially supported by the villages. Village health days are

organized so that all children are weighed regularly and vaccinated; each child's progress is
recorded on his/her growth card and in the village registry. Each quarter, village health workers
report on nutritional status (e.g. using faltering growth as an indicator) and numbers and
symptomatic "causes" of death of children in the village. The health committees review the
report, analyse the problems, and take follow-up action. Ibis information is compiled in steps
from the village to the district levels and used as a motivational and management tool.
The JNSP programme included a range of projects - such as immunization, ORT, potable water,
latrines, improved wood stoves, community-based day care, forestry, and credit for women - in
addition to community monitoring of nutrition status and child deaths. The 1983-1988
programme's success in reducing malnutrition, according to one evaluation, lay in increasing the
capacity of Tanzanians, from villagers to district government officials, to assess the situation,
analyse immediate and basic causes, and take action to improve conditions (JNSP, 1988),
applying, reiteratively, a "triple A cycle" assessment analysis and action.
The programme is being replicated, and is now operating in about 1,000 of a total of 8,300
villages in Tanzania. In Iringa, focus is shifting to dealing with problems such as malaria,
prenatal care, lessening woman's workloads and household food security, now that better
weaning practices, immunization and use of ORT are widespread. Moreover, this community
monitoring/action method is being applied in JNSP programmes in other countries, including
Mali, Ethiopia, Niger and Peru.
The sentinel site survey approach, which is described in Section IV, subsection 8, is particularly
well suited for providing rapid feedback to the local level.

5. Monitoring the Situation of Children and Women
Definition and purpose
Monitoring the situation of children and women represents a special application of monitoring
and calls for organizing the collection, analysis and dissemination of data to track changes in
their condition. To fulfill its mandate, UNICEF has a unique responsibility to monitor the
condition of children and women worldwide. The results are used for advocacy, social
mobilization, programme development and fundraising efforts by government officials and
UNICEF.
Monitoring the situation of children and women provides inputs to a Situation Analysis by
describing what is occurring. The Situation Analysis uses this and analyses why it is occurring.

Summary of Action
Primary responsibility for data collection on a national level lies with national governments.
UNICEF can take a leadership role in advocating for the collection, analysis and dissemination
of these data obtained from the surveillance of the situation of women and children. Usually it is
desirable to establish a focal point - someone in the Planning Ministry, National Statistics Office
or Ministry of Health, for example - who will take the lead in child monitoring. The first step,
selecting indicators to track, could well be linked with establishing national targets through the
year 2000. Another initial activity is taking an inventory and assessing the quality of existing
data sources and making recommendations to improve them.

The capacity of national institutions and local agencies to collect and analyse these data can be
strengthened through the provision of technical assistance, training, and computer hardware
and software. Many countries will need continued support for data collection, system operations,
and even printing the forms. Statistical agencies of the UN, other donors, national universities,
etc., may provide assistance. UNICEF offices should draw on these resources in working with
the government to increase its monitoring and evaluation capacity.
UNICEF staff and government officials should use the results of situation monitoring for
advocating better policies, leading towards a national policy for children and women, mobilizing
efforts of officials and private citizens and improving programming.

Sources of data for situation monitoring include:
a. population censuses
b. household surveys
c. vital registration statistics
d. routine administrative data, regular reports of government and non-government agencies
e. large-scale monitoring or surveillance programmes
f.

sentinel site reporting

g. technical publications, national and local studies, with information on child and maternal
poverty, health, nutrition, welfare, education, community participation
h. discussions with informed people/communities
i.

rapid appraisals

j.

evaluations

k. legislation and policy documents

Collection of data
Data collection from routine systems should be a continuous process. However, since it often
takes too long for information to come up through the various stages of service/programme
monitoring, and since coverage of services often extends only to a fraction of the population,
supplementary systems can be put into place for national situation monitoring. Possibilities
include the use of sentinel sites, rapid appraisals, or a sample household survey. Information
from these sources can cross-check national statistics from the above sources.
Generally, data are being gathered by many different programmes in a country. UNICEF should
play a role in bringing together the various groups such as the Planning Ministry, statistical
departments of other ministries, research centers, etc.

Analysis of data in situation monitoring
A common weakness, in many countries, is the failure to analyse, use and publish in an up-todate and compelling manner data that have already been collected. Another common
shortcoming is not disaggregating the information by socio-economic level, gender and age
group.

A. Disaggregating data by socio-economic group
UNICEF's mandate to help countries to reach the disadvantaged requires disaggregated
information to develop strategies for reducing disparities. When designing programmes, it is
important to know where the problem areas are in order to set priorities and determine where
limited resources can best be put to work.
UNICEF can promote the analysis of existing data, and its reporting, in ways that show
differences between regions (or provinces), urban and rural areas, ethnic or racial groups,
communities or families dependent on different occupations, persons of different education and
income levels, etc. Sometimes this requires collecting new data, usually by the addition of a few
simple questions to existing surveys, forms, or other data collection instruments.
Disaggregation also can be used to mobilize efforts of other donors, NGOs and church groups
by increasing their awareness of the needs of special groups. Also, tracking disparities can
assist in monitoring the impact of policy changes on disadvantaged groups.This is essential for
following the effects, for example, of structural adjustment and attempts at cushioning the
impact, known as "Adjustment with a Human Face" policies.
B. Disaggregating data by gender and age
Equally important, and for similar reasons of advocacy, programming and social mobilization,
data should be disaggregated by gender and age. Increasingly, those involved in development
work in agriculture, health and education are aware of the importance of considering women's
needs and resources and the effects of proposed activities on women. For example,
understanding intrahousehold divisions of labor, income and expenditures is necessary for
those planning agricultural and income-generating programmes. Menopausal women have
health needs different from teenage mothers. For national literacy rates to rise, special effort
must be directed to the schooling of women and girls.
UNICEF can promote this effort to track disparities between men and women, boys and girls, in
many ways, from requiring it on monitoring forms to publishing data collected by others. The
chief challenge is to convince those collecting and analysing data of the value of disaggregation,
since, generally speaking, it is neither difficult nor costly.
UNICEF's Programme Guidelines for Women (UNICEF Policy and Procedure Manual, Book D,
Chapter 1, Section 3) lists the kinds of information useful for assessing the condition of women:
•
•
•
•
•
•
•
•

literacy of adult population by sex;
women's access to education and training opportunities;
type of family structure and family headship patterns;
fertility levels and birth spacing;
household income in relation to women's income;
work patterns of women in subsistence and market production;
access to and ownership of productive resources;
mortality rates by age and causes (including maternal mortality);

•
•
•
•
•
•

major health and nutritional needs and problems (including those that are not maternity
related);
women's access to health services;
child care arrangements (accessibility of formal and informal arrangements);
perceptions of men and women, including those of older women (mothers-in-law);
female circumcision (where applicable);
violence against women.

Publication, Dissemination and Use of Results
Publication. The analysis of data on the condition of children and women should be
disseminated in ways that have greatest impact on decision-makers and public opinion. The
State of the World's Children demonstrates the impact a report can have. The presentation can
show trends over time and focus attention on disadvantaged groups. A number of countries
publish their own "State of the Nation's Children" report In Burkina Faso, a Child Monitoring
Bulletin is published twice a year. Publication and dissemination of the Situation Analysis
prepared as part of (or in conjunction with) the country programme can be useful in advocating
policy directions and formulating future national plans. Some countries print information on the
status of children in official publications, such as those of the Census Office or Ministry of
Health. The Annual Report of the Central Bank of Sri Lanka included a section on the state of
children. Well-designed graphs and charts can be used effectively in reports, press releases,
pamphlets, posters, and TV spots to increase awareness. Dissernination of reports provides
opportunities to improve child monitoring.
Advocacy and programme planning. A process should be established so that planners and
policy makers take into consideration the situation of children as they decide on policy,
programme and budget priorities. Effective advocacy convinces decision-makers to give high
priority to targets and efforts to improve the well-being of poor children. Monitoring results
should be brought together in the Mid-Term Programme Review and the Situation Analysis
every two or three years.
Social mobilization. Results of child monitoring efforts can be used to mount public awareness
campaigns. For example, the new Facts for Life initiative co-sponsored by UNICEF, WHO,
UNESCO and over 100 NGOs and professional associations can be individualized for each
country by including statistics about the problems affecting children in that country and adapting
the core health messages to the culture.

7. Monitoring Goals for Children and Development in the 1990s
UNICEF has undertaken to monitor the "Goals for Children and Development in the 1990s",
which were endorsed by the World Summit for Children in September 1990. This goes beyond
"Strengthening Monitoring", which treats monitoring from the point of view of the country.
Monitoring of the global goals will also have its base in countries, and will be of benefit to
countries, but it also has a larger dimension (Box III-1; see also Appendix F for complete list of
goals).

Box III-1: Major Goals for Child Survival, Development and Protection in the
1990s.

1. Between 1990 and the year 2000, reduction of at and under-5 mortality rate in all countries
by one-third or to 50 and 70 per 1000 live births respectively, whichever is less.
2. Between 1990 and the year 2000, reduction of maternal mortality rate by half.
3. Between 1990 and the year 2000, reduction of severe and moderate malnutrition among
under-5 children by half.
4. Universal access to safe drinking water and to sanitary means of excreta disposal.
5. By the year 2000, universal access to basic education and completion of primary education
by at least 80 per cent of primary school age children.
6. Reduction of the adult illiteracy rate (the appropriate age group to be determined in each
country) to at least half its 1990 level with emphasis on female literacy.
7. Improved protection of children in especially difficult circumstances.
Supporting goals, also important in their own right, include:
1. Global eradication of poliomyelitis by the year 2000.
2. Elimination of neonatal tetanus by 1995.
3. Elimination of guinea-worm disease (dracunculiasis) by the year 2000.
4. Virtual elimination of Vitamin A deficiency and iodine deficiency disorders.
5. Access by all couples to information and services to prevent pregnancies which are too
early, too closely spaced, too late or too many.
6. Growth promotion and its regular monitoring to be institutionalized in all countries by the end
of the 1990s.
7. Reduction by 95 per cent in measles deaths and reduction by 90 per cent of measles cases
compared to pre-immunization levels by 1995, as a major step to the global eradication of
measles in the longer run.
8. Reduction by 50 per cent in the deaths due to diarrhoea in children under the age of five
years; and 25 per cent reduction in the diarrhoea incidence rate.
9. Reduction by one-third in the deaths due to acute respiratory infections in children under five
years.
Source: UNICEF, Executive Board Document, E/ICEF/1990/L.5

The Plan of Action adopted by the Summit states:
"(v) As the world's lead agency for children, the United Nations Children's Fund is requested to
prepare, in close collaboration with the relevant specialized agencies and other United Nations
organs, a consolidated analysis of the plans and actions undertaken by individual countries and

the international community in support of the child-related development goals for the 1990s. The
governing bodies of the relevant specialized agencies and United Nations organs are requested
to include a periodic review of the implementation of the Declaration and this Plan of Action at
their regular sessions, and to keep the General Assembly of the United Nations, through the
Economic and Social Council, fully informed of progress to date, and additional action required
during the decade ahead" (para 35).
..."Furthermore the Secretary-General of the United Nations is requested to arrange for a middecade review, at all appropriate levels, of the progress being made towards implementing the
commitments of the Declaration and Plan of Action" (para 35 (iv)).
On the subject of how to monitor the goals, the Plan of Action states:
"(v) Each country should establish appropriate mechanisms for the regular and timely collection,
analysis and publication of data required to monitor relevant social indicators relating to the wellbeing of children such as neo-natal, infant and under-five, mortality rates, maternal mortality and
fertility rates, nutritional level, immunization coverage, morbidity rates of diseases of public
health importance, school enrolment and achievement and literacy rates - which record the
progress being made towards the goals set forth in this Plan of Action, and corresponding
national plains of action. Statistics should be disaggregated by gender to ensure that any
inequitable impact of programmes on girls and women can be monitored and corrected. It is
particularly important that mechanisms be established to alert policy makers quickly to any
adverse trends to enable timely corrective action. Indicators of human development should be
periodically reviewed by national leaders and decision makers, as is currently done with
indicators of economic development" (para 34).
The goals for children and development are listed in the Appendix to the Plan of Action. The
preamble states:
"...These goals are recommended for implementation by all countries where they are applicable,
with appropriate adaptation to the specific situation of each country in terms of phasing,
standards, priorities, and availability of resources, with respect for cultural, religious, and social
traditions. Additional goals that are particularly relevant to a country's specific situation should
be added in the national plan of action."
The goals all relate to outcomes, not processes. It follows that there is not necessarily a
separate programme to reach each goal. To take a major example, services of PHC/MCH/FP
would be a process leading towards the realization of many of these goals.
The global goals have to be "internalized" in country programming as far as they are relevant to
the country. Document CF/PD/PRO/1990-005 states:
"The Medium-Term Plan (1990-1993) provides suggestions on how, as Part of the country
programming process, countries can go about a systematic review of the goals and strategies
for the 1990s and their relevance to their respective countries (paras 15-1 7).
As requested by the Executive Board, the strategies for children should be situated within
general development strategies, and should take into account.- national capacity building,
poverty alleviation, people's - in particular women's - empowerment and participation in planning
and implementation, sustainability and environmental soundness, and coordinated and
intersectoral policies, recognizing the necessity for adequate resources to achieve those
objectives."

The monitoring of global outcomes has to be at country level, with aggregation to regional and
global levels. Thus it will be necessary to have a breakdown of country outcomes corresponding
to the list of global goals, but this may not correspond exactly to the country's programmes
(processes) and their expected outcomes to be monitored as described in subsections 4 and 5
of this section.
It will, therefore, be necessary to monitor additional outcomes and to support each country's
capacity to do this. However, the monitoring of all these outcomes will be of use to the country.
The methods of monitoring will, in general, be those discussed earlier in this section. However,
there are some points to be clarified, for example:
• how is the problem of missing data to be dealt with, e.g. where the outcome depends on
the base value in 1990;
• where are proxies to be used, e.g. passing grade 4 might be taken as a proxy for the
"achievement of primary education";
• there needs to be some uniformity of indicators to permit aggregation to global goals.
The indicators for some goals need more definition, e.g. universal access to sanitary
means of excreta disposal.
Headquarters has set up a task force to work on the identification of indicators. The Planning
Office is working on a Database on the Decade, and feasibility of achievement (as assessed by
each field office). In connection with this, CF/EXD/1990-009 Add 1 was issued in August 1990.
When these two instruments have been developed, a supplement to this chapter will be issued,
giving further information about methods of monitoring global goals, and agreed interpretations
and indicators. Meanwhile, when offices are helping countries to prepare programme
monitoring, they should take account as far as possible of the decade goals.

7. References and Bibliography
Monitoring methods
a. Pratt, Brian and Boyden, Jo, eds., The Field Director's Manual: An Oxfam Manual for
Development Workers, Oxford, UK Oxford University Press, 1985. Especially useful for
programmes in which UNICEF is co-operating with NGOs.
b. Norton, Maureen, and Pines Benoliel, Sharon, Guidelines for Data Collection, Monitoring,
and Evaluation Plans for A.I.D.-Assisted Projects, A.I.D. Program Design and Evaluation
Methodology Report No. 9., Washington, D.C., U.S. A.I.D., April 1987. For additional
information on preparing monitoring plans.
c. Kumar, Krishna, Indicators for Measuring Changes in Income, Food Availability and
Consumption, and the Natural Resource Base, A.I.D. Program Design and Evaluation
Methodology Report No. 12. Washington, D.C.: U.S. A.I.D., September 1989. Helpful
discussion about indicators for assessing impact of agricultural and rural development
programmes.
d. Cohen, Sylvie I., "Rapid Assessment Guide for Effective and Sustainable Social
Mobilization: An Application to the Goals of UCI by 1990," New York: UNICEF,
Communication Section, Programme Division, 5 May 1989. For additional information on
monitoring social mobilization.

e. World Health Organization, Diarrhoeal Diseases Control Programme, Communication, A
Guide for Managers of National Diarrhoeal Disease Control Programmes, Planning,
management and appraisal of communication activities, 1987.
f.

Bichmann, Wolfgang, et al., Towards the Measurement of Community Participation, World
Health Forum, Vol. 10, p.471.

UNICEF examples of community-based monitoring in Central America and
Tanzania
a. Andersson, Neil, "Monitoring and Evaluation of Child Survival: A Conceptual Framework for
Health Measurement Emerging from Two UNICEF Projects in Central America," UNICEF
Area Office for Central America and Panama, Unpublished paper, nd.
b. Andersson, Neil, et al, "Sentinel Site Surveillance in Health Planning: Preliminary
Experiences in Mexico and Central America," Unpublished paper, nd.
c. Joint WHO/UNICEF Nutrition Support Programme in Iringa, Tanzania, 1983-1988 Evaluation
Report, Dar es Salaam, United Republic of Tanzania, UNICEF, WHO, October 1988.
d. Ledogar, Robert J. and Andersson, Neil, 'Monitoring and Evaluation of Development
Programmes: An Epidemiological Approach," Unpublished paper, nd.

Monitoring Women's Programmes
a. UNICEF. Programme Guidelines. Women's Programmes. Vol 4. (New York: UNICEF, July
1989).
b. Wignaraja, Ponna. Women, Poverty and Access to Credit Innovative Approaches. Staff
Working Paper No. 1. (New York: UNICEF, Sept. 1988).

Section IV - Strengthening Evaluation
1. Scope
The evaluation process includes a number of distinct stages each of which may be carried out
by a different individual or group of individuals. UNICEF staff may be involved in any or all of
these stages:
• planning evaluations
• gathering or recording baseline data
• managing evaluations
• implementing evaluations
• using evaluations
•
This section outlines the principal tasks in each stage.

•
•
•
•
•

The principal planning tasks are determining the purposes, questions and methods of
the evaluation.
Making provisions for the gathering and recording of baseline data is a crucial
contribution to evaluation, through often not viewed as a part of "evaluation"
Managing evaluations entails negotiating with others to determine the evaluation plan
and prepare the Terms of Reference, selecting and supervising the evaluator(s), and
using the results.
Carrying out an evaluation essentially consists of collecting and analysing data,
formulating recommendations and writing a report.
Using the evaluation results ranges from disseminating findings and implementing
recommendations to planning future programmes and advocacy. The uses and
limitations of a number of evaluation methods are summarized in the last section.

2. Summary of Action
During programme preparation and start up it is necessary to specify general arrangements and
make budgetary provisions fur future evaluations, and arrange for gathering base-line data. It
usually depends on officials who are not responsible for evaluations, and perhaps for this
reason is far too often overlooked.
The evaluation plans should be outlined in the country programme plan of operations, annual
plans of action and annual reports for each programme and should clarify responsibilities of
country officials and UNICEF (described in Section II). The principal manager/supervisor of an
evaluation of a UNICEF-assisted programme may be a government official and/or UNICEF staff
member. An individual may assume different roles for different evaluations depending on what
is being evaluated, the skills and interests of the individual, and work schedules. For example,
the individual may be responsible for planning and managing an evaluation, participating in an
evaluation team, supervising a consultant or using the results of several evaluations.
Managers of evaluations usually have three principal responsibilities: (1) to determine the
evaluation purposes, questions and methods, and sum these up in the Terms of Reference
(TOR); (2) to select train and supervise the evaluation team; and (3) to use the evaluation
results. The steps involved are described below and summarized in a checklist in Appendix D.
Persons responsible for conducting evaluations should: (1 familiarize themselves with the
programme/project and refine the evaluation plan; (2) use the evaluation methods to gather
data; (3) analyse the data collected and formulate findings and recommendations; (4) write the
evaluation report; and (5) debrief the interested parties on the findings and recommendations
(checklist in Appendix D).

3. Laying the Basis for Evaluation
Objectives should be drafted sufficiently clearly in the plan of operations, to make evaluation
possible; often they will contain the indicators on which the success of the programme or
service is to be judged.
The plan should include provision for monitoring and evaluation. It should designate an
organization or person who will be responsible for evaluation. The organization may well be
influenced by who should use the results of the evaluation process. It should make budgetary
provisions and if necessary arrange for some training of the responsible people.
Unless the base-line data have been gathered during the situation analysis, arrangements
should be included for it to be gathered as the programme or augmented services start up.

These duties fall on the officials responsible for programme preparation, who will usually not be
professionals in the field of evaluation. There is in fact a serious risk of their being forgotten.
Assuming this is not allowed to happen, it may be useful to seek the advice of people
knowledgeable about evaluation, for the formulation of these sections of the plan of operations.

4. Planning Evaluations
The evaluation plan should identify:
•
•
•
•
•
•

Why the purposes of the evaluation - who can/will use the results
When the timing of evaluation in the programme cycle
What the scope and focus of the national plan or evaluation and questions for the
evaluation to answer
Who those responsible for managing and those responsible for carrying out the
evaluation, specifying whether the evaluation team will be internal or external or a
combination of both
How the methods of gathering data to answer the questions
Resources the supplies and materials, infrastructure and logistics needed for the
evaluation

Why do the evaluation? Decide on evaluation purposes
The first steps in planning an evaluation are to identify a manageable number of evaluation
purposes and to obtain agreement on priorities among them. This comes down to identifying
who needs what information and for what purpose. Busy managers may be tempted to assume
that the evaluation purposes and specific objectives do not need a lot of attention and simply
jump ahead to the methods. However, such an approach ensures that the evaluation report will
gather dust on shelves, remaining unused.
The best way to decide on the purposes of an evaluation is to identify all who might use the
evaluation and then discuss with them what they want the evaluation to do and how they would
use its results, what difference the information would make - that it would be possible to do
something about the problems disclosed. Potential evaluation users include responsible national
officials, policy makers or those with a stake in the project or programme and individuals in
organizations related to the activity.
When the evaluation purpose has been decided it must be clearly set forth in the Terms of
Reference (see Section IV, subsection 5).

Four common reasons for evaluations are:
a. improving design and/or performance of policies, services, programmes and projects
(formative evaluation) .Evaluations can identify problems that explain why certain activities
are not performing as expected. The evaluation process may also improve performance by
clarifying objectives, improving communication between all concerned parties, highlighting
obstacles, and it may lead to accelerated implementation.
b. making choices among activities (summative evaluation) .It is important to clarify at the
outset whether the primary purpose of the evaluation is to improve programme performance
or to make an overall judgement about programme effectiveness. Formative evaluation is

particularly useful to operating staff. Summative evaluation can help policy-makers in the
development of national policies and programmes by identifying efforts with the greatest
impact. It is also useful for making decisions whether to continue funding an activity (Patton,
1978).
c. learning lessons for future application . Evaluations can be conducted for the lessons they
provide to government agencies, UNICEF, other donors, and/or those responsible for similar
services/programmes. An evaluation may look at programmes to learn what fosters
replication (expanding a programme to a larger area, "going to scale") or sustainability. For
example, an evaluation can look at social mobilization in several UCI programmes to identify
what methods and strategies work best over time.
d. accountability . Evaluations are often required for accountability, to find out whether
programme managers are using limited resources in the ways planned and bringing about the
intended results. Government planning offices, donors and others who contribute to a
programme use evaluations to make funding decisions.
Evaluations usually have multiple purposes. Limiting the number of objectives, however, helps
make the work feasible and the results practical. The sponsors must decide on a few reasons
for the evaluation and among them, what takes priority. Identifying and delimiting the purposes
of the key users is essential for good evaluation design and influences the rest of the evaluation
process. Any single evaluation cannot address all issues.

When to do an evaluation?
The timing of a major evaluation is affected by:
a. its relation to the sector/ministry plan, within the country development plan;
b. its relation to the UNICEF programming cycle, which is usually based on the above;
c. a significant problem identified in the course of monitoring; and
d. a donor request.
Formal evaluations usually are scheduled (in the Planops) on completion of critical phases or at
the end of the project/programme period. Ongoing evaluation, especially self-evaluation by the
programme/project staff, occurs during implementation. Mid-term and annual reviews also
assess programme progress during implementation. Timeliness is critical to the degree of utility
of the results of a given evaluation.
Timing of evaluations depends on decision-making needs and the purposes of the evaluation.
Mid-term evaluations are useful to focus on programme effectiveness and management (inputsoutputs, processes). For example, the 1988 mid-term evaluation of UNICEF Emergency
Programmes in Mozambique presented recommendations on management issues and on
improving UNICEF supply operations in Maputo and New York. Final evaluations tend to
address impacts, sustainability, costs and broader policy issues. An example is the evaluation of
the Cash for Food programme in Ethiopia, described below. Plans of Action should specify
when evaluations are to take place.

When evaluations may not be needed

If project managers have carried out on-going evaluations and have well-documented project
accomplishments, a formal evaluation may be needed less frequently. Similarly, when
successes or failures are well-documented, and the reasons clearly understood, a formal
evaluation may be waived and a final report written using existing documentation.
When scheduling evaluations, consider the seasons and work cycles of people involved in the
activity. Visiting isolated communities may be impossible in the rainy season; many rural
beneficiaries are not available for interviews at harvest time; at the end of the fiscal year,
managers are swamped with other duties. Scheduling also may affect the findings directly. For
example, malnutrition and certain diseases are related to seasonal changes (Chambers, 1985).

What should be evaluated? Deciding the scope and focus of the evaluation;
shaping the specific evaluation questions
Determining the scope of an evaluation includes identifying the geographic area, type of activity,
and time period that the evaluation should cover. This may have been specified to be identical
with the programme/project during planning (e.g., in the PlanOps). Other options can be
considered, including looking at one activity in several programmes to compare the
effectiveness of various approaches (a thematic evaluation). Or looking at several projects in
one small area can provide insight into their interactions and relative effectiveness. These
options expand the scope from project to service/programme evaluation, emphasize learning,
and conserve evaluation resources. They help bridge the gap between project evaluation and
programme planning. For example, a 1987 evaluation that was a sector review of early
childhood development programmes in Mauritius assessed all pre-school child development
projects and programmes over the previous decade and outlined priorities for the following
decade.

What questions should evaluations answer?
The scope of work of those conducting evaluations should include description, analysis and
prescription. Evaluation is the process of observing, measuring, analysing and interpreting the
results of a development project, service/programme or policy to answer these questions:
Description: What happened and how does this compare with what was expected? Evaluations
should measure and describe what the activity or service has accomplished and compare this to
what it was intended to do.
Analysis: Why and how did it happen or not happen? Evaluations should analyse the reasons
for what happened or the changes that occurred
Prescription: What should be done about it? Evaluations should recommend actions for
decision-makers
to
take,
based
on
the
answers
to
these
questions.
All evaluations should answer these general questions. In addition, each evaluation design
should include questions specific for that evaluation.
An evaluation may focus on different levels of results of a service/programme or project inputsoutputs, processes, outcomes or impacts (see key terms in Section III, subsection 3). The chart

below illustrates how an evaluation may pose questions in different ways depending on the
focus of the evaluation (Box IV-1: Sample Questions for Health Projects).
Box IV-1: Sample Questions for Health Projects
Level of question What Is Happening?
Input-output
Administrative records show that health clinics have been built and staffed. Are health services
(e.g., pre-natal and young child clinics, nutrition, education, ORT, and immunization) being
provided, accessible and being used?
Outcome
Are the health- and nutrition- related behaviors (e.g., knowledge of ORT, use of growth
monitoring) of beneficiaries changing as anticipated? Is vaccination coverage increasing?
Impact
To what extent have maternal/infant/child nutrition status and morbidity improved?
Why and How?
Input-output
What factors are responsible for low utilization of health services?
Outcome
Why are some participants changing the nutrition and health behaviors and others not?
Impact
What factors have prevented greater improvements in health and nutrition status and morbidity?
Source: Norton and Benouel, Guidelines for Data Collection, Monitoring and Evaluation Plans
for A.I.D.-Assisted Projects, pp. 36-37
Evaluation of implementation at various levels (central, regional, district, community) is another
possible framework for organizing an evaluation and planning its design, data collection, and
analysis. The WHO programme review methodology stresses this approach, particularly for
making programme adjustments.

The following key issues provide another focus in formulating the main questions
an evaluation should address:
Evaluations examine changes and their significance in relation to effectiveness, efficiency,
relevance, impact and sustainability. Any single evaluation may not be able to examine each of
these comprehensively, but they should be taken into consideration. It is important to
differentiate broad questions relating to effectiveness, efficiency, relevance and impact from
more specific survey questions.

a. Effectiveness. Is the project or programme achieving satisfactory progress toward its stated
objectives? The objectives describe specifically what the project is intended to accomplish,
for example, improving the work of traditional midwives, attainig a certain percentage of
children vaccinated or training a specific number of community health workers to use ORS
correctly. Accomplishments on this level are sometimes referred to as project outputs (what
was done), and are assumed to be linked to provision of inputs (human, financial and
material resources contributed to achieve the objectives).
b. Efficiency. Are the effects being achieved at an acceptable cost, compared with alternative
approaches to accomplishing the same objectives? The project may achieve its objectives
at lower cost or achieve more At the same cost. This involves considering institutional,
technical and other arrangements as well as financial management. What is the costeffectiveness of the project? (see Glossary).
c. Relevance. Are the project objectives still relevant? What is the value of the project in
relation to other priority needs and efforts? Is the problem addressed still a major problem?
Are the project activities germane to the country's development strategy and plausibly linked
to attainment of the intended effects?
d. Impact. What are the results of the project? What are the social, economic, technical,
environmental, and other effects on individuals, communities, and institutions? Impacts can
be immediate and long-range, intended and unintended, positive and negative, macro
(sector) and micro (household). Impact studies address the question: what difference has
the project made to the beneficiaries? How many have been affected?
Some people distinguish between outcomes and impacts, referring to outcomes as shortterm results (on the level of purpose) and impacts as long-term results (on the level of
broader goals). Outcomes are usually changes in the way people do things as a result of
the project (for example, mothers properly treating diarrhoea at home), while impacts refer
to the eventual result of these changes (the lowered death rate from diarrhoeal disease).
Demonstrating that a project caused a. particular impact is usually difficult since many
factors outside the project influence the results.
e. Sustainability. Is the activity likely to continue after donor funding, or after a special effort,
such as a campaign, ends? Two essential aspects of sustainability for social development
programmes are: social-institutional and economic (for economic development projects,
environmental sustainability is a third consideration). Do the beneficiaries accept the
programme, and is the host-institution developing the capacity and motivation to administer
it? Do they "own" the programme? Can the activity become partially self-sustaining
financially? How can the host institution meet future expenses, especially recurrent costs?
At times, a narrower focus may be appropriate. Process evaluations, which examine how a
project functions and is managed, may be used when decision-makers know that
implementation is not progressing as it should but are not sure why.

What not to evaluate
It is not necessary to re-evaluate the scientific basis of a medical act e.g., does the timely
administration of Vitamin A prevent blindness or does oral rehydration prevent/treat dehydration.
Nevertheless, trials may be advisable to convince the establishment and community about new
treatments and test out alternatives. Apart from this aspect, what needs to be evaluated is the

administration of the treatment - is it reaching those needing it, and is it being performed
correctly.

Who will do the evaluation?
The evaluation sponsors are responsible for determining the evaluation purposes and
participants. People can participate in an evaluation in various capacities, as managers,
evaluators, or providers of information. Country officials should participate in each capacity;
beneficiaries should be involved, too (as described in "a co-operative approach" in Section II).
The evaluation plan should:
a. identify which entity (entities) will manage/supervise the evaluation. The evaluation
manager/supervisor refers to the person who engages the evaluator or evaluation team and
supervises the team leader. Thee designated team leader is responsible for supervising
team members and managing the process of carrying out the evaluation
b. indicate what type of person (s) should conduct it, i.e., internal or external evaluators or a
combination. Internal evaluations are carried out by persons associated with the
service/programme; external by individuals or organizations not associated with it
c. identify skills required in the evaluation team
d. clarify roles of the various participating organizations and of those responsible for the
service/programme
e. indicate whose views should be sought, such as:

i. service deliverers. Usually there is a hierarchy to be consulted - province, district, front-line
workers.
ii. civil administrators - governor's staff, district commissioner, local government, political
party hierarchy.
iii. service users and potential users, beneficiaries and nonparticipants.

Internal and external evaluators
One advantage of an organization carrying out its own evaluation is that often it can more easily
accept and put recommendations into practice. Internal evaluations usually are carried out by
the staff of the service or programme, who tend to focus on process. Internal evaluations serve
as important national capacity strengthening activities. By-participating in evaluations
government officials not only gain skills in evaluation but learn how their programmes can be
enhanced. Internal evaluators, however, may lack objectivity and therefore hesitate to be critical
of their own programme. Evaluators from outside the programme may provide additional insight,
greater technical expertise, and be more objective in formulating recommendations. On the
other hand, they are not as familiar with the service/ programme, its history, constraints, and the
personalities involved. Advantages and limitations of both types of evaluators are summarized
in Table IV-1.

View Table IV-1: Trade-Offs Between Internal and External Evaluators
A well-balanced combination of internal and external evaluators may be preferable for
many purposes. Increasingly, evaluations have both kinds of participants, to take advantage of
the strengths and counter the limitations of each. Careful consideration of the purposes of the
evaluation helps determine whether such a balance is best in a given case. External evaluators
may be more credible to donors and officials of the Ministry of Finance and/or Planning and
more appropriate when the evaluation is summative. But the national capacity strengthening
effect of government participation may outweigh this factor.
External evaluations generally benefit when experts from the country are included on the team.
They understand the economic, political, social and cultural context, and may provide the team
access to other officials and sources of information that persons from outside the country, even
expatriates who live there, may not have. Similarly, in the case of internal evaluations, the
advantages should be weighed of including persons with some distance from the project, such
as an official from another department in the ministry or a UNICEF staff person from a
neighboring country. Sometimes it is preferable that shortcomings be pointed out by someone
not closely involved with the service/programme.
The Terms of Reference should include the required qualifications of evaluators. Selection of
the evaluation team should be done after the various stakeholders in the evaluation have had
an opportunity to suggest individuals for the team.
How to answer the evaluation questions? Determining information needs is an initial step
First, existing data should be identified and its quality assessed. In the process, some questions
may be answered. Some available data can be used to measure service/project progress in
meeting its objectives, while other existing data may be helpful for developing standards of
comparison. Subsection 7 discusses data sources for evaluation more fully.
a. Monitoring documents (such as regular reports and field visit notes), previous evaluations
(including feasibility analyses), and other documentation (audits, mid-term/annual reviews)
of the project should be available in government or UNICEF country offices. Also,
administrative data kept by project managers can be useful.
b. Other data for developing standards of comparison may be found in government routine
reporting systems (for example, census reports, household surveys, sentinel sites), records
or evaluations of similar programmes in agency or donor offices (WHO, other ministries,
NGOs, etc.), and reports in universities, research institutions, etc. Look especially for data
that can be disaggregated by geographic area and groups targeted by the project being
evaluated.
Then identify the minimum amount of new information needed to answer the evaluation
questions. Considerations of cost, time, feasibility and usefulness require careful selection of
what data to collect. Extensive data gathering is time-consuming and expensive and can result
in mountains of unnecessary data and "so what" results.
The potential users of the evaluation should be asked to consider how they will use the
information for important decisions. For example, certain information can clarify areas of doubt
about alternative courses of action. Concentrate on factors responsive to intervention. Also,
general trends or direction of change, rather than precise measures, often are sufficient.
Effective evaluations concentrate on collecting timely, relevant and useable data.

Thus, some questions to help decide the minimum new information needed are:
a. When is the information needed?
b. How will the information be used?
c. What difference would it make if this information is or isn't obtained?
d. What level of precision is required?
f. What resources are available for the evaluation?

Decide what data should be used for the evaluation
While planning an evaluation, the sponsors should work to specify evaluable questions and
design a feasible data collection plan. This involves deciding which indicators to use to measure
progress (see discussion of indicators in section on the monitoring plan). For example, an
indicator to measure results of a water project might be the use of the new water supply by
various population groups. Or, a measure of effectiveness of a women's credit project might be
the loan repayment rate. Selected illustrative indicators are found in Appendix C. The "best"
indicator is one carefully tailored to a specific project, selected by or with the active participation
of those engaged in the effort.
Ideally, indicators should be spelled out during the programme planning process and progress
in meeting them tracked by monitoring. Those planning an evaluation, and later the evaluation
team, should identify the original project objectives and their indicators. If indicators were clearly
stated, then the evaluators, in addition to using them, should judge whether they are
appropriate. If they were not clearly stated, then the evaluators must decide which indicators
should be used to measure progress.

Decide on criteria to judge progress
Every evaluation should measure progress and compare it to some standard. One comparison
should be to programme/project objectives. But evaluators also make judgements about the
adequacy of these objectives and the programme/project design. For example, an evaluation
may conclude that a programme is progressing very well, despite the fact that the objectives are
not being met on schedule. Their judgement, in this case, could be that the original objectives
were overly ambitious or programme design flawed in some other way. The evaluators can
review whether or not service delivery managers (and not just programme directors or
department heads) participated in setting the objectives.
Comparisons may be made to past performance, national targets, baseline data, similar
services or project areas to help in judging whether or not the service/project has made "enough
progress.' To make before/after comparisons adequate baseline data must be available. If it is
not available, an evaluation can be designed to compare participants to similar non-participants
or comparable communities with/without service/project activities.
Before/after and with/without comparisons are important for measuring outcomes and impacts.
However, determining the degree to which service/programme or project activities brought
about the impact is difficult because multiple factors are almost always involved in change.
With/without comparisons can give better insight into whether progress can be attributed to the
project activity or to other factors than before/after comparisons. But finding comparable groups
is difficult. However, insights into causality may be gained without rigorous control groups and
by using qualitative data.

The evaluation plan should request that evaluators look into reasons for programme success
and failure as well as explain their standard standardforfor measuring it.
Evaluation planners must then decide on the collection of quantitative and qualitative data
during the evaluation. Quantitative indicators are measurable, objective and can be expressed
as percentages, rates or ratios. Quantitative methods offer a higher degree of precision,
consistency, and replicability of findings-reliability in evaluation terms. However, limitations of
quantitative data are well known. Their narrow focus on a few specific variables can generate
"so what" results. The indicator may be precise, but not measure what the evaluation intends to
measure - and be of questionable validity or meaningfulness (Patton, 1978).
Qualitative methods of observing and interviewing as well as empathy with participants and their
situation are useful for gaining a holistic understanding, especially of complex socio-economic
changes. Qualitative research uses specialized techniques for obtaining in-depth responses
about what people think and how they feel. It enables programme managers to gain insights into
attitudes, beliefs, motives and behaviours of the target population. But qualitative methods do
not generate strictly representative data, so generalizations cannot always be drawn; they
cannot be replicated precisely for comparative analysis, and so may lack credibility with policymakers. Qualitative methods can result in information that is overly subjective, even biased, if it
is not backed up by empirical evidence as well. Qualitative techniques should be used with
quantitative techniques in a manner that is interrelated and complementary. (Source: A seminar
in qualitative research/July 1986/prepared for the Academy for Educational Development by
Needham Porter Novelli).
Quantitative data may be used for measuring "what happened" and qualitative for analysing
"how and why" it happened. Qualitative, for exploring a problem and possible solutions;
quantitative for showing the extent of a problem or population characteristic. In addition,
evaluators can use the two types of information to cross-check and balance one other. Most
evaluations use a combination of quantitative and qualitative data. Another consideration in
deciding which type of data to collect is determining how credible each kind of data is to those
who will use evaluation results.
A good example of the effective use of some of the qualitative indicators by an evaluation team
is seen in a 1987 impact evaluation of several UNICEF-supported programmes for street
children in Brazil. The team listed indicators for measuring child development in four major
areas: democratic skills, work skills, individual growth and moral values. Examples from the list
of indicators for each area respectively are co-operation, job-seeking skills, self-esteem and
sense of justice. Evaluation team members - Brazilian university professors - then employed
unobtrusive, participant observation methods to record what they judged as examples of each
trait among the children enrolled in the programmes. Qualitative data can be used in rapid
assessments such as this one to learn about peoples' perceptions, motivations, attitudes and
behaviors; these perceptions, etc., may also be measured quickly using quantitative indicators
and methods.

Match data-collection methods with evaluation purposes
Although the evaluation planner may not actually participate in the data collection (that is the
role of the evaluator), he or she will help to select the methods. There are four fundamental
ways of obtaining information:
1. collecting, tabulating, reviewing already available data;

2. questioning people through interviews, focus groups;
3. conducting surveys; and
4. observing people and things though field visits.
These techniques are often combined in methods such as Rapid Assessment Procedures
(RAP); Knowledge, Attitude, Practice (KAP) surveys; and case studies. Each of these methods
described in sub-section 8 of this section can be used for obtaining baseline, progress or impact
data. The methods selected should always match the evaluation objectives and focus. They
should flow from the questions asked and correspond with the purposes and uses of the data
(see Tables IV-2 to IV-4).
View Table IV-2: Data Gathering Method Related to Type of Question
View Table IV-3: Types of Data, Uses, and Collection Method for Different Evaluation
Focuses
View Table IV-4: Overview of Evaluation Instruments and their use: the example of EPI
Source: WHO Evaluation and monitoring of Maternal Immunization Programme (1986)

What resources are needed and available for evaluation?
Early in planning an evaluation make an estimate of its costs. Costs depend on the nature and
size of the service/programme and design of the evaluation. They include salaries, per diem,
travel costs and cost of reproduction of the report, office supplies and field equipment, costs of
surveys and data processing, translators, drivers, secretaries, overhead costs if other
organizations are contracted, etc. Also, evaluation planners should consider the non-financial or
indirect costs of the evaluation, the time and effort that people involved must contribute, when
away from their regular work.
The amount of resources available, determined during programme planning, should be built into
the budgets of all the entities supporting the programme financially, including UNICEF, other
donors, and the government department implementing the service/ programme. Certainly the
amount available influences the scope and methods of the evaluation.

Technical Assistance for Evaluation Design
It may be useful in designing an evaluation, particularly when of a technical nature, to obtain
specialized expertise early in the planning process. Advance planning of evaluations can help to
assure that appropriate expertise is available. Within UNICEF technical assistance can be
obtained from both the Regional Offices and the New York Evaluation Office and its consultant
registry.

5. Managing Evaluations
Managing evaluations principally consists of negotiating the evaluation plan, preparing the
Terms of Reference, selecting and working with the evaluation team.

Negotiating the evaluation plan with others
Planning evaluations requires negotiating skills as much as understanding how to design
evaluations. In most cases, many layers of persons/institutions have some responsibility for
various components of the service/programme being evaluated. The various roles must be

taken into account in deciding who should be consulted in determining evaluation objectives and
methods. These persons include ministry policy-makers, department supervisors,
service/programme/project managers at various levels, provincial/regional authorities,
representative primary service units users, beneficiaries, donor representatives, NGOs involved,
other UNICEF staff, etc.
The first challenge may be to convince officials and managers of the value of doing an
evaluation. One of the best ways to 'sell evaluation' is to convince those responsible for a
service/programme that evaluation can serve them and meet their needs. A long-range goal is
to build up a constituency of support for evaluation in government departments and agencies at
all levels. Scheduling evaluations as a part of the plan of operations can facilitate this process.
Donors can also assist by emphasizing the value of evaluation in their negotiations with country
officials.
Those in charge of planning the evaluation must set aside sufficient time for communicating with
staff, officials, and interested parties to determine their interests in the evaluation and insights
into the evaluation plan. This includes anyone who could use the evaluation results. Generally a
range of uses/users of the evaluation results can be identified. It is important to uncover the
various explicit and implicit interests and agendas regarding the priority purposes of the
evaluation. This step is often overlooked in preparing an evaluation.
Negotiating agreement on priority objectives of the evaluation consists of the same process as
building coalitions among groups for any purpose: identifying common ground, and persuading
those who have pressing, but unshared interests, to let them go for this activity to succeed. Of
course, this democratic, participatory process may not be possible in all instances. Certain
parties have overriding authority, for example, the government official responsible for the
agreement with UNICEF must approve an evaluation. Or a major funder may require an
external evaluation as a requirement for further funding. But, within these limits, often much
latitude about the specific evaluation objectives and methods exists.
The most common omission during the evaluation planning stage is not consulting soon enough
with the persons who could best use the evaluation results. If the results do not meet their
needs, it should not be surprising that they do not implement the evaluation recommendations
or incorporate findings in future planning. Detailed discussion before formulating the plan avoids
wasting time and resources collecting data that are not needed, or seeking answers to
questions of interest only to a few.
Identifying common ground on the purposes of the evaluation helps determine priorities in order
to Emit the number of evaluation questions. Various group process methods can be used to
help the interested parties determine priorities. For example, each may be asked to rank the list
of proposed evaluation purposes (uses) and/or questions (objectives) in numerical order of their
preferences. The totals can be the basis for further discussion and agreement. Agreement on
other aspects of the evaluation plan, such as who should conduct it and what methods should
be used, is easier to reach once the major parties agree on a few key purposes of the
evaluation.

Drawing Up the Terms of Reference
Before finalizing the Terms of Reference (TOR), it is important to obtain agreement of the
evaluation sponsors. These may include donors of the activity and other agencies in addition to
UNICEF and government ministry officials responsible for the programme being evaluated. One

way to proceed is to invite the sponsors and major stakeholders in the evaluation to a meeting
to discuss the draft evaluation plan. The agenda could include the following:
a. Present a one-page summary of the activity to be evaluated (service/project title, objectives,
strategies, time frame, funding sources and amount, and brief status report).
b. Present a draft one-page summary of the proposed evaluation (goals, scope, focus, time
frame, resources available, and possible evaluation questions). Alternately, this information
can be presented orally.
c. Discuss the evaluation design. Listen to all points of view and attempt to reach agreement
on priority goals for the evaluation (if this has not been done already), whether the
evaluators should be external or internal or both, data collection methods, and resources
each will contribute to the evaluation.
After the meeting, prepare a revised onepage summary of the evaluation plan to distribute to
interested persons.
The Terms of Reference, sometimes called a Scope of Work (SOW), set out the formal
agreements about the evaluation. It is like a contract with the evaluators spelling out what they
are to accomplish. It should clearly spell out what the evaluation is to do. (see Box IV-2). All
these topics should fit together logically, and can be concisely detailed in a few pages. The TOR
for a major evaluation should be prepared four to six months before the evaluation to allow
adequate time for planning. A good TOR paves the way for a good evaluation.
Box IV-2: Suggested Contents of the Terms of Reference
1. Background and Purpose
• Summarize in a paragraph the project, programme or service being evaluated (problem
addressed, objectives, strategies funding, accomplishments, etc.).
• Describe the purpose(s) of the evaluation, including who will use the evaluation results
and how, its scope, focus, timing, and who will conduct it.
•
2. Evaluation Questions or Objectives
• List the 24 major questions the evaluation should answer. The questions should be
related to the purpose and precisely stated so the evaluator knows what data to collect
and the information needs they meet.
• Specify the area and population to be considered, and, if possible, the kinds of
measurement (indicators).
• Some prefer using an "objectives" format. In this case, change questions into phrases
following "The objectives of the evaluation are to" (assess..., analyse..., advise..., etc.).
3. Evaluation Methods
• Describe the overall evaluation approach and data collection methods. Provide as much
guidance as possible, for example, specifying data collection instruments, sampling size
and procedures, or type of data analysis.
• Identify sources of available data.
4. Composition of the Evaluation Team
• Identify the skills and experience required to carry out the evaluation plan (e.g.,

•
•
•
•

education, field and evaluation experience, knowledge of the subject and project area,
and language proficiency), number of team members, and willingness to work with
nationals.
Some distinguish between desired and mandatory skills, or provide that certain
conditions be met by at least one team member. Multi-disciplinary teams are often
appropriate.
Specify the roles of others: project managers, government officials, donor
representatives, etc.
Assign skills required for individuals sponsored by each funding source.
Consider how the team will contribute to building national capacity.

5. Schedule of Major Tasks
• State the specific tasks the evaluator(s) are responsible to carry out and a preliminary
schedule of when they should be done. For example, suggest that evaluators meet with
certain officials early on, or present preliminary findings and recommendations to a
steering committee.
6. Deliverables
• List products to be delivered, to whom and when.
• Specify what should be in the report (see Box IV-4). For example, identify to whom
recommendations should be directed.
• Other deliverables could be a copy of the data collection instruments or the completed
data sets (filled out questionnaires or surveys).
• Indicate if the report should be translated.
7. Financial Requirements and Logistical Support
• Give the budget for the evaluation, or the amount for team salaries, per diem, travel,
expenses, etc.
• Include administrative reporting requirements about financial matters, if needed.
• Identify logistical support (vehicles, office space, secretarial services, etc.) available to
the team.
Specification of evaluation methods should be in the TOR. However, it is appropriate for the
evaluators (those carrying out the evaluation) to prepare the detailed evaluation design.
Sometimes, evaluators are asked to assist in drawing up the TOR If this is not possible during
the time the evaluation plan is being prepared, then they should be given a chance to review
and comment on the TOR before beginning the assignment.
Example:
The TOR for an 1988 evaluation of the Ethiopia Cash-for-Cash-for-FoodFood (CFF) Programme
clearly spelled out the uses and users of the evaluation and the task of evaluators. The $5.5
million pilot programme, begun in 1983 during the midst of drought, gave cash to famine victims
to purchase food in local markets. Able-bodied recipients, excluding women with small children,
were required to work on community projects such as irrigation, terracing, reforestation, and
feeder road construction. The evaluation purposes were clearly stated:
•

to assess the accomplishments and problems met in implementing the CFF approach as
a relief intervention and as a entry point to rehabilitation programmes; and

•

to suggest alternative courses of action for further improvements to the CFF approach
with a view of possible expansion or replication in other areas.

•
The first purpose was summative, helping policy-makers decide among alternative approaches
to a problem (comparing traditional food aid, including food-for-work programmes, to the pilot
cash-for-food programme as an emergency response to drought). The second purpose was
formative, helping improve implementation. Thus, the evaluation users (planners, sponsors)
were concerned with broad policy issues--impacts, costs, sustainability, and replicability-- as
well as improving implementation. Therefore, in the TOR they alerted the evaluators to seek out
unplanned results, unanticipated benefits and negative consequences of the programme. They
requested that the team give special attention to the following issues:
•
•
•
•
•

extent to which the project met its major objective as a relief intervention:
advantages/disadvantages of cash assistance as opposed to grain distribution;
cost effectiveness of the cash-for-food approach;
how cash assistance affected the economy of the project site and the adjacent surplus
area; and
whether activities generated by CFF reflected the felt needs of the community and
contribute to recovery.

The methodology selected was consistent with the objectives. The team would review existing
documentation from various administrative sources (not just the programme monitoring
documents) and use "rapid assessment" techniques. The team was asked to interview groups
of beneficiaries and non-beneficiaries, including female heads of household separately, and
local officials from various ministries at several sites (some implementing aspects of the
programme, many not involved with the programme). The four-person team consisted of two
Ethiopians from Addis Ababa University and two UNICEF officials based outside of Ethiopia
(from NY and Nairobi). Thus all four were "external" to the Programme.

Recruiting and selecting the evaluation team
The purposes and methods of the evaluation determine whom to recruit. The size of the
programme evaluated also affects the amount that can be spent on evaluation, and thus the
number of evaluators. Larger programmes are increasingly assessed by multi-disciplinary
teams. Often members can be recruited to bring different expertise and experience to a team. At
least one team member should be experienced in using the specific evaluation methods;
another with the technology or technical areas involved in the project.
Experience in the country and culture is an asset; it is important to include people from the
country on the team. If women are the subject of the study, recruiting women team members
usually helps ensure that the data gathered will more reliably reflect their views. When donor
representatives are included, their skills should mesh appropriately with other team members
and their qualifications should be suitable to the terms of reference. The size of the team
depends also on the duration of the evaluation: for "short" evaluations larger teams may be
necessary to ensure full coverage in the time allotted.
A "consultant profile" can be a helpful recruiting tool. Assistance with recruitment of consultants
can be sought from universities, UNICEF headquarters, where a consultancy registry is
maintained, and regional offices, as well as other donors, local institutions such as NGOs, etc.

Normally recruitment and selection takes 23 months. Following up mailings with phone calls
may expedite the process.

Selection
One advantage of working with a steering committee for the evaluation is that committee
members can help recruit, screen, interview (to the extent possible), and check references of
the most promising applicants. Finally, discuss the Terms of Reference with those selected and
make any mutually agreed upon modifications before initiating contractual arrangements.

Working with the evaluation team: orientation to the evaluation
The person(s) designated to manage the evaluation and supervise the evaluation team should
meet with the team before it begins work to give instructions, review the work plan and answer
questions. Often it is best to send background materials to team members before the orientation
meeting, including project/programme/service documentation (e.g., proposal, regular reports
and other evaluations), information about the evaluation (evaluation proposal or summary,
resumes of team members), and other helpful materials (such as evaluations of similar
programmes, information about the country context or baseline data).
Team orientation should include meetings with:
a. sponsors of the evaluation to obtain a better understanding of how they intend to use the
results
b. government policy-makers responsible for the programme
c. government managers implementing the programme
d. funders of the project and of similar programmes.
Time spent on team-building is well spent. The most important output of the orientation,
however, is a detailed team work plan and schedule. Also, the team could prepare a draft
outline of the final report (see Box IV-4, subsection 6). Arrangements for logistical support
should be worked out during the orientation.

Supervising the evaluation team
Good supervision requires that the evaluation manager keep in regular contact with the team
leader to see that work is progressing as planned. Maintaining frequent communication and a
cooperative relationship enables problems to be identified early, tackled together, and the work
plan modified accordingly. The team should limit data collection to the time allotted to ensure
that formulation of recommendations receives the adequate attention and that sufficient time is
allowed to write the report (most people can write 5 to 7 double-spaced pages a day). If the
team falls behind schedule, consider alternatives to ensure that the final steps are done
carefully. ideally, the supervisor and the government users of the report should discuss the
findings and recommendations, and comment on the draft report to the team before it is
finalized. The report must be completed at the end of the evaluation. Many evaluations have lost
their impact, because the report was not completed when the team broke up and the exercise
was over.

Finally, supervisors should evaluate the work of the evaluation team and assess the quality of
the evaluation and report by:
a. providing feedback to team members, individually or in a group. Identify what they did well
and areas where improvement is needed;
b. asking team members to offer their suggestions for improving the process next time; and
c. assessing the evaluation report and discussing its strengths and limitations with the team.

5. Conducting Evaluations: Tasks of Evaluators
Persons responsible for carrying out evaluations should: (1) familiarize themselves with the
service/programme/project and refine the evaluation plan; (2) gather data; (3) analyse existing
data and those collected to formulate findings and recommendations; (4) write the evaluation
report and (5) debrief the interested parties on the findings and recommendations (Appendix C).
Officials responsible for managing the evaluation, government personnel and/or UNICEF field
staff, may also participate in collecting data. Often they select an evaluator, or evaluation team,
and limit their own role to facilitating the work of the evaluator(s). In order to assess the
evaluation, however, they must have some familiarity with professional evaluation practice.

Refining the evaluation plan: sampling and site selection
Sampling is used when it is not feasible to include everyone in a study and so a smaller group is
chosen to represent the total. Evaluators must choose the type of sample, the sites and sample
size when using surveys, interviews or questionnaires. These selections occur in almost all
evaluations and are done more or less formally. Evaluators should explain the rationale for their
selection of sites and persons they interview in the evaluation report.
a. Probability samples are either random, giving each unit an equal chance of being selected,
or systematic, selecting every nth unit. They can be relied upon to be representative in order
to generalize for an entire population. But they also require a listing of every unit in the
population (a sampling frame) to draw the sample, using a table of random numbers. Often
comprehensive demographic records are not available and creating a frame is timeconsuming. Stratified random samples narrow the field by subdividing the population into
different layers or sections of a community (e.g., nurses, farmers, population of a given
village or area). Then, a table of random numbers can be used. If possible, this should be
done in advance by the project staff as preparation can be very time consuming. If done
correctly and the criteria for sample selection are valid and explained to the review team
members they should not have much trouble accepting the sample. At a minimum, the
project management should collect the necessary information (census data, tax rolls, etc.)
so that a sample can be drawn quickly by the team.
b. Non-probability samples include selecting cases on purpose to meet specified criteria (e.g.,
rural households with less than 2 acres of land) or through membership in some group (e.g.,
mothers' club) or accidentally (e.g., whomever is available at a particular site that day).
Probability and non-probability samples can be combined in two stage surveys, for example
by selecting areas or villages on purpose but persons within each village at random (or vice
versa) . Various "cluster" surveys concentrate a number of interviews in a limited number of
sites chosen at random, saving time and expense involved in these twostep approaches.

c. In general, data collection methods using stratified and purposive samples are most useful
because they are quicker and less costly than standard random samples. They can more
easily provide data disaggregated by income level and gender. For identifying problems and
possible solutions, looking at extremes (e.g., villages with successful and unsuccessful
latrine programmes) may be more useful than a random, representative sample (e.g., of all
villages with latrine programmes). While the randomized, control trial is the standard basis
for evaluating new medical interventions, such as vaccines, this research method presents
logistical and ethical problems for community-based interventions, inasmuch as it would
involve purposely denying immunization or some other needed intervention to the control
group.

d. Site selection is required for most evaluations, and should be done carefully so that sites are
representative. Evaluators should visit areas and activities that are typical of the range in the
programme being assessed, actively countering the tendency to see the most successful
and convenient sites. Robert Chambers argues convincingly in favor of overcoming
antipoverty biases, such as "dry season" and "tarmac" biases. If researchers visit only
during the dry season and groups nearest paved roads, they may miss the neediest people.
Other antipoverty biases: project areas tend to be better off than non-project areas;
outsiders tend to meet the more powerful and active people in a locale.
e. The size of the sample should depend on the frequency and variability of what is studied in
the population and the degree of precision needed. To identify the occurrence of a relatively
rare disease, the sample size must be larger to find those with the disease. If the condition
is widespread, sample size can be smaller in a homogeneous area than one with a diverse
population. For example, a commonly used sample size for reviewing immunization
coverage is 210, a cluster survey of 7 respondents in each of 30 sites. This example is a
minimum, possible because there is one variable yes/no (vaccinated/not vaccinated). To
take a more complicated example, for malnutrition it is usual to distinguish 3 degrees
(moderate, severe, very severe) and chronic (height/ age), acute (temporary--weight/height)
malnutrition, a much larger sample would be required. Larger samples permit a higher
degree of confidence and less probability of error (instructions for calculating sample size
and identifying random samples are found in WHO "Household Survey Manual," 1986;
USAID, 1985; and Feuerstein, 1986.

A major portion of the evaluators' time will be absorbed in data collection. Data collection
methods are presented in subsection 7 of this section. During the course of the evaluation,
evaluators should periodically review the Terms of Reference to ascertain that appropriate data
are being collected.
After collecting information, whenever possible the team should meet again with community
leaders or local officials to describe the initial findings and invite their reactions and insights.
This courtesy not only provides the community leaders information in return for their
cooperation, but also allows evaluators to cross-check their early analysis.
Analysing Data entails summarizing information gathered and organizing it in order to draw
conclusions. Statistical methods are appropriate for quantitative data; however, analyses of both
qualitative and quantitative data should be integrated in the findings.

A. Preliminary steps in analysing quantitative data are to check the questionnaires for
consistency and errors, exclude those that contain unreliable or incomplete data, and count
the answers to each question. For open-ended questions, group the responses into a few
discrete categories before counting. Tallying the responses can be done by hand or by
computer.
The next step is to list the results, often expressing them in percentages, and display them
in tables or figures, for example, bar, line or pie graphs (see Feuerstein, Chapter 5 for
practical suggestions on displaying data in tables and graphs) which:
•
•
•
•

help to show key information quickly;
make it easier to show comparisons;
illustrate patterns and trends; and
take up less room than narrative text.

Present data using more commonly understood kinds of statistical analysis, such as
percentages, averages, frequency distributions and cross-tabulations. Complex statistical
analysis, such as multiple regression analysis, usually should be avoided for policy makers.
Descriptive statistics describe quantities and situations and analytical statistics enable a
conclusion to be drawn from the numbers. It may be helpful to review a statistics textbook.
Computer software programmes, such as the Statistical Package for the Social Sciences
(SPSS) and Statistical Analysis System (SAS), can be used.
B .Analysis of qualitative data also organizes information into categories and displays it
systematically. Review interview notes and select the key points by looking for patterns and
relationships. It may be helpful to organize information around the items in the interview
schedule or evaluation objectives. Categorize responses numerically, for example, "Five of
six directors of health clinics interviewed identified..."
To organize and present qualitative data it may be helpful to develop matrices and
categories. A checklist matrix collects similar information about several different groups or
sites and reports it on a chart (e.g., views of directors, staff and beneficiaries on different
aspects of an activity; list of a number of physical and social characteristics at a number of
project sites). A conceptual matrix can suggest relationships between two dimensions of a
problem, each with varying values (e.g., the level of participation of farmers in maintaining an
irrigation system--high, medium or low--and where their land is positioned, at the head or tailend of the system). This kind of matrix helps explain relationships and reasons for a situation
(White, 1986).
Another technique to organize qualitative data is to develop subcategories of the most
important issues or variables and identify factors that exert a positive, negative or neutral
influence on each subcategory. For example, reasons given to interviewers about why
parents do/do not want their daughters to attend school can be listed in order of importance
or frequency mentioned and compared to a similar list of reasons given for their sons'
attendance or nonattendance. Then, display information graphically in tables (e.g., matrixes)
or figures (e.g., flow charts or diagrams showing interactions).
Qualitative data have advantages and uses in explaining reasons for a situation. For example,
qualitative data can be used to explain perceptions of beneficiaries and outlooks of community
leaders without using costly large-scale surveys.

C. Cost analysis may be required by the evaluation TOR (see Box IV-3: Cost Analysis).

Box IV-3: Cost Analysis
The main purposes of cost analysis are:
(a) to examine the efficiency of activities to ensure that the best use is being made of available
resources and identify where costs can be reduced or benefits extended for the same cost; and
(b) to assess the financial viability and sustainability of activities To identify ways to ensure that
needed resources will be available in the future.
Costs are economic values of all resources used to carry out an activity, and include financial and
nonfinancial inputs. Nonfinancial costs are in-kind contributions of materials or labor and
opportunity costs (value of a resource if not used for the project e.g. working time lost while
attending a clinic) . The value of these inputs can be calculated using project records or small scale
surveys.
The various categories of cost often have their own significance, and should thus be shown
separately. For example, the key constraint to the government budget may be recurring costs or
purchase of foreign exchange. At the community level, money cost may be a more severe
constraint than cost in time or supplies, if they have a low opportunity cost.
The cost of inputs can be divided by the number of direct outputs (e.g.,number of services
provided), beneficiaries, and outcomes (e.g., deaths averted). Cost effectiveness analysis
compares unit costs between two or more activities, and is the most relevant economic analysis for
social development programmes. Cost benefit analysis compares the costs to the economic value
of the benefits of an activity, which is difficult in the social sector (how do you calculate an
economic value on outcomes like deaths averted?). Simple Cost benefit analysis is useful in
assessing economic projects, such as women's income generating projects.
Despite the increased appreciation for Cost effectiveness studies, few adequate ones have been
done in the social field, principally because records on all costs of an effort are not available. Nor is
sufficient time given to calculating these costs during evaluations. Outside donors generally keep
careful track of their own expenditures, but are not too concerned about other expenditures and
costs. Complete records of the government's and communities' contributions, such as officials'
time, often do not exist. This problem weakened the cost analysis of the Iringa JNSP evaluation
and comparability of most EPI programme cost studies (Brenzel, 1989). Consequently, total costs
often are understated and constraints to sustainability overlooked. A useful analysis may be made
of costs that are more constraining than others, e.g. inputs that have to come from outside the
community may be more constraining than what the community or the users can supply. A
software package, "EPI Cost", is available from WHO.
An example of an evaluation focusing on cost issues, in which the evaluation team obtained
sufficient cost information using rapid assessment techniques and existing records, is the
evaluation of the Cash for Food Programme in Ethiopia. Team members queried officials from the
ministries of agriculture, finance, education; local and regional government administrators; peasant
association leaders, etc. and reviewed records they supplied. The team was able to estimate the
cost/per beneficiary/per year of the CFF programme ($38) and compare it to the cost/per
beneficiary/per year of the World Food Programme wheat delivered to the same sites ($72).By

interviewing participants and nonparticipants at several sites they also estimated the uses that
participants made of the cash (about one-third was spent on productive assets, such as animals,
rather than food),consequences of the community programmes undertaken by participants, and
effects of their spending on market prices. These estimates, while not providing definitive data,
were sufficiently clear to make programme recommendations and answer the evaluation questions

Formulating Findings
Findings answer the evaluation questions. They describe the situation, compare it to what was
expected, explain the reasons for the situation and its consequences for achieving programme
goals. For findings to be credible and persuasive, they must flow from the data gathered and be
backed up by the evidence collected. Findings are often better accepted when the positive
precedes the negative.
But what if the data is partial and information gathered is conflicting? First, judgments need to
be made despite the fact the data gathered will always be partial; so avoid sweeping
generalizations. Good evaluators always note the limitations of their data and conclusions.
Secondly, conflicting information can be handled by cross-checking data from different sources
or by giving more importance to views of more credible sources. However, do not suppress
conflicting data that cannot be resolved satisfactorily by cross-checking and weighing. Multiple
causes or factors influence a situation. Sometimes the most important findings are not expected
or emerge from "minority" information.
Findings may be called conclusions or lessons learned. However, some people distinguish
between findings (the facts, for example, the numbers or percentages), conclusions (what they
mean), and lessons learned (their relevance for the future of other programmes) . The findings
should:
a. describe project/programme/service results;
b. compare them to what was planned and/or some other standard;
c. judge whether "enough progress" was made; and
d. identify major reasons for successes, failures and constraints.
Do not limit a finding to describing a fact--include "so what," its effect on programme goals, and
"why," what accounts for it. Be sure to describe unexpected and unplanned consequences of
the programme, positive and negative. Identify major reasons for the accomplishments and
failures; even more important is to describe continuing constraints that affect future activity.
Usually some constraints are outside the control of the managers, and these should also be
noted because they might be relevant for other decision makers. For example, one of the
reasons why a particular health programme might not be working well is because of the
government's reduction in the staff, or freezing salaries, of the health service. Evaluators must
make a judgment about the programme/service, considering all the evidence and comparing
progress to some standard.
Focus the findings on issues related to the purposes of the evaluation and select the most
important and interesting ones. At this point, evaluators should review the original evaluation
questions in the TOR If the data do not enable evaluators to answer some of these questions,

they should acknowledge this in the findings. Using quantitative data facilitates comparisons,
since findings can be expressed in percentages and compared to the project objectives or
national targets. But qualitative data best provide the reasons for accomplishments and
problems, answering the how, why, and "so what" questions. Identifying cause-effect
relationships is difficult since so many factors are involved, but it is possible to identify a number
of contributing factors, noting where there is consensus and divergence on their importance.

Developing recommendations
The next step is to make recommendations based on the findings, interpreting the implications
of the findings for decision-makers. Recommendations should be practical, feasible suggestions
in response to the question: What should be done? In general, recommendations should
suggest ways to build on achievements, solve problems and counter constraints.
a. When drawing up recommendations, start with the findings. Make several possible
suggestions for action based on each. Identify to whom each recommendation is directed.
Then, check to see if the recommendations address the evaluation objectives.
b. Recommendations should be directed to different kinds of decisionmakers, such as project
managers; official policymakers, such as department heads and staff of planning/budget
departments; donor representatives and UNICEF. Make recommendations for potential
users of the evaluation. For example, suggestions can go beyond the service and include
policy recommendations to extend its benefits.
c. Avoid vague, general and impractical recommendations. Sometimes it may be beneficial to
provide options for decision makers or alternative suggestions to deal with the same issue.
In this case, the lead suggestion may be general (e.g., devise ways to improve the ability of
community health workers to make follow-up home visits), followed by several specific,
alternative ways of doing this. While impractical recommendations damage the credibility of
the evaluation, some important suggestions may be feasible only in the long run. Thus, it
may help to specify which suggestions are for the short-term, which for the mid term and
which for the long-term.
d. Always list the recommendations in priority order. If recommendations are grouped for
different types of decision makers, prioritize recommendations within each group as well as
overall. Answer the question "What is the most important thing that I should do?" for the
potential users. Prioritizing helps them understand which of many recommended actions will
have the most beneficial effect.
e. When possible, include estimates of the financial and organizational costs to implement the
suggestions. This makes the recommendations more usable by those to whom they are
directed. Assigning costs, including staff time, to each recommendation also helps the
evaluators be more realistic and aware of the need to select recommendations carefully.
Sometimes, if exact figures cannot be given, recommending percentages that represent
total increased allocations for each activity helps clarify priorities. Or, the evaluators may
recommend that total funding remain constant, but certain activities should be reduced, so
that others may receive increased resources.
Review findings and recommendations with users in a debriefing session before completing the
report. Evaluators may find it helpful to present a group of potential users of the evaluation the
results of the analysis of qualitative and quantitative data (on one sheet) and the evaluation

questions to be answered. Ask them to discuss what the data suggest to them, what is
significant or surprising, and what they think about conflicting data. Then give them a draft of the
preliminary findings/conclusions to discuss.
Reviewing the draft recommendations with those who must implement them is important. First,
this may help the evaluators improve the recommendations. The implementers might provide
additional information/insight which requires revising the analysis, suggest recommendations
the evaluators had not thought of, convince them to revise other recommendations or their
priority order. However, the evaluators must maintain professional objectivity and base their
recommendations on the available evidence. Secondly, if project managers and other decision
makers are involved in the interpretation of the data and formulation of the recommendations,
they will be more apt to accept the findings and act on the recommendations.

Preparing the Evaluation Report
A report is needed to communicate the findings and recommendations to decision-makers.
While project managers can use detailed, technical information about project implementation,
policy-makers want a short summary of the findings and recommendations. Thus, an executive
summary should always be prepared. A suggested outline of the evaluation report is contained
in Box IV-4.

Box IV-4: Suggested Contents of Evaluation Report
1. Title page
•
•
•
•

name of the service, programme or project evaluated
name and address of the organization the report is submitted to
names and organizations of the evaluators
date

2. Table of Contents
3. Acknowledgments (optional)
•

Identify those who contributed to the evaluation.

4. Executive Summary
•

Summarize the activity evaluated, the purpose of the evaluation and the methods used,
the major findings, and the recommendations in priority order.

•

2-3 pages (usually) and able to "stand alone" without reference to the rest of the report.

5. Introduction
•

Describe the activity being evaluated (the setting and problem addressed, objectives and
strategies, funding).

•

Summarize the evaluation context (purposes, sponsors, composition of the team,
duration).

6. Evaluation Objectives and Methodology.
•

List the evaluation objectives (the questions the evaluation was designed to answer).

•

Describe fully the evaluation methods (e.g., what data were collected; specific methods
used to gather and analyse them; rationale for visiting selected sites).

7. Findings (sometimes called findings and conclusions)
•

State findings clearly with data presented graphically in tables and figures. Include
effects of the findings on achievement of programme goals.

•

Explain the comparisons made to judge whether "enough progress" was made.

•

Identify reasons for accomplishments and failures, especially continuing constraints.

8. Recommendations
•

List the recommendations for different kinds of users in priority order. Include costs of
implementing them.

•

Link recommendations explicitly with the findings, discussing their implication for
decision-makers.

•

A proposed timetable for implementing/reviewing recommendations

9. Lessons Learned (optional)
•

Identify lessons learned for those planning, implementing or evaluating activities other
than those under consideration based on this evaluation.

10. Appendices
•

List of persons interviewed and sites visited

•

Data collection instruments (copies of questionnaires, surveys, etc.).

•

Terms of Reference.

•

Abbreviations.

•

Cost of the Evaluation.

•

Country data.

7. Using Evaluation Results

Attention to this final step--using evaluation results--helps the evaluation make a
difference.
Managers of evaluations can promote the use of evaluation results in many ways in addition to
those already outlined. They may encourage discussion of draft findings and recommendations;
distribute copies of the evaluation findings and recommendations to those who can use the
results; implement recommendations; advocate for greater support of the programme and
incorporate the evaluation results into planning future work.
Evaluation results should be used to:
a. improve organization and management
b. improve planning
c. assist decision-making
d. assist policy-making
e. indicate where action is needed
f.

improve monitoring

g. indicate where technical assistance and training are needed
h. indicate where further research is needed
i.

provide information for advocacy, social mobilization and fundraising

j.

update information on the condition of children and women.

Payoff for a well-focused evaluation
The time spent carefully planning the evaluation so that it meets needs of decisionmakers pays
off at this point. In Ethiopia government officials and UNICEF needed the evaluation results
comparing the trial Cash-for-Food programme to traditional food relief, because at the time of
the evaluation, drought and famine were beginning to reappear in many parts of Ethiopia.
Experience with two different approaches to famine relief and rehabilitation was evaluated.
Results in hand, they decided to use the new approach in selected areas adjacent to those
where food was grown. The evaluation helped bring about policy change, programme
improvement and renewed donor funding.

Providing Feedback
One of the first tasks after the evaluation is completed is to disseminate its results to potential
users. However, those who commissioned the evaluation should verify that the evaluation has
produced timely and credible information and well-based recommendations. Poor evaluations
should be shelved, but only after a careful analysis of what went wrong.

To whom

First, disseminate results to the evaluation requestors and then to other potential users. These
include implementers of the activity evaluated as well as of similar projects, such as those of
PVOs, policy makers, donor representatives, researchers interested in the topic (including
students); representatives of advocacy and interest groups, and the media. Returning findings
to the communities affected by the service/programme is often difficult, but is extremely
important.

How
Evaluation results can be disseminated in many forms, depending on the potential audience
(Table IV-5: Dissemination of Evaluation Plans, Procedures and Results). The entire report can
be distributed to administrators and funders. The executive summary can be distributed more
widely, for example to other policy-making staff, political bodies, or others involved in similar
programmes. Sometimes a one-page summary of the evaluation, with instructions for obtaining
the entire report, aids distribution of the findings and recommendations. The report should either
be written in or translated into the language of the country. A press release and radio or
television interviews can disseminate results widely. Articles can be written on the study and
published in professional journals. When evaluations are shared with organizations outside the
country, consider attaching a UNICEF Country Data Sheet to the report.

Table IV-5: Dissemination of Evaluation Plans, Procedures, and Results
______________________________________________________________________
Likely communication form
Potential Audience
Funding agencies for program or evaluation
Program administrators
Other relevant management level staff
Board members, trustees
Technical advisory committees
Relevant political bodies (for example, legislatures, city councils)
interested community groups
Current clients (guardians where appropriate)
prospective clients
Providers of program service (for example instructors, counselors, distributors)
Professional colleagues of evaluator(s)
Organizations or professions concerned with Program content

Local, state, regional media
national media
Other
____________________________________________________________________________
Source: Scarvia B. Anderson and Samuel Ball The Profession and practice of Program
evaluation. (SF: Jossey Bass Publications, 1978)
Meetings are another way to communicate evaluation results. First, the evaluation manager and
any team members available should meet with the requestors and others closely involved with
the activity evaluated. Later, seminars or workshops can be held at the national, regional or
local level and outcomes discussed with community groups, health workers, and others.
Discussion of the evaluation can be added to the agenda of already scheduled meetings.
The dissemination plan should consider the various audiences for the evaluation findings and
develop a strategy for reaching each (Table IV-6: Who Needs to Get the Results, Why and
How). Those responsible for the evaluation need to decide what to share, with whom and how.
View Table IV-6: Who needs to get the Results, Why and How

Following up on recommendations
A systematic effort should be made to implement recommendations made to managers and
policy- makers. Inclusion of a timetable for action and review in the report could help this.
Recommendations should be considered first by the managers and staff, who can:
a. meet alone and then with the evaluation requesters to discuss the findings and
recommendations,
b. schedule a staff retreat to consider issues raised and ways to solve problems identified by the
evaluation,
c. schedule staff training identified in the report and,
d. if appropriate, cooperate in implementation.
The manager of the service or programme should prepare a time line for implementing
recommendations. The government officials responsible for the programme/project and UNICEF
staff should check progress with project managers regularly. Some find it helpful to schedule a
meeting in 90 days to review implementation of recommendations. Often this follow-up requires
strengthening the monitoring system.
The recommendations directed to policy makers or others should be handled in a similar way.
Meet with interested persons within the appropriate ministries or agencies to discuss the
evaluation, expand on the recommendations if appropriate, and devise a plan for implementing
them. Then, keep in regular communication and schedule another meeting to review progress.

Using results to obtain support

Evaluations can be a tool to obtain support for a service or programme, even if they point out its
weaknesses. (If they do not point out weaknesses and/or constraints, and recommendations to
strengthen the programme, evaluators are not doing their job). By documenting what has been
achieved, evaluators help project leaders obtain support of government officials, increase
credibility in the community and raise funds from donors. If the evaluation affirms that the project
goals remain valid this lends credibility to programme managers' argument for the need for the
programme. By pointing out what needs to be done and ways to strengthen the programme, the
evaluation helps managers argue for obtaining the resources needed to implement the
recommended changes. Only if the evaluation recommends terminating funding are managers
prevented from using it for seeking additional programme support.
If the evaluation is well done and recommends policy-changes, programme managers can use it
as a tool for advocacy. Good evaluations forcefully demonstrate the potential beneficial impact
of suggested policy changes.

Using results for planning
Evaluation results should be fed into the next planning phases of the programming cycle of the
country and of UNICEF. The annual or mid-term reviews and preparation of the annual review
and Plan of Action for the next year often provide the first opportunities. But the findings should
also be considered in subsequent reviews and programming for the next cycle, especially the
strategy meetings leading to preparation of the Planops. Evaluation results provide a baseline
for future overall country-level, as well as individual programme/ project-level, planning. By
comparing evaluations of various programmes in the country, planners understand better what
works and why, which sector activities have made the greatest impact (in relation to amount of
funding), and which most need additional support. They thus have some objective basis for
making decisions of where to increase or cut funds, should that be necessary.
Evaluation results also can be used by country planners on the policy-level. Of course, a good
evaluation will only be one of many influences on policy; but a poor evaluation has no chance of
making a difference.
Evaluations sent to UNICEF headquarters become part of an institutional memory that
contributes to learning about effective programmes on a regional and global level. An expanding
data bank in the Evaluation Office and a regular Evaluation Newsletter are helping demonstrate
that this learning process can be useful for country staff to plan better
programmes.

8. Data sources for Evaluation
The following data collection methods may be used, alone or in various combination with others,
to gather data for monitoring and evaluation.

Using available information
Existing data includes primary and secondary data, data from the service or project and from
external sources. Sources of information on the programme include monitoring documents,
such as progress reports and field trip notes, and evaluations, including midterm reviews.
Relevant outside information includes evaluations of similar programmes, special studies

researching the problem in the country and data from government census, surveys, sentinel
sites, or routine reporting. Existing data may be used to help design an evaluation, to answer
some evaluation questions, to cross-check other data or to provide a standard of comparison.
Interviews may be conducted with individuals or groups. Interviews or discussions with persons
associated with a service/project and beneficiaries are usually part of evaluations. Interviewing
key informants, officials and leaders in the community, can provide valuable information from
different perspectives. Group interviews can be useful to obtain a range of views quickly. There
are two main types of group interview: focus group interviews and community interviews.
Selected participants of small focus groups talk with one another under the guidance of a
moderator who lists the main subtopics and issues to be discussed. In community interviews,
which usually take the form of community meetings open to all, interaction is between the
interviewer and the participants (Kumar, 1987).
Group interviews are useful for:
a. understanding the family and community context, beliefs, perceptions and customs;
b. securing background information for planning;
c. generating ideas and hypotheses for design of services or programmes;
d. getting feedback from beneficiaries;
e. assessing responses to recommended innovations;
f. interpreting available quantitative data;
g. investigating implementation problems;
h. monitoring; and
i. evaluation (Kumar, 1987).
Interviews yield qualitative data through free and guided interviews or quantitative data through
standardized interviews. In a free interview the only guidelines are the evaluation objectives;
thus, it is a good tool for exploring the opinions of a respondent and uncovering unexpected
factors. In a guided interview, the respondent is asked to provide information about items on a
prepared checklist. Because they aim to collect quantitative data, standardized interviews have
a questionnaire with a fixed number of questions and sometimes a preselected range of
possible answers. The more open-ended the interview the more deeply the respondents'
feelings and perspectives can be understood; the more structured the interview the more
comparable the data.
Less structured interviews rely on judgement and experience, and are often used by decision
makers because they are less costly and can provide richer, timely, useful information.
Standardized interviews generate precise, quantitative data as do surveys. They also minimize
subjective bias and maximize comparability with other cases. However, they tend to reduce the
potential scope of responses.

A good questionnaire is short, avoids ambiguity and leading questions, includes cross-checking
questions and avoids questions that depend on memory and cannot be verified in any way.
(Many helpful suggestions about constructing and administering questionnaires are found in
Feuerstein, Chapter 4). One technique for accurate translation is to ask a second person to
translate the draft back into the original language.
Currently, questionnaires tend to be overused, maybe because they yield quantitative results
that can be tabulated by computer. Administering a questionnaire may make respondents feel
awkward, not a good atmosphere for eliciting information about past and potential changes in
behaviour. The greater richness of group discussions, home-visit discussions, discussions with
wise people more than compensates for their lesser quantitative content. Many reports based
on questionnaires provide a classical array of facts (percentages, breakdowns) but little
enlightenment about what makes people "tick" and motivations that could be built on to improve
Practices.

Data collection instruments should be pretested
Open-ended questions can be used in a pretest to develop closed ended questions, which are
easier to pre-code. Also, observing behavior can verify what people say. After identifying
specific observation items and devising a format to record each, dc a practice run and modify
the form if needed. Distinguish between observations and interpretation of what is seen.
Often it is preferable to have female staff interview female respondents, and male staff interview
male respondents. Sometimes rural women may be more comfortable expressing themselves in
a group meeting than in individual interviews. Questionnaire designers should seek
disaggregated data and adapt their methods to enable women to express their opinions freely.
People may be more willing to discuss what others do than what they do themselves.
Before beginning interviews in a site, explain to community leaders the reasons for the
evaluation, what data is being collected and how it will be used. Official backing can be a help or
a hindrance. Such backing promotes confidence in the evaluators, but not necessarily among
respondents. Outside evaluators should exhibit sensitivity, refrain from promising benefits they
cannot deliver and accept local customs.
Asking follow-up or probing questions to understand a response is important in less structured
interviews (e.g., ask respondents for more details to clarify or elaborate a point). Interviewers
should be open to hearing new or conflicting information. Informal conversations outside of
group interviews are one way to cross-check information or to identify areas of agreement and
disagreement. Asking a series of related questions in a group meeting is particularly effective for
identifying dimensions of a problem and feasible solutions (Honadle, 1982). Interviewers should
avoid making"snap judgements", encourage everyone to speak out, seek those with opposing
views and remain open to the unexpected.

Recording information in a systematic way is essential for later data analysis
Interviewers can repeat back their understanding of answers to help clarify a response. Some
people take notes during the interview. Time should be set aside to write down or clarify notes
each day. It is important to be systematic in recording observations, to record words or actions
exactly, descriptively rather than analytically, and try not to let one's own perceptions interfere
while recording.

Surveys
Censuses and sample surveys collect data about a population. A census is a complete
enumeration of all the units in a population, while a sample survey uses a randomly drawn
representative subgroup from which researchers generalize about the whole population. Largescale, technically complex surveys should be avoided in programme/project evaluations. They
are expensive, time-consuming and more appropriate for academic research or special studies.
Rapid assessments often include informal, small scale surveys. A survey with a small sample
size and only a few variables may be more feasible and useful than a large complex survey
(unless what is being studied is a rare occurrence). For example, informal surveys, with a small
number of questions (1020) administered to a small number of people (3050), based on nonrandom sampling (such as purposeful selection of people in different categories, or on the basis
of easy accessibility) and which permit interviewers more flexibility in the field, also generate
quantitative data which can be statistically analysed (Kumar, 1987). These non-random sample
surveys can provide helpful information, for example, about use of and access to health
services. Specific decisions about sampling methods, size and sites should be made by the
evaluators and are discussed in subsection 5.
A survey is only as good as the people administering it, so care should be given to selecting,
training and supervising surveyors. University students, health workers, teachers, government
officials and community workers may be selected. They should come from the same area as the
population to be surveyed. They should not only speak the local language fluently, but also be
respected by the people and have good listening and writing skills to record responses
accurately.
During the first day of field work, the supervisor (usually an evaluation team member) should
accompany the assistants, check their questionnaires and answer questions. Scheduling
regular meetings helps solve problems and keeps motivation high. The supervisors should
check that assistants are at work, forms are filled out correctly and truthfully, missing or
misunderstood data are corrected and good relationships are maintained with the people (Kearl,
1976).
Deciding between using a self reporting method (survey) or an interview depends on a number
of factors. In an interview the respondent does not have to be literate; however, questionnaires
must be administered to non literate people by someone who reads the questions aloud and
writes the responses. Some people may be reluctant to speak in a mixed group or to outsiders
about topics considered private (such as contraception), but are not reluctant to give an
anonymous response on a written survey. Surveys may be cheaper to use per person
contacted, can be distributed to a large number of people by persons with minimal training, and
answers can be quickly recorded. This method is useful for seeking mostly quantitative data.
But the information is limited to the specific questions and it is hard to know whether the
answers are reliable. The interview method may be more expensive and time consuming if the
same number of persons will be contacted, requires training interviewers, and the interviewer
may influence the responses. But the interviewer can explain questions that are not clear, probe
for more information, check answers which are not clear and see the circumstances in which the
responses are given (Feuerstein, 1986).
Observation is a good method for collecting data on behaviour patterns and physical conditions.
For example, observation is the best method to verify if ORS are being prepared correctly or
project inputs, such as ORS packets, have actually been delivered as planned. In these cases,
quantitative data may be collected. Observation can also be used to collect qualitative data, to

get a general feeling for local conditions or specific in-depth information. However, since people
are selective in what they see, remember, and interpret, observation must be systematic and
carefully recorded.
Participant observation, a technique first used by anthropologists, requires that the researcher
stay for some time with the group being studied. This method gathers insights that might
otherwise be overlooked but is time consuming. Also, generalizations made from data gathered
at one site tend to be unreliable.
Another type of observation, laboratory testing, is appropriate for some types of indicators. For
example, laboratory methods can be used to test water quality or proper storage of vaccines or
drugs.

Rapid Assessments
"Rapid appraisal" or "reconnaissance" was developed to obtain timely, relevant, accurate and
useable information for development decision makers in a cost-effective way (Chambers, 1985).
While it was first used in the late 1970s and early 1980s for agriculture and rural development,
more recently it has been applied to health and other areas by UNICEF (see Box IV5: UNICEF's
Use of Rapid Assessments). Rapid Assessment Procedures (RAP)can be used to provide
information needed by decision-makers at every stage of the programming cycle-- design,
monitoring and final evaluation.
Major rapid lowcost data collection methods are key informant interviews, focus group
interviews, community interviews, direct observation and informal surveys (Kumar, 1987;
Honadle, 1982). RAP methods may collect qualitative and quantitative data.

Box IV-5: UNICEF's use of rapid assessment
UNICEF's increasing focus on community participation, including financial participation and
community management as proposed in Alma-Ata and in the Bamako Initiative, requires a
methodology that obtains beneficiary views. Rapid Assessment Procedures do this.
A series of rapid assessments of national EPI programmes in Nigeria, Burkina Faso, Turkey,
Cameroon, Senegal and the Dominican Republic were undertaken from 19851989 to draw
lessons which could improve immunization programmes and promote sustainability.
UNICEF, the United Nations University and the University of California sponsored development
of a rapid anthropological approach to assessing the effectiveness of primary health care
programmes that resulted in a 1987 publication by Scrimshaw and Hurtado, Rapid Assessment
Procedures for Nutrition and Primary Health Care. It suggests techniques for interviewing,
observing, record keeping, and conducting focus group sessions and includes data collection
guides for the community, households and health care providers. At a workshop in Zagreb,
Yugoslavia in June 1988, UNICEF staff met with persons developing RAP in the health field to
refine it and increase its practical application to programming. A videotape introducing the
method is now available.

A major advantage of RAP is that it involves obtaining the outlook of beneficiaries. Additional
advantages are ease of supervision and understanding by managers. Representatives of

development agencies are usually better able to supervise evaluations based on RAP than ones
based on formal methods. Their understanding of the data and rationale behind the
recommendations helps them use RAP results with greater conviction than those of highly
technical, specialized studies. However, rapid methods are not best in every situation. They do
not necessarily produce representative data, so generalizations cannot always be drawn.
Case studies can be used to find out why and how changes did or did not occur. They are
particularly useful for assessing pilot projects or innovative activities. Cases are designed to
analyse in an in-depth way some aspect of a particular organization, community, business or
other unit and trace it over time. Case studies provide insights into causality and topics that are
difficult to quantify or involve complex processes, such as technology transfer. Case study
methods collect detailed, often descriptive, data on one group, and are good for making before/
after
comparisons.
They
usually
take
more time than other evaluations and allow the use of rigorous research methods, but may
result in voluminous reports that few people read. It is often necessary to conduct several
related case studies in order to obtain comparable data and identify patterns and differences
among them.
Sentinel sites were developed by WHO for surveillance of communicable diseases and first
used in the campaign to eradicate smallpox. Health workers at selected health facilities collect
and report information, for example on new cases of target diseases or on services provided,
such as immunization. The data reach the centre much more quickly than comprehensive
routine reporting. They are analysed to monitor such things as disease trends and programme
efficacy and impact. The system can be used to supplement routine reporting systems and
collect information on demographic trends. But care must be exercised in selecting sentinel
sites and in making generalizations based on data obtained from sentinel sites since they may
not be representative of the country's population as a whole. Health planners in Central America
adapted this methodology to gather data among people who do not go to health posts.

Surveys using "sentinel sites"
The concept of sentinel sites is used somewhat differently in this approach.
This sentinel site survey technique for monitoring and evaluation has been adapted in Central
America, where a large portion of the population does not have any contact with health care, to
monitor the situation of children and measure the impact, coverage and costs of primary health
care activities (Andersson, Ledogar). Sentinel sites, in this case, are population clusters
purposely selected to be representative of an area. The people in a stratified sample of
communities are surveyed periodically to obtain information, for example, to check the effects of
various service programmes and to compare risks under different conditions. Health personnel,
nurses and nurses assistants--and even traditional midwives--are trained to conduct house-tohouse surveys.
Health officials at the local and provincial levels learn to process and analyse this information
using portable computers and obtain rapid feedback. The statistical method of cross-tabulations
is used to show the effects of some variables (such as, immunization, nutrition status, the
presence of latrines and potable water, education of mothers) on others and on child mortality
and morbidity. Findings can be fed back to the community to motivate its members to participate
in health activities.

The methodology has been used for a number of purposes and in a range of settings in five
Central American countries and Mexico since 1984. Surveys may take only two weeks from
data collection through its analysis. Thus, this technique is able to measure changes in health
status periodically, rapidly and in a more cost-effective manner than traditional methods
Knowledge, Attitude and Practice (KAP) studies may be used for monitoring or evaluation, since
changes in knowledge, attitudes and practices of health providers and beneficiaries are often
programme objectives. The phrase, however, refers more to content ("what" is examined) than
to an evaluation method ("how" to find out information); these studies use a variety of the
methods described above. KAP studies are often used to measure project outcomes or
establish baseline data. The following features help to increase their usefulness: a close
partnership between government colleagues and other collaborators throughout; careful
pretesting of data collection instruments; thorough training and quality control during the
administration of surveys or interviews; and actionable recommendations.
Growth monitoring can screen individual children and monitor changing nutr to decisionmakers.
While project managers can use detailed, technical information about project implementation,
policy-makers want a short summary of the findings and recommendations. Thus, an executive
summary should always be prepared. A suggested outline of the evaluation report is contained
in Box IV-4.
In verbal autopsies a trained health worker visits the home after death of an infant is reported
and interviews the parents. The health worker records their responses to questions on a
standard form listing signs and symptoms prior to death in a way that can be recognized in the
culture. After reviewing this form, a doctor or health professional familiar with the locale can
assign a probable cause (or contributing causes) of death. These diagnoses of causes of death
based on postmortem interviews have been found to be sufficiently accurate for planning health
programmes and assessing their impact (Kalter, Gray and Black, nd). Surveillance for cause of
death is used to determine priorities among health activities and to assess the impact of
disease-specific interventions on child mortality.

References and Bibliography
Basic references
a. Feuerstein, Marie-Therese, Partners in Evaluation: Evaluating Development and Community
Programmes With Participants, London, Macmillan, 1986,194 pp. Excellent overview of
evaluation by the community. Extensive bibliography.
b. Pratt, Brian and Boyden, Jo, eds., The Field Director's Manual: An Oxfam Manual for
Development Workers, Oxford, UK, Oxford University Press, 1985. Manual with examples
from Oxfam. Good for those working with national or international NGOs/PVOs.
c. Rugh, Jim, Self-Evaluation: Ideas for Participatory Evaluation of Rural Community
Development Projects, Oklahoma City, Oklahoma, World Neighbors, 1986, 42 pp. How-to
guide for those working with community groups. Bibliography.

d. University of California at Los Angeles, Center for the Study of Evaluation, Program
Evaluation Kit, Beverly Hills, CA, Sage Publications, 1987. Nine paperback volumes on
different aspects of evaluation methodology. Good for those evaluating large, government
programmes or those experienced in evaluation.

Sector specific evaluation methods
a. Boot, Marieke, Training Course, Evaluating Water Supply and Sanitation Projects, Course
Modules, The Hague, The Netherlands, UNICEF/International Reference Centre, August
1987,104 pp.
b. Buzzard, Shirley and Edgcomb, Elaine, eds., Evaluating Small Business Projects: A Stepby-Step Guide for Private Development Organizations, New York, Private Agencies
Collaborating Together, 1987, 262 pp.
c. World Health Organization, Diarrhoeal Diseases Control Programme, training materials for
national health staff:
Evaluating Progress and Course Summary. Supervisory Skills, revised 1987, 59 pp.
Sections on monitoring use of health services and end-of-year evaluation.
Evaluation. CDD Programme Management--A Training Course, 1988, 29 pp. Training
module, one in a series.
Household Survey Manual: Diarrhoea Morbidity, Mortality and Treatment Practices, 1986.
Loose-leaf folder with sections on planning and conducting the survey and analysing the
results.
d. World Health Organization, Expanded Programme on-- Immunization, Evaluation and
Monitoring of National Immunization Programmes, 1986.

Further reading on evaluation methods
a. Andersson, Scarvia B. and Ball, Samuel, The Profession and -- Practice of Program
Evaluation, San Francisco, CA, Jossey Bass Publications, 1978. See section on
communicating evaluation -- plans and results.
b. Johnson, Allan G., Social Statistics Without Tears, New York, McGraw Hill, 1977, 270 pp.
College textbook on statistical analysis -- of quantitative data.
c. Kearl, Bryant, ed., Field Data Collection in the Social Sciences: Experiences in Africa and
the Middle East, New York, -- Agricultural Development Council, 1976, 200 pp. Excellent
suggestions for collecting field data.
d. Kumar, Krishna, Conducting Group Interviews in Developing Countries, A.I.D. Program
Design and Evaluation Methodology Report No. 8., Washington, D.C., U.S. A I.D., April
1987, 44 pp. Especially helpful for those conducting rapid assessments, suggestions on how
to be more systematic and professional while -- interviewing beneficiary groups.
Bibliography.

e. Norton, Maureen, and Pines Benoliel, Sharon, Guidelines for Data Collection, Monitoring,
and Evaluation Plans for A.I.D.-Assisted -- Projects, A I.D. Program Design and Evaluation
Methodology Report No. 9, Washington, D.C., U.S. A I.D., April 1987, 51 pp, -- excluding
annexes. Helpful suggestions for setting up data collection systems. Bibliography
f.

Patton, Michael Quinn, Utilization-Focused Evaluation, Beverly -- Hills, CA, Sage
Publications, 1978. Influential text, more advanced reading. Stresses the need to consult
with evaluation users -- throughout planning and implementing evaluations so that the
results will, in fact, be used.

g. Patton, Michael Quinn, Qualitative Evaluation Methods, Beverly Hills, CA, Sage
Publications, 1980. Presents rationale for use of qualitative data and suggestions for its
more rigorous and professional use.
h. U.S. A I.D., A.I.D. Evaluation Handbook, A I.D. Program Design and Evaluation
Methodology Report No. 7, Washington, D.C., U.S. A I.D., April 1987, 53 pp. Concise
overview of AlD's evaluation policies and procedures. Helpful for those working with
programmes funded by AID. Bibliography.
i.

U.S. A I.D. and Bureau of the Census, Selecting Data Collection Methods and Preparing
Contractor Scopes of Work, A I.D. Program Design and Evaluation Methodology Report No.
3, Washington, D.C., U.S.A I.D., August 1985, 46 pp. Suggests wording for Scopes of Work
for several data collection approaches.

j.

Needham Porter Novelli, A seminar in qualitative research, July 1986, prepared for the
Academy for Educational Development.

Further reading on rapid assessment methods
a. Cohen, Sylvie I., "Rapid Assessment Guide for Effective and Sustainable Social
Mobilization: An Application to the Goals of UCI by 1990," New York, New York, UNICEF,
Communication Section, Programme Division, 5 May 1989, pp. 85.
b. Chambers, Robert, "Shortcut Methods of Gathering Social Information for Rural
Development Projects," in Michael M. Cernea, ed. Puffing People First--sociological
Variables in Rural Development, New York, New York, Oxford University Press, 1985, pp.
399414. The guru of rapid appraisal.
c. Honadle, George, "Rapid Reconnaissance for Development Administration: Mapping and
Moulding Organizational Landscapes," World Development, Vol 10. No. 8., 1982, pp.
633649.
d. Kumar, Krishna, Rapid, Low-Cost Data Collection Methods for A.I.D., A I.D. Program Design
and Evaluation Methodology Report No. 10, Washington, D.C., U.S. A I.D., Dec. 1987, 34
pp. Basic information about five major types of rapid, low-cost methods: key informant
interviews, focus group interviews, community interviews, direct observation and informal
surveys.
e. Scrimshaw, Susan C. M., and Hurtado, Elena, Rapid Assessment Procedures for Nutrition
and Primary Health Care, Los Angeles Calif., UCIA, 1987, 70 pp. Bibliography. Described in
Box 7-1.

Further reading on impact evaluations
a. Gray, R.H., "Report of a Workshop on Health Impact Evaluation-- October 9-10, 1986,"
Institute for International Programs and The Johns Hopkins University School of Hygiene
and Public Health, Baltimore, Maryland.
b. Kumar, Krishna, Methodologies for Assessing the Impact of Agricultural and Rural
Development Projects: A Dialogue, A I.D. Program Design and Evaluation Methodology
Report No. 11, Washington, D.C., U.S. A I.D., June 1989, 48 pp. Bibliography. Helpful
discussion of indicators for measuring impacts.
c. White, Louise G., An Approach to Evaluating the Impact of A.I.D. Projects, A I.D. Program
Design and Evaluation Methodology -- Report No. 5, Washington, D.C., U.S.A I.D., March
1986, 55 pp. Bibliography.

Examples of UNICEF evaluations cited
a. Brenzel, Logan, The Cost of EPI: A Review of Cost and Cost-Effectiveness Studies (197987), Washington, DC, AID, April 1989.
b. Bosnjak, Vesna, et al., "Mozambique UNICEF Emergency Programme: Mid-Term
Evaluation," Maputo, Mozambique, June 1988.
c. Firme, Thereza Penna, et al., "A Avaliacao do Impacto de Programas Alternativos nos
Meninos de Rua," Brasilia, DF, Brazil, UNICEF, Setembro de 1987.
d. Landers, Cassie, "Mauritius: An Evaluation of Activities and Programmes for the Pre-School
Child: Summary of a UNICEF Consultancy, October 15-November 20, 1986," New York,
UNICEF. March 1987.
e. Relief & Rehabilitation Commission and Government of Ethiopia in cooperation with
UNICEF/Addis Ababa, Quick Assessment: Cash-For-Food Programme UNICEF/RRC
Emergency Intervention, Addis Ababa, Ethiopia, UNICEF, 820 February 1988.

Other references cited
a. Kalter, Henry D., Gray, Ronald H., and Black, Robert E., "Validation of Postmortem
Interview to Ascertain Selected Causes of Death in Children," Baltimore, MD, The Johns
Hopkins School of Hygiene and Public Health, nd.
b. Sohm, Earl D., "Glossary of Evaluation Terms," Geneva, Switzerland, Joint Inspection Unit
of United Nations, November 1978, 22 PP, excluding 14 pages of annexes.

Appendices
A. Glossary
B. Acronyms and Abbreviations
C. Indicators for Some Sectors of UNICEF Activity
D. Checklist for Evaluation Manager
Checklist for Evaluation Team
D. Field Trip Report
E. Goals for Children and Development in the 1990s
G. Annual report Form for Listing Evaluations and Studies

Appendix A: Glossary
Audit Determining whether, and to what extent, the measures, processes, directives, and
organizational procedures ...conform to norms and criteria set out in advance.
Monitoring A management function which uses a methodical collection of data to determine
whether the material and financial resources are sufficient, whether the people in charge have
the necessary technical and personal qualifications, whether activities conform to work plans,
and whether the work plan has been achieved and had produced the original objectives.

Types of evaluation
Appraisal ex-ante The critical examination of the identification report, which selects and ranks
the various solutions from the standpoints of: relevance; technical, financial and institutional
feasibility; socio-economic profitability. The appraisal stage immediately precedes that of the
approval, by the authorities, of the proposed action.
Evaluation An examination as systematic and objective as possible of an on going or completed
project or programme, its design, implementation and results, with the aim of determining its
efficiency, effectiveness, impact, sustainability and the relevance of the objectives. The purpose
of an evaluation is to guide decision-makers.
Mid-term An evaluation carried out during implementation.
evaluation Its principal goal is to draw conclusions for administering the project. Sometimes
referred to as "ongoing" project evaluations.
Ex-post evaluation Evaluation of an intervention after it has been completed. Its purpose is to
study how well the aid served its purposes, and to draw conclusions for similar interventions in
the future.

Information for evaluation
Input The set of means (resources and activities) mobilized to produce the output.
Outputs The result of the aid. The set of concrete results to be produced through sound
management of inputs. Intermediate results necessary to achieve the purpose. Goods, services
produced or directly controlled by the programme personnel.
Indicator A measure which is used to demonstrate the change or result of a programme.
Baseline study The analysis describing the situation prior to receiving aid, which is used to
determine the results and accomplishments of an activity, and which serves as an important
reference for the ex-post evaluation.
General evaluation criteria
Effectiveness A measure of the extent to which an aid programme attains its objectives.
Efficiency An economic term which means that the aid uses the least costly resources
necessary to achieve its objectives.
Impact A term indicating whether the project has had an effect on its surroundings in terms of
technical, economic, sociocultural, institutional and environmental factors.
Sustainability The extent to which the objectives of an aid activity will continue after the project
assistance is over; the extent to which the groups affected by the aid want to and can take
charge themselves to continue accomplishing its objectives.
Cost-effective-analysis An economic or social costbenefit analysis that quantifies benefits
without translating them into monetary terms. This type of analysis allows one to compare
alternative ways to accomplish one's objectives (and) select among alternative activities the one
that will accomplish the objectives at the least cost.
Source: Based on Organization for Economic Cooperation and Development (O.E.C.D.).
Development Assistance Committee, Expert Group on Aid Evaluation, Evaluation in Developing
Countries: A step in a dialogue. (Paris, France: O.E.C.D., 1988).
Programme In UNICEF, "programme" is used in two ways 1) "country support programme"
defined as the whole field in which UNICEF is cooperating in the country, e.g., the subject of a
programme recommendation to the Board; and 2) cooperation in a sector or a geographical
area, e.g., a "health programme" or an "area-based programme". Programmes are designed to
have a specified outcome(s) or impact, and are detailed in a Plan of Operations or a Master
Plan of Operations (MPO), if it consists of a number of programmes.
Project Frequently a programme consists of a set of projects, which in turn are made up of
activities. A project is usually related to one main implementing agency, therefore to one sector
(e.g. health) or part of a sector (e.g. EPI) or one field (e.g. women's activities). It can be defined
as "a planned undertaking composed of a group of interrelated activities whereby specified
inputs are designed with the purpose of achieving specified outputs (or changes) within a
specified time-frame". It is detailed in an annual plan of action. It is therefore convenient to have
a project for each important management unit in the government structure so that those who are
responsible for implementation know their responsibilities, and are committed to them. Projects

forming part of a programme should be interrelated wherever appropriate (EPI is often such a
case).
Source: UNICEF Policy and Procedure Manual, Book D Programme Operations. Chapter 3.
Section 1.

Appendix B: Acronyms and Abbreviations
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•

CDD Control of Diarrheal Disease
CHW Community Health Worker
CPR Country Programme Recommendation
CFF Cash for Food Programme
DPT Diptheria, Pertussis, Tetanus (vaccines)
ed education
EPI Expanded Programme of Immunization
FAO Food and Agriculture Organization (UN)
GFSS Global Field Support System
GNP Gross National Product
JNSP Joint WHO/UNICEF Nutrition Support Programme
KAP Knowledge, Attitudes, and Practices
MCH Maternal and Child Health
mgr - manager
mo - month
na - not available
NGO Non-Government Organization
ORS Oral Rehydration Solution, Oral Rehydration Salt
ORT Oral Rehydration Therapy
PlanOps Country Plan of Operations
PVO Private Voluntary Organization
RAP Rapid Assessment Procedures
SOW Scope of Work
TOR Terms of Reference
UCI Universal Child Immunization
UN United Nations
UNESCO United Nations Educational, Scientific and Cultural Organization
UNICEF United Nations Children's Fund
USAID U.S. Agency for International Development
WES Water and Environmental Sanitation (programme)
WHO World Health Organization (UN)

Appendix C: Indicators for Some Sectors of UNICEF Activity
1. Selected Indicators for Child Survival and Development in the 1990s
The UNICEF indicators for Child Survival are constantly undergoing refinement. The choice of
specific indicators will vary according to country goals and resources available for monitoring
and evaluation. The following maternal and child health indicators suggested by WHO can be
used as a point of reference:
a. Anthropometric Indicators Prevalence of Wasting (weight/height) and Stunting (height/age)
by age group .

b. Prevalence and Duration of Breastfeeding: Percentage at 3, 6, and 12 Months of Age
c. Perinatal Mortality Number of Late Fetal, Perinatal and Neonatal Deaths (in first 30 days
after birth)
d. Low Birthweight (under 2.5 kg), and Pre-Term Gestation
e. Maternal Mortality Rates per 100,000 Live Births
f.

Coverage of Maternity Care: Percentage of Women Receiving Prenatal, Trained Attendant
deliveries, and Post-Natal Care

g. Gender Differences in Mortality During Infancy and Childhood
h. Immunization Coverage and Disease Incidence
i.

Control of Diarrheal Diseases Estimated ORT and ORS Use Rates

j.

Environmental Health: Water and Sanitation Coverage

k. Mortality Statistics According to ICD Classification
In the following sections, frequently used indicators in UNICEF priority health programme areas
are suggested. Indicators for other activities are in the nine UNICEF Program Guidelines.
2. Indicators of Nutritional Status
Nutritional status may be measured by several indicators which reflect current malnutrition or
long-term growth and development. Problems may be associated with food availability, health
and infection status, availability of health services, and nutritional practices. Indicators of
nutritional status can be used to compare individuals and populations. The US National Center
for Health Statistics standards on nutritional status have been accepted by WHO. Nutritional
status is measured through anthropometry using height, weight, and age.
a. Weight-for-age. The most commonly used indicator on standard growth charts shows
nutritional status, and can be used to identify cases of nutritional deprivation. It reflects the
impact of acute infections as well as food consumption on nutritional status. It is especially
valuable as a measure of growth increments (growth faltering over time). The growth curve,
showing progression over time, is more significant than where particular isolated readings
are observed in relation to standard norms. It shows the direction of change in overall
nutritional status.
b. Height-for-age. This measure reflects overall social conditions and chronic nutritional
problems, but cannot be acted on except through weight-for-height. It is commonly reported
as the proportion of children having height for age less than two standard deviations below
the median WHO norms at specified ages within a given population. Therefore this
measure has been suggested as an indicator of overall social deprivation.
Stunting. A term used for deficits in height-for-age. It reflects a slowing of skeletal growth.

c. Weight-for-height. This measure reflects current health status of an individual. It can
demonstrate current malnutrition in a stunted child. Besides its value in project and

programme monitoring and evaluation, it can also be used to screen for undernourished
children in emergencies.
Wasting A term used for deficits in weight-for-height. It reflects loss of tissue and fat mass.
d. Arm circumference is sometimes used as a proxy for weight in children aged one to five
years. This measure is not sensitive to change, but simpler to use than weight.
e. Birth weight. In many countries the single most important predictor of mortality in babies is a
birth weight below 2500 grams. It is a good indicator of the general condition of mothers and
the level of social development of a population. Retarded fetal growth reflects the nutrition of
the pregnant mother and is measured best as smallness-for-gestational-age. Prematurity is
a second major cause of low birth weight and is influenced especially by maternal infections,
ill health or exposure to toxic agents.

In the 1980s, malnutrition in children and women has been exacerbated in Africa where many
countries have had to cut back on government services to meet requirements of foreign lenders.
Economic reforms and austerity programmes, components of structural adjustment, may thus
have reduced services to the poor without the economy improving for them. The decline in
nutritional status has created a special need to track the impact of these economic reforms on
children and women. The following indicators have been selected to measure the nutritional
status of women and children.
a. Birth weight is an indicator of health and nutrition of the mother and a predictor of the
chances that the baby will survive.
b. Weight-for-age, especially of children between 12-24 months, is a sensitive indicator of
trends and distribution of nutrition problems.
c. Height-for-age at five years, or school entry. This is recommended by WHO as an indicator
of the overall development of young children. It has not yet been widely applied.
In addition to these core indicators, other indicators reflect or predict future nutritional stress. In
Ghana, the indicators included prevalence of second and third degree malnutrition, cases of
kwashiorkor and marasmus, deaths from measles--which reflected malnutrition. The list of
indicators tracked also included area cropped and crop conditions, food prices in regional
markets, and rainfall--which could be used as a early warning system for future nutritional
problems.

3. ORT/CDD Indicators
The goal of Oral Rehydration Therapy (ORT) is to reduce diarrhoea deaths due to dehydration.
The goal of Control of Diarrhoeal Disease (CDD) is to improve case management and prevent
diarrhoea. The primary objective in promoting ORT is to change practices in the home and in
health facilities through education and training. WHO selected the following indicators to
determine the extent to which appropriate knowledge, attitudes, and practices have been
transmitted to health workers and parents. A comprehensive list of indicators is provided from
which items can be selected depending on local conditions.
Training or education of health personnel can be determined through a review of records or site
visits:

a. Training of supervisors- proportion of current health staff with supervisory responsibilities
who have been trained.
b. Health facility staff trained in diarrhea case management -proportion of current staff who
have been trained.
c. Community Health Workers trained in diarrhea case management -proportion of current
community health workers who have been trained.
Changes in the Knowledge, Attitudes, and Practices of parents are normally determined through
routine (government) reporting, surveys of health facilities, and household surveys.
a. ORS Access Rate - proportion of the population less than five years with reasonable
parental access to a trained and supplied provider of ORS packets.
b. ORT Use Rate - proportion of diarrhea episodes in children less than 5 years treated with
ORS and/or recommended home fluid in amounts to compensate for fluid loss.
c. ORS Use Rate - proportion of all cases of diarrhea in children less than five years treated
with ORS in amounts adequate to compensate for fluid loss.
d. Continued feeding rate - proportion of diarrhea episodes in children less than five years
during which children are given normal or increased amounts of food.
e. Proportion of households with correct knowledge of when to seek treatment outside the
home for a child with diarrhea.
f . Households able to correctly prepare a recommended home fluid.
g. Households able to correctly prepare ORS - proportion of mothers (or other household
members) who can demonstrate the correct preparation of ORS.
h. Cases correctly assessed - proportion of diarrhea cases among children less than five years
treated at health facilities who are correctly assessed for the Presence of dehydration or
other complications.
i.

Cases correctly rehydrated - proportion of diarrhea cases among children less than five
years treated at health facilities who are correctly rehydrated (orally or IV).

j.

Cases whose mothers were correctly advised on treatment at home - proportion of diarrhea
cases treated at health facilities whose mothers were given correct advice on treatment at
home including continued and compensatory feeding, ORT, and when to seek further
treatment.

k. Dysentery cases given appropriate antibiotics- proportion of dysentery cases among
children less than five years treated at health facilities who were given appropriate
antibiotics.
All these indicators have one common theme: proportion. This means that in each case the size
of the target or reference group must be known. For example, to determine the proportion of the
current community health workers (CHW) who have been trained (numerator), it is necessary to

know the total number of CHWs (denominator). The accuracy of these parameters is critical in
determining the value of the results.

4. Indicators for Immunization Programmes
Indicators for immunization programmes (Expanded Programme of Immunization, EPI) fall into
two broad categories: indicators of coverage and indicators of morbidity and mortality from the
diseases covered. The EPI programme currently includes tuberculosis, diptheria, pertussis,
tetanus, polio, and measles.
Indicators of EPI coverage for children are expressed as the percent of children under one year
who have been vaccinated with:
a. BCG
b. DPT 1st Dosage 2nd Dosage 3rd Dosage
c. Polio 1st Dosage 2nd Dosage 3rd Dosage
d. Measles
The indicator of tetanus immunization coverage for women is expressed as the percent of
women 15-45 years or of pregnant women who have received tetanus toxoid. International
attention to coverage should not obscure the fact that reducing morbidity and mortality is the
primary goal of immunization programmes.
To monitor immunization coverage, data concerning the number of doses of the various
vaccines administered by type of vaccine and by age can usually be obtained from the routine
reporting system. The birth rate is used to estimate the number of expected newborns per year,
per month or per quarter. Comparing the expected newborns with the number of first doses of
DPT administered gives an estimate of access to immunization services. Drop out rates are the
proportion of children who do not complete the vaccination series. Estimates of monthly,
quarterly, or yearly dropout rates can be calculated. These data can be used for policy
formulation, advocacy, and social mobilization.
Coverage data can also be obtained through sample surveys if the routine reporting system is
inadequate. In addition, reporting results should be validated by surveys. The WHO/EPI uses a
simple and inexpensive cluster sampling method to estimate coverage. It requires that 210
children from 12-23 months old (younger children may not have had sufficient time to complete
the immunization schedule) be selected, divided among 30 "clusters". This sampling method
has a 95% chance of being within 10 percentage points of the true value and is adequate for
most purposes. However, extreme care must be exercised in designing the survey and selecting
the clusters to assure representatives. In most countries technical advice for developing cluster
samples will be available from statisticians in the government, local universities, UN agencies,
or UNICEF. In general, the higher the coverage, the more refined the analysis can be (12
months, "full" immunization for a given age, etc.).
Because the EPI diseases have different levels of infectiousness, they require different levels of
immunity in the population before transmission will be prevented. For example, immunization
levels above 95% seem to be necessary to stop transmission of measles. Another important
factor is high population density which increases transmission rates and requires higher
coverage. Also, survey information may yield indicators of knowledge, attitudes and practices

which explain why mothers have not had children immunized. Such findings are of value in
designing education programmes.
A national survey presents a single aggregate figure but does not show variations by districts
and regions. Local or regional surveys with appropriate cluster samples may be necessary to
identify problem areas.
Reduced disease incidence is the ultimate measure of success in an immunization programme
and is measured through routine records, sentinel sites and household surveys. Disease
incidence rates and case fatality rates are used to estimate cases and deaths prevented from
immunizable diseases. The first requires reporting the number of cases for all ages; age-specific
incidence is desirable but not normally reported. The standard method of gathering data on
disease incidence is through routine reporting of the government health system. Data from
health centers are compiled to obtain district and then national totals. Disease incidence is often
difficult to measure because of under-reporting (of people not in contact with health centers). As
a result, in some countries reporting could concentrate first simply on reporting outbreaks. The
introduction of an immunization programme will reduce incidence and make reporting of cases
salient and feasible.
Sentinel reporting sites may simplify gathering data on disease incidence. Sites where data are
known to be complete and accurate are selected as part of a national surveillance system. Sites
should also be chosen to be representative. Household surveys may also provide indicators of
morbidity. However, these tend to be costly unless the information can be piggybacked on
another survey.

5. indicators for Water and Sanitation Programmes
Many variables exist for water and sanitation, but most evaluations examine their functioning,
use, and impact.
•

Determining whether facilities are functioning as intended can be easy or complicated.
Verifying functioning of simple facilities may entail a site visit and completing a checklist
of specifications (such as, rates of water flow, actual production of running pump, status
of holding tanks, taps, etc.) More complex facilities will require an appropriate level of
technical expertise.

•

A second issue for evaluating a water or sanitation facility regards its use: are the
facilities being utilized properly? Are the facilities being used or accessed in the manner
which was intended or predicted? In the wet season, when other sources are closer?
Differences between predicted and actual utilization may reveal oversights or flaws in
project design. For example, local cultural factors may have been ignored.

•

A third focus is impact: Are the optimum health, social, and economic impacts being
obtained?

The following are the most common indicators used to evaluate water supply projects:
(a) Water quantity. Project designs normally set targets in terms of location of the water in
relation to its users. For example, a communal stand post (20-40 liters per person per day),
a yard connection (40-80 l/c/d), and or house connection (5-150 l/c/d) may be constructed
depending on the local climatic conditions and availability. The evaluator should review: the

human and animal population, predicted levels of demand and wastage of water, actual
quantity of water produced daily, actual duration of operation of the facility daily/weekly, and
any pattern in the availability of water (seasonal variations in capacity and demand) and the
capacity of the facility (cubic meters/hour or day). Differences between the theoretical and
actual demand could indicate a need to redesign parts of the project and should be
confirmed by a water use study of a sample of households. If production is close to capacity,
the reports might include steps which should be taken to increase capacity, avoid waste, or
control access. On the other hand low actual demand might suggest a need to promote the
project and attract users, change user rates (if applicable), or improve the distribution
system.
(b) Water quality depends on the source, its protection (e.g. from animals), the way the water is
drawn from wells, habits of transport and storage at home. It is assessed in terms of its
microbiological, chemical, and physical properties. These need to be assessed at the water
source, and at the point of consumption. These characteristics are normally determined
through sanitary surveys and water analysis. An evaluation team should have results of
quality control studies if these are not to be carried out as part of the evaluation. The team
should include a specialist who can interpret raw data, such as absence of coliforms,
discoloration, odors or tastes. Recommended corrective actions may include chlorination,
sand filters, removal of nitrates and/or excess fluoride or education campaigns to change
behavior (hygiene) or to promote new standards.
(c) Reliability of the water supply. Usually several factors combine to undermine the reliability of
a water system: poor maintenance, poor design, and poor construction. The evaluation team
should obtain operational records to detect stoppages, their duration, frequency,
seasonality, etc. Proposed remedial actions could include promoting community
responsibility, clearer definition and assignment of maintenance tasks, identification of
reliable sources for funding for maintenance, adoption of technology which is more suited to
the social and economic conditions of the users, training of operators, or redesigning and
reconstructing parts of the facility.
(d) Convenience of water points. Assessing the distribution system includes measuring the
distance between water points and households. In interpreting the significance of the
findings, distance must be qualified by considerations such as the presence of traditional
sources of water, terrain to water points, and availability of draft animals to transport the
water.

The following are the most common indicators used to evaluate sanitation projects:
(a) Proportion of household's that have improved latrines. The evaluation team should establish
the number of households who have access to improved latrines through a house-to-house
survey. Evaluators should try to determine the reasons for decisions not to install latrines, in
order to learn about flaws in the design, the need for promotion or education, or cultural
constraints
(b) Sanitation hygiene. The aim of this measure is to determine the extent to which latrines are
designed to curtail insect breeding, odor and fouling. The evaluation team must inspect the
latrines and render judgments. The team should inspect a few facilities together to establish
a common range (good-acceptable-bad-very bad). Such assessments should reflect what

the community thinks of the "new" facilities as compared to what may have existed
traditionally.
(c) Sanitation reliability. This indicator refers to whether specifications have been implemented
faithfully regarding design, construction, and maintenance. Depending on the technology
involved, the evaluators must determine which aspect of the design, maintenance, or
construction will be looked at during an inspection.

Appendix D: Checklist of Evaluation Manager
Planning
•
Discuss evaluation purposes with others
•
Decide on evaluation questions
•
Select evaluation methods based on questions
•
Prepare Terms of Reference
•
Recruit members of evaluation team
•
Select members of evaluation team
•
Identify existing materials for team
Implementation (while the team conducts the evaluation)
Provide orientation to members of evaluation team
•
Supervise and provide ongoing support
•
Attend meeting for preliminary discussion of findings and recommendations
•
Review draft report and give comments to team
•
Schedule debriefing session with team and sponsors
•
Meet with team to evaluate the evaluation
•

Follow-Up
Send copies of report to interested parties
•
Meet with project managers to discuss implementation
•
Meet with others to discuss implementation and follow up
•
Use results in future programme planning
•

Checklist for Evaluation Team
Planning
•
Review Terms of Reference with manager
•
Meet with evaluation sponsors
•
Refine evaluation design
•
Contact local and regional authorities
•
Conduct preliminary investigation
•
--review project documents
•
--review other existing material
•
--meet with project managers and beneficiaries
•
Select sample and sites
•
Prepare draft data collection instruments
•
Pre-test data collection instruments
Implementation
If required, recruit, hire, and train survey assistants
•
Collect qualitative and quantitative data: administrative records, interviews, surveys,
observation, etc.
•

•
•
•
•

Analyse data
Draft report
Review preliminary findings and recommendations with sponsors and/or implementers
Revise report based on review comments

Follow-Up
Debrief sponsors and others

•

Appendix E: Field Trip Report
Field trip Report
Date(s) of Visit ________________
Name of visitor ________________ Date Of Report __________________
Cost (If Applicable) ______________
1. Program/Project/Activity visited:
2. Objective of Visit:
3. Location(s) visited:
4. Activities Undertaken:
5. Persons Interviewed:
6. Overall, Implementation Has Been:
Implementation Is / Is Not On Schedule (Circle One)
7. List Significant Achievements Expected / Unexpected:
8. List Problems and/or Principal Constraints:
9. Followup Action Recommended:
A. UNICEF (specify Staff Member/Section):
B. Government :
C. Others:
10. Additional Comments:
11. Name, Comments, Signature of Supervisor:

Appendix F: Goals for Children and Development in the 1990s
The U.N. Convention on the Rights of the Child embodies the most comprehensive listing of
goals for the well-being of children. Full implementation of the Convention is the ultimate
objective of programmes for children and development. The following goals, derived through a
process of extensive international consultation, are essential means to achieve that ultimate
objective.
I. MAJOR GOALS FOR CHILD SURVIVAL, DEVELOPMENT AND PROTECTION

(a) Between 1990 and the year 2000, reduction of infant and under-5 child mortality rate in all
countries by one-third or to 50 and 70 per 1000 live births respectively. whichever is less.
(b) Between 1990 and the year 2000, reduction of maternal mortality rate by half.
(c) Between 1990 and the year 2000, reduction of severe and moderate malnutrition among
under-5 children by half.
(d) Universal access to safe drinking water and to sanitary means of excreta disposal.
(e) By the year 2000, universal access to basic education and completion of primary education
by at least 80% of primary school age children.
(f) Reduction of the adult illiteracy rate (the appropriate age group to be determined in each
country) to at least half its 1990 level with emphasis on female literacy.
(g) Improved protection of children in especially difficult circumstances.
II. SUPPORTING/SECTORAL GOALS
A. Women's Health and Education
(h) Special attention to the health and nutrition of the female child, and pregnant and lactating
women.
(ii) Access by all couples to information and services to prevent pregnancies which are too
early, too closely spaced, too late or too many.
(iii) Access by all pregnant women to prenatal care, trained attendants during child birth and
referral facilities for high risk pregnancies and obstetric emergencies.
(iv) Universal access to primary education with special emphasis for girls, and accelerated
literacy programmes for women.
B. Nutrition
(i) Reduction in severe as well as moderate malnutrition among under-5 children by half of
1990 levels.
(ii) Reduction of the rate of low birth weight (2.5 kg or less) to less than 10 per cent.
(iii) Reduction of iron deficiency anaemia in women by one-third of 1990 levels.
(iv) Virtual elimination of iodine deficiency disorders.
(v) Virtual elimination of vitamin A deficiency and its consequences, including blindness.
(vi) Empowerment of all women to exclusively breast-feed their child for four to six months and
to continue breast-feeding with complementary food well into the second year.

(vii) Growth promotion and its regular monitoring to be institutionalised in all countries by the
end of the 1990s.
(viii) Dissemination of knowledge and supporting services to increase food production to ensure
household food security.
C. Child Health
(j) Global eradication of poliomyelitis by the year 2000.
(ii) Elimination of neonatal tetanus by 1995.
(iii) Reduction by 95 per cent in measles deaths and reduction by 90 per cent of measles cases
compared to pre-immunization levels by 1995, as a major step to the global eradication of
measles in the longer run.
(iv) Maintenance of a high level of immunization coverage (at least 90 per cent of children under
one year of age by the year 2000) against diphtheria, pertussis, tetanus, measles, poliomyelitis,
tuberculosis and against tetanus for women of child bearing age.
(v) Reduction by 50 per cent in the deaths due to diarrhoea in children under the age of five
years; and 25 per cent reduction in the diarrhoea incidence rate.
(vi) Reduction by one-third in the deaths due to acute respiratory infections in children under five
years.
D. Water and Sanitation
(i)

Universal access to safe drinking water.

(ii)

Universal access to sanitary means of excreta disposal.

(iii)

Elimination of guinea-worm disease (dracunculiasis) by the year 2000.

E. Basic Education
(i)
Expansion of early childhood development activities including appropriate lowcost
family- and community-based interventions.
(ii)

Universal access to basic education, and achievement of primary education by at least
80 percent of primary school age children through formal schooling or non-formal
education of comparable learning standard, with emphasis on reducing the current
disparities between boys and girls.

(iii)

Reduction of the adult illiteracy rate (the appropriate age group to be determined in each
country) to at least half its 1990 level, with emphasis on female literacy.

(iv)

Increased acquisition by individuals and families of the knowledge, skills and values
required for better living, made available through all educational channels, including the
mass media, other forms of modern and traditional communication, and social action,
with effectiveness measured in terms of behavioural change.

F. Children in Difficult Circumstances

Provide improved protection of children in especially difficult circumstances and tackle the root
causes leading to such situations.

Unicef guide 4 monev

More Related Content

What's hot (20)

Similar to Unicef guide 4 monev (20)

More from budhi mp (20)

Recently uploaded (20)

Unicef guide 4 monev