Program Evaluation 1

By: Sourav Goswami
Moderator:
Dr Abhishek Raut
Dept Of Community
Medicine, MGIMS,

Framework
Evaluation – Basic Concepts
Steps for program evaluation
Framework for Program Evaluation
 Systems framework
 Logical framework
Types of Evaluation
A Case Study
References

“ One of the great mistakes
is to judge policies and
programs by their intentions
rather than their results ”
- Milton Friedman
(1912-2000),
Economist

What is Program Evaluation?
It is the process of determination of merit, worth,
significance of a program” Michael Scriven
“A systematic method for collecting, analyzing
and using information to answer questions about
projects, policies and programs, particularly
about their effectiveness and efficiency”
# term “program” = “any set of organized activities
supported by a set of resources to achieve a specific
and intended result.”

Research VS Evaluation
1. Creating evidence VS Assessing the effectiveness of
evidence
2. Researcher derived questions VS Program
derived questions
3. More controlled setting VS Action setting
4. Published VS Often Not Published

“ Research seeks to prove,
evaluation seeks to
improve”
- Michael Quinn
Patton (Born on 1945),
Program Evaluation Consultant,
Former President of the
American Evaluation Association

MONITORING EVALUATION
TIMING Continuous throughout
The project
Periodic review at significant
point of project progress – end of
project, mid point of project,
change of phase
SCOPE Day to day activities,
indicators of process,
progress, sometimes
outputs
Assess overall delivery of outputs
& progress towards objectives &
goals
MAIN
PARTICIPANT
S
Project staff, project
users
External evaluators/ facilitators,
project users, project staff,
donors
PROCESS Regular meetings,
interviews, monthly,
quarterly reviews etc
Extraordinary meetings,
additional data collection
exercises etc
WRITTEN
OUTPUTS
Regular reports &
updates to project
users, management &
donors
Written reports with
recommendations for change to
project – presented in workshops
to different stakeholdersSurveillance - tracks disease or risk behaviors

Why to Evaluate Programs?
To gain insight about a program and its operations
– to see where we are going and where we are coming from,
and to find out what works and what doesn‘t
To improve practice – to modify or adapt practice to
enhance the success of activities
To assess effects – to see how well we are meeting
objectives and goals, how the program benefits the
community, and to provide evidence of effectiveness
To build capacity – increase funding, enhance skills,
strengthen accountability

The Program Management
PLAN
EVALUATE IMPLEMENTATION

Steps for doing the evaluation
1. Engage Stakeholders
2. Describe the program
3. Focus the evaluation design
4. Gather Credible Evidence
5. Justify Conclusions
6. Ensure use and share lessons learned

CDC Framework for Program
evaluation

Understanding the STANDARDS
1. Utility
2. Feasibility
3. Propriety
4. Accuracy
Utility
•Ensures that the information
needs of intended users are met.
# Who needs the evaluation findings?
# What do the users of the evaluation need?
# Will the evaluation provide relevant
(useful) information in a timely manner?
Feasibility
•Ensures that evaluation is
realistic, prudent, diplomatic,
and frugal.
# Are the planned evaluation activities
realistic given the time, resources, and
expertise at hand?
Propriety
Ensures the evaluation is conducted
legally, ethically, & with due regard for
the welfare of those involved
and those affected.
# Does the evaluation protect the rights of
Individuals & protect the welfare of those
involved?
# Does it engage those most directly affected by
the program & by changes in the program,
such as participants or the surrounding
community?
Accuracy
Ensures that the evaluation revealsEnsures that the evaluation reveals
and conveys technically accurateand conveys technically accurate
information.information.
# Will the evaluation produce findings that# Will the evaluation produce findings that
are valid and reliable, given the needs ofare valid and reliable, given the needs of
those who will use the results?those who will use the results?

System framework
Inputs Process Outcomes
Output Effect Impact
Targets
(immediate results
in numbers)
Objectives
(Trend)
Goals
(Planned
Impacts)

When to conduct a Prog Evaluation?
Planning
a NEW
Program
Assessing a
DEVELOPING
Program
Assessing
a STABLE,
MATURE
Program
Assessing a
Program after
It has ENDED
 > >>Start End
Stages of program Development
Influences the reason for program evaluation

Managing through the
project life cycle

Logical Framework Approach (LFA)
Impac
t
Impac
tInput Process Output Outcome> >>>
Planning Logic
Implementation Logic
A log frame is a tool for improving the planning,
implementation, management, monitoring and
evaluation of projects.
The log frame is a way of structuring the main elements
in a project and highlighting the logical linkages
between them.

Log Frame – Filling the matrix

(e.g. - Measles immunization)
SUMMARY INDICATORS EVIDENCE / SOURCE
OF VERIFICATION
ASSUMPTIONS
GOAL Decrease incidence
and mortality from
measles
OBJECTIVE Increased coverage of
measles immunization
(MV)
OUTPUTS (1) Increased
awareness about MV
(2) Increased number
of vaccination days
(3) Increased
availability of MV at
health centers

SUMMARY INDICATORS EVIDENCE ASSUMPTIONS
ACTIVITIES 1.1 Train CHW
about importance of
MV
CHW apply the training
1.2 Conduct IEC
campaign for MV
Community responds to
IEC
2.1 Recruit CHWs CHWs come regularly to
work
Incentives adequate for
accepting job
2.2 Inform
community about
additional
vaccination day
Community responds to
additional day
3.1 Establish
functional cold chain
Cold chain is maintained
3.2 Train health
center staff in better
logistic mgmt.
Health workers apply the
training
AND

SUMMARY INDICATORS EVIDENCE ASSUMPTIONS
GOAL Decrease incidence and
mortality from measles
PURPOSE Increased coverage of
measles immunization
(MV)
MV is effective
OUTPUTS (1) Increased awareness
about MV
(2) Increased number of
vaccination days
(3) Increased
availability of MV at
health centers
MV is culturally
acceptable
Additional vaccination
days made use of by
communities
Additional vaccines
used
AND
THEN

Types of Evaluation
Process (Formative) Evaluation
Outcome ( Summative ) Evaluation
Impact Evaluation
Economic Evaluation

Formative Evaluation
It occurs while a training program is forming or occurring
Its main purpose is to find out whether the program is
needed, and if so, whether it realistically can be improved.
Its key feature is that it is designed to bring about
improvement of the course, curriculum or educational
package while it is still possible to do so, i.e while the
material/program has not yet been put into its operational
form
This type of evaluation often is helpful for pilot projects
and new programs, but can be used for progress
monitoring of ongoing programs too.

Summative Evaluation
 Here, programs or projects are assessed at the end of an
operating cycle.
 Findings typically are used to help decide whether a program
should be adopted, continued, or modified for improvement.
 The object of such evaluation is to determine whether it meets
its design criteria, ie whether it does the job for which it was
designed.
 It may also be carried out in order to compare one program,
course, curriculum etc with another.
 It also evaluates the attitudes and information learned after the
program has been conducted, or determining how the
information provided is used back .

Impact Evaluation
“ Impact” is sometimes used synonymous to “
outcome”
“Impact” is perhaps better defined as longer
term and deeper outcomes.
It also tells us, what degree program/service
outcomes are as a result of the program and to
what degree other factors influenced the
outcome.

How do we plan an Impact Evaluation?
 Lets say, we like to evaluate a program that reduces infant
mortality in Melghat area of Maharashtra
 Now, it calls for 5 important questions that need to be
answered before we begin IE. Those are:
1.Is IE the best way to answer the question about the
program?
2.What are the key evaluation questions?
3.What is the “ Theory of Change” ?
4.How to procure and build the “evaluation team”?
5.Who will be involved in making decisions and how
will the evaluation process be managed?

Economic Evaluation of a Program
It may also be necessary to conduct an economic
evaluation to demonstrate “value for money” and
possible cost savings for government by investing in
prevention.
Economic evaluation answers the following question:
 Does spending Rs “ x ” on programme A represent a
better investment than spending Rs “ y ” on programme
B?
Economic evaluation is based on the comparison of
alternatives in terms of their “costs” and
“consequences”
 #“consequences” = an outcome of value

Methods used in Economic Evaluation
of a Program
There are various forms of economic evaluation
that can be conducted
A common element across all forms of economic
evaluation is that they involve measuring “costs”
The important methods of doing an economic
evaluation are:
1.Cost Effectiveness analysis (CEA)
2.Cost-Utility analysis (CUA)
3.Cost-Benefit analysis (CBA)

Measurement of costs and consequences in
economic evaluation

What all can we Evaluate?
Direct service interventions
Community mobilization
Research initiatives
Surveillance systems
Outbreak investigations
Laboratory diagnostics
Training and educational services etc.

Qualitative Methods in
Program Evaluation
The methods used for each type of evaluation
will vary.
Both qualitative and quantitative methods can
be used within the design of an evaluation.
Qualitative methods may be employed for
formative and process evaluations, e.g. focus
groups, short- answer or open-ended
questionnaires, in-depth interviews,
participatory methods, verbal autopsies etc.

A Case Study
Community Led Initiative In Child Survival
(CLICS) Project
The CLICS Project was implemented in the
eastern Maharashtra, district of Wardha and
reached a population of approximately 90,000
in 67 villages.
Goal: To bring about sustainable improvement
in the health status and well being of the
children under the age of three ; and women
of reproductive age (15-44 years)

A Case Study
(CLICS) ProjectObjectives:
1.Provide affordable, high quality health care
through effective partnerships at the village level;
2.Build the capacity of coalitions of local partners to
sustain child survival activities and health gains;
3.Refine and test a social franchising model for the
delivery of child survival interventions; and
4.Document, disseminate and share key program
lessons and results to facilitate adaptation,
replication and policy advocacy.

A Case Study: CLICS Project
 file://localhost/Users/souravgoswami/Desktop/29.06.17/Screen Shot 2017-06-29 at 9.31.13

A Case Study
(CLICS) ProjectLog Frame Analysis of CLICS Project
file://localhost/Users/souravgoswami/Desktop/29.06.

To sum up..
Evaluation should be seen as an integral
component of any program.
An evaluation plan needs to be determined at
the beginning of a program development, so
that a plan for data collection for this purpose
is built into project implementation.
Information on the effectiveness of a program,
will help identify if there are any problems in
running a program.

To sum up.... continues
The aims of the program must be reflected in
the evaluation plan and the performance
indicators selected. This will help to decide how
best to carry out the evaluation.
Depending on the program aim and availability
of resources, method of evaluation should be
decided.
Last but not the least, the results of the
evaluation should be shared with the
appropriate parties, and the results should be
used in the planning of future programs.

References
 Centers for Disease Control and Prevention. Introduction to
program evaluation for public health programs: A self-study
guide. Atlanta, GA: Oct 2011.
 Gertler PJ, Martinez S, Premand P, Rawlings LB, Vermeersch
CMJ. Impact Evaluation in Practice. The World Bank.
Washington DC; 2011.
 Issel LM. Health Program Planning and Evaluation: A practical,
systematic approach for community Health. John & Bartlett
Learning. Burlington MA; 2014.
 Drummond MF, Sculpher MJ, Claxton K, Stoddart GL,
Torrance GW. Methods for the Economic Evaluation of Health
Care Programmes.4th
ed. Oxford University Press. United
Kingdom;2015

References contd
 Khandker SR, Koolwal GB, Samad HA. Handbook on impact
evaluation: Quantitative methods and practices. The World
bank. Washington D.C. 2010
 World Health Organization. WHO Evaluation Practice
Handbook. Switzerland. 2013.
 USAID. Technical Note. The logical framework. Number 2:
Version 1.0; Dec 2012.
 The World Bank. Monitoring & Evaluation: Some tools,
methods & approaches. Washington D.C.2004
 Bhalwar R, Vaidya R, Tilak R, Gupta R, Kunte R. Text Book Of
Public Health and Community Medicine. Armed Force Medical
College. Pune.
 Kishore J. National Health Programs of India: national polices
and programs related to health. Century Publications. New
Delhi. 2017.Chapter 30: 560-64.

Program Evaluation 1

More Related Content

What's hot (20)

Similar to Program Evaluation 1 (20)

More from sourav goswami (15)

Recently uploaded (20)

Program Evaluation 1

Editor's Notes