Monitoring and evaluation for hiv
Download as PPT, PDF1 like1,037 views
This document provides an overview of monitoring and evaluation for HIV/AIDS programs. It discusses key concepts like monitoring, evaluation, and surveillance. It also describes the different levels of measurement from inputs and processes to outcomes and impacts. The document provides examples of core indicators that can be used to monitor HIV/AIDS programs and discusses best practices for developing a strong monitoring and evaluation system, including having clear goals and objectives, a set of indicators, a data collection and analysis plan, and a dissemination plan.
Monitoring and evaluation for hiv
- 1. MONITORING AND EVALUATION FOR HIV/AIDS PROGRAMS PDMES SEPTEMBER 2014
- 2. Monitoring AND Evaluation Monitoring: What are we doing? Tracking inputs and outputs to assess whether program are performing according to plans (e.g. people trained, condoms distributed) Evaluation: What have we achieved? Assessment of impact of the programme on behaviour or health outcome (e.g. condom use at last risky sex, HIV incidence) Surveillance: monitoring disease Spread of HIV/STD (e.g. HIV prevalence among pregnant women)
- 3. Program Components • Program inputs refer to the set of resources (i.e., financial, policies, personnel, facilities, space, equipment and supplies) that are the basic materials of the program. • Program processes refer to the set of activities in which program inputs are utilized to achieve the results expected from the program. • Program outputs are the results obtained at the program level through the execution of program activities using program resources. • These may be divided into the following three components: functional outputs, service outputs and service utilization.
- 4. Program outputs • Functional outputs are the direct result of program activities in six key functional areas: policy, training, management, commodities and logistics, research and evaluation, and information, education, and communication (IEC). Examples of • Functional outputs include the number of people trained in the last year, number of IEC messages aired on the radio over the last quarter, and existence of a management information system. • Service outputs are the results of program activities aimed at improving the service delivery system. These are measured in terms of quality, accessibility of services, and program image and acceptability. • Service utilization is the result of making services more accessible and satisfactory to potential clients. This result is generally measured at the program level.
- 5. Program outcomes and impacts • Program outcomes and impacts are the set of intermediate and longer term results expected to occur at the population level due to program activities and the generation of program outputs. • Program outcomes are the intermediate results at the population level that are closely linked to program activities and program-level results. These intermediate results, or outcomes, are generally achieved in two to five years. • Program impacts are the results at the population level that are long term in nature and are produced only through the action of intermediate
- 6. Levels of Measurement • Inputs, process, and outputs relate to activities and results at the program level and are usually measured with program-based or facility-based data. • Program-based data come from routine data collection (e.g., service statistics, client and other clinic records, administrative records, commodities shipments, sales) as well as information that is collected on site whether services are delivered (e.g., provider surveys, observation of provider-client interaction, retail audits, mystery clients) or from a follow-
- 7. Program Outcomes • Outcomes are usually measured with population- based biological and behavioral data. • Population-based data refer to information obtained from a probability sample of the target population in the catchment area for the program. • This may be a country, a region, or a particular subgroup of the population. • The data are generally collected from surveys, such as the Demographic and Health Survey, Behavioral Surveillance Survey or the Young Adult Reproductive Health Survey. • Biological-based data are generally collected through sentinel surveillance systems.
- 8. SELECTING AND USING INDICATORS • Good indicators for the monitoring and evaluation of HIV/AIDS/STI programs should be clear about the purpose they are to serve. Once this is established, efforts should be made to ensure that the indicator is well defined, feasible to collect, easy to interpret, and able to track changes over time. Selecting Indicators Features of a good indicator, more specifically, • should actually measure the phenomenon it is intended to measure (valid), • produce the same results when used more than once to measure precisely the same phenomenon (reliable)
- 9. Cont’ • measure only the phenomenon it is intended to measure (specific), • reflect changes in the state of the phenomenon under study (sensitive), and • be measurable or quantifiable with developed and tested definitions and reference standards (operational). • Most importantly, an indicator should be relevant. If one cannot make decisions based on an indicator or group of indicators, there is no point in collecting the information.
- 10. Using Indicators Criteria to consider in choosing among performance indicators at the program level: •Is the indicator oriented toward the targeted results (objective) and is it at the appropriate level? •It is important to include at least one indicator relating to the desired results, appropriate to the scale of the intervention. • How easy is it to obtain the information, how often is the information updated, and what are the sources of the information?
- 11. • What is the quality of the data? Effort should be given to design or select indicators of high priority which involve minimal difficulty in measurement. Naturally, priority should be given to indicators based on measures of known quality (i.e., strong reliability and validity).
- 12. • How comparable are the results from the indicator? Because of the need to monitor the performance of health interventions across a number of programs simultaneously and given the new evaluation methods for HIV/AIDS/STI programs, priority should be given to those indicators that offer comparable results
- 13. • How responsive to change is the indicator? An indicator should change in response to program interventions. Indicators that are responsive to underlying intervention efforts in a short period of time (3-5 years) are to be preferred over, but should not displace, those requiring a longer lag time (e.g., HIV prevalence).
- 14. Responsiveness also depends on sample size, confidence intervals, and normal variation over time. This last factor, together with the expected change due to the intervention, should determine the frequency of data collection. For example, if an indicator is only expected to change over a five-year period, it does not make sense to measure it every year. It is necessary to first obtain a baseline value on the indicator so that subsequent values can be compared to determine if change or improvement has occurred.
- 15. Input Process Output Outcome Impact A FRAMEWORK for Monitoring and Evaluation People money equipment policies etc. Training Logistics Management IEC/BCC etc. Services Service use Knowledge Behaviour; Safer practices (population level) HIV/STI transmission Reduced HIV impact
- 16. Input Process Output Outcome Impact DATA COLLECTION for Monitoring and Evaluation HIV/STI surveillance Household Surveys Facility surveys Programme Monitoring
- 17. Input Process Output Outcome Impact Did the National Response Make the Difference? 1 HIV prevalence changing!2 Can the changes in HIV prevalence be attributed to changes in behaviour?? 3 Can the changes in behaviour be attributed to interventions / programs?
- 18. Input Process Output Outcome Impact The components of AIDS programmes Voluntary counselling and testing Reduction of mother-to-child transmission IEC programs: knowledge, attitudes Condom promotion and distribution School programs: adolescent KAP Targeted interventions Control of STDs Blood safety, prevention nosocomial transmission Care & support programs (including ARV)
- 19. Lesson Learned: 5 Elements of a Good Monitoring and Evaluation System 1 Presence Monitoring and Evaluation unit 2 Clear goals and objectives of the program 3 A core set of indicators and targets 4 A plan for data collection and analysis 5 A plan for data dissemination
- 20. Clear goals and objectives • National strategic plan has no specific goals and objectives • No system of ongoing assessment with programs reviews and built-in evaluation • Limited coordination with districts and regions • Limited coordination between sectors • Donor-driven M&E system • Well-defined national programme goals and targets - M&E plan • Regular reviews/evaluations of the progress of the implementation of the national programme plans • Guidelines and guidance to districts and regions or provinces for M&E • Guidelines for linking M&E to multiple sectors • Co-ordination of national and donor M&E needs Not so good GOOD
- 21. A set of indicators (and targets) • No indicators or indicators that cannot be measured • Indicators that cannot be compared with past indicators or with other countries • Indicators are only used for donors and each donor has its own set of indicators • Indicators are irrelevant to those who collect the data • Each district or sector uses its own indicator • A set of priority indicators and additional indicators that cover programme monitoring, programme outcomes and impact - M&E plan • Selection of indicators through process of involving multiple stakeholders and maintaining relevance and comparability • Utilization of past and existing data collection efforts to assess national trends (e.g. DHS) Not so good GOOD
- 22. Data collection and analysis plan • M&E is an ad hoc activity without a plan, mostly driven by donors • Data are collected but not analysed sufficiently / utilized • There is no systematic monitoring of programme inputs and outputs • An overall national level data collection and analysis plan, linked to the national strategic plan • • A plan to collect data and analyse indicators at different levels of M&E (programme monitoring) • Second generation surveillance, where behavioural data are linked to HIV/STI surveillance data Not so good GOOD
- 23. Data dissemination plan • Dissemination is ad hoc and not planned or coordinated • Annual surveillance report is much delayed not user friendly and not well disseminated • Dissemination to the districts and regions is not done • Dissemination activities are donor driven • Overall national level data dissemination plan • Well-disseminated informative annual report of the M&E unit • Annual meetings to disseminate and discuss M&E and research findings with policy-makers and planners • Clearinghouse / Resource centre at national level Not so good GOOD
- 24. Overview of Indicators Bi-annualBi-annualProgrammeProgramme reportsreports /modelling/modelling CS3CS3 The number and percent of persons with advanced HIVThe number and percent of persons with advanced HIV infection receiving ART (UNGASS)infection receiving ART (UNGASS) AnnualAnnualProgrammeProgramme reportsreports CS2CS2 The percent of districts with at least one health facilityThe percent of districts with at least one health facility providing ARTproviding ART AnnualAnnualProgrammeProgramme reportsreports CS1CS1 The number of individuals receiving HIV testing andThe number of individuals receiving HIV testing and counselling in the last 12 monthscounselling in the last 12 months a)a) The number of individuals who received HIV testingThe number of individuals who received HIV testing b)b) Percent of those tested who received pre-test counsellingPercent of those tested who received pre-test counselling c)c) Percent of those tested who were positivePercent of those tested who were positive d)d) Percent of those tested who received their resultsPercent of those tested who received their results through post-test counselling servicesthrough post-test counselling services FrequencyFrequencyMethodsMethodsCore IndicatorsCore Indicators
- 25. Overview of Indicators, con’t... Every otherEvery other yearyear Interviews /Interviews / recordrecord reviewreview CS4CS4 The existence of comprehensive HIV/AIDS care andThe existence of comprehensive HIV/AIDS care and support policies, strategies and guidelinessupport policies, strategies and guidelines Every 2-4 yearsEvery 2-4 yearsHealthHealth facilityfacility surveysurvey CS7CS7 Percent of health facilities that have the capacity andPercent of health facilities that have the capacity and conditions to provide advanced level HIV care andconditions to provide advanced level HIV care and support services, including provision and monitoring ofsupport services, including provision and monitoring of ARTART Every 2-4 yearsEvery 2-4 yearsHealthHealth facilityfacility surveysurvey CS6CS6 Percent of health facilities that have the capacity andPercent of health facilities that have the capacity and conditions to provide basic level HIV testing and HIV/AIDSconditions to provide basic level HIV testing and HIV/AIDS clinical managementclinical management Every 2-4 yearsEvery 2-4 yearsHealthHealth facilityfacility surveysurvey CS5CS5 The percent of facilities that either provideThe percent of facilities that either provide comprehensive care and support services onsite forcomprehensive care and support services onsite for people living with HIV or through an effective referralpeople living with HIV or through an effective referral systemsystem FrequencyFrequencyMethodsMethodsCore IndicatorsCore Indicators
- 26. Overview of Indicators, con’t... To beTo be determineddetermined HealthHealth facilityfacility survey /survey / special labspecial lab studystudy CS8CS8 The percent of designated laboratories with theThe percent of designated laboratories with the capacity to monitor ART according tocapacity to monitor ART according to national/international guidelinesnational/international guidelines Every 2-4 yearsEvery 2-4 yearsHouseholdHousehold surveysurvey CS10CS10 The percent of orphans and vulnerable children lessThe percent of orphans and vulnerable children less than 18 years whose households received free basicthan 18 years whose households received free basic external support in caring for the childexternal support in caring for the child Every 2-4 yearsEvery 2-4 yearsHouseholdHousehold surveysurvey CS9CS9 The percent of persons aged 15-59 who have beenThe percent of persons aged 15-59 who have been chronically ill for 3 or more months in the last 12 monthschronically ill for 3 or more months in the last 12 months whose households received free basic external support inwhose households received free basic external support in caring for the chronically ill personcaring for the chronically ill person FrequencyFrequencyMethodsMethodsCore IndicatorsCore Indicators
- 27. Overview of Indicators, con’t... Every otherEvery other yearyear Interview /Interview / recordrecord reviewsreviews CS-A1CS-A1 The existence of national monitoring andThe existence of national monitoring and evaluation capacity for HIV/AIDS care and supportevaluation capacity for HIV/AIDS care and support programmesprogrammes Every 2-4 yearsEvery 2-4 yearsHealthHealth facilityfacility surveysurvey CS-A2CS-A2 The percent of health facilities with record keepingThe percent of health facilities with record keeping systems for monitoring of HIV/AIDS care and supportsystems for monitoring of HIV/AIDS care and support FrequencyFrequencyMethodsMethodsAdditional IndicatorsAdditional Indicators
Editor's Notes
- #25: Overall there are 10 core indicators covering a broaf range of areas including: Testing and counselling (CS+) Coverage (CS 2 and 5) UNGASS (CS3)
- #26: Areas, con’t… Existence of policies and guidelines (CS4) Capacity (CS 7,8,9) ART (CS2 and 5- coverage, 3- UNGASS, and capacity- CS7)
- #27: Areas, con’t… OVCs (CS10)
- #28: Two additional indicators are also included that look at M&E capacity and record keeping systems