Prt 595 week 2 lecture 2
- 1. Evaluation Context & Questions PRT 595 Dr. Bob Barcelona
- 2. Evaluation Context There is no “one size fits all” approach to evaluation Good evaluations must fit the circumstances Perspectives of stakeholders Purpose of evaluation efforts Key guiding questions Practical yet rigorous social science methods
- 3. Who Determines Purpose & Questions? Program stakeholders (e.g. evaluation sponsor, funders, program management) Expect different perspectives (or conflict!) from different stakeholders How do evaluators deal with this? Incorporate conflicting perspectives within the evaluation design Focus on only one perspective Facilitate negotiation of perspectives – build shared understanding Balance stakeholder interests – see the case study by Morris (2003) and the commentary by Leviton (2003)
- 4. Involving Stakeholders Identify who your stakeholders are Establish a structure for stakeholder involvement Involve them early Involve them continuously Involve them actively Reineke (1991)
- 5. Questions for Stakeholders Why is an evaluation needed? What is the evaluation’s purpose? What are the program’s goals and objectives? What are the important questions for the evaluation to answer? Are questions tied to a program’s theory or logic model?
- 6. What’s the Purpose? Formative Evaluation: providing information that helps to guide program improvement Summative Evaluation: providing information that helps to determine whether program resources are spent efficiently and effectively, and whether programs produce intended benefits Knowledge Generation: providing information to describe the nature and effects of an intervention as a contribution to knowledge
- 7. Purpose Statement Example The purpose of this evaluation is to understand the impact of housing relocation on the quality of life, well-being, and economic self sufficiency of former and current residents of a severely distressed public housing development in Greenville, SC.
- 8. Good Evaluation Questions Help to focus the evaluation Facilitate the design of data collection procedures Must be reasonable and appropriate – they are tied to the goals and objectives of the program Must be answerable – specific, concrete, practical, and measurable Set criteria for performance (i.e. 75% of children… a majority of residents…)
- 9. Seventy-five percent (75%) of regular attendees will improve levels of fitness relative to the previous year by participating in health enhancing physical and recreational activities each day 75% or more of the regular attendees will achieve at least 60 minutes of moderate or vigorous physical activity each day Selected evaluation criteria, GoalPOST: Goal Oriented Performance in Out-of-School-Time Program , Clemson University
- 10. Types of Evaluation Questions Needs Assessment Assessment of Program Process Assessment of Program Theory Impact Assessment Efficiency Assessment
- 11. Answering Questions Sound social science methodology! Some examples (HFRP, 2004): Surveys and Questionnaires Interviews and Focus Groups Observations Tests and Assessments Document Reviews and Analysis Secondary Sources and Data Reviews Each has strengths and weaknesses Advantages of using multiple methods
- 12. In Sum…Good Evaluations…. Have defined purposes Accurately identify and involve stakeholders Ask clear, relevant, and answerable questions Are focused on assessing program need, developing program theory, assessing program process, assessing program impact, or assessing program efficiency Use practical and rigorous social science methodology and data collection techniques to answer questions Are used by stakeholders in some tangible way
Editor's Notes
- #3: Focus on the questions to be answered – can’t answer EVERY question – the purpose of the evaluation and its subsequent set of guiding questions need to be SPECIFIC in its purpose Questions should drive the methods – we use good social science to answer our evaluation questions – quantitative, qualitative, mixed methods, etc. But ultimately, the practicality of the social science methods matters – it needs to be DOABLE and USABLE by stakeholders! Must identify what the relationship is between stakeholders and the evaluator – how will you be interacting with your stakeholders?
- #4: Any of these are legitimate responses – but all have potential downsides – 1) difficult, 2) can alienate key stakeholders or destroy credibility, 3) takes skill, time, and resources to accomplish Case study – deals with an evaluation of a life skills program designed to deal with sexual risk-taking – a hot button issue!
- #6: In all of these cases – it is the evaluator’s job to clarify ambiguous, vague, or untenable goals, objectives and questions!
- #7: Formative evaluations – focused on program improvement; results are used immediately – focusing on program need, design, implementation, impact, efficiency…can be informal Summative evaluation – focused on accountability – what the program does – usually happens at the end of a program – decisionmakers – usually formal Knowledge generation – interventions – does a particular program or intervention work? Usually conducted by academic researchers, government agencies, foundations… Robert Wood Johnson Foundation’s Active Living Program
- #8: Could fit into any of the three categories – formal purpose statements often tell us little about the actual purpose of the evaluation, who the primary audience for the evaluation results will be, and how the results will be used. It is important that we understand this at the beginning of the evaluation process!
- #9: The key here is that evaluation questions are reasonable and appropriate – they are tied to the goals of the program – DMS example – how did afterschool enrichment programs impact grades in core subjects? Not reasonable to expect that students who take karate or hip hop dance after school will automatically do better on tests or in core subjects. What are the goals of the program? Evaluation questions should stem directly from the program’s goals. Also – must be answerable within the scope of the evaluation and its resources – did housing redevelopment create economic impact? Too soon to answer that question! Criteria for performance are determined by a number of things – needs/wants of target population, goals and objectives, professional standards, legal requirements, past performance, baseline levels….This is critical because unlike academic research, we need to be able to judge the PRACTICAL VALUE of a program – understand whether it met standards or not…
- #10: 75% - based on program goals Regular attendees – those attending 80% or more of the program time 60 minutes of moderate or vigorous physical activity – CDC guidelines for childhood physical activity
- #11: Needs Assessment – questions about the social conditions a program is intended to ameliorate and the need for the program Assessment of Program Theory (Evaluability) – questions about the program’s conceptualization and design Assessment of Program Process – questions about program process, operations, implementation, and service delivery Impact Assessment – questions about program outcomes and impact Efficiency Assessment – questions about program cost and cost-effectiveness
- #12: We’ll discuss social science methods throughout the course – but it is helpful to understand the broad range of techniques that can be used to collect data to help answer evaluation questions!