School of ManagementProgram EvaluationMPA 513Week 3.docx
- 1. School of Management Program Evaluation MPA 513 Week 3 School of Management Policy in the NewsReview Needs Assessment / StakeholdersProcess EvaluationsExercise:Performance MonitoringExercise: City Stat exampleQuestions and Conclusions Class Overview and Objectives * School of Management In the News Public Administration Evaluation in the News * School of Management Logic Models
- 2. Stakeholders Review * School of Management Involving StakeholdersGain broader perspective, avoid blind spots, try to ensure utilization of resultsKey stakeholders:Those served or affected by activityThose involved in program operationsThose in a position to make decisions about the activityFor a manageable process, the list of stakeholders must be narrowed to primary intended users School of Management Evaluating Internal Processes “Now that this is the law of the land, let’s hope we can get our government to carry it out.” John F. Kennedy School of Management What is a Process Evaluation? Process (formative) evaluations are aimed at enhancing your program by understanding it more fully, and whether it is functioning as intended. Process evaluations study what is being done, and for whom these services are provided
- 3. * Evaluators often distinguish between process/implementation/formative vs. outcome/impact/summative evaluations. School of Management Process vs. Outcome Evaluation Process (Formative) – program managers, front-line staff, program designers, evaluation professionals and other internal and external entities focused on wanting to know why the program (or class of programs) is or is not working and what sort of program adaptations are appropriate. Outcome (Summative) measures – legislators, accounting entities, interest groups, other levels of government, and other external entities focused on accountability or accreditation. Evaluators often distinguish between: process or implementation (formative) vs. outcome or impact (summative): * School of Management Illustration of Process Evaluation Formative Evaluation Research Examines Inside The Program
- 4. At “The Process” (1)Jablonski, J.R. Total Quality Management. Technical Management Consortium Albuquerque, NM. * School of Management Organization Change and Process EvaluationProcess evaluation supports a program administrator’s desire to correct program deficiencies.Problem-solving orientation is different from evaluations that are more outcome-focused.Process intervention model provides a framework for planned organizational change. “We are interested not so much in whether X causes Y as in the question , if Y is not happening, what is wrong with X.” –Sylvia et al, p.70 * Sylvia, Sylvia, and Gunn. 1997. Program Planning and Evaluation for the Public Manager. Waveland Press. * School of Management Activities Inputs Outputs Intermediate Effects/ Outcomes
- 5. Short-term Effects/ Outcomes Long-term Effects/ Outcomes Context Assumptions Stage of Development Illustration Using the Logic Model (1)CDC Presentation on Process Evaluation: www.cdc.gov/vaccines/programs/progeval *Lots of this will have been decided when you “staked the claim” but worth considering again.How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions. In that case, then a need to reconcile where to put the resources.Key questions include:Ripple effect or reasonableness: This one should have been answered in “staking the claim”. The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact. Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”Accountability: This is what the major stakeholders are expecting of you. As mentioned, they might be
- 6. “wrong” and err in either a too ambitious or not ambitious enough direction. Still, in designing an evaluation, the information of most importance to the stakeholder is a major consideration.Measurability: As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect. School of Management Activities Inputs Outputs Intermediate Effects/ Outcomes Short-term Effects/ Outcomes Long-term Effects/ Outcomes Context Assumptions Stage of Development What the program and its staff actually do Results of activities: who, what will change
- 7. * School of Management Activities Inputs Outputs Intermediate Effects/ Outcomes Short-term Effects/ Outcomes Long-term Effects/ Outcomes Context Assumptions Stage of Development Sphere of control Sphere of influence
- 8. * School of Management Activities Inputs Outputs Intermediate Effects/ Outcomes Short-term Effects/ Outcomes Long-term Effects/ Outcomes Context Assumptions Stage of Development Outcome evaluation Process evaluation *
- 9. School of Management Why Study Implementation?Implementation is the most over- looked phase of the process approach to program change Change is frequently resisted and nearly always disruptive Critical components of planned change: Clear agenda and chain of command; Proper specificity for assigned responsibilities; and System for managing and monitoring implementation * School of Management When Can Process Evaluation be Useful?When expected outcomes are not observed, process evaluation can suggest reasons. When outcomes are positive, process evaluation may help understand which components are especially important. When an activity is newly implemented and it is too early to expect changes in outcomes. When an outcome evaluation is not feasible due to resource constraints. School of Management Look at Coverage and BiasHow many persons are receiving
- 10. services?What proportion of those who might need the service(s) actually used it/them?Has the service, or network of services served the intended clients?What were the demographics and clinical characteristics of clients? Are certain groups over/under represented? Is there creaming?What proportion of clients completed treatment (or received full services) and what were the characteristics of those who dropped out?Are clients aware of the program? Coverage deals with whether the program is reaching the appropriate population, and answers such evaluation questions as: * In general, process evaluations pose questions about coverage and process School of Management Common Coverage Problems“Self-selection” – bias in coverage due to voluntary over-participation or “creaming” Under-coverage due to limited funds, poor public education, poor access, or poor design – e.g., population too broadly defined, such as all poor mothers Over-coverage – reaching more than the intended group, e.g., Sesame Street, or special education * School of Management Look at Organizational FunctionAre they receiving the proper
- 11. amount, type, and quality of services? What are waiting times?Is staffing sufficient in numbers and competencies for the function?Is the program well-organized? Do staff work well together?Are resources, facilities, and funding adequate?Is the program in compliance with applicable professional and legal standards? Are participants satisfied with services?Is there appropriate discharge and follow up? Process questions deal with whether service delivery and support functions are consistent with program design specs and standards: * School of Management Common Implementation Failures“Non-programs”, poor quality, or incomplete intervention, e.g., Title I implementation had no proof of services for targeted audience of poor children Wrong or inappropriate intervention, e.g., something that is too sophisticated to go to scale or resisted Unstandardized intervention, e.g., Community Action Program and “maximum feasible participation” on the part of the poor * School of Management Coverage vs. Function Is the program serving clients it was intended to serve? How many direct contact hours does each client receive?
- 12. What is the average age of program participants? What percent of clients complete the program? From Workbook 4, Process Evaluations, WHO http://www.unodc.org/docs/treatment//process_evaluation.pdf Test your knowledge. Is the following a coverage question or a function question (1): * School of Management Process Evaluation MethodsQuantitativeQualitativeCombination School of Management Qualitative MethodsCase studiesStructured or semi-structured interviewsFocus groupsDirect observationReviews of program meeting minutes, progress reports School of Management Quantitative MethodsSurveysInformation collected from program participantsAgency records School of Management Summary Points Program process evaluation tells you if your program operates as intended;Service utilization issues typically break down into
- 13. questions about coverage and organizational functionCoverage problems can occur due to self-selection, under-coverage, and over-coverage Common implementation failures are incomplete interventions, delivery of the wrong intervention, and unstandardized or uncontrolled interventions * Design and Measurement Rubric Objectives Excellent Good Poor Unacceptable 1. Research Design Appropriate analytical method/design(s) have been chosen, with complete justification of the choice(s). If a process evaluation, describes the standard for comparison. If an outcome study, identifies the design (experimental, quasi-/ non-experimental). (3.0-2.9 points) Appropriate analytical method(s) were chosen, but justification and explanation could be better developed.
- 14. (2.8-2.7 pts) Appropriate analytical methods were chosen, but justification and explanation is incomplete or poorly developed. (2.6-2.5 pts) Inappropriate analytical methods were chosen and/or justification of those choices is incomplete or weakly developed. (2.4 - 0 pts) 2. Measurement and Data Collection Plan Thorough explanation of data collection and sampling plan, including (where appropriate), the standards for comparison, who the treatment and control group are, how large they are and
- 15. how they are selected Analytical methods thoroughly and clearly described, including using established scales where appropriate, discussing when and where measures will be taken, and how they will be used. (3.0 – 2.9 points) Explanation of data collection and sampling plan thorough, but could be more strongly tied to results and conclusions. Analysis plan is complete, but minor errors in application of chosen analysis method(s). (2.8-2.7 pts) Explanation of data collection and sampling plan incomplete and/or weakly tied to results and conclusions. Analysis plan is complete but significant or numerous errors in application of chosen analysis method(s) detract from paper. (2.6-2.5 pts) Explanation of data collection and sampling plan very weak or missing.
- 16. Analysis is inaccurate and/or incomplete. Does not use correct analytical method(s) and/ or they are incorrectly applied and interpreted. (2.4 - 0 pts) 3. Strengths and Limitations of Approach Lists all the merits and limitations of the design, and steps taken to minimize any threats. (2.0-1.9- points) Merits and limitations of the design and steps taken to minimize threats are incomplete. (1.8-1.7 pts) Missing more than two of limitations of the design and steps to minimize them. (1.6-1.5 pts) Fails to list the merits and limitations of the design, and steps taken to minimize any threats. (1.4 pts) 4. Quality of Writing and Grammar. Scholarly style that is also highly readable. Grammar, word choice, spelling or punctuation errors are very rare, no more often than once every 5-6 pages.
- 17. (2.0 – 1.9 points) Generally readable but with a few awkward or unclear passages. Grammar, word choice, spelling and punctuation errors occur, but are rare and do not detract from the paper. (1.8 – 1.7 pts) Awkward or unclear passages occur as often as once per page. Grammar, word choice, spelling and punctuation errors occur as often as once per page and detract. (1.6-1.5 pts) Awkward or unclear passages occur more often than once per page. Grammar, word choice, spelling and punctuation errors occur more often than once per page and substantially detract from the paper. (1.4– 0 pts) Total Objective = 10 (100%)
- 18. Program Brief Selected Program to be Evaluated The program selected for the proposal paper is the Summer Youth Employment Program (SYEP) implemented by the Mosholu Montefiore Community Center (MMCC). MMCC is a very well-known Center in New York City that receives millions of dollars’ worth of public funds, every year, to provide youth, senior and community development programs throughout the City (MMCC, 2017). Benefits, Beneficiaries & Scope of the Program The SYEP program provides young adults, ages 17 through 24, and youth as young as 14 to 16 years old, with proper working papers, an opportunity to gain employment for up to eight weeks during the summer months of July and August. MMCC is able to provide these services through City, State and Federal funders including NYC Department of Youth & Community Development (DYCD) and the NYS Department of Labor (DOL). The program must meet certain funding criterion to continuously receive the funding. One of the goals of MMCC’s SYEP program is to “provide a valuable work experience at job sites that are safe and well supervised” (NYC DYCD, 2015), while decreasing youth unemployment and poverty for at-risk youth. This goal encompasses assisting youth in distinguishing what their career interests may be, while exposing them to different industries within the workforce. The program’s main objective is to establish good work-habits, professional development, financial literacy and eventually, long-term employment. Problems to be Addressed by the Program While the youth-employee applicant is selected via a lottery,
- 19. there are specific eligibility requirements in place, to assure that the youth that come from lower-income, impoverished homes and neighborhoods, have the opportunity to obtain gainful employment through this program. It is pertinent that programs like the SYEP program, continue to be evaluated, critiqued, then continuously improved. Although cliché, the statement that “youth is the future”, is a very true statement. It is important to have programs work to create opportunities for quality education and gainful employment for young people, who may not be able to easily access it on their own. A decrease in quality of these programs could result in at- risk youth engaging in risky behavior, remaining uneducated and financially illiterate, resulting in an increase in the unemployment and poverty rates. The SYEP program works to address what could be, lack of examples of professional behavior or steady income, if a youth, for example, comes from a single-parent home where the parent may be unemployed. The program partners with the schools and financial institutions to provide education, training and knowledge for long-term success for the young candidates, including professional development and financial management literacy. References NYC Department of Youth & Community Development (2015). Jobs & Internships. 2014 NYC SYEP Annual Summary. Retrieved from http://www.nyc.gov/html/dycd/html/jobs/jobs.shtml Mosholu Montefiore Community Center (2017). Youth Employment. Retrieved from https://www.mmcc.org/youth-employment/ Rationales
- 20. Assumptions Employment-based programs can induce learning and help close the “opportunity gap” between youth professional development and employer demand Youth employment can be a critical facilitator of positive youth development and preparation for adulthood. It can improve educational outcomes, minimize learning loss and contribute to the family budget. Contribution to the family budget can help reduce the poverty rate and contribute to the local economy through steady work and steady pay. Youth in low income neighborhoods want to work in meaningful jobs while contributing
- 21. to family income, but may lack support and access to opportunities and ways to display their skillsets, Resources 5 Training Developers/Facilitators 5 Recruiters 1 Program Director 1 Intern 2 Financial Advisors Activity Groups Outreach/Recruitment Professional Development Matching Post-Placement Education & Support.
- 22. Outputs 50 Interviews per week 40 hours/1 week of Professional Development. Onsite, job placement visits and follow-up Financial advisors from local banks provide financial literacy workshops, once a week. Intermediate-Term Outcomes Work readiness for participating youth. Successful placement of 70% of participants. Increased school attendance for at least 60% of participants.
- 23. Pre-employment skills. Participants maintain steady employment for at least 6 months. Short-Term Outcomes Obtain skill-set and confidence to interview for employment. Recruit applicants, from targeted, low- income neighborhoods Gain temporary employment. Learn future job seeking skills Learn money management and investment techniques Youth participants have an increase desire to go and stay in school. Problem Statement There is a high unemployment rate, high poverty, low-school attendance and barriers to quality financial education for youth in low income neighborhoods.
- 24. Logic Model Diagram: MMCC's Youth Employment Logic Model Goal Increase youth employment, school attendance and financial literacy and planning, while decreasing the poverty rate for youth, through job placement, training and educating. Long-Term Outcomes Increase family income Decrease of unemployment rate in low-income neighbhorhoods Higher high school graduation rates. Increase in participants confidence resulting in decrease of fear about not obtaining employment. Long-term, permanent employment for at least through the duration of college years. Please look at attachments first before agreeing. I will need this in 6 hours. Attachments are as follows: · Assignment Instructions (assignment: Data Collection Plan for future evaluation) · Lecture Slides · Brief of Program to be evaluated (for your reference) · Rubric/Goals for assignment If you find that a couple more hours are needed, you should let me know. I prefer QUALITY over speed.