You Can Do Program Evaluation
- 1. You Can Do Program Evaluation November 19, 2010 Stephen B. Johnson, PhD Diane M. Talley, MA James A. Penny, PhD Castle Worldwide
- 2. NOCA 1100 5.3 The certificate provider shall conduct periodic program evaluations to assess program quality and effectiveness and implement future improvements. ANSI 6.2.10.1 The evaluation shall measure the quality, effectiveness, and value of the certificate program against stated program performance objectives. 6.2.10.4 The evaluation shall include mechanisms to monitor and identify regularly the need for changes to the program’s purpose, score, or learning outcomes …. Program evaluation standard Definition of evaluation: Systematic investigation of the worth of something to gain direction for future improvement (Joint Commission - Program Evaluation Standards page 3). Why are we here? Plato and Aristotle in The School of Athens fresco, by Raphael. ICE 2010 Conference Atlanta Georgia
- 3. Simplified program evaluation model 1. Identify a problem. 2. Generate and implement alternatives to relieve the stress. 3. Evaluate those alternatives. 4. Adopt those most likely to reduce problem. ICE 2010 Conference Atlanta Georgia
- 4. Good program evaluation is a tool 1. Improve your program 2. Illuminate/explain how/why a program works 3. Challenge assumptions 4. Avoid the doldrums 5. For certificate programs – focuses on the learner 6. Takes program design from: •Trial and error •Based on what is known/assumed •Heuristic design (planned trial and error) 7. To a planned program design that might surprise ICE 2010 Conference Atlanta Georgia
- 5. The Wright model ICE 2010 Conference Atlanta Georgia
- 6. The Wright flyer ICE 2010 Conference Atlanta Georgia
- 7. Wright wind tunnel ICE 2010 Conference Atlanta Georgia
- 8. General Model of Program Evaluation ICE 2010 Conference Atlanta Georgia
- 9. Questions about evaluation Why evaluate? Purpose? Who will do it? Where do the questions come from? What questions to ask? Importance (and number) of questions? Methods to use? How will the results be used/implemented? Time and budget? What is the fall back if something breaks? ICE 2010 Conference Atlanta Georgia
- 10. Evaluating education/training programs Appraise the worth of educational undertaking, against a desired standard, for the purpose of decision making typically around changing elements of the program (Popham, Scriven et. al.) Focus on learner behavior/outcomes MOST concerned with post-instruction outcomes directly related to the learner ICE 2010 Conference Atlanta Georgia
- 11. Novice errors A single evaluation – inevitably flawed Too many questions Expensive, inconclusive, problems with implementing solutions Choose easy but trivial issues The change is not worth the cost The identified problem may actually be a symptom – the problem may be deeper ICE 2010 Conference Atlanta Georgia
- 12. Going deep ICE 2010 Conference Atlanta Georgia
- 13. Hubble deep field image ICE 2010 Conference Atlanta Georgia
- 14. Our advice 1. Start with 1 or 2 practical problems. 2. Be aware of confusing the symptom with the problem. 3. Plan for the long term (with flexibility). 4. Get deeper over time. Will most likely lead to innovative solutions and program outcomes including different clients, products, or services 1. Remember the Wright model. ICE 2010 Conference Atlanta Georgia
- 15. Questions? Stephen B. Johnson sjohnson@castleworldwide.com Diane M. Talley dtalley@castleworldwide.com James A. Penny jpenny@castleworldwide.com 919.572.6880 www.castleworldwide.com
Editor's Notes
- #3: Our goal here today is to: Make program evaluation something tangible/understandable that you can think of in terms of your own program rather than an abstract concept See program evaluation not as a burden, another hurdle to learn but something to give your program wings.
- #4: Life is never this simple – too many assumptions such as you know the problem
- #5: Becalmed, a study on the east coast of Australia [sailing ship, possibly a grain ship, taken while Hurley was en route to Antarctica with the Australasian Antarctic Expedition in 1911] [picture] : [Australia] / [Frank Hurley] National Library of Australia online catalog http://catalogue.nla.gov.au/Record/90519
- #6: 1901 Wright Glider
- #7: 1903 Wright Aeroplane (Kitty Hawk NC)
- #8: Model of Wright Wind tunnel – Kitty Hawk NC
- #9: General Model of Evaluating Training Programs: Proposed by Stephen Brian Johnson 2004. Levels refer to Donald Kirkpatrick (1979). Level I: Training Reaction: At the end of the training program, participants are asked to assess the value of the training they received. Level II: Learning: Assesses participant learning through testing, skill practices, role-plays, simulations, group evaluations and other assessment tools. Level III: Behavior: After completing the training program, participants should demonstrate a change or improvement in on-the-job performance. Level IV: Business Results: Focuses on the actual business results achieved by program participants as they successfully apply the program education. Metrics can include output, quality, costs, time, and customer satisfaction. Measurement at this level will often compare the program’s monetary benefits with the program’s costs Donald Kirkpatrick (1979)
- #11: A special case Aim to appraise the worth of educational undertaking, against a desired standard, for the purpose of decision making typically around changing elements of the program. Reporting the results is NOT evaluation Evaluation of educational programs should focus on learner behavior Sure you can evaluate the impact of process variables (e.g., learner background, instructional time), and what happened during the instruction (e.g., engagement), but we are MOST concerned with post-instruction outcomes directly related to the learner. As a result you need to have a theory of action that defines what post-instruction learner changes will have occurred. Could be behavioral, attitudinal or other affect. But, let’s face it, for training purposes we looking to change affect (e.g., attitudes) and/or knowledge and skills so there is an impact on the learner’s behavior to
- #13: Hubble deep field location
- #14: Seeing more than you expect