Measuring learning impact & effectiveness
2 likes1,147 views
This document discusses strategies for measuring learning effectiveness. It outlines key principles for measurement including being relevant, credible, simple, consistent and efficient. It describes different levels of measurement including reaction, learning, behavior and results. It provides guidelines for evaluating learning at each level. These include assessing knowledge gained, skills developed, attitudes changed and on-the-job application of skills. The document also references the Kirkpatrick-Phillips model of evaluation and proposes instruments to measure factors like background knowledge, self-regulation, learning content, style and time on task.
Measuring learning impact & effectiveness
- 2. MEASUREMENT STRATEGY CORE PRINCIPLES RELEVANT CREDIBLE & COMPELLING SIMPLE & CONSISTENT EFFICIENT
- 3. MEASUREMENT KEY IMPACT QUESTION What will the participants be doing differently if this programme works? What impact will it have on those around them? What specific business /department KPI’s would you want to see improve as a result of the programme? What will success look like for the learners and the business? INDIVIDUAL GROUP UNIT CORPORATE
- 4. MEASUREMENT CATEGORY PREDICTIVE MEASUREMENT FORMATIVE MEASUREMENT BASELINE MEASUREMENT IN-PROCESS MEASUREMENT RESTROSPECTIVE MEASUREMENT
- 13. KIRKPATRICK – PHILLIPS MODEL Knowledge, skills, & attitude LEVEL 2 LEARNING
- 14. KIRKPATRICK – PHILLIPS MODEL LEVEL 2 LEARNING LEVEL 1 REACTION LEVEL 3 BEHAVIOR LEVEL 4 RESULTS VALUE OF INFORMATION FREQUENCY OFUSE DIFFICULTYOF ASSESSMENT Least valuable mostvaluable frequent easy infrequent difficult
- 15. LEVEL 1 : REACTION Evaluate trainee’s reactions to the program: Did they like the program? Did they like the facilitators? Did they like accommodation & facilities? Guidelinesfor evaluatingreaction 1. Determine what you want to find out 2. Design a form that will quantify reactions 3. Encourage written comments & suggestion 4. Get 100% immediate response 5. Get honest response 6. Develop acceptable standards 7. Measure reactions againts standards, & take apropiate action 8. Communicate reaction as appropiate
- 16. LEVEL 2 : LEARNING Measuring learning means determining one or more of the following: What knowledge was learned? What skills were developed or improved? What attitude were changed? Guidelinesfor evaluatinglearning 1. Use control group if practical 2. Evaluate knowledge, skills and/or attitude both before and after the program 3. Use a paper-and-pencil test to measure knowledge 4. Use performance test to measure skills 5. Get 100% immediate responses 6. Use the results of evaluation to take appropriate action
- 17. LEVEL 3 : BEHAVIOR • The frequency of application of new skills/ knowledge/ attitudes (on the job) • The effectiveness of the skills/knowledge/attitudes (as applied on the job) Guidelinesfor evaluating BEHAVIOR 1. Use control group if practical 2. Allow time for behavioral change and application to take place 3. Evaluate both before and after program if practical 4. Survey and/or interview one or more of the following: trainees, their immediate supervisor, their subordinates, and others who often observe their behavior 5. Get 100% response or a sampling 6. Repeat the evaluation at appropiate times 7. Consider soct versus benefits
- 18. LEVEL 4 : RESULT • Indicate the extent to which you think this program has influenced each of these measures in your work unit, department, or busines unit: • Productivity / quality / customer response time / cost control / employee satisfaction / customer satisfaction Guidelinesfor evaluating RESULT 1. Use control group if practical 2. Allow time results to be achieved 3. Evaluate both before and after program if practical 4. Repeat the evaluation at appropiate times 5. Consider soct versus benefits 6. Be satisfied with the evidence if prrof is not possible
- 19. MEASUREMENT OF LEARNING EFFECTIVENESS ITEM INSTRUMENT ; REMARKS Background knowledge, prior skills, abilities Conduct a pre-test Self-regulation in learning context (Toering, Elferink- Gemser, Jonker, Van Heuvelen, & Visscher, 2012) Use of tools that support self-regulation; Collect Self-regulation of Learning and Self-Report Scale prior and post learning intervention. The online form is available at https://docs.google.com/spreadsheet/viewform?formkey=dHFwNnBiRjY1QWM0eE5ielA5X0pVd Xc6MQ. Learning content Analysis of learning content provided for the learner; Learning support tools Use of auxiliary support tools online, on the device; usage by the learner; measurement of learner engagement; learner analytics Learning transfer Conduct post-tests, apply knowledge to specific tasks that map to learning objectives; these tests include, quizzes, case studies, written assignments and peer review Learner feedback Qualitative feedback and textual analysis Learning style/preferences Learning modality preference inventory and Kiersey Temperament Inventory. It is anticipated that the learning style has second order effects on learning. (see also (Neuhauser, 2002)). Learning style may have stronger influence on learner engagement and attention span. Time on task Record of tasks completed and the use of tools. This data may be collected on the device (aka MoLE). Instructional method Types used and the nature of the tasks Teacher effects Controlled experiment
- 26. Thank you JOKO SUSENO CORPORATE CULTURE MANAGEMENT joko.suseno@live.com