Scorecards, Learning Metrics and Measurement Strategies
9 likes3,600 views
This document provides information about a webinar on scorecards, learning metrics, and measurement strategies. It includes details about joining the webinar via teleconference or computer speakers. It lists the speakers and moderator for the webinar. It also provides information on how to use the question and answer and polling features during the webinar. Finally, it answers some frequently asked questions and closes by introducing the moderator and speakers.
Scorecards, Learning Metrics and Measurement Strategies
- 1. Scorecards, Learning Metrics and Measurement Strategies You can listen to today’s webinar using your computer’s speakers or you may dial into the teleconference. If you would like to join the teleconference, please dial 1.866.469.3239 and enter access code xxx xx xxx. You will be on music hold until the seminar begins. #CLOwebinar
- 2. Scorecards, Learning Metrics and Measurement Strategies Speaker: Dennis Bonilla Senior Vice President GP Learning Solutions Group Andrew Snoey Senior Program Manager, Operations Microsoft Corporation Moderator: Kellye Whitney Managing Editor Chief Learning Officer magazine #CLOwebinar
- 3. Tools You Can Use • Q&A – Click on the Q&A panel (?) in the bottom right corner – Type in your question in the space provided – Click on “Send.” #CLOwebinar
- 4. Tools You Can Use • Polling – The poll will appear on the right side of your screen – Select the best option for each question – Click on “Submit” #CLOwebinar
- 5. Frequently Asked Questions 1. Will I receive a copy of the slides after the webinar? YES 2. Will I receive a copy of the webinar recording? YES Please allow up to 2 business days to receive these materials #CLOwebinar
- 6. Scorecards, Learning Metrics and Measurement Strategies Kellye Whitney Managing Editor Chief Learning Officer magazine #CLOwebinar
- 7. Scorecards, Learning Metrics and Measurement Strategies Dennis Bonilla Senior Vice President GP Learning Solutions Group Andrew Snoey Senior Program Manager, Operations Microsoft Corporation #CLOwebinar
- 8. LEARNING METRICS, SCORECARDS MEASUREMENT STRATEGIES
- 9. POLL QUESTION Where does your role fit in your Learning and Development Organization? • Not at all • Instructional Designer • Instructor • Training Coordinator • Training Manager • In the Line of Business 9
- 10. POLL QUESTION Does your organization formally track and report learning metrics? • Not at all • Sometimes • I Don’t Know • Yes 10
- 11. POLL QUESTION Does your organization formally track and report learning metrics? • Not at all • Sometimes • I Don’t Know • Yes If you answered yes to the previous poll, are the metrics: • Just Learning Process and Event Statistics (butts in seat, courses completed, etc….) • Tied to Organizational Business Outcomes • Both 11
- 12. SECTION 1 – CRITERIA FOR DEVELOPING EFFECTIVE SCORECARDS 12
- 13. SCORECARD OPERATING PRINCIPLES • Drive Business Leadership • Aim for Simplicity • Set Realistic, But Stretched, Targets • Ensure Consistency in Taxonomy and Calculations • Ensure Similar Metric Types have Similar Thresholds • Align Related Scorecards’ Metrics and Targets 13
- 14. IS LEARNING MAKING A DIFFERENCE? • Are business objectives being met? • Business leaders are demanding more linkages to business outcomes and measurements of results, yet it’s the outcome least often measured. • Huge confusion between internal process metrics and actual results. • Sponsors want to see evidence of IMPROVED BUSINESS PERFORMANCE, not internal training operations metrics. 14
- 15. SECTION 2 – DISCUSS THE EVOLUTION OF LEARNING MEASUREMENT FRAMEWORKS 15
- 16. POLL QUESTION How does your organization / company measure the quality of the training programs for your key strategic initiatives? • Not at all • Data collection measuring attendee satisfaction • Data collection measuring attendee competency • Data collection measuring manager satisfaction in attendee performance • Data collection measuring business impact after course completion 16
- 17. WHAT METHODS ARE USED IN EVALUATION? • Evaluation is traditionally represented as the final stage in a systematic approach to training • More recent Instructional Systems Design (ISD) models incorporate evaluation throughout the process, not as end state • There are six general approaches to educational evaluation with Goal-based and Systems-based approaches as the predominant modes used in the evaluation of training programs • Various frameworks have been proposed under the influence of these two approaches • The most influential framework has come from Kirkpatrick which follows the Goal- based approach and the most commonly used today 17
- 18. KIRKPATRICK LEVELS But Level IV is the only place RESULTS BUSINESS IMPACT AS A where we can easily perform LEVEL IV CONSEQUENCE OF TRAINING an ROI calculation TRANSFER STUDENTS’ BEHAVIORAL CHANGE RESULTING Assessing Level II and III takes LEVEL III IN PERFORMANCE AT A HIGHER LEVEL more time and effort –this is where our certification program is located LEARNING STUDENT UNDERSTANDS AND CAN RECALL LEVEL II PRIMARY CONCEPTS FROM THE TRAINING REACTIONS STUDENT PERCEPTIONS OF THE COURSE Most organizations have LEVEL I AND ITS VALUE Level 0 to I ATTENDANCE BASELINE INFO SUCH AS LEVEL 0 STUDENT PRESENCE AND PARTICIPATION 18
- 19. SECTION 3 –ALIGNING TRAINING MEASURES WITH BUSINESS MEASURES 19
- 20. TALK ABOUT A DISCONNECT WHAT EXECUTIVES WANT 4 3 2 1 0 BUSINESS LINKAGE REGISTRATION & PROGRAM DELIVERY MEASUREMENT ALIGNMENT Data from the Forum Corporation, Customer Driven Training Organization 20
- 21. TALK ABOUT A DISCONNECT WHAT EXECUTIVES WANT CURRENT EFFORT ALLOCATION 4 3 2 1 0 BUSINESS LINKAGE REGISTRATION & PROGRAM DELIVERY MEASUREMENT ALIGNMENT Data from the Forum Corporation, Customer Driven Training Organization 21
- 22. MEASUREMENT STRATEGIES • 1st Question that should always be asked: “Does this metric actually measure an outcome for which the training program was created?” • For some, it could be a behavior change; for others, improved perception by stakeholders, and others a quantitative measure of business impact (revenue, market share, market position, etc.). • Measure what matters to the customer in a way that leads to informed action. • Don’t pick a process before defining the purpose. Let form follow function! 22
- 23. MEASUREMENT STRATEGY CORE PRINCIPLES • Relevant • Credible and Compelling • Simple and Consistent • Efficient “For every complex problem there is an answer that is clear, simple and wrong!” – H.L. Mencken 23
- 24. POLL (TRIVIA) QUESTION When was the first, documented, formal use of an educational program evaluation conducted in the USA? • 1792 • 1815 • 1897 • 1916 • 1958 24
- 25. “NOT EVERYTHING THAT CAN BE COUNTED, COUNTS.” So What Counts? • Outcomes that are of interest to the business to which training contributed in a demonstrable way. • Sponsors may not be as wedded to proof of financial ROI as many HR professionals assume. • An evaluation that provides sound evidence to support informed decisions that are in the best interest of the organization. • They prove to an appropriate level of certainty that the program did or did not achieve the desired outcomes and they provide insights to improve subsequent courses or iterations; actionable Information (Bersin) 25
- 26. PRESENTING TRAINING RESULTS Power of Narrative • “Stories are easier to remember – because in many ways, stories are how we remember.” Pink (2005): A Whole New Mind • Stories bring statistics to life • Stories will be repeated and retold Sell the Sizzle: It doesn’t matter how good your results are if no one knows about them. Results x Marketing = Perception of Value 26
- 27. SECTION 4 – SHARING EXAMPLES OF MICROSOFT’S EFFORTS 27
- 28. PROJECT PICASSO In the Fall of 2009, the Microsoft training organization to applied our measurement principles to our course assessment survey. Remember • Relevant: questions refer to key course success metrics à eliminated non-action- oriented questions • Simple: shortened the question quantity to improve response rate • Consistent: aligned format, wording, and calculations à minor wording revisions OLD SURVEY NEW SURVEY OLT 12 + 4 6+4 cILT 11+7 12 + 4 Compelling: added business impact questions vILT 11+7 12 + 4 • Manager engagement on the material • Different actions will the employee take due to the course 28
- 29. PROFESSIONAL CERTIFICATION TECHNICAL EMPLOYEES SALES & MARKETING EMPLOYEES • Microsoft has an extensive • Sales staff was taught the certification program for internal and “soft” (professional) skills, but how external candidates using third-party do you measure the knowledge? testing organizations • In January 2010, Microsoft started • Our technical staff can be certified in an effort to train Sales & 18 tracks; when looking across the Margeting staff through an technologies and levels, resulting in academy program that would 64 discrete certifications have certifications associated with certain skills http://www.microsoft.com/learning/en/us/certification/cert-default.aspx 29
- 30. SALES AND MARKETING ACADEMY Our Sales & Marketing Academy is part of our People focus Purpose • Accelerate sales and marketing staff readiness • Promote management coaching and re-enforcement Basic Competent Master Business Management A101 A201 A301 Options considered Customer Relationship • Certification of the growth of a skill Management B101 B201 B301 • Certification of the growth of a skill set C101 Products & Services -- C301 • Separate certifications for each skill C102 Organizational A101 B201 • Certify high-priority skills Awareness A102 D201 -- … E101 B201 -- Built F301 … F101 F201 F302 • Curriculum roadmaps • Course knowledge assessments – demand pass rate of 80% • Single certification activity across all courses in a level involving case study/simulation, live work product review, role play, oral review panel 30
- 31. MEASURING TRAINING IMPACT Almost two years ago, Microsoft started an effort to measure the business impact of our core sales training curriculum (CST) Key questions asked • Does CST increase the Account Manager’s annual quota attainment? • Does CST increase Customer Satisfaction? • Does CST increase the Account Manager’s pipeline velocity? Tried to control for • Region / country • Tenure / experience level • Account Manager’s satisfaction level with CST courses • Account Manager's self-assessed impact of CST courses 31
- 32. ALIGNING TRAINING TO A BUSINESS SCORECARD DEVELOP SALES & QUARTERLY CORPORATE MARKETING SCORECARD SCORECARD Metric G EXECUTES TO RESULTS Sell $ STRATEGIC Color? Share % INITIATIVES Cust Sat # Y/R Feedback Loops TRAINING ASSESS OTHER OTHER Root TRAINING PLANS Cause? ADOPTION Course Metric A Metric B Metric C Metric D Metric E Sell… X X X FORMULATE RESPONSE PLAN Market… X X 1. Drive existing course(s) Present… X 2. Bolster manager reinforcement 3. Re-vamp existing course(s) Create… X 4. Develop new course(s) Close… X X 32
- 33. SUMMARY • Align Your Scorecard with the Desired Business Results • Do Not Abandon Measurement at the Smiley Face • Count Training Results, Not Effort 33
- 34. LET’S CONTINUE TALKING Dennis Bonilla Senior Vice President, Learning Solutions General Physics Corporation v-deboni@microsoft.com 425-636-3410 office 702-885-5814 mobile Andrew Snoey Senior Program Manager, Operations Microsoft Corporation asnoey@microsoft.com 425-722-8510 office
- 35. Join Our Next CLO Webinar To the Agile Go the Spoils: Why Learning Must Evolve Tuesday, June 21, 2011 CLO Webinars start at 2 p.m. Eastern / 11 a.m. Pacific Register at www.clomedia.com/events Join the CLO Network: http://network.clomedia.com/ #CLOwebinar