Learning and Business Impact: Making the Case through Metrics and Analytics
2 likes1,066 views
The document outlines the processes and tools for measuring the impact of learning and development (L&D) initiatives in organizations, with insights from Scott Schaffer, a global head of learning measurement and analytics at Ericsson. It emphasizes the importance of aligning learning objectives with business goals, utilizing metrics to evaluate effectiveness, and understanding the quantitative and qualitative impacts of training on organizational performance. Various methodologies, measurement strategies, and case studies are discussed to illustrate how to demonstrate the value of L&D to stakeholders.
![Scott Schaffer, Learning measurement and analytics, 2017
Source: Measurement map derived from work of Kaufman and Keller; and R. Smith
Learning
Program
Major Objective
• Topic
• Topic
• Topic
• Topic
Major Objective
• Topic
• Topic
• Topic
Performance
Objectives
Performance 1
Performance 3
Performance 5
Performance 4
Performance 2
Leading
Indicators
Improved
presentations, calls,
proposals
Reduced errors,
increased volume
Business
Results
Client Satisfaction
Productivity
Strategic
Goals
Cost Reduction
Gross Margins
Individual learning programs are designed to drive business value
The impact map identifies links between learning objectives and business objectives
From left to right, ask “How do we achieve these objectives?”
From right to left, ask “Why are we doing this [intervention or measurement]?”
Performance 6](https://image.slidesharecdn.com/3-170308160340/85/Learning-and-Business-Impact-Making-the-Case-through-Metrics-and-Analytics-18-320.jpg)
![Scott Schaffer, Learning measurement and analytics, 2017
Business
Goal
Learning Impact @ the Solution Level
Impact occurs when:
A training-acquired skill [S] is applied to improve performance in some part of a job that
produces a Result [R] that contributes to a Business Goal (adapted from R. Brinkerhoff).
Learning
Intervention
S
S
S
S
S
S
S R
Individual or Team
Behaviors/Performance
20
Skills are applied
while in the role…
Improved KPIs;
Correlations
between S & R](https://image.slidesharecdn.com/3-170308160340/85/Learning-and-Business-Impact-Making-the-Case-through-Metrics-and-Analytics-20-320.jpg)
![Scott Schaffer, Learning measurement and analytics, 2017
Business
Goal
The Anatomy of Learning Impact:
When Learning works, what happens?
Impact occurs when:
A training-acquired skill [S] is applied to improve performance in some part of a job that
produces a Result [R] that contributes to a Business Goal
Learning
Intervention
S
S
S
S
S
S
S R
Individual or Team Job
This is the
Finish Line](https://image.slidesharecdn.com/3-170308160340/85/Learning-and-Business-Impact-Making-the-Case-through-Metrics-and-Analytics-21-320.jpg)
![Scott Schaffer, Learning measurement and analytics, 2017
The Anatomy of Learning Impact:
When Learning works, what happens?
Impact occurs when:
A training-acquired skill [S] is applied to improve performance in some part of a job that
produces a Result [R] that contributes to a Business Goal
Learning
Intervention
S
S
S
S
S
S
S R
Individual or Team Job
This is the
Finish Line
Not this!
Business
Goal](https://image.slidesharecdn.com/3-170308160340/85/Learning-and-Business-Impact-Making-the-Case-through-Metrics-and-Analytics-22-320.jpg)
Learning and Business Impact: Making the Case through Metrics and Analytics
- 1. #CLOwebinar Sponsored by The presentation will begin at the top of the hour. A dial in number will not be provided. Listen to today’s webinar using your computer’s speakers or headphones. Welcome to the webinar!
- 2. #CLOwebinar Sponsored by Tools You Can Use Audio Control – A dial in number will not be provided. – Adjust the volume by sliding the indicator in the Media Player box to the right. – Also check your computer’s volume for external speakers or headsets.
- 3. #CLOwebinar Sponsored by Tools You Can Use Speaker Bios Resources Media Player Q&A Slides Twitter Help Post Event EvaluationShare This Group Chat HRCI & SHRM CLO Webinars
- 4. #CLOwebinar Sponsored by 1. May I receive a copy of the slides? YES! Click on the resource list located on the top right portion of your screen. 2. May I review the webinar recording at a later date? YES! You may log in again using today’s link to review the presentation on-demand. 3. Is this webinar HRCI or SHRM certified? YES! The HRCI and SHRM certification codes will appear in the box to the right of the slides after the required watch- time has elapsed. Frequently Asked Ques6ons
- 5. #CLOwebinar Sponsored by Sco% Schaffer Global Head of Learning Measurement & Analy=cs Ericsson Learning and Business Impact: Making the Case through Metrics and Analy6cs
- 6. Learning and Business Impact: Making the Case through Metrics and Analytics Scott Schaffer, PhD Twitter: @schaffer3999 Global Head of Learning Measurement & Analytics
- 7. Scott Schaffer, Learning measurement and analytics, 2017
- 8. Aligning Measurement with Learning and Business objectives Scott Schaffer, PhD Global Head of Learning Measurement & Analytics › Introduction to Ericsson › Measuring Business Impact › Measurement Maturity › Analytics & Learning Measurement
- 9. Public | © Ericsson AB 2014 | 2014-12-5 | Page 2 37,000 40% 5 busd 1 Billion 2.5 Billion 65,000 28 busd 180 111,000 Subscribers managed Subscribers supported Services professionals Net Sales 2016 Countries Employees Patents Mobile traffic In R&D Ericsson at a glance MOBILE INFRASTRUCTURE OPERATIONS & BUSINESS SUPPORT SYSTEMS TELECOM SERVICES TV PLATFORMS
- 10. Scott Schaffer, Learning measurement and analytics, 2017 Our Strategic direction FUTURE/ EMERGING PRESENT/ LARGE Excel in Core Business Establish leadership in targeted areas Expand Business In New Areas1 2 3 Radio, Core & Transmission Telecom Services IP Networks TV & Media OSS & BSS Industry & Society Cloud
- 11. Scott Schaffer, Learning measurement and analytics, 2017 CLO CEO Ericsson leaders want to know that we are planning to Measure L&D Metrics Is it money well spent? What is the impact on our business? What does it cost to achieve the competence shift I need? Was the learning any good? How does that compare to other people in the Unit or Firm? What learning has been completed by my team?
- 12. Scott Schaffer, Learning measurement and analytics, 2017 What gets Measured gets Managed
- 13. Scott Schaffer, Learning measurement and analytics, 2017 Xerox Internal Use Only Learning Measurement & Analytics Maturity What key metrics link learning activities to organization goals, and how will those metrics be captured and reported? DataModern Learning Insights
- 14. Scott Schaffer, Learning measurement and analytics, 2017 Key Global Business Requirements Key Business Requirements List the desired bottom-line business impact(s) Key Drivers Describe the actions that must be taken to drive the Key Business Requirement Key Metrics List the necessary metrics to track the listed Key Drivers. Growth Growth acceleration Sales Profitability Cost efficiency Profit Margins Order-to-Cash Capital efficiency; Cash flow WCD; high-margin sales and cash collection Pricing Transformation Commercial excellence New – Retained Customers Key Global Learning Requirements Key Business Requirements List the desired bottom-line business impact(s) Key Drivers Describe the actions that must be taken to drive the Key Business Requirement Key Metrics List the necessary metrics to track the listed Key Drivers. L&D Business Alignment Competency analysis CCRG L&D Spend Efficiency Cost efficiency Avg $ spend per employee L&D Effectiveness Human capital readiness Success cases; KPI improvement L&D Learning Excellence - Key learning programs Access to right learning assets, JIT, right format Value, Quality
- 15. Scott Schaffer, Learning measurement and analytics, 2017 Data Sources and People Data Description Describe what data is needed to deliver the Key Metrics listed in Section 1 Data Type Describe where the data listed is stored and how it is formatted. Access Describe how the data will be accessed and credentials needed (if applicable) Refresh Rate Describe how “real-time” the data needs to be Prep Needed Describe how the data may need to be prepared, transformed or join with other data described in this table DRIVERS Describe role(s) of those that drive the Key Metrics listed in Section 1. QTY LEADERS Describe role(s) of those that review the Key Metrics listed in Section 1 QTY PREPARERS Describe the role of those that prepare the Key Metrics listed in Section 1. QTY Data – what metrics and measures are associated with each requirement? People – Who is accountable for results? Who will verify data?
- 16. Scott Schaffer, Learning measurement and analytics, 2017 Align Business Requirements and Learning Investments Priority Business Goals-L&D Programs 2016 Actual 2017 Plan 1 Profit: Increase Cost Efficiency by X % Corporate Goal = Order to Cash Business Acumen for Managers 5% 8% 2 Growth: Growth Acceleration of X % Corporate Goal = Sales Negotiation Sample Program 8% 10% 3 Pricing Tr: New / Retained Customers X% Corporate Goal = Consultative Selling Sample Program 10% 20% Adapted from TDRp, Dave Vance
- 17. Scott Schaffer, Learning measurement and analytics, 2017 Outcomes are aligned to business goals and separated into individual contributors to success Business outcome: What are the contributors to success? Increase Working Capital Days by 10% 3% 5% 12% 3% 7% 1 Increase cash flow Increase in customer satisfaction Increase in billed CR Reduce WIP 30% 25% 20% 15% 10% 5% 0% Causal outcome: Increase OTC efficiency by 30% Increase in process efficiencies ILLUSTRATIVE
- 18. Scott Schaffer, Learning measurement and analytics, 2017 Source: Measurement map derived from work of Kaufman and Keller; and R. Smith Learning Program Major Objective • Topic • Topic • Topic • Topic Major Objective • Topic • Topic • Topic Performance Objectives Performance 1 Performance 3 Performance 5 Performance 4 Performance 2 Leading Indicators Improved presentations, calls, proposals Reduced errors, increased volume Business Results Client Satisfaction Productivity Strategic Goals Cost Reduction Gross Margins Individual learning programs are designed to drive business value The impact map identifies links between learning objectives and business objectives From left to right, ask “How do we achieve these objectives?” From right to left, ask “Why are we doing this [intervention or measurement]?” Performance 6
- 19. Scott Schaffer, Learning measurement and analytics, 2017 Impact MaPs Define Targets Increase revenue • <current> • <target> Financial Objectives Strategic Objectives Tactical Objectives Performance Objectives Learning Objectives Increase in new and expanded accounts • Measure – new accounts sales $ Increase profits from all accounts • Measure - increment per account$ Maximize deal value • Cost of remediation Develop partner network • Measure xxxx Expand sales team footprint • Measure xxxx Consultative selling • Measure xxx Better Presentations • Measure xxx Develop accounts • Measure xxx Cross-functional selling • Measure xxx Negotiate better deals • Price discounts • Value of concessions Opportunity › Increase 2016 Sales volume – Q1 2015 = $154.2 million – 6% percent increase over prior quarter – 28 percent increase over 2014 Q1 Ask why? Ask how? Networking Skills • Level 0,1 Sales Negotiation Skills • Level 0,1
- 20. Scott Schaffer, Learning measurement and analytics, 2017 Business Goal Learning Impact @ the Solution Level Impact occurs when: A training-acquired skill [S] is applied to improve performance in some part of a job that produces a Result [R] that contributes to a Business Goal (adapted from R. Brinkerhoff). Learning Intervention S S S S S S S R Individual or Team Behaviors/Performance 20 Skills are applied while in the role… Improved KPIs; Correlations between S & R
- 21. Scott Schaffer, Learning measurement and analytics, 2017 Business Goal The Anatomy of Learning Impact: When Learning works, what happens? Impact occurs when: A training-acquired skill [S] is applied to improve performance in some part of a job that produces a Result [R] that contributes to a Business Goal Learning Intervention S S S S S S S R Individual or Team Job This is the Finish Line
- 22. Scott Schaffer, Learning measurement and analytics, 2017 The Anatomy of Learning Impact: When Learning works, what happens? Impact occurs when: A training-acquired skill [S] is applied to improve performance in some part of a job that produces a Result [R] that contributes to a Business Goal Learning Intervention S S S S S S S R Individual or Team Job This is the Finish Line Not this! Business Goal
- 23. Scott Schaffer, Learning measurement and analytics, 2017 End to end support for L&D that provides reporting, dashboards, and analytic insights into the specific impact of learning on individual performance, business units and the organization. Learning Measurement and Analytics as a service
- 24. Scott Schaffer, Learning measurement and analytics, 2017 1. Completely! We don’t communicate learning impact at the executive level. 2. Somewhat. We measure consumption but don’t have a compelling impact story. 3. We’re Covered. We tie learning metrics to business goals and communicate our success at the executive level. Poll 1: 62% of cLO’s feel Executive Pressure to use Learning Analytics to show impact How exposed are you?
- 25. Scott Schaffer, Learning measurement and analytics, 2017 › The seven levels represent the chain by which learning creates value in large organizations. › Learning is consumed on an individual basis, begins to add value when it is retained, adds further value when the individual takes it back to the workplace, and adds more value when it is integrated into the larger team performance. This process is built from the bottom up. Impact Measurement and the Learning Value Chain Business Outcomes Operational Results Application Learning Learner Corporate Dashboard Usage However, business results are produced when learning interacts with many other factors — factors that are commonly managed from the top down.
- 26. Scott Schaffer, Learning measurement and analytics, 2017 levels of Measurement maturity across Outcomes Traffic, Activity, Smart Sheets, Quizzes, Tests Behaviors, KPIs Correlate behaviors with metrics; causal relationships MeasurementMeasurementMeasurementMeasurementMeasurement Application Consumption Learner Learning Business & Societal Outcomes Operational Results Cost Value Individual Outputs & Outcomes Indiv & Team Outcomes Organizational & Societal Outcomes • Types of Learning Products • Learning Reach • Completions • Time on task • Learner reactions • Reflections • Projects • Cognitive/ Skills Testing • Design Quality • Formative and Summative Assessment • Learning x Learner Measures • Transfer surveys • Transfer Interviews • Focus Groups • Observations • Experiential/ Work-based learning • Checklists KPI Metrics • Quality • Productivity • Customer Satisfaction/ Retention • Margins • Spend Business Measures • Revenue • Profit • Growth Sustainable Development • End extreme poverty • Fight inequality and injustice • Halt climate change • Tech for Good
- 27. Scott Schaffer, Learning measurement and analytics, 2017 Measurement starts with a Research question Traffic, Activity, Smart Sheets, Quizzes, Tests Behaviors, KPIs Correlate behaviors with metrics; causal relationships MeasurementMeasurementMeasurementMeasurementMeasurement Application Consumption Learner Learning Business & Societal Outcomes Operational Results Cost Value Individual Outputs & Outcomes Indiv & Team Outcomes Organizational & Societal Outcomes What are the most popular learning channels? Are the most expensive courses also highly rated? What percentage of people believe the learning experience was a good investment? What percentage of people would recommend this training to a peer How much do we spend on L&D investments? What specific behaviors lead to better sales margins? Did the learning solution transfer to the job? Which employees were most successful when attempting to transfer learning to the job? What enhancers/barriers affect learning transfer? Which learning investments had the highest return? Is learning associated with high performance? What specific learning factors predict increased profits? What role does learning play in sustainability and development goal attainment?
- 28. Scott Schaffer, Learning measurement and analytics, 2017 An example of a Measurement Strategy Phase Product Will be used to measure… Method/Tool Status Analyze Align Plan Impact Map Measurement Plan Alignment of learning investment with business results; intention to measure PC completes/Template PC completes/Template Part of new process Impact maps are part of SCM projects Readiness Checklist Organization, Program, and Learner Readiness PM completes/Checklist Part of new process Engage (L 0,1) Activity/Consumption Completions, activities, log-ins for a particular program ITM, Promote, Domo, Traffic Too slow. ITM reporting unwieldy. Domo could help. Need use cases. End of Course Survey Perceived value, relevance, confidence, intent to apply ITM, Promote, Netigate Awaiting Learning Leadership approval. Needs to be built. Learning (L 2) Survey/Tests/ Projects Learning/competence gain. Formative and summative. ITM, Promote Not enough data to comment. Transfer & Early Impact (L 3,4) Survey, Interview Successes, Unrealized Potential, Non-starters; Barriers; Manager Support ITM, SCM, Netigate PC completes Conducting 4 SCM studies through Q3 Business impact case Evaluation Study Early business impact of the offerings (Qualitative); Learning effect by comparing participant and control groups or trend analysis. Interview transcripts/Email Evaluator consults Domo, Tableau, Excel, This is a Q3 project. Could use SSN or SPIN Predict & Optimize (L 5,6) Prescriptive Predictive Optimization Report Correlations between learning and performance; cause & effect relationships, e.g., increase $ spend predicts increased performance Evaluator consults Domo, Plateau, (Vestrics) Q3 – Q4 project dependent upon previous phase. Need analysis tool.
- 29. Scott Schaffer, Learning measurement and analytics, 2017 Analytics Involves Multiple sources and owners of Data!C › Full accountability and governance process › Simplified data architecture reduces risk of critical events › Cost savings from limited duplication › Easy reconciliation and auditing Single enterprise data warehouse with tools for data visualization and analysis Data Sources: Operational & Learning Platforms Downstream Users Single golden source EDW Formal Blended Informational / Social Extraction tools: Domo, Excel, SPSS, Tableau SuccessFactors Pluralsight AdobeConnect Promote Traffic Kaltura Operation s Finance BW HRMS
- 30. Scott Schaffer, Learning measurement and analytics, 2017 Moving from Data to Reporting to insight is the Essence of the Analytics journey AnalyticsData management HIGH LOW LOW HIGH Degree of Intelligence CompetitiveAdvantage Raw data Clean data Standard reports Ad hoc queries Selective Drill Down Alerts Prescriptive Optimization Predictive Analysis Diagnostics Descriptive Analysis Advanced Analytics Standard Reporting Data Standard Analytics Source: McKinsey & Company
- 31. Scott Schaffer, Learning measurement and analytics, 2017 q Very Mature! We make decisions based on insights gained from evidence based on rigorous data collection. q Mature. We use statistical analyses to analyze data that was collected in a systematic way. q Somewhat. We have standard analytics and self-service reporting in place. q We’re just getting started. We struggle with reporting and data management. Poll 2: Measurement and Analytics Maturity. Where do you fit on the curve?
- 32. Scott Schaffer, Learning measurement and analytics, 2017 Use Cases for Learning Impact questions • Core Dashboards for Function, Regions, Business Units • Learning transfer surveys 60-90 days post How are we doing on key metrics compared to last year? Is Learning impacting performance? • Top Sales Winners have 20% more learning credits peers • Top performers have significantly more learning hours than others Does overall learning activity positively relate to customer satisfaction? Promotability? • High manager engagement is positively associated with better performance What factors differentiate high performers from others?
- 33. Scott Schaffer, Learning measurement and analytics, 2017 Dashboards for all
- 34. Scott Schaffer, Learning measurement and analytics, 2017 Does Learning Consumption correlate with higher performance?
- 35. Scott Schaffer, Learning measurement and analytics, 2017 60-90 days post-workshop survey Identify Success and Non-Success Cases Analyze Data Success Case Method – Identify top performers and What Differentiates Them from others Insights & Recommendations Manager and Employee Interviews Impact Map* Collect KPI Measures (if available)
- 36. Scott Schaffer, Learning measurement and analytics, 2017 Four Impact Measurement Projects were identified as priorities based on their alignment to our business strategy, reach and impact OTC WCD Sales-related Impact Measurement Projects Consultative sales related 1 2 3 4 Prioritized HIGH HIGH MED MED HIGH MED HIGH HIGH
- 37. Scott Schaffer, Learning measurement and analytics, 2017 Learning Program Fundamentals • Sales Process • Ericsson Business • Business Drivers • Customer Focus • PGM/PD Roles Project Management • Key CPM Actions • Changes that Maximize Margins • Assure Customer Acceptance • Decision-making that adds Value • Establishing Billing Milestones Performance Objectives Analyze customer needs Agreement to milestones and fulfillment criteria Identify & negotiate add-on sales /change requests Release billing milestones early Consult & negotiate agreement with customers Leading Indicators Increased cash flow Increase margins on project work Reduced WIP Increased project pipeline Improved billed CR Increased cash flow Business Results Increased revenues per customer Increased ROC Increased WCD Improved CPE Increased customer retention Increased operating cash flow Strategic Goals Profits Growth How (This Global Program) will drive business value Align learning, performance and business objectives From left to right, ask “How do we achieve these objectives?” From right to left, ask “Why are we doing this?” Control project costs
- 38. Scott Schaffer, Learning measurement and analytics, 2017 Success Case Stories What I learned How I applied it Results I have created “Negotiation was the best part for me – tension & concession – give and take – I’m now always doing that! “ “How can I anticipate argumentation on the part of the customer?” “Methods for consciously applying high value to customer, low cost for us.” “It works really well in Renewal – the challenge is customer price erosion – not reducing the opex; they ask for discounts as efficiency is improved.” “I scope clients using value argumentation. What are the needs? Why do they need it? Once we have this information we can quantify the value of the deal.” “We won an OSS/BSS deal… but the customer wanted to renegotiate focused on price. I now had to use negotiables – to give them something.” “I avoided a 10% discount in exchange for training: saved $x5K/year for two years – i.e. $xx130K by providing training worth $20K = deal value of $xxx,000.” “By arguing the numbers: What do you want? What can we give away that wont cost much? These projects have become prioritized and we sacrifice others where we make less $.” “Instead of giving a x % saving – we gave training + x% saving that we in turn negotiated from the PPP/supplier, This was a $xxx K deal saving = $xxK”
- 39. Scott Schaffer, Learning measurement and analytics, 2017 Early Impact: 74% Indicate Learning had Significant-Moderate Impact on Performance *Survey completed by a small sample of mid-low success cases
- 40. Scott Schaffer, Learning measurement and analytics, 2017 What Factors Differentiate Higher Performers from others? Relationship of Manager Engagement to Impact. Copyright Brinkerhoff Evaluation Institute 2016
- 41. Scott Schaffer, Learning measurement and analytics, 2017 Prescriptive Analytics: what investments should we Double Down on and which ones should be Discontinued? 41 • “Evidence of impact” chain links investments to business results • “What-‐if” predic=ve modeling both up and down the chain • Predic=ve modeling ROI to op=mize investment decisions
- 42. Scott Schaffer, Learning measurement and analytics, 2017 March 22 – CLO Week Silicon Valley : Driving L&D Leadership Through Measurement: Shaping the Impact Discussion July 17 – Corporate Learning Analytics, Nashville: Solving for Talent: Developing the role of the learning analytics leader For More on This topic!
- 44. #CLOwebinar Sponsored by Register for the next webinar! One Left Behind: Driving Performance within a Deskless Workforce Tuesday, March 28, 2017 Webinars start at 2 p.m. Eastern / 11 a.m. Pacific Register for all upcoming Chief Learning Officer Webinars at events.clomedia.com/webinars OR click the icon on the widget bar!