Digital Learning Sample Metrics
Download as PPTX, PDF1 like255 views
The document discusses metrics for measuring engagement and success in digital learning. It provides examples of different types of metrics to capture at various stages, from awareness through application. These include metrics on involvement, interaction, intimacy, influence, content consumption, social engagement, and real-world application. Tracking the right metrics at each stage can provide insights into how to improve the learning experience and better meet learner needs.
Digital Learning Sample Metrics
- 1. Digital Learning Sample Metrics What you should be looking for to know if you’ve created something truly engaging.
- 2. It’s not just about completions. It’s about so much more. The new ecosystem you are creating will give you access to much more data than you’re used to from your traditional LMS. With a proper measurement strategy, you’ll have a much better understanding of your users and will be able to analyze needs faster, as well as make better content and design decisions. Remember, every organization has different needs and the ecosystems you create will be different. You may not be able to capture all of the information listed in these examples, but having an idea of what measurements you will need early on will help you make more informed decisions going forward.
- 3. 3 Past and Present As you lead your organization through this digital transformation, another thing to keep in mind is how you measure success. How we measure success today is different than we used to in measure it in the past, so it’s important to adjust your mindset for that. Here’s a quick reminder. Physical presence in class Completed an eLearning (clicked to the end) Created an action plan to use on the job after an ILT Passed a Quiz Improvement in scores from Pre and Post tests How we used to measure learning How we measure learning now Engagement with content and peers Meeting completion criteria (points, badges, etc.) Did users demonstrate evidence of on the job application during the experience? Were users able to generate and share personal insights from the content? Did users create a working job deliverable?
- 4. Use Web Analytics Involvement, Interaction, Intimacy and Influence define “engagement” in Web Analytics, and translate well for engagement in digital experiences. Category Definition Metric Involvement Measurable aspects of an individual learner’s connection to the digital experience • Frequency of visits • Average time of session Interaction Depth / breadth of learner connection to the digital experience • Content page views per visit • Number of pieces of content accessed • Return rate per piece of content • Search volume Intimacy Direct sentiment of a learner towards the digital experience • Net promoter score • Feedback collected Influence Indicators that a learner will encourage the learning of others • Discussion forum participation rate • Public posting of assignments • Number of “likes” given to content
- 5. Learner Activity Another way to look at measuring engagement is to follow the flow of learner activity and measure every step along the way. Invitation to join/register (email) Acceptance/ Registration action completed (email/other system) Course launch email sent/ auto enrollment link sent Learner accesses site/class and logs in Learner clicks on content but doesn’t complete it Learner completes at least one content item Learner accesses multiple types of content Learner interacts with others (discussions, projects, feedback, support) Learner meets completion criteria Learner applies on the job Awareness andInterest Consumption Social Completion Application
- 6. Awareness and Interest Here are some things you can track, and what you can do with that information. What to Track Why track it Total # of logins How many people have logged in vs. the total population? This could be helpful in seeing if your communications or marketing campaign was effective. If the logins are low, perhaps you need a different round of communications, or a high profile leader to push your initiative so that it reaches more people. Think of new ways to attract people. # of logins per day / avg. length of visit How often are users logging in? Multiple times a day? Once a week? Once a month? How long are they staying once they are there? This type of information can be helpful in multiple ways. If people aren’t logging in, why? Poor communication or marketing of the program? For users accessing multiple times per day, what are they using it for? Time of day for peak usage Being able to see the time of day that users access content the most can be a powerful tool. If there is always a spike in usage around 10am, then that may be the best time to release new content. It will get the most eyes on it and generate more engagement that is likely to hold it over longer for others to see it later when they log in.
- 7. Consumption Metrics Here are some things you can track, and what you can do with that information. What to Track Why track it Most viewed or completed content Knowing what content is getting the most views is important data for your larger data story. What’s the topic of this content? What type of content is it (article, video, etc)? Were users required to complete this content or was it organic? It’s difficult to use required content as a look into engagement. If you’re forcing users to complete something, then you’re not measuring their actual interest in the content, and that could lead you down the wrong path. Likes, dislikes, comments, and shares This will give you an idea of how well a content is being received. Likes and dislikes will give you a quick look at perception of the videos. Compare these with the # of views and see what the % of views vs likes is. A high like % is most likely telling you that content is what users need for that topic. Then explore the comments to see if you uncover additional needs. “This was great. I’d love to see more about … too.” or “I wish this would have talked about ...” Type of content What types of content are your using viewing the most? Videos, articles, podcasts, links, etc? Identifying trends like this can tell you the types of content your users prefer most and you can cater to those preferences as you curate or development new content.
- 8. Social Metrics Here are some things you can track, and what you can do with that information. What to Track Why track it How deeply are people interacting with each other? We know how many views a piece of content has, but let’s go one layer deeper. Are people engaging and interacting with each other? How many posts are there on a topic? How many people are replying to other comments (actual conversations)? How many projects have been submitted? How many peer reviews have been completed?
- 9. Application Metrics Here are some things you can track, and what you can do with that information. What to Track Why track it How are individual learners actually changing their behavior on the job? Before course launch, identify behaviors that you want to see changes in. How is the class as a whole changing their output as a result of this course? Identify a deliverable learners should be able to create on the job? Design the program around step by step creation of the deliverable and allow for peer to peer learning and feedback.
- 10. The stories are in the data. This is only a small sample of things to measure but you can see how individual data points can give you valuable insight. What’s even more powerful is when you combine the data. Don’t just look at one piece, compare several pieces to get the full story. See appendix for some high impact stories, and visit the Intrepid by VitalSource Learning Library to find more detailed case studies.
- 11. Situation • Consumer goods organization transforming from a holding company to an operating company • Goal: cascade the core components of in-person and virtual leadership program delivered to top executives to the entire company Solution • Blended approach embedding Harvard Business Publishing content in the Intrepid platform with client context and applied learning • Combination of live webinars and self-directed learning paths with recorded videos, Harvard content, discussion threads, and individual and group missions Results • Very high satisfaction, program impact, and applicability to job ratings (4.5/5, higher than traditional classroom training historically offered) from learners and strong support for the blended learning approach from the CEO • Program participant: “Once everyone is through the program and has engaged in the content we will operate as a ‘well-oiled machine’.” • Learners report the platform activities made the content come alive and made the company- specific context clear • 800 participants through the program globally across roles in 8 languages • Program has won 2 CLO Learning in Practice Gold Awards and 2 Brandon Hall Excellence in Learning Awards Story: Company Wide Leadership Development
- 12. Situation • Highly distributed global locations • Rigorous, 18-week role-based training • 100% in-person training model • Large team of skilled facilitators required to travel constantly to locations Solution • Blended learning model enables new hires to learn at their own pace & connect regularly with facilitator for VILT events. Includes role-specific training. Results • ~10,000 new hires onboarded • 15%+ reduction in onboarding cycle time for call center new hires. Thousands of hours of productive time added to the business. • Increased instructor productivity – reduce teaching time & over-time for admin; increase course prep & development time. • Increased training availability – now available world-wide regardless of new hire numbers or geographic location. • Accelerated adoption across multiple roles. Story: Health Insurance Call Center & Operations Onboarding
- 13. Situation • Enterprise PLM software rollout and change management strategy to drive efficiency and throughput across the organization. • Training required for 70+ roles • Initial approach was an 18 month software rollout/migration and change management plan that included 50+ ILT events tailored for the unique roles – at massive time, expense and opportunity cost. Solution • Blended learning online + classroom approach of 34 courses delivered to 8,000 learners in 4 countries and in 2 languages. “Tell Me, Show Me, Let Me, Help Me” format guides self-paced training with same training site used for ongoing performance support. Applied PLM practice through real-world assignments. Results • PLM software is live and learners are using the site for just-in-time performance support. The PLM training program received positive participant and leadership reaction. VP of Customer Service: “This is the best training and only approach we should use ongoing.” Story: Aerospace Manufacturing Software Rollout
- 14. Situation • Firm sought to broaden partner awareness and knowledge related to top strategic themes impacting their global clients • Partners traditionally met every other year for development • Most sophisticated, time-starved, senior-level executives in firm Solution • Developed 10 MOOC-like courses on strategic themes as pre-work leading up to 2 in-person global events • 1600 learners; ~ 5 hours/course Results • 600 client conversations • 1100 partners took 2+ courses • Vast majority of senior partners rated online learning “very effective preparation” for live learning events • Based on results, program being rolled out to broader audience Story: Management Consulting Senior Partner Development
- 15. Situation • 2 year program to accelerate leadership development of entry-level talent capable of meeting challenges of global supply chain operations. Longitudinal program includes 3 eight- month rotations. • Global program needed an integrated learning experience that supports the three rotations. Solution • Developed two 3-month MOOCs that enables collaboration and real-world application for new cohort. Phase 1 focused on Onboarding and Phase 2 focused on supply chain. New MOOC being developed for 3nd rotation on leadership and a ‘refresh’ of first MOOC for winter new hires. Results • 85% of participants earned 80% of possible points • 35% of participants completed all content and earned extra credit • 43,300 unique content views • Global Program Manager: “We have been so happy with the design and performance of the Intrepid platform. The discussion forums have been the most powerful aspect, and I love the simplicity and openness of the format—our high-potential leaders are able to have great conversations amongst themselves easily.. It’s so simple to use but so powerful for our learners, offering them a dynamic experience that really increases the impact of our program. It’s invaluable to our company’s future.” Story: Continuous Learning Journey for Leadership