Data Fluency for Dummies

Editor's Notes

• #4: Why is there so much confusion over what data science is? Hype cycle VCs and startups! (data science in general a few years back, now AI, Deep Learning and ML) The problem is that it’s become a catch all for a fast growing set of competencies.
• #5: You could spend a lifetime trying to find a standard definition of data science. Try it for yourself on Google, Quora, or even at a bar with a bunch of data scientists I like to ask companies and data science leaders how THEY define it when I’m going to work with them. There are some common themes though A combination of business analysis, computer science, statistics, and general mathematics A growing segment of data science is veering into specific artificial intelligence skill sets as well
• #6: Data Scientists are people with some mix of coding and statistical skills who work on making data useful in various ways. In my world, there are two main types: Type A Data Scientist:  The A is for Analysis. This type is primarily concerned with making sense of data or working with it in a fairly static way… Type B Data Scientist:  The B is for Building. Type B Data Scientists share some statistical background with Type A, but they are also very strong coders and may be trained software engineers.
• #9: You can think of learning analytics from the perspectives of a particular learning experience, of a learner of group of learners, or of a learning program where a group of learners work through a collection of resources and experiences. Within each of those categories, you can do anything from simple measurement of what’s happening right up to complex predictive and prescriptive analytics. This can also be seen as a maturity model and we encourage people to start with measurement in just one category and get value from that, before expanding into other categories or moving outwards in the triangle. So how do you get started?
• #11: Then thinking about complexity, here’s some of the questions you should ask at each level. Notice how they build on one another, so you can’t just dive straight in to predictive and prescriptive; you have to start with measurement. I’ll give you a moment to read those questions. [PAUSE for 30 seconds] Also bear in mind that each of these questions could be something that’s answered automatically by analytics software, but they could also be questions that are answered outside of the software. For example, your analytics software might tell you what people are doing and how that compares to benchmarks, and then you do more qualitative research of talking to the people involved and reflecting to cover the higher levels of analysis.
• #12: Here’s an example of moving through the levels of complexity around the length of videos. In this example, you could imagine reporting on video length, number of times each video is watched and the point at which people drop off watching the video. Again, I’ll give you a moment to read that example yourselves. [PAUSE for 30 seconds]
• #13: So when you’re designing learning, whether that’s a single course or a whole program, start with a business goal in mind. Then, working with your subject matter experts, identify what people need to do differently or better to meet that goal. Then identify what they need to learn and the training and resources required for them to learn that. As you can see from the bottom half of the slide, designing the learning in this way gives you clear measures you can report on at each stage of your logic from the training to the business goal.
• #15: In medical training, evaluation of training effectiveness is especially important. The real-world tasks clinicians are being trained to perform are high stakes and need to be done right first time. Clinicians can’t practice on real patients while they figure it all out; to do so would put lives at risk. Instead, they must reach competency in the training environment, before applying that training in the real world. Not only do the clinicians have to be competent, but MedStar needs to be confident in their competence. Assessments need to be rigorous and effective so MedStar knows which clinicians are ready. Another unique aspect of medical training is that the skills learnt relate to very practical and physical tasks. Training and assessments, therefore, also need to be practical and physical. While it is important to understand the theory of how the heart works, what really matters is to have the skills to perform the tasks needed to resuscitate the patient. 
• #16: Here we see an example of how we can learn more from the data by asking further questions of it. The top row shows examples of questions that straightforward measurements can answer. To continue going further to evaluate the learning we need to ask questions that help us understand what are data describes. 
• #17: A measurement and evaluation example: This dashboard is constructed to display activity measurements of Instructor Led Training in a physical classroom environment. If we focus on the chart on the left hand side, we can initially see the measurement of the NPS scores for 5 different courses. By applying a benchmark (identified by the broken horizontal line), we’re able to add context to this data so we can evaluate it. We now know whether what we are looking at is good or bad. Looking at this chart, it would be reasonable to investigate further into Courses 3 and 5 to learn why people are becoming increasingly dissatisfied with them.
• #18: Another measurement vs evaluation example: This line chart displays data measuring the revenue generated by two different groups of sales associates: those who have completed a particular sales training course and those who have not. At first glance, it’s easy to think that the training was ineffective because the green group, who did not complete the training, outperform the blue group.
• #19: However, in order to truly evaluate the effectiveness, we need to understand the complete context. If we know that the training was only completed by the Blue Group in May, and we show that they were already underperforming, we can demonstrate with confidence that the training was effective at bringing the underperforming group up to parity with their peers. In this example, the product in question was discontinued at the end of August, hence the large drop off.
• #23: An example – Use your data to see what leaners on the platform are actively looking for. In this example (again, Degreed but we can generalize) you can see the growth in what learners are searching for year over year. Is everyone aware of these trends? Are we making sure to keep up with them as far as content, training, etc?
• #25: In Degreed, we found that the majority of the content learner’s are using comes from “open” sources (free from across the internet). It’s not just about formal learning – we can use this insight to help learner’s find good content from all over. We’d never know if we didn’t look into thedata
• #26: An example on recommendations We can also use data to better understand how users learn as well as to improve the learning product itself. This is the goal of many Data Science teams. At Degreed we use data to feed back into the recommendation models, but also to see why certain recommendations do better than others. In this example, content that ends up in a learner’s feed (the home screen on Degreed) are far more likely to be clicked on if they made the feed due to a social connection of the learner. Average CTRs LinkedIn 0.025% https://www.clickz.com/what-is-a-good-click-through-rate-for-ppc/44829/ Facebook 0.090% http://www.wordstream.com/blog/ws/2017/02/28/facebook-advertising-benchmarks Twitter 1.64% https://www.signupto.com/news/permission-marketing/twitter-marketing-what-results-should-you-expect-infographic/ Google 1.91% (intent) http://www.wordstream.com/blog/ws/2016/02/29/google-adwords-industry-benchmarks
• #28: An example from Degreed data, but we can generalize – Compare an the skill level of employees in your organization to an industry or set of competitors. Track these over time, not just once! Can talk through data collection on basic visualization in an Excel chart