Future Learn Academic Networks 02/12/15

Editor's Notes

• #2: Hi, I’m Tim O’Riordan. I’m a member of WAIS at the University of Southampton and I’m talking about the digital tools I use to support my Learning Analytics research. I carry out content analysis of MOOC comment data using a range of methods – and use a number of tools to help to rate, explore, look for correlations and visualise my findings.
• #3: Why am I interested in this? Finding and evaluating online learning resources is a significant hurdle to overcome - We know that striking online gold can be a hit or miss operation. Web objects are rarely effectively tagged and people don’t know the pedagogical usefulness of what they upload to the Web. Maybe there’s some automatic way that can help?
• #4: Learning Analytics analyses and makes visible the trace data that people leave behind on the web in order to understand how people learn – and there is evidence of links between positive sentiment , ‘creative capacity’ and betweeness centrality, and knowledge acquisition in discussion forums.   But it’s recognised that methods are inaccurate, need refinement – and don’t really get to the heart of the matter – that is, are people actually engaged in learning?
• #5: So I’ve been looking a pedagogical frameworks, toolkits and methods that help educators design learning interventions to see if they can help spot evidence of pedagogical activity in the comments people make about things online. Can pedagogical content analysis identify this activity in a reliable way, and can this be used to design and automatic detection system?
• #6: So, I looked at applying a pedagogical approach to content analysis of computer mediated communication to identify levels of engagement with online objects. It turned out I was alone in doing this and that many studies had been using Bloom and Community of Inquiry methods to explore CMC.
• #7: First lot of data - 6 week Portus MOOC. > 1,800 ‘social’ learners - 20,000 comments containing 1,000,000 words were made available 600 comments from 12 steps to 4 rating methods, categorised and scored learning activity.
• #8: So – that was archaeology. But do people write the same way in other disciplines? And do other people apply the rating methods in similar ways? To find out I’ve been looking at 3 other MOOCs which focus on language learning, oceans and contract management. Or rather I’ve employed 8 research assistants to rate a selection of about 1500 comments according to both Bloom and CoI.
• #9: Looking at the digital tools I’ve been using or trying to use.
• #10: MS stuff is provided free by the University, and comment data is provided in csv format – so excel is the weapon of choice for early data cleaning and number crunching. So – lots of sum average and mode calculations - and I’ve also used it to calculate Euclidean distance and to create network graphs using NodeXL. Currently my main workstation is a mac – so using nodexl is a problem. I have yet to get the alternative – Gephi working.
• #11: Excel is useful in preparing data for analysis elsewhere. So, all 1500 comments in my current study have to be exported from excel cells into individual text files. Excel’s developer tab facilitates this with a nice bit of visual basic code.
• #12: The resultant text files can then be analyses in NVIVO and other software. Initially I used Nvivo to find patterns in Word frequency in my data. And I did find patterns. Needed other tools to help look for significant patterns. Supplied by the University – so no cost.
• #13: The Linguistic Inquiry and Word Count application is useful for – basically counting words and word types, based on studies into the ‘psychological meaning’ of words. It counts the occurrence of over 80 word text characteristic, including long words, pronouns, preposition, positive and negative sentiment. It works well enough and has enabled me to show strong correlations between word types associated with cognition and the rating methods I’ve used. Basically to show that there may well be strong patterns in how people write that indicate depth of knowledge. Chief limitation is that it only really works on texts containing more than 100 words. Unless you aggregate there are problems dealing with the brief statements that are common in CMC. This is an inexpensive tool with an academic discount and free upgrades.
• #14: The Linguistic Inquiry and Word Count application is useful for – basically counting words and word types, based on studies into the ‘psychological meaning’ of words. It counts the occurrence of over 80 word text characteristic, including long words, pronouns, preposition, positive and negative sentiment. It works well enough and has enabled me to show strong correlations between word types associated with cognition and the rating methods I’ve used. Basically to show that there may well be strong patterns in how people write that indicate depth of knowledge. Chief limitation is that it only really works on texts containing more than 100 words. Unless you aggregate there are problems dealing with the brief statements that are common in CMC. This is an inexpensive tool with an academic discount and free upgrades.
• #15: As I was taking on research assistants to help with rating comments – I needed to train them. Socrative is a free online audience response tool that enables you to create polls and provides excel spread sheets containing results. I used this to ask my raters to consider comments then vote on how they would grade them. We could then see how everyone had voted and have a discussion on how decisions where taken. I did this for all categories of Bloom and CoI - and ran the results through SPSS reliability tests. This provided an early indication of how people where applying the content analysis methods and led me do a short test run with just 60 comments to see how the raters performed on their own. This test provided results that gave me more confidence in reliability.
• #16: This tool is paid for by the university. I initially used SPSS Scatter plot and linear regression functions to look for correlations. As content analysis requires tests of reliability I’ve now started to use Reliability Analysis (ICC) and Krippendorff’s α tools. Very useful tool – but could do better on visualisation – which is why I use…
• #17: Tableau to create Scatter plot and lines of regression with nice bold shapes and colours. This is a result from the research I did on the Portus MOOC showing strong, significant correlation between my coding of comments using Bloom and Community of Inquiry. This is a free tool for students.
• #18: I’ve just started with this free Machine learning work bench will probably use some form of Naïve Bayes algorithm. However, I’ve immediately run into trouble, by failing to get data into arff format using the simple command line. But – and this is important for questions about what works and what doesn’t…
• #19: A big positive aspect of working with these tools is that I’m part of a lab with people who support and encourage my work – and have much more experience than I do in them. For example Adrianna is using Weka and has offered to help my over this early bump.
• #20: So – to sum up… I should re-iterate that while my work has benefitted from access to MOOC data, my research isn’t all about MOOCs. The idea is that the outcomes will be applicable all over the web. Regarding tools – with the current exception of Weka - they all work for me and are really helpful in moving my research forward BUT It’s very easy in this type of analysis to get lost in the process – so I also need to talk to real people about what it all means – does any of this relate to people actual experience of engaging in web based environments?
• #21: http://eprints.soton.ac.uk/376609/