Theory-based Learning Analytics
Download as PPTX, PDF6 likes3,925 views
The document discusses the concept of theory-based learning analytics, emphasizing that any analytic is inherently tied to underlying theories and models of data interpretation. It highlights examples from educational institutions such as Open University and English schools, showcasing how predictive and proactive measures based on analytics can enhance student learning outcomes. Additionally, it addresses the importance of sensemaking in complex educational environments, suggesting that analytics design should support effective discourse and collaborative learning.
Theory-based Learning Analytics
- 1. Learning Analytics & Knowledge 2011, Banff, Canada Theory-based Learning Analytics: Notes & Examples from Learning & Sensemaking Simon Buckingham Shum Knowledge Media Institute Open University UK http://simon.buckinghamshum.net http://open.edu @sbskmi http://creativecommons.org/licenses/by-nc/2.0/uk 1
- 2. KMi: near future (3-5 years) infrastructure for Open U, and other knowledge-intensive orgs http://kmi.open.ac.uk
- 3. Theory and Technology at the intersection of Hypertext and Discourse http://projects.kmi.open.ac.uk/hyperdiscourse Projects span many disciplines, ranging from basic research to applications: 3
- 4. theory ~ analytics examples 4
- 5. theory ~ analytics 5
- 6. ? ―theory-based analytics‖ ? 6
- 7. Premise: ANY analytic is an implicit theory of the world, in that it is a model, selecting specific data and claiming it as an adequate proxy for something more complex ―theory-based analytics‖ 7
- 8. So for ―Theory‖, let‘s include assumptions, as well as evidence-based findings, statistical models, instructional methods, as well as more academic ―theories‖ ―theory-based analytics‖ The question is whether this has INTEGRITY as a meaningful indicator, and WHO/WHAT ACTS on this data 8
- 9. A theory might tell you WHAT to attend to as significant/interesting system behaviour. ―theory-based analytics‖ The analytics task is to meaningfully MINE from data, or ELICIT from learners, potential indicators in a computationally tractable form 9
- 10. A mature theory will tell you WHY a given pattern is significant system behaviour. ―theory-based analytics‖ This might help in guiding how to MEANINGFULLY, ETHICALLY PRESENT ANALYTICS to different stakeholders, aware of how they might react to them 10
- 11. A mature theory validated by pedagogical practices will tell you APPROPRIATE INTERVENTIONS to take given particular learner patterns ―theory-based analytics‖ If formalizable, analytics might then be coupled with recommendation engines or adaptive system behaviours 11
- 12. A theory can shed new light on familiar data ―theory-based analytics‖ This might equate to reinterpreting generic web analytics through a learning lens 12
- 13. A theory might also predict future patterns based on a causal model ―theory-based analytics‖ This might be formalizable as a predictive statistical model, or an algorithm in a rec-engine 13
- 14. Example RAISEonline: learning analytics in English schools 14
- 15. RAISEonline (English schools) Literacy, Numeracy & Science can be measured in controlled, written examinations, designed by experts for the whole country ―theory-based analytics‖ Analytics can be generated from test scores in order to meaningfully compare schools in national league tables, accounting for different contexts 15
- 16. Learning analytics in English schools 16
- 17. Learning analytics in English schools 17
- 18. Example Open U‘s analytics 18
- 19. OU Analytics Years of data reveal significant patterns between student study history, demographics and outcomes ―theory-based analytics‖ We can formalize these, eg. as organisational good practices around student support, or as statistical models to reflect on actual vs predicted course outcomes 19
- 20. OU Analytics service: Effective Interventions Proactive measures targeted at specific points in the student journey are associated with improved retention and progression Telephone contact with students considered to be potentially „at risk‟ before the start of their first course is associated with around a 5% improved likelihood of course completion. Additional tutor contact mid-way through a course is associated with between 15% to 30% improved likelihood of course completion. Additional tutor contact around course results is associated with between 10% to 25% improved likelihood of registering for a further course. Contact with students intending to withdraw before course start is associated with retaining 4% of students on their current course. OU Student Support Review 20
- 21. OU Analytics service: Predictive modelling Probability models help us to identify patterns of success that vary between: student groups areas of curriculum study methods Previous OU study data – quantity and results – are the best predictors of future success The results provide a more robust comparison of module pass rates and support the institution in identifying aspects of good performance that can be shared and aspects where improvement could be realised OU Student Statistics & Surveys Team, Institute of Educational Technology 21
- 22. Example sensemaking in complex problems 22
- 23. It‘s all about Sensemaking ―Sensemaking is about such things as placement of items into frameworks, comprehending, redressing surprise, constructing meaning, interacting in pursuit of mutual understanding, and patterning.‖ Karl Weick, 1995, p.6 Sensemaking in Organizations
- 24. Sensemaking in complexity Individual and collective cognition has known limitations in dealing with complexity and information overload ―theory-based analytics‖ Collective intelligence tools and analytics should be designed specifically to minimise breakdowns in sensemaking 24
- 25. Sensemaking in complexity Complexity ~ sensemaking Focus for analytics/rec-engines? phenomena Risk of entrained thinking from experts who fail • Pay particular attention to exception cases to recognise a novel phenomenon • Scaffold critical debate between contrasting perspectives on common issues Complex systems only seem to make sense • Use narrative theory to detect and analyse retrospectively knowledge-sharing/interpretive stories • Coherent pathways through the data ocean are important Much of the relevant knowledge is tacit, shared • Scaffold the formation and maintenance of through discourse, not formal codifications quality learning relationships Many small signals can build over time into a • Highlight important events and connections significant force/change aggregation and emergent patterns • Recommend based on different kinds of significant conceptual/social connections Buckingham Shum, S. and De Liddo, A. (2010). Collective intelligence for OER sustainability. In: OpenEd2010: 25 Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Eprint: http://oro.open.ac.uk/23352
- 26. Sensemaking in complexity Complexity ~ sensemaking Focus for analytics/rec-engines? phenomena Risk of entrained thinking from experts who fail • Pay particular attention to exception cases to recognise a novel phenomenon • Scaffold critical debate between contrasting perspectives on common issues Complex systems only seem to make sense • Use narrative theory to detect and analyse retrospectively knowledge-sharing/interpretive stories • Coherent pathways through the data ocean are important Much of the relevant knowledge is tacit, shared • Scaffold the formation and maintenance of through discourse, not formal codifications quality learning relationships Many small signals can build over time into a • Highlight important events and connections significant force/change aggregation and emergent patterns • Recommend based on different kinds of significant conceptual/social connections Buckingham Shum, S. and De Liddo, A. (2010). Collective intelligence for OER sustainability. In: OpenEd2010: 26 Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Eprint: http://oro.open.ac.uk/23352
- 27. Sensemaking in complexity Complexity ~ sensemaking Focus for analytics/rec-engines? phenomena Risk of entrained thinking from experts who fail • Pay particular attention to exception cases to recognise a novel phenomenon • Scaffold critical debate between contrasting perspectives on common issues Complex systems only seem to make sense • Use narrative theory to detect and analyse retrospectively knowledge-sharing/interpretive stories • Coherent pathways through the data ocean are important Much of the relevant knowledge is tacit, shared • Scaffold the formation and maintenance of through discourse, not formal codifications, and quality learning relationships trust is key to flexible sensemaking when the environment changes Many small signals can build over time into a • Highlight important events and connections significant force/change aggregation and emergent patterns • Recommend based on different kinds of significant conceptual/social connections Buckingham Shum, S. and De Liddo, A. (2010). Collective intelligence for OER sustainability. In: OpenEd2010: 27 Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Eprint: http://oro.open.ac.uk/23352
- 28. Sensemaking in complexity Complexity ~ sensemaking Focus for analytics/rec-engines? phenomena Risk of entrained thinking from experts who fail • Pay particular attention to exception cases to recognise a novel phenomenon • Scaffold critical debate between contrasting perspectives on common issues Complex systems only seem to make sense • Use narrative theory to detect and analyse retrospectively knowledge-sharing/interpretive stories • Coherent pathways through the data ocean are important Much of the relevant knowledge is tacit, shared • Scaffold the formation and maintenance of through discourse, not formal codifications quality learning relationships Many small signals can build over time into a • Highlight important events and connections significant force/change aggregation and emergent patterns • Recommend based on different kinds of significant conceptual/social connections Buckingham Shum, S. and De Liddo, A. (2010). Collective intelligence for OER sustainability. In: OpenEd2010: 28 Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Eprint: http://oro.open.ac.uk/23352
- 30. ―We are preparing students for jobs that do not exist yet, that will use technologies that have not been invented yet, in order to solve problems that are not even problems yet.‖ ―Shift Happens‖ http://shifthappens.wikispaces.com 30
- 31. Learning Power There is a set of generic dispositions and skills characteristic of good learners. If learners can be taught a language for these, they can get better at ‗learning-to-learn‘ across many contexts ―theory-based analytics‖ Analytics can be generated from a self-report questionnaire, validated through good mentoring — and possibly from online behaviour, demonstrating improvements in L2L 31
- 32. Learning to Learn: 7 Dimensions of ―Learning Power‖ Expert interviews + factor analysis from literature meta-review: identified seven dimensions of effective “learning power”, since validated empirically with learners at many stages, ages and cultures (Deakin Crick, Broadfoot and Claxton, 2004) Being Stuck & Static Changing & Learning Data Accumulation Meaning Making Passivity Critical Curiosity Being Rule Bound Creativity Isolation & Dependence Learning Relationships Being Robotic Strategic Awareness Fragility & Dependence Resilience www.vitalhub.net/index.php?id=8
- 33. Learning to Learn: 7 Dimensions of Learning Power www.vitalhub.net/index.php?id=8
- 34. Learning to Learn: 7 Dimensions of Learning Power www.vitalhub.net/index.php?id=8
- 35. Analytics tuned to Learning Power: ELLI ELLI: Effective Lifelong Learning Inventory (Ruth Deakin Crick, U. Bristol) A web questionnaire generates a spider diagram summarising the learner‟s self- perception: the basis for a mentored discussion and interventions Changing and learning Critical Learning Curiosity relationships Meaning Making Strategic Awareness Creativity Resilience 35 Professional development in schools, colleges and business: ViTaL: http://www.vitalhub.net/vp_research-elli.htm
- 36. Analytics tuned to Learning Power: EnquiryBlogger (National Learning Futures programme: learningfutures.org) Wordpress multisite plugins tuning it for learning-to-learn dispositions and enquiry skills http://people.kmi.open.ac.uk/sbs/2011/01/digital-support-for-authentic-enquiry 36
- 37. 37
- 38. EnquiryBloggers with teacher status can view their cohorts plugins in their Dashboard 38
- 39. Example discourse analytics 39
- 40. Discourse analytics Effective learning conversations display some typical characteristics which learners can and should be helped to master ―theory-based analytics‖ Learners‘ written, online conversations can be analysed computationally for patterns signifying weaker and stronger forms of contribution 40
- 41. Socio-cultural discourse analysis (Mercer et al, OU) • Disputational talk, characterised by disagreement and individualised decision making. • Cumulative talk, in which speakers build positively but uncritically on what the others have said. • Exploratory talk, in which partners engage critically but constructively with each other's ideas. Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social mode of thinking. Journal of Applied Linguistics, 1(2), 137-168. 41
- 42. Socio-cultural discourse analysis (Mercer et al, OU) • Exploratory talk, in which partners engage critically but constructively with each other's ideas. • Statements and suggestions are offered for joint consideration. • These may be challenged and counter-challenged, but challenges are justified and alternative hypotheses are offered. • Partners all actively participate and opinions are sought and considered before decisions are jointly made. • Compared with the other two types, in Exploratory talk knowledge is made more publicly accountable and reasoning is more visible in the talk. Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social mode of thinking. Journal of Applied Linguistics, 1(2), 137-168. 42
- 43. Analytics for identifying Exploratory talk Elluminate sessions can It would be useful if we could be very long – lasting for identify where learning seems to hours or even covering be taking place, so we can days of a conference recommend those sessions, and not have to sit through online chat about virtual biscuits Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within 43 Synchronous Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
- 44. Analytics for identifying Exploratory talk Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within 44 Synchronous Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
- 45. Discourse analysis with Xerox Incremental Parser (XIP) Detection of salient sentences based on rhetorical markers: BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION: Recent studies indicate … ... new insights provide direct … little is known … evidence ... … role … has been elusive … the previously proposed … ... we suggest a new ... approach ... Current data is insufficient … … is universally accepted ... ... results define a novel role ... CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING: … unorthodox view resolves … studies ... have provided The goal of this study ... paradoxes … important advances Here, we show ... In contrast with previous Knowledge ... is crucial for ... Altogether, our results ... hypotheses ... understanding indicate ... inconsistent with past findings valuable information ... from ... studies GENERALIZING: SURPRISE: ... emerging as a promising We have recently observed ... approach surprisingly Ágnes Sándor & OLnet Project: Our understanding ... has grown We have identified ... unusual http://olnet.org/node/512 exponentially ... The recent discovery ... suggests ... growing recognition of the intriguing roles importance ...
- 46. Human and machine annotation of literature Document 1 19 sentences annotated 22 sentences annotated 11 sentences = human annotation Document 2 71 sentences annotated 59 sentences annotated 42 sentences = human annotation Ágnes Sándor & OLnet Project: http://olnet.org/node/512
- 47. Analyst-defined visual connection language De Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf. 47 Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
- 48. — discourse-centric analytics De Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
- 49. — discourse-centric analytics Does the learner compare his/her own ideas to that of peers, and if so, in what ways? De Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
- 50. — discourse-centric analytics Does the learner act as a broker, connecting the ideas of his/her peers, and if so, in what ways? De Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
- 51. LAK 2015?... New learning theories have emerged which are dependent on the enabling capabilities of online analytics to provide data on a huge scale… ―theory-based analytics‖ While in tandem, new categories of learning analytic and rec-engine have emerged in specific response to the need to test emerging theories with computational models and datasets 51
Editor's Notes
- #36: Finally, the ELLI approach has now developed a body of practitioner knowledge about the kinds of interventions that can be made to help stretch a learner in different dimensions. This includes knowledge of which dimensions tend to be most important or dependent. This know-how may be formalisable as patterns that can help an ELLI mentor advise, or help a learner to decide what to do next. These ELLI profiles can of course also be be embedded in learner profiles in order to match with other similar (or dissimilar) peers.