SAAIR: Implementing learning analytics at scale in an online world: lessons learned from the Open University UK
1 like161 views
The document discusses the implementation of learning analytics at scale in the online educational context by the Open University UK, highlighting key insights from an open access book that integrates research on educational technologies and high-quality education. It emphasizes the importance of management support, faculty engagement, and evidence-based practices for successful innovation in learning analytics. Additionally, it addresses the ethical considerations and ownership of data in learning analytics.
SAAIR: Implementing learning analytics at scale in an online world: lessons learned from the Open University UK
- 1. @DrBartRienties Professor of Learning Analytics All papers referred to in this presentation can be accessed via https://iet.open.ac.uk/people/bart.rienties Implementing learning analytics at scale in an online world: lessons learned from the Open University UK
- 4. Now freely Available • This open access book provides state-of-the-art contemporary research insights into key applications and processes in open world learning. • It explores how the application of open world and educational technologies can be used to create opportunities for open and high- quality education. • Presenting ground-breaking research from an award winning Leverhulme doctoral training programme, the book provides several integrated and cohesive perspectives of the affordances and limitations of open world learning. • The book brings together a range of innovative uses of technology and practice in open world learning from 387,134 learners and educators learning and working in 136 unique learning contexts across the globe • Free to attend online event on Wednesday 9 February 2022 1400-1600, Open World Learning, Institute of Educational Technology, The Open University. • https://www.routledge.com/Open-World-Learning-Research-Innovation- and-the-Challenges-of-High-Quality/Rienties-Hampel-Scanlon- Whitelock/p/book/9781032010915
- 5. What we have learned in 10 years of implementing learning analytics and learning design at the OU Change is slow, but can be enhanced with: 1. Clear senior management support 2. Bottom-up support from teachers and researchers who are willing to take a risk 3. Evidence-based research can gradually change perspectives and narratives 4. You quickly forget about the small/medium/large successes and fail to realise that you are making a real impact 5. Large-scale innovation takes substantial time and effort 6. It is all about people… Rienties., B. (2022). Implementing learning analytics at scale in an online world: lessons learned from the Open University UK. Liebowitz, J. (Ed). Online learning analytics. Taylor & Francis.
- 6. Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of Educational Technology & Society, 15(3), 58-76.
- 7. Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of Educational Technology & Society, 15(3), 58-76.
- 10. Leading global distance learning, delivering high-quality education to anyone, anywhere, anytime The Open University Largest University in Europe No formal entry requirements enter with one A-level or less 33% 38% of part-time undergraduates taught by OU in UK 173,927 formal students 55% of students are 'disadvantaged' FTSE 100 have sponsored staff on OU courses in 2017/8 60% 66% of new undergraduates are 25+ 1,300 Open University students has a disability (23,630) 1 in 8 Students are already in work 3 in 4 employers use OU learning solutions to develop workforce
- 12. Predictive learning analytics (since 2013) Learning Design (since 2005) Rienties., B. (2022). Implementing learning analytics at scale in an online world: lessons learned from the Open University UK. Liebowitz, J. (Ed). Online learning analytics. Taylor & Francis. Rienties, B., Herodotou, C. (2022). Making sense of learning data at scale. Sharpe, R., Bennett, S., Varga-Atkins, T. (Eds). Handbook for Digital Higher Education. Edward Elgar Publishing
- 13. Predictive analytics to identify whether students are going to make the next assignment Kuzilek, J., Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU Analyse: analysing at-risk students at The Open University LACE Learning Analytics Review (Vol. LAK15-1). Milton Keynes: Open University. Kuzilek, J., Hlosta, M., & Zdrahal, Z. (2017). Open University Learning Analytics dataset. Scientific Data, 4, 170171. doi: 10.1038/sdata.2017.171 Wolff, A., Zdrahal, Z., Herrmannova, D., Kuzilek, J., & Hlosta, M. (2014). Developing predictive models for early detection of at-risk students on distance learning modules, Workshop: Machine Learning and Learning Analytics Paper presented at the Learning Analytics and Knowledge (2014), Indianapolis.
- 14. Start FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS Pass Fail No submit TMA-1 time VLE opens Start Activity space FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS
- 15. FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS Start FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS Pass Fail No submit TMA-1 time VLE opens Start VLE trail: successful student FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS
- 16. FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS Start FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS FS F RFS OFS ORF N O S RF R OF OR ORS ORFS OS RS Pass Fail No submit TMA-1 time VLE opens Start VLE trail: student who did not submit
- 17. Probabilistic model: all students time TMA1 VLE start
- 18. Herodotou, C., Rienties, B., Hlosta, M., Boroowa, A., Mangafa, C., Zdrahal, Z., (2020). Scalable implementation of predictive learning analytics at a distance learning university: Insights from a longitudinal case study. Internet and Higher Education, 45, 100725.
- 19. Herodotou, C., Rienties, B., Hlosta, M., Boroowa, A., Mangafa, C., Zdrahal, Z., (2020). Scalable implementation of predictive learning analytics at a distance learning university: Insights from a longitudinal case study. Internet and Higher Education, 45, 100725. Amongst the factors shown to be critical to the scalable PLA implementation were: Faculty's engagement with OUA, teachers as “champions”, evidence generation and dissemination, digital literacy, and conceptions about teaching online.
- 21. Rets, I., Herodotou, C., Bayer, V., Hlosta, M., Rienties, B. (2021). Exploring critical factors of the perceived usefulness of a learning analytics dashboard for distance university students. International Journal of Educational Technology in Higher Education. 18 (46). Mixed method with 22 undergraduate business students The majority of participants found the Study recommender useful for two reasons: a) to remind them of the learning material they had missed, and b) as a means of directly accessing content (e.g., as opposed to going through the VLE). Perceived usefulness was influenced by • Trustworthiness of learning analytics dashboard • Peer comparison • Academic self-confidence • Change in study patterns • “Good” vs “not-so-good” students
- 23. Predictive learning analytics (since 2013) Learning Design (since 2005) Rienties., B. (2021). Implementing learning analytics at scale in an online world: lessons learned from the Open University UK. Liebowitz, J. (Ed). Online learning analytics. Taylor & Francis. Rienties, B., Herodotou, C. (2021). Making sense of learning data at scale. Sharpe, R., Bennett, S., Varga-Atkins, T. (Eds). Handbook for Digital Higher Education. Edward Elgar Publishing
- 24. Magic of learning design (does not come easy) “Research on the relationship between learning design and learning analytics has also been a focus in European research in recent years. For example, in their research at the Open University UK, Toetenel and Rienties combine learning design and learning analytics where learning design provides context to empirical data about OU courses enabling the learning analytics to give insight into learning design decisions. This research is important as it attempts to close the virtuous cycle between learning design to improve courses and enhancing the quality of learning, something that has been lacking in the research literature. For example, they study the impact of learning design on pedagogical decision-making and on future course design, and the relationship between learning design and student behaviour and outcomes (Toetenel and Rienties 2016; Rienties and Toetenel 2016; Rienties et al. 2015).” Wasson, B., & Kirschner, P. A. (2020). Learning Design: European Approaches. TechTrends, 1-13.
- 25. Assimilative Finding and handling information Communication Productive Experiential Interactive/ Adaptive Assessment Type of activity Attending to information Searching for and processing information Discussing module related content with at least one other person (student or tutor) Actively constructing an artefact Applying learning in a real-world setting Applying learning in a simulated setting All forms of assessment, whether continuous, end of module, or formative (assessment for learning) Examples of activity Read, Watch, Listen, Think about, Access, Observe, Review, Study List, Analyse, Collate, Plot, Find, Discover, Access, Use, Gather, Order, Classify, Select, Assess, Manipulate Communicate, Debate, Discuss, Argue, Share, Report, Collaborate, Present, Describe, Question Create, Build, Make, Design, Construct, Contribute, Complete, Produce, Write, Draw, Refine, Compose, Synthesise, Remix Practice, Apply, Mimic, Experience, Explore, Investigate, Perform, Engage Explore, Experiment, Trial, Improve, Model, Simulate Write, Present, Report, Demonstrate, Critique Conole, G. (2012). Designing for Learning in an Open World. Dordrecht: Springer. Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Open University Learning Design Initiative (OULDI)
- 26. Toetenel, L., Rienties, B. (2016). Analysing 157 Learning Designs using Learning Analytic approaches as a means to evaluate the impact of pedagogical decision-making. British Journal of Educational Technology, 47(5), 981–992.
- 28. Constructivist Learning Design Assessment Learning Design Productive Learning Design Socio-construct. Learning Design VLE Engagement Student Satisfaction Student retention 150+ modules Week 1 Week 2 Week30 + Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Communication
- 29. Assessment activities Constructivist Learning Design Assessment Learning Design Productive Learning Design Socio-construct. Learning Design VLE Engagement Student Satisfaction Student retention Week 1 Week 2 Week32 + Communication & Assessment Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028.
- 31. Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. 69% of what students are doing in a week is determined by us, teachers!
- 34. Rizvi, Saman; Rienties, Bart; Rogaten, Jekaterina and Kizilcec, René F. (2022). Beyond one-size-fits-all in MOOCs: Variation in learning design and persistence of learners in different cultural and socioeconomic contexts. Computers in Human Behavior, 126, article no. 106973. Rizvi, Saman; Rienties, Bart; Rogaten, Jekaterina and Kizilcec, René F. (2020). Investigating Variation in Learning Processes in a FutureLearn MOOC. Journal of Computing in Higher Education pp. 162–181. Rizvi, Saman (2022) Inclusiveness in Online Learning Designs: Geo-Cultural and Socioeconomic Perspectives. PhD Thesis, Open University UK
- 35. Further reflections 1. Who owns the data? 2. What about the ethics? 3. What about professional development? 4. Are we optimising the record player?
- 36. What we have learned in 10 years of implementing learning analytics and learning design at the OU Change is slow, but can be enhanced with: 1. Clear senior management support 2. Bottom-up support from teachers and researchers who are willing to take a risk 3. Evidence-based research can gradually change perspectives and narratives 4. You quickly forget about the small/medium/large successes and fail to realise that you are making a real impact 5. Large-scale innovation takes substantial time and effort 6. It is all about people… Rienties., B. (2022). Implementing learning analytics at scale in an online world: lessons learned from the Open University UK. Liebowitz, J. (Ed). Online learning analytics. Taylor & Francis.
- 37. @DrBartRienties Professor of Learning Analytics All papers referred to in this presentation can be accessed via https://iet.open.ac.uk/people/bart.rienties Implementing learning analytics at scale in an online world: lessons learned from the Open University UK