ASCILITE Webinar: A review of five years of implementation and research in aligning learning design with learning analytics at the Open University UK
Download as PPTX, PDF2 likes2,453 views
The document reviews five years of research on the integration of learning design and learning analytics at the Open University UK, highlighting the transformative impact this has had on student engagement, satisfaction, and performance. It discusses the need for effective pedagogical strategies and the significant role of data in informing educational practices. Key findings emphasize the importance of learning analytics in empowering both teachers and students, while advocating for further exploration in this field.
ASCILITE Webinar: A review of five years of implementation and research in aligning learning design with learning analytics at the Open University UK
- 1. A review of five years of implementation and research in aligning learning design with learning analytics at the Open University UK ASCILITE SIG LA Webinar 20 September 2017 @DrBartRienties Professor of Learning Analytics
- 2. A special thanks to Avinash Boroowa, Shi-Min Chua, Simon Cross, Doug Clow, Chris Edwards, Rebecca Ferguson, Mark Gaved, Christothea Herodotou, Martin Hlosta, Wayne Holmes, Garron Hillaire, Simon Knight, Nai Li, Vicky Marsh, Kevin Mayles, Jenna Mittelmeier, Vicky Murphy, Quan Nguygen, Tom Olney, Lynda Prescott, John Richardson, Jekaterina Rogaten, Matt Schencks, Mike Sharples, Dirk Tempelaar, Belinda Tynan, Lisette Toetenel, Thomas Ullmann, Denise Whitelock, Zdenek Zdrahal, and others…
- 3. Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of Educational Technology & Society, 15(3), 58-76.
- 6. 1. Increased availability of learning data 2. Increased availability of learner data 3. Increased ubiquitous presence of technology 4. Formal and informal learning increasingly blurred 5. Increased interest of non-educationalists to understand learning (Educational Data Mining, 4profit companies) 6. Personalisation and flexibility as standard
- 7. The power of learning analytics: is there still a need for educational research? 1. How can learning analytics empower teachers? 2. How can learning analytics empower students? 3. How to join us…
- 8. Big Data is messy!!!
- 9. Learning Design is described as “a methodology for enabling teachers/designers to make more informed decisions in how they go about designing learning activities and interventions, which is pedagogically informed and makes effective use of appropriate resources and technologies” (Conole, 2012).
- 10. Assimilative Finding and handling information Communication Productive Experiential Interactive/ Adaptive Assessment Type of activity Attending to information Searching for and processing information Discussing module related content with at least one other person (student or tutor) Actively constructing an artefact Applying learning in a real-world setting Applying learning in a simulated setting All forms of assessment, whether continuous, end of module, or formative (assessment for learning) Examples of activity Read, Watch, Listen, Think about, Access, Observe, Review, Study List, Analyse, Collate, Plot, Find, Discover, Access, Use, Gather, Order, Classify, Select, Assess, Manipulate Communicate, Debate, Discuss, Argue, Share, Report, Collaborate, Present, Describe, Question Create, Build, Make, Design, Construct, Contribute, Complete, Produce, Write, Draw, Refine, Compose, Synthesise, Remix Practice, Apply, Mimic, Experience, Explore, Investigate, Perform, Engage Explore, Experiment, Trial, Improve, Model, Simulate Write, Present, Report, Demonstrate, Critique Conole, G. (2012). Designing for Learning in an Open World. Dordrecht: Springer. Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Open University Learning Design Initiative (OULDI)
- 12. Merging big data sets • Learning design data (>300 modules mapped) • VLE data • >140 modules aggregated individual data weekly • >37 modules individual fine-grained data daily • Student feedback data (>140) • Academic Performance (>140) • Predictive analytics data (>40) • Data sets merged and cleaned • 111,256 students undertook these modules
- 13. Toetenel, L., Rienties, B. (2016). Analysing 157 Learning Designs using Learning Analytic approaches as a means to evaluate the impact of pedagogical decision-making. British Journal of Educational Technology, 47(5), 981–992.
- 14. Nguyen, Q., Rienties, B., & Toetenel, L. (2017). Unravelling the dynamics of instructional practice: a longitudinal study on learning design and VLE activities. Paper presented at the Proceedings of the Seventh International Learning Analytics & Knowledge Conference, Vancouver, British Columbia, Canada, pp. 168- 177
- 15. Constructivist Learning Design Assessment Learning Design Productive Learning Design Socio-construct. Learning Design VLE Engagement Student Satisfaction Student retention Learning Design Week 1 Week 2 Week30 + Rienties, B., Toetenel, L., Bryan, A. (2015). “Scaling up” learning design: impact of learning design activities on LMS behavior and performance. Learning Analytics Knowledge conference. Disciplines Levels Size module
- 17. Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028.
- 18. Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028.
- 19. Cluster 1 Constructive (n=73)
- 20. Cluster 4 Social Constructivist (n=20)
- 21. Model 1 Model 2 Model 3 Level0 -.279** -.291** -.116 Level1 -.341* -.352* -.067 Level2 .221* .229* .275** Level3 .128 .130 .139 Year of implementation .048 .049 .090 Faculty 1 -.205* -.211* -.196* Faculty 2 -.022 -.020 -.228** Faculty 3 -.206* -.210* -.308** Faculty other .216 .214 .024 Size of module .210* .209* .242** Learner satisfaction (SEAM) -.040 .103 Finding information .147 Communication .393** Productive .135 Experiential .353** Interactive -.081 Assessment .076 R-sq adj 18% 18% 40% n = 140, * p < .05, ** p < .01 Table 3 Regression model of LMS engagement predicted by institutional, satisfaction and learning design analytics • Level of study predict VLE engagement • Faculties have different VLE engagement • Learning design (communication & experiential) predict VLE engagement (with 22% unique variance explained) Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341
- 22. Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. • VLE engagement per module significantly predicted by Communication • VLE engagement per week significantly predicted by Communication (with 69% unique variance explained)
- 23. Model 1 Model 2 Model 3 Level0 .284** .304** .351** Level1 .259 .243 .265 Level2 -.211 -.197 -.212 Level3 -.035 -.029 -.018 Year of implementation .028 -.071 -.059 Faculty 1 .149 .188 .213* Faculty 2 -.039 .029 .045 Faculty 3 .090 .188 .236* Faculty other .046 .077 .051 Size of module .016 -.049 -.071 Finding information -.270** -.294** Communication .005 .050 Productive -.243** -.274** Experiential -.111 -.105 Interactive .173* .221* Assessment -.208* -.221* LMS engagement .117 R-sq adj 20% 30% 31% n = 150 (Model 1-2), 140 (Model 3), * p < .05, ** p < .01 Table 4 Regression model of learner satisfaction predicted by institutional and learning design analytics • Level of study predict satisfaction • Learning design (finding info, productive, assessment) negatively predict satisfaction • Interactive learning design positively predicts satisfaction • VLE engagement and satisfaction unrelated Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341
- 24. Model 1 Model 2 Model 3 Level0 -.142 -.147 .005 Level1 -.227 -.236 .017 Level2 -.134 -.170 -.004 Level3 .059 -.059 .215 Year of implementation -.191** -.152* -.151* Faculty 1 .355** .374** .360** Faculty 2 -.033 -.032 -.189* Faculty 3 .095 .113 .069 Faculty other .129 .156 .034 Size of module -.298** -.285** -.239** Learner satisfaction (SEAM) -.082 -.058 LMS Engagement -.070 -.190* Finding information -.154 Communication .500** Productive .133 Experiential .008 Interactive -.049 Assessment .063 R-sq adj 30% 30% 36% n = 150 (Model 1-2), 140 (Model 3), * p < .05, ** p < .01 Table 5 Regression model of learning performance predicted by institutional, satisfaction and learning design analytics • Size of module and discipline predict completion • Satisfaction unrelated to completion • Learning design (communication) predicts completion Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341
- 25. Constructivist Learning Design Assessment Learning Design Productive Learning Design Socio-construct. Learning Design VLE Engagement Student Satisfaction Student retention 150+ modules Week 1 Week 2 Week30 + Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. Communication
- 26. So what happens when you give learning design visualisations to teachers? Toetenel, L., Rienties, B. (2016) Learning Design – creative design to visualise learning activities. Open Learning: The Journal of Open and Distance Learning, 31(3), 233-244.
- 27. Toetenel, L., Rienties, B. (2016) Learning Design – creative design to visualise learning activities. Open Learning: The Journal of Open and Distance Learning, 31(3), 233-244.
- 30. Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU Analyse: analysing at-risk students at The Open University. Learning Analytics Review, 1-16.
- 31. Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU Analyse: analysing at-risk students at The Open University. Learning Analytics Review, 1-16.
- 32. Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU Analyse: analysing at-risk students at The Open University. Learning Analytics Review, 1-16.
- 33. Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU Analyse: analysing at-risk students at The Open University. Learning Analytics Review, 1-16.
- 34. So what happens when you give learning analytics data about students to teachers? 1. How did 240 teachers within the 10 modules made use of PLA data (OUA predictions) and visualisations to help students at risk? 2. To what extent was there a positive impact on students' performance and retention when using OUA predictions? 3. Which factors explain teachers' uses of OUA?
- 35. Usage of OUA dashboard by participating teachers 3 Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M., & Naydenova, G. (2017). Implementing predictive learning analytics on a large scale: the teacher's perspective. Paper presented at the Proceedings of the Seventh International Learning Analytics & Knowledge Conference, Vancouver, British
- 36. 36 Which factors better predict pass and completion rates? Regression analysis Student characteristics Age Gender New/c ontinu ous Disability Ethnicity Educat ion IMD band Best previous score Sum of previous credits Teacher characteristics Module presentations per teacher Students per module presentation OUA usage module design Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M. (Submitted: 01-08-2017). Using Predictive Learning Analytics to Support Just-in-time Interventions: The Teachers' Perspective Across a Large-scale Implementation.
- 37. 37 Significant model (pass: χ2= 76.391, p < .001, df = 24). Logistic regression results (pass rates) ●Nagelkerke’s R2 = .185 (model explains 18% of the variance in passing rates) ● Correctly classified over 70% of the cases (prediction success overall was 70.2%: 33.5 % for not passing a module and 88.7% for passing a module). ●Significant predictors of both pass and completion rates: ●OUA usage (p=.006) ●Best previous module score achieved (p=.005) ● All other predictors were not significant. Best predictors of pass rates OUA usage Best previous score Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M. (Submitted: 01-08-2017). Using Predictive Learning Analytics to Support Just-in-time Interventions: The Teachers' Perspective Across a Large-scale Implementation.
- 38. 1. How can learning analytics empower teachers? 38 Learning analytics can enhance and facilitate teaching practice, especially within distance learning contexts Strong variation in teachers’ degree and quality of engagement with learning analytics/design. Lack of consensus about intervention strategies
- 39. Conclusions and moving forwards 1. Learning design and teachers strongly influences student engagement, satisfaction and performance 2. Visualising learning design and learning analytics to teachers lead to more interactive/communicative designs and improved student retention
- 40. Conclusions and moving forwards 1. Learning analytics approaches can help researchers and practitioners to test and validate big and small theoretical questions 2. Giving students access to learning analytics data and insight next frontier
- 41. A review of five years of implementation and research in aligning learning design with learning analytics at the Open University UK ASCILITE SIG LA Webinar 20 September 2017 @DrBartRienties Professor of Learning Analytics
Editor's Notes
- #11: Explain seven categories
- #12: For each module, the learning design team together with module chairs create activity charts of what kind of activities students are expected to do in a week.
- #13: 5131 students responded – 28%, between 18-76%
- #16: Cluster analysis of 40 modules (>19k students) indicate that module teams design four different types of modules: constructivist, assessment driven, balanced, or socio-constructivist. The LAK paper by Rienties and colleagues indicates that VLE engagement is higher in modules with socio-constructivist or balanced variety learning designs, and lower for constructivist designs. In terms of learning outcomes, students rate constructivist modules higher, and socio-constructivist modules lower. However, in terms of student retention (% of students passed) constructivist modules have lower retention, while socio-constructivist have higher. Thus, learning design strongly influences behaviour, experience and performance. (and we believe we are the first to have mapped this with such a large cohort).
- #26: Cluster analysis of 40 modules (>19k students) indicate that module teams design four different types of modules: constructivist, assessment driven, balanced, or socio-constructivist. The LAK paper by Rienties and colleagues indicates that VLE engagement is higher in modules with socio-constructivist or balanced variety learning designs, and lower for constructivist designs. In terms of learning outcomes, students rate constructivist modules higher, and socio-constructivist modules lower. However, in terms of student retention (% of students passed) constructivist modules have lower retention, while socio-constructivist have higher. Thus, learning design strongly influences behaviour, experience and performance. (and we believe we are the first to have mapped this with such a large cohort).
- #38: Same results for completion rates.