2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
2 likes182 views
1. The document discusses three conceptualizations of multimodal learning analytics (MMLA): MMLA to automate human tasks, augment teaching and learning practices, and as a research methodology. 2. It examines what modalities of data are used in MMLA, including video/audio data, eye tracking data, physiological sensors, and location sensing. Machine learning has been applied to MMLA tasks like classifying collaboration. 3. Challenges of MMLA include connecting findings to learning theory, addressing ethics concerns like privacy and surveillance, and determining what behaviors are considered good or bad in education. Students have mixed reactions to being analyzed by MMLA.
![Human
Control
Automation through AI
Most traditional
Educational Technology
HA = Humans internalise MMLA
models H à A
Analytics and GOFAI: Changing
the operations and
representations of thought
H[A] = Human cognition (H)
extended with MMLA (A), tightly
coupled human and artificial
systems.
H[A] > (H) + (A)
AH = Human tasks are replaced
by MMLA H ß A
Initial vision of AIED: Early
Promises of Intelligent Tutoring
Systems
High
Low High
Low
2) Human-centred
3) Trustworthy
MMLA: A vision for the future
1) Learning sciences/
learning design-driven](https://image.slidesharecdn.com/emadridcukurova-221115085007-d8a00bf3/85/2022_11_11-The-promise-and-challenges-of-Multimodal-Learning-Analytics-34-320.jpg)
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
- 1. Dr. Mutlu Cukurova University College London m.cukurova@ucl.ac.uk @mutlucukurova The Promise and Challenges of Multimodal Learning Analytics
- 2. Three conceptualisations of MMLA 1. MMLA to Automate Human Tasks 2. MMLA to Augment Teaching and Learning Practices 3. MMLA as a Research Methodology
- 3. “tools that have been designed and developed to replace human tasks through analysis of multiple modalities of data, and prediction of the best value for a designated outcome variable, which is conveyed through a user interface.” Multimodal Learning Analytics as:
- 4. What Modalities of Data We Use? 0 5 10 15 20 25 30 35 40 Presenter’s pose, gaze direction, visual attention Audio, dialogue’s characteristics Facialdata and emotions Posture and gesture, body movement, distance, motion, Position Learners’ concentration level, visual attention and habits Learners’ gaze in off- screen activity Student’s physiological data, arousal and EDA Students’ cognitive load Posture and gesture Location, position and duration, movement Sitting position EEG data, brain activity Video and audio Eye-tracking Skin sensing In depth camera Location sensing Pressure sensing EEG sensor Number of published articles 2010 - 2020 Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2021). The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review. https://doi.org/10.35542/osf.io/sd23y
- 5. MMLA as Machine Learning Tools Human Control Automation through AI Most traditional Educational Technology AH = Human tasks are replaced by MMLA H ß A Initial vision of AIED: Early Promises of Intelligent Tutoring Systems High Low High Low
- 7. Machine Learning Classification of CPS Competence Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of Computer Assisted Learning, 34(4), 366-377.
- 8. Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of Computer Assisted Learning, 34(4), 366-377. The promise of MMLA
- 9. R. Kawamura et al., Detecting Drowsy Learners at the Wheel of e-Learning Platforms With Multimodal Learning Analytics, IEEE Access, vol. 9, pp. 115165-115174, 2021, doi: 10.1109/ACCESS.2021.3104805
- 10. “to support learning experiences that may be collaborative, hands-on, and face-to-face, de-emphasizing the computer screen as the primary form or object of interaction.” Multimodal Learning Analytics: Worsley, M., Martinez-Maldonado, R., & D'Angelo, C. (2021). "A New Era in Multimodal Learning Analytics: Twelve Core Commitments to Ground and Grow MMLA." Journal of Learning Analytics 8.3, 10-27. 2. MMLA to Augment Teaching and Learning Practices
- 11. MMLA to Augment Teaching and Learning Practices Human Control Automation through AI Most traditional Educational Technology HA = Humans internalise MMLA models H à A Analytics and GOFAI: Changing the operations and representations of thought AH = Human tasks are replaced by AI H ß A Initial vision of AIED: Early Promises of Intelligent Tutoring Systems High Low High Low
- 12. 17 Cukurova, M., Kent, C., & Luckin, R. (2019). Artificial Intelligence and Multimodal Data in the Service of Human Decision-making: A case Study in Debate Tutoring, British Journal of Educational Technology, 50 (6), 3032-3046.
- 13. Models as learning affordances for humans Kent, C., Chaudhry, M., Cukurova, M. Bashir, I., Pickard, H., Jenkins, C., du Boulay, B., Luckin, R., (2021). Machine learning models and their development process as learning affordances for humans. International Conference of Artificial Intelligence in Education, Springer, Cham. What do successful students look like? What are their behaviour and personality traits? Mental model : diagram of functionality Initial student profiles
- 14. Active Active Passive Kasparova, A., Celiktutan, O., & Cukurova, M. (2020). Inferring Student Engagement in Collaborative Problem Solving from Visual Cues. Companion Publication of the 2020 International Conference on Multimodal Interaction. https://doi.org/10.1145/3395035.3425961
- 15. From machine observations to learning sciences constructs of collaboration Interpretationsof Collaboration Competence from Ma Observables Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solvin students' physical interactions. Computers & Education, 116, 93-109. Some important components of collaboration may be interpreted through automated machine observable data: • synchrony, • individual accountability, • equality, • mutuality. Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solving from students' physical interactions. Computers & Education, 116, 93-109. Cukurova, M. (2018). A syllogism for designing collaborative learning technologies in the age of AI and multimodal data. In European Conference on Technology Enhanced Learning (pp. 291-296). Springer, Cham.
- 16. Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solving from students' physical interactions. Computers & Education, 116, 93-109. 14
- 17. Cukurova, M., Mavrikis, M., & Luckin, R. (2017). Evidence-Centered Design and Its Application to Collaborative Problem Solving in Practice-based Learning Environments. Analytics4Learning. Stanford Research Institute, Menlo Park, USA.
- 18. Learning sciences should drive the features. Learner/Teachers’ needs and expectations should be met. Kent, C., & Cukurova, M. (2020). Investigating Collaboration as a Process with Theory-driven Learning Analytics. Journal of Learning Analytics, 7(1), 59-71. Cukurova, M., Kent, C., & Akanji, A. (2022). Identifying tertiary level educators' needs and understanding of the collaboration process analytics. https://doi.org/10.35542/osf.io/2d3ty
- 19. MMLA as a Research Methodology 3. MMLA as a Research Methodology
- 20. Ouyang, F., Xu, W., Cukurova, M. (2022). An Artificial Intelligence-driven Learning Analytics Method to Examine the Collaborative Problem-solving Process from the Complex Adaptive Systems Perspective, International Journal of Computer Supported Collaborative Learning. http://arxiv.org/abs/2210.16059v1 Interactive Peer interaction through communications (Int-C) Peer interaction through behaviours (Int-B) Cognitive Superficial-level knowledge (KS) Medium-level knowledge (KM) Deep-level knowledge (KD) Behavioural Resource management (RM) Concept mapping (CM) Observation (OB) Regulative Task understanding (TU) Goal setting and planning (GSP) Monitoring and reflection (MR) Socio-emotional Active listening and respect (ALR) Encouraging participation and inclusion (EPI) Fostering cohesion (FC)
- 21. • Behaviour-oriented collaborative pattern (Type 1) - Medium-level performance • Communication-behaviour-synergistic collaborative pattern (Type 2) – High-level performance • Communication-oriented collaborative pattern (Type 3) – Low-level performance Ouyang, F., Xu, W., Cukurova, M. (2022). An Artificial Intelligence-driven Learning Analytics Method to Examine the Collaborative Problem-solving Process from the Complex Adaptive Systems Perspective, International Journal of Computer Supported Collaborative Learning. http://arxiv.org/abs/2210.16059v1
- 23. • Peer-interaction focussed collaborative pattern (PIF Type) • Resources-interaction focusses collaborative pattern (RIF Type) Zhou, Q., Suraworachet, W., Celiktutan, O., & Cukurova, M. (2022). What Does Shared Understanding in Students’ Face-to-Face Collaborative Learning Gaze Behaviours “Look Like”?. In International Conference on Artificial Intelligence in Education (pp. 588-593). Springer, Cham.
- 24. 1. Methodological - ML AI is not that intelligent - AIP 2. Logistical/practical/financial 3. Connection to learning theory and learning design Short-term Challenges of MMLA Cukurova, M., Giannakos, M., & Martinez-Maldonado, R. (2020). The Promise and Challenges of Multimodal Learning Analytics. British Journal of Educational Technology, 51(5), pp. 1441-1449, https://doi.org/10.1111/bjet.13015. Zhou, Q., Suraworachet, W., Cukurova, M. (2021). Different modality, different design, different results: exploring self-regulated learner clusters’ engagement behaviours at individual, group and cohort activities. https://doi.org/10.35542/osf.io/u3g4n
- 25. “More Significant” Challenges of MMLA – Particularly for the first conceptualisation Normativity 1. What behaviours are good/bad in education? 2. Inference based on history 3. Flourishing Diversity/ in uncertainty Prediction 1. What is it good for/intervention? 2. Fairness 3. Transparency 4. Accountability Human Agency 1. Balancing exploration and exploitation 2. The reward function 3. Privacy/ surveillance Ethics- Human Values Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2022). The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature Review. In The Multimodal Learning Analytics Handbook (pp. 289-325). Springer International Publishing. doi:10.1007/978-3-031-08076-0_12
- 26. Students have mixed reactions to MMLA Zhou,Q., Suraworachet, W., Pozdniakov, S., Martinez-Maldonado, R., Bartindale, T., Chen, P. Richardson, D., & Cukurova M. (2021). Investigating Students’ Experiences with Collaboration Analytics for Remote Group Meetings. International Conference of Artificial Intelligence in Education, Springer, Cham. Zhou, Q., Suraworachet, W., Cukurova, M. (2021). Different modality, different design, different results: exploring self-regulated learner clusters’ engagement behaviours at individual, group and cohort activities. https://doi.org/10.35542/osf.io/u3g4n - Were not concerned (or faded) due to: the module domain, invisibility, not parts of sum assessment - More concerning for low contributors - Motivational for increased accountability for high contributors
- 27. Nazaretsky, T., Cukurova, M., Ariely, M., & Alexandron, G. (2021). Confirmation bias and trust: Human factors that influence teachers' attitudes towards AI-based educational technology, CEUR Workshop Proceedings (Vol. 3042). Cukurova, M., Luckin, R., & Kent, C. (2019). Impact of an Artificial Intelligence Research Frame on the Perceived Credibility of Educational Research Evidence. International Journal of Artificial Intelligence in Education, 1-31. Teachers have confirmation bias and unrealistic expectations from AI-based EdTech. "AI framing effect": when people (including teachers) are presented with research evidence framed as coming from AIED, they tend to judge it as less credible compared to educational psychology and neuroscience. Teachers appear to be more critical
- 28. How do we gain teachers’ trust in MMLA? • Developed a survey instrument that revealed eight dimensions. Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2021). The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review. https://doi.org/10.35542/osf.io/sd23y Nazaretsky, T., Cukurova, M., Alexandron, G. (2022). An Instrument for Measuring Teachers’ Trust in AI-Based Educational Technology. Learning Analytics and Knowledge Conference. ACM: New York. 1. Perceived Benefits 2. Lack of Human Characteristics 3. Lack of Transparency 4. Anxieties 5. Self-efficacy 6. Required Shift in Pedagogy 7. Means to Increase Trust 8. AI-EdTech vs Human Advice Strongly not agree Not agree Strongly agree Agree Neutral
- 29. How would a CPD to increase teachers’ trust in intelligent technologies look like? Nazaretsky, T., Ariely, M., Cukurova, M., Alexandron, G. (2022). Teachers’ Trust in AI-powered Educational Technology and a Professional Development Program to Improve It, British Journal of Educational Technology, DOI: 10.1111/bjet.13232
- 30. What do we mean by transparency? Transparency for whom? Chaudhry, M. A., Cukurova, M., & Luckin, R. (2022). A Transparency Index Framework for AI in Education. Artificial Intelligence in Education. https://arxiv.org/abs/2206.03220 .
- 31. Cukurova, M., Zhou, Q., Spikol, D., & Landolfi, L. (2020). Modelling collaborative problem-solving competence with transparent learning analytics: is video data enough?. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 270-275). Listening Speaking Transparent decision trees to model Collaborative problem solving competence Watching Making Collaborative problem-solving competence
- 32. The Golden Triangle for Value Alignment Teachers & Learners Tech Developers Academic researchers ENABLER 3 Engage educators, trainers, researchers and AI developers in co- development ENABLER 1 Train educators and trainers ENABLER 2 Train AI developers Data, evidence and research Cukurova, M., Luckin, R., & Clark-Wilson, A. (2019). Creating the golden triangle of evidence-informed education technology with EDUCATE. British Journal of Educational Technology, 50(2), 490-504. Luckin, R., & Cukurova, M. (2019). Designing educational technologies in the age of AI: A learning sciences-driven approach. British Journal of Educational Technology, 50(6), 2824-2838. doi:10.1111/bjet.12861
- 33. Human-centred design for value alignment: How Can We Design Analytics Interfaces Aligned to Teachers' Inquiry? Pozdniakov, S., Martinez-Maldonado, R., Shan-Tsai, Y., Cukurova, M., Bartindale, T., Chen, P., Harrison, M., Richardson, D., & Gasevic, D. (2022). The Question-driven Dashboard: How Can We Design Analytics Interfaces Aligned to Teachers' Inquiry?. Learning Analytics & Knowledge, ACM.
- 34. Human Control Automation through AI Most traditional Educational Technology HA = Humans internalise MMLA models H à A Analytics and GOFAI: Changing the operations and representations of thought H[A] = Human cognition (H) extended with MMLA (A), tightly coupled human and artificial systems. H[A] > (H) + (A) AH = Human tasks are replaced by MMLA H ß A Initial vision of AIED: Early Promises of Intelligent Tutoring Systems High Low High Low 2) Human-centred 3) Trustworthy MMLA: A vision for the future 1) Learning sciences/ learning design-driven
- 35. Thank you Dr. Mutlu Cukurova m.cukurova@ucl.ac.uk @mutlucukurova PhD students and Collaborators:, Wannapon Suraworachet, Qi Zhou, Ali Chaudhry, Haifa Al-Wahaby, Rose Luckin, Carmel Kent, Benedict Du Boulay, Juan I. Asensio-Pérez, Daniel Spikol, Manolis Mavrikis, Yannis Dimitriadis, Stanislav Pozdniakov, Roberto Martinez-Maldonado, Tom Bartindale, Peter Chen, Dan Richardson, Fan Ouyang, Xi Wu, Michali Giannakos, Zacharoula Papamitsiou, Oya Celiktutan, Anjelika Kasparova, Cristina Villa-Torrano, Abayomi Akanji