Charting the Design and Analytics Agenda of Learnersourcing Systems
- 1. Charting the Design and Analytics Agenda of Learnersourcing Systems Dr Hassan Khosravi The University of Queensland Brisbane, QLD, Australia h.khosravi@uq.edu.au Professor Shazia Sadiq The University of Queensland Brisbane, QLD, Australia shazia@itee.uq.edu.au Professor Dragan Gasevic Monash University Clayton, Vic, Australia Dragan.Gasevic@monash.edu A/Prof Gianluca Demartini The University of Queensland Brisbane, QLD, Australia g.demartini@uq.edu.au
- 2. Charting the Design and Analytics Agenda of Learnersourcing Systems 2 Challenges in Teaching Large Classes Learnersourcing is emerging as a viable and pedagogically justified approach of addressing these challenges. Help students develop creativity, critical thinking, communication and collaboration skills Provide timely and authentic feedback Provide learning content and instructions that cater to the diverse academic ability of the learners https://httpessaysahmyookhs.wordpress.com/21st-century-skills/
- 3. Charting the Design and Analytics Agenda of Learnersourcing Systems 3 Overview Data Driven reflections and lessons learned from design, development and deployment of RiPPLE Suggestions for developing learnersourcing system The RiPPLE system Learnersourcing
- 4. Charting the Design and Analytics Agenda of Learnersourcing Systems 4 Learnersourcing: a pedagogically supported form of crowdsourcing that mobilises the learner community as experts-in-training to contribute to teaching and learning activities while being engaged in meaningful learning experiences themselves. Learnersourcing Kim, J, 2015. Learnersourcing: improving learning with collective learner activity. Ph.D. Dissertation. Massachusetts Institute of Technology. Khosravi, H. and Demartini, G, Sadiq, S., & Gasevic D. (2021). Charting the Design and Analytics Agenda of Learnersourcing Systems. Lak’11 (in press)
- 5. Charting the Design and Analytics Agenda of Learnersourcing Systems 5 Learnersourcing in Action
- 6. Charting the Design and Analytics Agenda of Learnersourcing Systems 6 Benefits • Promotes critical learning, creativity, collaboration and communication. • Recognises them as partners in learning. • Helps them develop evaluative judgement. • Leads to the development of large repositories study material. Challenges • Assessing quality of contributions • Incentivising students to contribute • Enabling Instructors to provide oversight with minimal additional work. Learnersourcing Benefits and Challenges
- 7. Charting the Design and Analytics Agenda of Learnersourcing Systems 7 Problem, Aim and Contribution Problem best practices for design and development of learnersourcing systems are unknown Aim Contribute to development of best practices for implementation of learnersourcing systems https://blog.marketo.com/2014/02/how-to-set-goals-you-can-meet.html https://www.searchenginejournal.com/question-improve-your-problem-solving/327995/#close Contribution Share data-driven reflections and lessons learned from design, implementation and deployment of a learnersourcing system
- 8. Overview Data Driven reflections and lessons learned from design, development and deployment of RiPPLE Learnersourcing The RiPPLE system Suggestions for developing learnersourcing system
- 9. Charting the Design and Analytics Agenda of Learnersourcing Systems 9 The RiPPLE System 9 Key Features Partnering with students to develop a repository of high-quality study material. Learnersourcing Adaptive learning Recommending learning resources to each student based on their learning needs. Engaging students with interactive in-class or self- paced learning activities Peer learning Helping students find effective study partners that suit their availability and learning needs. Additional features Interactive learning activities RiPPLE is an open access educational platform that aims to provide a personalised, active and social learning experience. Khosravi et al. in the Journal of Learning Analytics (2019) For more info visit http://ripplelearning.org/
- 10. Charting the Design and Analytics Agenda of Learnersourcing Systems 10 Adoption and Recognition RiPPLE is increasingly being recognised as an exemplar tool for using AI in education. 15 Peer-reviewed articles 2 best paper nomination Featured as an exemplar educational technology in EDUCAUSE Horizon 2019 Report Academics and researchers from over 50 universities have accessed RiPPLE After six semesters, RiPPLE has been adopted by courses in Medicine, Pharmacy, Psychology, Education, Business, Computer Science & Biosciences. 60 Course offerings 14000+ Users 1,000,000+ Resources studied 30,000+ Study resources created
- 11. Charting the Design and Analytics Agenda of Learnersourcing Systems 11 Learnersourcing in RiPPLE (1) Students author learning resources. (2) Resources are moderated by peers and tutors. (3) Based on the moderations a decision is made. (4) RIPPLE flags questionable resources for spot checking.
- 12. Overview Suggestions for developing learnersourcing system Learnersourcing The RiPPLE system Data driven reflections and lessons learned from design, development and deployment of RiPPLE
- 13. Charting the Design and Analytics Agenda of Learnersourcing Systems 13 2. Incentivising high quality contributions 1. Quality control of student generated content 3. Empowering instructors with explainable and actionable insights Data Driven Reflections and Lessons Learned
- 14. Charting the Design and Analytics Agenda of Learnersourcing Systems 14 Can Students Accurately Evaluate the Quality of Learning Resources? • Data presence 64k moderations from 2.4k learners and 28 instructors on 15K resources. • Overall, ratings provided by students strongly correlate with ratings from experts Abdi et al. in IEEE Transactions on Educational Technologies (2021) • When instructors reject a resource, the chance of students also rejecting is 16%. • Identifying low-quality resources based on peer review is challenging
- 15. Charting the Design and Analytics Agenda of Learnersourcing Systems 15 1.1 Developing Accurate and Explainable Consensus Approaches Arrieta, A et al. (2020). Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82-115. How can we accurately, transparently and fairly evaluate student-generated content? TPR TNR AUC TPR TNR AUC Majority Vote 0.96 0.20 0.58 0.99 0.10 0.54 Mean 0.96 0.20 0.58 1.00 0.14 0.57 Median 1.00 0.20 0.60 1.00 0.14 0.57 Expectation Maximisation 0.94 0.4 0.67 1.00 0.33 0.67 High Trust Propagation 0.81 0.62 0.71 0.88 0.52 0.70 Medium Sentiment-Alignment 0.94 0.53 0.74 1.00 0.29 0.64 Relatedness - BERT 0.94 0.60 0.77 0.98 0.24 0.61 Length×Relatedness 0.92 0.73 0.83 0.91 0.76 0.84 Bagged Decision Tree 0.89 0.53 0.71 0.91 0.81 0.86 Random Forest 0.91 0.6 0.75 0.91 0.67 0.79 Boosting 0.87 0.6 0.73 0.91 0.67 0.79 RPART 0.79 0.87 0.83 0.90 0.86 0.88 Comment-based Models Type Baseline Models Ensemble Models Explainable ? Very High Very Low Low Method INFS NEUR Probabilistc models Darvishi et al., to appear in the proceedings of the Learning at Scale conference (2020)
- 16. Charting the Design and Analytics Agenda of Learnersourcing Systems 16 1.2 Employing Human-in-the-loop and Responsible AI Based on: https://hai.stanford.edu/news/humans-loop-design-interactive-ai-systems Full Automation: AI as a Big Red Button Designing with a Human in the Loop Benefits ü Significant gains in transparency ü Effective incorporation of human judgment ü Less pressure on building “perfect” algorithms. Challenges Data and algorithmic bias Lack of transparency and trust in decision making
- 17. Charting the Design and Analytics Agenda of Learnersourcing Systems 17 1.2 Employing Human-in-the-loop in RiPPLE Challenging the outcome of a moderation
- 18. Charting the Design and Analytics Agenda of Learnersourcing Systems 18 1.3 Employing Appropriate Criteria for Evaluating Resources Criteria: Alignment, Correctness, Clarity Alignment, Correctness + Clarity, Difficulty, Critical thinking Gyamfi et al. in Assessment and Higher Education (2021)
- 19. Charting the Design and Analytics Agenda of Learnersourcing Systems 19 Quality control of student generated content Empowering instructors with explainable and actionable insights Data Driven Reflections and Lessons Learned Incentivising high quality contributions
- 20. Charting the Design and Analytics Agenda of Learnersourcing Systems 20 Participation inequality. In most online communities, 90% of users are lurkers who never contribute, 9% of users contribute a little, and 1% of users account for almost all the action. Incentivising high quality contributions Many streaming, gaming, and social media platforms are designed with the prime intention of increasing engagement without considering the quality of the engagement. So how can learnersourcing systems be designed to encourage a large portion of the student population to contribute high-quality learnersourcing activities? Photo by camilo jimenez on Unsplash Source: https://www.nngroup.com/articles/participation-inequality/
- 21. Charting the Design and Analytics Agenda of Learnersourcing Systems 21 2.1 Modelling students in Learnersourcing Systems Abdi et al. in proceedings of AI in Education (2020 and 2021) 1. Discouraging learnersourcing contributions. 2. Missing the opportunity to leverage learnersourcing data in modelling students 3. Advancing the belief that learnersourcing does not contribute to learning.
- 22. Charting the Design and Analytics Agenda of Learnersourcing Systems 22 2.1 Open Learner Models for Learnersourcing Systems Abdi et al. in proceedings of AI in Education (2020 and 2021)
- 23. Charting the Design and Analytics Agenda of Learnersourcing Systems 23 • Using RiPPLE Formatively • Engagement following 90-9-1 rule • Low amount of high-quality engagement 2.2 Tying Learnersourcing to Assessment • Using RiPPLE Summatively • High amount of low quality engagement. • Requires supporting assessment logistics
- 24. Charting the Design and Analytics Agenda of Learnersourcing Systems 24 2.3 Employing Gamification Mechanisms Profile page Leaderboard Google Analytics
- 25. Charting the Design and Analytics Agenda of Learnersourcing Systems 25 Quality control of student generated content Incentivising high quality contributions Data Driven Reflections and Lessons Learned Empowering instructors with explainable and actionable insights
- 26. Charting the Design and Analytics Agenda of Learnersourcing Systems 26 3.1 The Analytics Toolbox
- 27. Charting the Design and Analytics Agenda of Learnersourcing Systems 27 3.2 Spot checking in RiPPLE Spot checking in RiPPLE • Identifying resources that have passed moderation but are likely to be incorrect or ineffective. • Uses human-driven metrics (e.g., high-disagreement in moderation evaluations) and data- driven metrics (e.g., low discrimination) to recommend resources for inspection.
- 28. Charting the Design and Analytics Agenda of Learnersourcing Systems 28 3.2 Explainable Spot Checking in RiPPLE
- 29. Overview Learnersourcing The RiPPLE system Data driven reflections and lessons learned from design, development and deployment of RiPPLE Suggestions for developing learnersourcing system
- 30. Charting the Design and Analytics Agenda of Learnersourcing Systems 30 Suggestions for Developing Learnersourcing Systems Develop accurate and explainable consensus approaches Enable users to raise concerns against automated decisions Reflect on the criteria used in the evaluation student generated content Use open learner models to present student contributions Support logistics to tie leanersourcing to assessment Employ various gamification mechanisms Empower instructors with actionable and explainable insights conduct rigorous empirical studies to evaluate the platform.
- 31. Charting the Design and Analytics Agenda of Learnersourcing Systems 31 1. Abdi, S., Khosravi, H., Sadiq, S., & Demartini, G. (2021). Evaluating the Quality of Learning Resources: A Learnersourcing Approach. IEEE Transactions on Learning Technologies, 14(1), 81-92. 2. Abdi, S., Khosravi, H., & Sadiq, S. (2020, July). Modelling Learners in Crowdsourcing Educational Systems. In International Conference on Artificial Intelligence in Education (pp. 3-9). Springer, Cham. 3. Arrieta, A et al. (2020). Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82-115. 4. Darvishi, A., Khosravi, H., & Sadiq, S. (2020, September). Utilising Learnersourcing to Inform Design Loop Adaptivity. In European Conference on Technology Enhanced Learning (pp. 332-346). Springer, Cham. 5. Gyamfi, G., Hanna, B. E., & Khosravi, H. (2021). The effects of rubrics on evaluative judgement: a randomised controlled experiment. Assessment & Evaluation in Higher Education, 1-18. 6. Jiang, Y., Schlagwein, D., & Benatallah, B. (2018, June). A Review on Crowdsourcing for Education: State of the Art of Literature and Practice. In PACIS (p. 180). 7. Kim, J. (2015). Learnersourcing: improving learning with collective learner activity (Doctoral dissertation, Massachusetts Institute of Technology). 8. Khosravi, H., Kitto, K., & Williams, J. J. (2019). RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities. Journal of Learning Analytics, 6(3), 91-105. References