Towards Strengthening Links between Learning Analytics and Assessment
6 likes946 views
The document discusses the need to strengthen the connections between learning analytics and assessment, highlighting deficiencies in feedback loops between students and educators. It emphasizes using analytics for personalized feedback to enhance student retention and success while addressing challenges in the validity and generalizability of learning analytics results. Furthermore, the text suggests that continuous data streams and advancements in data science can improve educational assessments, although measurement imperfections must be tolerated.
Towards Strengthening Links between Learning Analytics and Assessment
- 1. Towards Strengthening Links between Learning Analytics and Assessment Dragan Gašević @dgasevic ETCPS 2017 November 16, 2017 Iowa City, USA
- 2. Feedback loops between students and instructors are missing/weak!
- 6. Data in education not new, but… Real-time insights and longitudinal nature
- 7. Open ended and general purpose environments Different from custom made environments (e.g., ITS)
- 8. Assessment and feedback for learning
- 9. Student retention 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% Year 1 Year 2 Year 3 Year 4 Course Signals No Course Signals Arnold, K. E., & Pistilli, M. D. (2012, April). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267-270).
- 10. Current state Understanding and supporting learning Moving away from deficit models
- 12. Analytics-based feedback Pardo, A., Jovanovic, J. Dawson, S., Gasevic, D., Miriahi. N. (in press). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology
- 13. Analytics-based feedback Pardo, A., Jovanovic, J. Dawson, S., Gasevic, D., Miriahi. N. (in press). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology
- 14. Analytics-based feedback Pardo, A., Jovanovic, J. Dawson, S., Gasevic, D., Miriahi. N. (in press). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology
- 15. High interest in adoption of learning analytics
- 16. Is everything all that shiny?
- 18. Challenge Validity of learning analytics Messick, S. (1994). Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning. ETS Research Report Series, 1994(2), i-28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x
- 20. Open Academic Analytics Initiative http://nextgenlearning.org/grantee/marist-college
- 21. Consequentiality Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using Signals for appropriate feedback: Perceptions and practices. Computers & Education, 57(4), 2414-2422. Can teaching be improved?
- 22. Inconsistent associations of network centrality on performance External validity Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016, April). Translating network position into performance: importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 314-323). ACM.
- 23. Purposeful measurement Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting learning success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002 How can we act based on the count of logins?
- 24. Purposeful measurement Who makes decisions about instrumentation?
- 25. Structural validity Do existing* measures correspond to trace-based measures? *mostly self-reported Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017). Detecting Learning Strategies with Analytics: Links with Self-reported Measures and Academic Performance. Journal of Learning Analytics, 4(2), 113–128.
- 26. Understanding learning strategies Detection of learning tactics Detection of learning strategy Fincham, E., Gašević, D., Jovanović, Pardo, A. (2017). Seeing the Invisible: Learning Analytics to Measure the Effect of Interventions on Learning Strategies. IEEE Transactions on Learning Technologies (submitted).
- 27. Progression How do we measure progression?
- 28. Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., Siemens, G. (2016). Towards Automated Content Analysis of Discussion Transcripts: A Cognitive Presence Case,” Proceedings of the 6th International Conference on Learning Analytics & Knowledge (pp. 15-24). Cognitive presence
- 29. DIRECTIONS
- 30. Strengthening links between learning analytics and assessment
- 31. Critical dimensions Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the Learning Analytics Puzzle: A Consolidated Model of a Field of Research and Practice. Learning: Research and Practice, 3(2), 63-78. doi:10.1080/23735082.2017.1286142
- 32. Gašević, D., Dawson, S., Rogers, T., Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of course-specific technology use in predicting academic success. The Internet and Higher Education, 28, 68–84. Generalizability Instructional conditions shape learning analytics results
- 33. Purposeful measurement Extraction of theoretically informed traces Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11
- 34. Purposeful measurement Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11
- 35. Structural validity Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413– 419. doi:10.1016/j.learninstruc.2012.03.004 Achievement goal orientation (2x2)
- 36. Quid pro quo
- 37. Quid pro quo
- 38. External validity Network centrality with weak ties creates advantage only Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016, April). Translating network position into performance: importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 314-323). ACM.
- 39. Measurement of engagement Joksimović, S., Poquet, O., Kovanović, V., Dowell, D., Mills, C., Gašević, D., Dawson, S., Graesser, A. C., Brooks , C. (2017). How do we measure learning at scale? A systematic review of the literature. Review of Educational Research (in press).
- 40. Tracking progression Trace data based measures of the crowd-sourced learning skill E.g., Dreyfus model of skill acquisition Milligan, S. (2015). Crowd-sourced learning in MOOCs: learning analytics meets measurement theory. In Proceedings of the 5th International Conference on Learning Analytics And Knowledge (pp. 151-155). ACM.
- 41. Tracking progression Topic modeling to extract Guttman scales from online discussions He, J., Rubinstein, B. I., Bailey, J., Zhang, R., Milligan, S., & Chan, J. (2016). MOOCs Meet Measurement Theory: A Topic-Modelling Approach. Proceedings of the 30th AAAI Conference on Artificial Intelligence (pp. 1195-1201).
- 42. FINAL REMARKS
- 43. Opportunities afforded by continuous streams of data
- 44. Data science methods can be helpful but not sufficient von Davier, A. A. (2016). Computational psychometrics in support of collaborative educational assessments. Journal of Educational Measurement, 54(1), 3-11.
- 45. Links with methods from assessment and psychometric needed
- 46. Tolerance to some measurement imperfections
- 47. Generalizability in machine learning is an open research challenge too
- 48. Towards Strengthening Links between Learning Analytics and Assessment Dragan Gašević @dgasevic ETCPS 2017 November 16, 2017 Iowa City, USA