Can learning analytics offer meaningful assessment?
2 likes1,018 views
The document discusses the challenges and opportunities of using learning analytics for meaningful assessment in educational environments. It emphasizes the importance of valid and reliable measures of engagement and progression, while critiquing current practices and measurement methods. Furthermore, it highlights the need for purposeful measurement and transparency in learning analytics to enhance the educational assessment process.
Can learning analytics offer meaningful assessment?
- 1. Can learning analytics offer meaningful assessment? Dragan Gašević @dgasevic EARLI SIG Assessment August 29, 2018 Helsinki, Finland
- 2. Feedback loops between students and instructors are missing/weak!
- 6. Data in education not new, but… Real-time insights and longitudinal nature
- 7. Analytics for custom made environments e.g., intelligent tutoring systems
- 8. Moment by moment learning Baker , R. S., Hershkovitz, A., Rossi, L. M., Goldstein, A. B, & Gowda, S. M. (2013) Predicting Robust Learning With the Visual Form of the Moment by- Moment Learning Curve, Journal of the Learning Sciences, 22(4), 639-666, DOI: 10.1080/10508406.2013.836653
- 9. Open ended and general purpose environments Different from custom made environments
- 10. Learning analytics purpose Understanding and supporting learning
- 11. Current state in learning analytics Moving away from deficit models
- 12. High interest in adoption of learning analytics
- 13. Is everything that shiny?
- 15. Scrutinize learning analytics against assessment properties
- 18. Framework Unified theory of construct validity Messick, S. (1994). Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning. ETS Research Report Series, 1994(2), i-28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x
- 19. Content Engagement is primarily discussed in learning analytics
- 20. Content But, how is engagement defined and measured?
- 21. Content
- 22. Content Are simple counts of clicks measures of engagement?
- 23. Content Who makes decisions about instrumentation?
- 24. Purposeful measurement Can we trust some of measures? Time on task Kovanovic, V., Gašević, D., Dawson, S., Joksimovic, S., & Baker, R. S. (2016). Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings. Journal of Learning Analytics, 2(3), 81–110. https://doi.org/10.18608/jla.2015.23.6
- 25. Purposeful measurement Can we trust some of measures? Network centrality Fincham, E., Gašević, D., Pardo, A. (2018). From Social Ties to Network Processes: Do Tie Definitions Matter?. Journal of Learning Analytics, 5(2), 9–28.
- 26. Consequential (aka actionable) Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting learning success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002 How can we act based on the count of logins?
- 27. Consequential Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using Signals for appropriate feedback: Perceptions and practices. Computers & Education, 57(4), 2414-2422. Can teaching be improved?
- 28. Generalizability Difficulty to replicate findings
- 29. Open Academic Analytics Initiative http://nextgenlearning.org/grantee/marist-college
- 30. Inconsistent associations of network centrality on performance External validity Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016). Translating network position into performance: importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 314-323). ACM.
- 31. Structural validity Do existing* measures correspond to trace-based measures? *mostly self-reported Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017). Detecting Learning Strategies with Analytics: Links with Self-reported Measures and Academic Performance. Journal of Learning Analytics, 4(2), 113–128.
- 32. Understanding learning strategies Detection of learning tactics Detection of learning strategy Fincham, E., Gašević, D., Jovanović, Pardo, A. (2018). From Study Tactics to Learning Strategies: An Analytical Method for Extracting Interpretable Representations. IEEE Transactions on Learning Technologies (in press), https://doi.org/10.1109/TLT.2018.2823317
- 34. Challenge How do we measure progression?
- 35. Challenge Does increase in number of clicks means progression?
- 36. Increase in activity ≠ increase in learning Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89. https://doi.org/10.1016/j.iheduc.2015.06.002
- 37. Increase in activity ≠ increase in learning Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89. https://doi.org/10.1016/j.iheduc.2015.06.002
- 38. Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33, 74–85. Increase in activity ≠ increase in learning
- 39. Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33, 74–85. Increase in activity ≠ increase in learning A
- 40. Negative predictors of learning Student-content (counts) Student-teacher (time) Joksimović, S., Gašević, D., Loughin, T. M., Kovanović, V., & Hatala, M. (2015). Learning at distance: Effects of interaction traces on academic achievement. Computers & Education, 87, 204–217. https://doi.org/10.1016/j.compedu.2015.07.002
- 41. Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., Siemens, G. (2016). Towards Automated Content Analysis of Discussion Transcripts: A Cognitive Presence Case. Proceedings of the 6th International Conference on Learning Analytics & Knowledge (pp. 15-24). Cognitive presence
- 42. DIRECTIONS
- 43. Strengthening links between learning analytics and assessment
- 44. Critical dimensions Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the Learning Analytics Puzzle: A Consolidated Model of a Field of Research and Practice. Learning: Research and Practice, 3(2), 63-78. doi:10.1080/23735082.2017.1286142
- 46. Content Extraction of theoretically informed traces
- 47. Purposeful measurement Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11
- 48. Purposeful measurement Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11
- 49. Purposeful measurement Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11
- 50. Purposeful measurement Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11
- 51. Purposeful measurement Transparency in measurement Kovanovic, V., Gašević, D., Dawson, S., Joksimovic, S., & Baker, R. S. (2016). Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings. Journal of Learning Analytics, 2(3), 81–110. https://doi.org/10.18608/jla.2015.23.6
- 52. Gašević, D., Dawson, S., Rogers, T., Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of course-specific technology use in predicting academic success. The Internet and Higher Education, 28, 68–84. Generalizability Instructional conditions shape learning analytics results
- 53. Structural validity Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413– 419. doi:10.1016/j.learninstruc.2012.03.004 Achievement goal orientation (2x2)
- 54. Quid pro quo
- 55. Traced measures and quid pro quo
- 56. External validity Network centrality with weak ties creates advantage only Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016, April). Translating network position into performance: importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 314-323). ACM.
- 57. Measurement of engagement Joksimović, S., Poquet, O., Kovanović, V., Dowell, D., Mills, C., Gašević, D., Dawson, S., Graesser, A. C., Brooks , C. (2018). How do we measure learning at scale? A systematic review of the literature. Review of Educational Research, 88(1), 43-86.
- 59. Progression Trace data based measures of the crowd-sourced learning skill E.g., Dreyfus model of skill acquisition Milligan, S. (2015). Crowd-sourced learning in MOOCs: learning analytics meets measurement theory. In Proceedings of the 5th International Conference on Learning Analytics And Knowledge (pp. 151-155). ACM.
- 60. Progression Topic modeling to extract Guttman scales from online discussions He, J., Rubinstein, B. I., Bailey, J., Zhang, R., Milligan, S., & Chan, J. (2016). MOOCs Meet Measurement Theory: A Topic-Modelling Approach. Proceedings of the 30th AAAI Conference on Artificial Intelligence (pp. 1195-1201).
- 61. Tracking progression Ferreira, R., Kovanović, V., Gašević, D., & Rolim, V. (2018). Towards Combined Network and Text Analytics of Student Discourse in Online Discussions. In Proceeding of the 19th International Conference on Artificial Intelligence in Education (pp. 111-126).
- 62. Tracking progression Ferreira, R., Kovanović, V., Gašević, D., & Rolim, V. (2018). Towards Combined Network and Text Analytics of Student Discourse in Online Discussions. In Proceeding of the 19th International Conference on Artificial Intelligence in Education (pp. 111-126). Integration Exploration Triggering event
- 63. Tracking progression Ferreira, R., Kovanović, V., Gašević, D., & Rolim, V. (2018). Towards Combined Network and Text Analytics of Student Discourse in Online Discussions. In Proceeding of the 19th International Conference on Artificial Intelligence in Education (pp. 111-126).
- 65. FINAL REMARKS
- 66. Opportunities afforded by continuous streams of data
- 67. Data science methods can be helpful but not sufficient von Davier, A. A. (2016). Computational psychometrics in support of collaborative educational assessments. Journal of Educational Measurement, 54(1), 3-11.
- 68. Links with methods from assessment and psychometric needed
- 69. Tolerance to some measurement imperfections
- 70. Generalizability in machine learning is an open research challenge too
- 72. Can learning analytics offer meaningful assessment? Dragan Gašević @dgasevic EARLI SIG Assessment August 29, 2018 Helsinki, Finland