Untubing AI in Assessment: A Primer for Future’s Sake
1 like187 views
The document summarizes a presentation on AI in assessment given by Professor Mark Brown. The presentation addressed three main points: 1) Educators should avoid AI-centric thinking and own the narrative around how AI can help address existing assessment challenges; 2) Guidelines are needed for both allowing and prohibiting student use of AI tools in assessments; 3) Assessments need to be redesigned to emphasize authentic processes over products and evaluate uniquely human skills that AI cannot replicate. The presentation argued for augmenting human expertise with AI rather than automating assessments and discussed principles and actions needed to shape regulations, review practices, and support educators on the path to assessment change.
Untubing AI in Assessment: A Primer for Future’s Sake
- 1. The Mini-Summit: AI in Assessment Professor Mark Brown Institute of Education Dublin City University 26th September 2023
- 2. Two Truths and a Lie I’m no expert but I have a growing research interest in AI in education I have a strong track record of research on digital forms of Assessment I haven’t taught in the schooling sector for over 25 years
- 5. Ubiquitous
- 6. A few health warnings Learning the art of brushing both ways Untubing AI in Assessment: Developing positive habits for healthy minds
- 7. Key messages 1. Avoid the trap of AI-centric thinking. 2. Proactively own, shape, and influence the narrative regarding the assessment challenges that AI is attempting to solve. 3. Recognise that all was not well with Assessment (and Feedback) practices before the emergence of Generative Artificial Intelligence. Educators need to…
- 8. • Training memories • Limited relevance • Lack of challenge • Promoting transfer • Academic integrity • Culture of banking • Playing the game A c q u i s i t i o n P a r t i c i p a t i o n Knowledge Creation
- 9. 1. A few health warnings
- 10. 1. A few health warnings • 92% of OER content in English • Only 10 of 7,000 languages have been used by GenAI • Latest GenAI tools are less prone to fake knowledge • AI generated text can’t always be easily detected • Adoption of authentic assessment is not AI proof • Bad for the environment with ChatGPT-3 generating 85,000kg of CO2 emissions • GenAI is not freely accessible • ChatGPT is not just a big database • ChatGPT tends to pose itself falsely as a real person • Few of us understand the ‘black box’ and can keep up with developments • There are serious biases, discrimination, privacy and ethical issues • Risk of propagating inequity and marginalising those with already marginalised voices
- 11. 2. Learning the art of brushing both ways https://unsplash.com/photos/IW1uAv88Z0o
- 12. 2. Learning the art of brushing both ways ‘Design In’ ‘Design Out’ • Students can use GenAI tools in their assessments • Clear guidelines must be provided to students (and parents and caregivers) • Potential biases, discrimination and ethical issues have been considered as well as they can be • Students should NOT use GenAI tools in their assessments • Some assessments may need to be redesigned to minimize their vulnerability • Students should not be asked to undertake things that can be easily done with GenAI tools
- 14. https://www.bbc.co.uk/newsround/66107886 Risk of short-termism Evidence of knee-jerk reactions Photo by Henrik Lagercrantz on Unsplash
- 15. • In the face of AI, should we try to defend our existing Assessment practices? Straw poll… • Is it possible to avoid spillage across ‘design in’ and ‘design out’ approaches, especially given AI’s ubiquity? • To what extent do we risk a potential loss of confidence locally and nationally in existing assessment practices if we fail to appropriately respond to AI?
- 17. • From onerous to feasible • From discrete to continuous • From uniform to adaptive • From inauthentic to authentic • From antiquated to modern What are the assessment problems we face? https://doi.org/10.1016/j.caeai.2022.100075
- 18. Henderson (2023) asks, “What labours are involved that can be offloaded or co-constructed with AI?” https://www2.ed.gov/documents/ai-report/ai-report.pdf
- 19. “If our educators’ evaluative expertise through the application of the assessment criteria and standards cannot distinguish between raw AI outputs compared with what students bring to the task (with or without those AI) – then we have a problem with our assessment designs – not with AI. (Professor Michael Henderson, 2023) “WHY SHOULD WE ASSESS SOMETHING THAT AI CAN DO JUST AS WELL OR BETTER?” https://der.monash.edu/defending-assessment-from-ai
- 20. Designing assessment for an AI world… • Re-visiting what we are trying to assess • Emphasising the process - product is a poor proxy of learning • Finding ways of assessing the uniqueness of human processing • Empowering learners to own the assessment process • Focusing more on creative assessment designs – that elicit evidence of learning that matters for ‘a life worth living’
- 22. Augmenting Rather than Automating • Learning to ask the right questions • Learning how to make good use of a devil’s advocate • Learning to make judgments of what counts as quality • Learning that effortlessness is not always the goal to strive for • Learning to find the light through the gaps/cracks https://unesdoc.unesco.org/ark:/48223/pf0000385146
- 23. 3. Developing positive habits for healthy minds Photo by Jelleke Vanooteghem on Unsplash
- 24. 3. Developing positive habits for healthy minds https://lx.uts.edu.au/collections/artificial-intelligence-in-learning-and-teaching/resources/five-principles-for-effective-ethical-use-generative-ai/ Five principles…
- 27. https://www.unesco.org/en/digital-education/ai-future-learning/competency-frameworks
- 28. Actions… • Shaping regulations and policies • Critical review of Assessment practices • Actively engaging in communities of practice • Supporting teachers’ professional development • Developing, implementing and researching pilot initiatives How?
- 32. Two Truths and a Lie I’m no expert in AI but I have a growing research interest in the area I have a strong track record of research on digital forms of Assessment I haven’t taught in the schooling sector for over 25 years
- 33. Go raibh maith agaibh! Thank you