Engaging learners in computer-based summative exams: Reflections on a participant-informed assessment design
- 1. Engaging learners in computer-based summative exams University of York, UK Richard Walker & Zoe Handley ALT-C Shaping the Future of Learning Together 8th – 10th September 2015
- 2. Computer-based testing in HE: A review Most research has focused on the differential impact of computer-based vs. pen-and-paper assessments on student achievement (Kingston, 2008; Leeson, 2006; Mead & Drasgow, 2008) Attention has been directed to the relationship between individual differences (gender, race, digital literacy) and performance on computer-based assessments (Leeson, 2006) Typically this has been case study research / surveys with a focus on the UG experience Only a limited number of studies on students’ attitudes (e.g. Dermo, 2009; Hillier, 2014; Walker, Topping & Rodrigues, 2008)
- 3. Our challenge Designing a computer-based assessment for international postgraduate students in research methods (MA Language Learning & Teaching) Developing an assessment to address criticality & higher order thinking (combining MCQ & open questions) Delivering an assessment successfully across multiple test venues simultaneously for 150+ PG international students, with no prior exposure to computer-based testing at the University Managing the assessment within a ‘greenfield’ institutional context, with no established policy / protocols in place.
- 4. An evolutionary approach: participative- informed design • VLE hosted testing & exam (2013-2014) • VLE exam as a supported service (2014- 15) • Benchmarking & technology review (2012- 2013) • QMP trials (2009-2012) Formative tests – low stakes Technical review & dummy tests Formative testing informing higher stakes exam Formative & summative exams – high stakes
- 5. QMP trials (2009 – 2012) – LOW STAKES Strategic IT Projects Group (sponsor) E-Learning & IT Services E-learning, IT networking & desktop services Education & Management School E-learning, IT networking & desktop services Identification of QMP as software solution Configuration & performance challenges (missing data) Configuration & performance challenges (connection speed & submission problems)
- 6. Benchmarking & technology review (2012-2013) Strategic IT Projects Group (sponsor) E-learning / IT & Education Department E-learning, IT networking & desktop services Education Department Education Department; IT and e-learning services; Exams Office Students Reappraisal of Blackboard VLE’s assessment engine Creation of VLE Exam instance – with locked down desktop Issues over exam design: question volume & balance
- 7. VLE hosted exams (2014 – 2015) – HIGH STAKES Education Department; IT & e-learning services Exams Office Students Education Department; IT & e-learning services Exams Office Students Education Department; IT & e-learning services Exams Office Students Review of question-set: Ratio of MCQ & open questions; randomisation & sequencing Successful delivery of exam (160 students)
- 8. Exploratory research framework 2013-14 cohort (n= 155) Survey after summative exam (Jan 2014) Focus group (n=5) after summative exam, before release of results Qualitative content analysis on transcripts & free text survey comments: Focus on reception of computer- based testing methods (attitudes / experiences) 2014-15 cohort (n=160) Surveys after formative (mid- term) and summative exam (Jan 2015) Focus groups (n=18) after summative exam, before release of results Qualitative content analysis on transcripts & free text survey comments: repeated & combined with 2013-14 data to form rich picture of students’ reception of assessment methods
- 9. Engagement Issues: Socialisation of Learners Prior exposure to e-assessment: rationale and perceived fairness & equity Issues “It’s kind of fair for most students because using a computer is almost a necessity for us and especially for our generation but it is not as much fair, as other generation.” “Some of our classmates after they have had some experience … they return to school to get more experience in teaching. It could be some difficulty for them to use a computer in typing when they attend the examination, so it could take them longer time to get used to the system, so I think it could unfair for them.” Post-Test Focus Group 2014-15
- 10. Engagement Issues: Students’ preparation for assessment Keyboard proficiency under exam conditions: familiarisation with exam environment & controls Issues “I’m not used to using the keyboard because it’s different from laptop keyboard.” Post-Test Focus Group 2014-15 “I feel in the real exam, I found there’s no correction tools for you to correct.”
- 11. Engagement Issues: Exam technique Online exam craft - question selection, time management… Issues 2014-2015 Pre-Test Questionnaire “I do not like not having the ability to circle questions I am unsure about or make notes to myself about which questions to come back to. During written assessments, I often write all over my test questions with arrows, circles, and other brainstorming sketches and it is difficult to work through the online assessment without these techniques” “In terms of time management…when we are doing the handwriting exam, I know what questions to I have, but in e- exam I just didn’t know what I am currently facing and I don’t know what kind of questions, you know closed or open question or is coming next ” Post-Test Focus Group 2014-15
- 12. Engagement Issues: Revision strategy Management of self-studyIssues “In China we will focus on the memory so we try to remember the long answers to these questions……for the Chinese exam, I will remember all of the answers, long sentences, but I will not do this for this module.” Post-Test Focus Group 2014-15
- 13. Engagement Issues: Assessment Design and Interface Organisation and presentation of question-set, preparation of user interface Issues “It’s no sense to put an open question for ten points at the beginning, so because our brain doesn’t work at the beginning to write/type so much.” “Random questions for each student don’t represent the level of difficulties, for some students could encounter long answer question at Q1 which gives little confidence of students to move on. More it could also waste time in trying to answer that question and therefore time is not enough” Post-Test Focus Group 2014-15 Questionnaire 2013-2014
- 14. Questions? richard.walker@york.ac.uk zoe.handley@york.ac.uk z Richard Walker & Zoe Handley University of York, UK
- 15. Discussion points 1. How does our experience compare with your experience (with UGs / PGs, in a greenfield / mature institutional context)? 2. What recommendations would you make based on our results / your experience? 3. What developments do you foresee in computer- based testing in the next 5 years? What are the implications for learner engagement?
- 16. Recommendations Our research highlights the importance of: – Socialisation of learners focusing on the aims and rationale for eAssessment – Providing students the opportunities for practice necessary to develop IT proficiency for eAssessment and test-taking strategies for eAssessment – Asessment interfaces should be flexible and intuitive to accommodate a range of test-taking strategies – ‘one-interface’ does not fit all students
- 17. The LEeAP framework • Orientation of students to assessment methodsSocialisation • Guidance and preparation of students for e- assessment: Digital skills; Exam technique; Revision strategy Preparation of students for assessment • Organisation and presentation of question- set, preparation of user interface and assessment venue: Assessment design; Design of assessment interface; Preparation & management of assessment centre(s) Assessment design and interface
- 18. The LEeAP framework Read our working paper: – Walker, R. & Handley, Z. (in preparation). Designing for Learner Engagement with eAssessment Practices: The LEeAP Framework. For submission to ALT-J. At: – http://tinyurl.com/LEeAP All comments and feedback welcome!
- 19. References Dermo, J. (2009). e-Assessment and the student learning experience: A survey of student perceptions of e-assessment. British Journal of Educational Technology, 40(2), 203-214 Hillier, M. (2014). The very idea of e-Exams: Student (pre) conceptions. . In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 77-88). Kingston, N. M. (2008). Comparability of computer- and paper- administered multiple- choice tests for K-12 populations: A synthesis. Applied Measurement in Education, 22(1), 22-37 Leeson, H. V. (2006). The mode effect: A literature review of human and technological issues in computerized testing. International Journal of Testing, 6(1), 1-24 Mead, A. D. & Drasgow, F. (1993). Equivalence of computerized and paper and pencil cognitive ability tests: A meta-analysis). Psychological Bulletin, 114, 449-458. Walker, D. J., Topping, K., & Rodrigues, S. (2008). Student reflections on formative e- assessment: Expectations and perceptions. Learning, Media and Technology, 33(3), 221-234.
- 20. Thank you! richard.walker@york.ac.uk zoe.handley@york.ac.uk z Richard Walker & Zoe Handley University of York, UK