Out of the long shadow of the NSS: TESTA's transformative potential
Download as PPTX, PDF1 like73 views
This document summarizes the key points from a presentation on TESTA (Transforming the Experience of Students Through Assessment), a project that aims to address three major problems with assessment and feedback in higher education: 1) variations in assessment patterns leading to uncertainty about why some things are going well and others not, 2) an over-reliance on high-stakes summative assessment over formative assessment, and 3) disconnected feedback that does not engage students in a dialogue. The presentation provides evidence from various studies on these problems and strategies that institutions and educators can adopt based on TESTA's work, such as reducing summative assessment, increasing meaningful formative tasks, improving feedback cycles, and involving students in co-creating assessment
Out of the long shadow of the NSS: TESTA's transformative potential
- 1. Out of the long shadow of the NSS: TESTA’s transformative potential @solentlearning @tansyjtweets Tansy Jessop TESTA Project Leader 16 November 2017
- 2. Today’s session • TESTA’s USP • Three problems in A&F in HE • Evidence, strategies, action
- 3. What is the main assessment and feedback challenge you face? Your thoughts
- 5. Research and change process Programme Team Meeting Assessment Experience Questionnaire (AEQ) TESTA Programme Audit Student Focus Groups
- 7. TESTA…. “…is a way of thinking about assessment and feedback” Graham Gibbs
- 9. Three problems TESTA addresses 1. Things going awry but not sure why 2. Curriculum design challenge 3. Academic reading and writing challenge
- 10. It was heavy, tons of marking for the tutor. It was such hard work. It was criminal. Media Course Leader I’m really bad at reading feedback. I’ll look at the mark and then be like ‘well stuff it, I can’t do anything about it’ Student, TESTA focus group 1. Things going awry, not sure why
- 11. Wow! Our students love History! Fantastic!
- 12. Whoops there’s a little problem here
- 13. Fix it! Ok, we’ll look especially at polishing up our feedback. Students seems to find that the least best thing.
- 14. Problem 2: Curriculum design challenge
- 15. Does IKEA 101 work for complex learning?
- 17. The best approach from the student’s perspective is to focus on concepts. I’m sorry to break it to you, but your students are not going to remember 90 per cent – possibly 99 per cent – of what you teach them unless it’s conceptual…. when broad, over-arching connections are made, education occurs. Most details are only a necessary means to that end. http://www.timeshighereducation.co.uk/features/a-students- lecture-to-rofessors/2013238.fullarticle#.U3orx_f9xWc.twitter A student’s lecture to her professor
- 18. Problem 3: Academic reading and writing
- 19. Is assessment environment prompting it enough? Significant learning gains for students who 1) Read > 40 pages a week of academic writing 2) Write > 20 pages per semester for each unit
- 21. TESTA evidence and strategies for… 1. Variations in assessment patterns 2. High summative and low formative diets 3. Disconnected feedback 4. Confusion about goals and standards
- 22. TESTA definitions Summative: graded assessment which counts towards the degree Formative: Does not count: ungraded, required task with feedback
- 23. 1. Huge variations • What is striking for you about this data? • How does it compare with your context? • Does variation matter?
- 24. Assessment features across a 3 year UG degree (n=73) Characteristic Range Summative 12 -227 Formative 0 - 116 Varieties of assessment 5 - 21 Proportion of examinations 0% - 87% Time to return marks & feedback 10 - 42 days Volume of oral feedback 37 -1800 minutes Volume of written feedback 936 - 22,000 words
- 25. Typical A&F patterns Characteristic Low Medium High Volume of summative assessment Below 33 40-48 More than 48 Volume of formative only Below 1 5-19 More than 19 % of tasks by examinations Below 11% 22-31% More than 31% Variety of assessment methods Below 8 11-15 More than 15 Written feedback in words Less than 3,800 6,000-7,600 More than 7,600
- 26. Actions based on evidence a) Reduction in summative b) Increase in formative c) Streamlined varieties d) More or less feedback depending… e) Quantifiable f) Every time a coconut with each feature
- 27. Theme 2: High summative: low formative • Summative ‘pedagogies of control’ • Circa 2 per module in UK • Ratio of 1:8 of formative to summative • Formative weakly understood and practised
- 29. A lot of people don’t do wider reading. You just focus on your essay question. In Weeks 9 to 12 there is hardly anyone in our lectures. I'd rather use those two hours of lectures to get the assignment done. It’s been non-stop assignments, and I’m now free of assignments until the exams – I’ve had to rush every piece of work I’ve done. CONSEQUENCES OF HIGH SUMMATIVE
- 30. It was really useful. We were assessed on it but we weren’t officially given a grade, but they did give us feedback on how we did. It didn’t actually count so that helped quite a lot because it was just a practice and didn’t really matter what we did and we could learn from mistakes so that was quite useful. WHAT ABOUT FORMATIVE?
- 31. If there weren’t loads of other assessments, I’d do it. It’s good to know you’re being graded because you take it more seriously. BUT… If there are no actual consequences of not doing it, most students are going to sit in the bar. The lecturers do formative assessment but we don’t get any feedback on it.
- 32. So, how do engage students in meaningful formative? Go to www.menti.com and use the code 24 59 70 Choose the strategies you think work best
- 33. Case Study 1 • Systematic reduction of summative across whole business school • Systematic ramping up of formative • All working to similar script • Systematic shift, experimentation, less risky together
- 34. Case Study 2 • Problem: silent seminar, students not reading • Public platform blogging • Current academic texts • In-class • Threads and live discussion • Linked to summative
- 36. Principles of good formative 1. Reduce summative to increase formative 2. Whole programme approach 3. Give meaningful feedback 4. Public domain 5. Multi-stage, linked formative and summative 6. Risky, creative, challenging tasks 7. Collaborative
- 37. Theme 3: Disconnected feedback
- 38. System-wide problems • Modular structures impede feedback • Students don’t feel known • There is very little formative feedback • Lecturers do too much judging; students too little
- 39. Mass HE is squeezing out dialogue with the result that written feedback, which is essentially a one-way communication, has to carry almost all the burden of student-teacher interaction David Nicol 2010
- 41. Irretrievable breakdown… Your essay lacked structure and your referencing is problematic Your classes are boring and I don’t really like you
- 42. An impoverished dialogue The many diverse expressions of dissatisfaction with feedback can be taken as symptoms of an impoverished and fractured dialogue David Nicol 2010
- 43. Take five • Choose a quote that strikes you. • What is the key issue? • What strategies might address this issue?
- 44. The feedback is generally focused on the module Because it’s at the end of the module, it doesn’t feed into our future work. If It’s difficult because your assignments are so detached from the next one you do for that subject. They don’t relate to each other. I read it and think “Well, that’s fine but I’ve already handed it in now and got the mark. It’s too late”. STRUCTURAL
- 45. It was like ‘Who’s Holly?’ It’s that relationship where you’re just a student. Because they have to mark so many that our essay becomes lost in the sea that they have to mark. Here they say ‘Oh yes, I don’t know who you are. Got too many to remember, don’t really care, I’ll mark you on your assignment’. RELATIONAL
- 46. Ways to be dialogic • Conversation: who starts the dialogue? • Cycles of reflection across modules • Quick generic feedback • Feedback synthesis tasks • Peer feedback (especially on formative) • Technology: audio, screencast and blogging • From feedback as ‘telling’… • … to feedback as asking questions
- 47. Students feedback to us
- 48. Students to lecturers: Critical Incident Questionnaire Stephen Brookfield’s Critical Incident Questionnaire http://bit.ly/1loUzq0
- 49. Theme 4: Confusion about goals and standards • Consistently low scores on the AEQ for clear goals and standards • Alienation from the tools, especially criteria and guidelines • Symptoms: perceptions of marker variation, unfair standards and inconsistencies in practice
- 50. We’ve got two tutors- one marks completely differently to the other and it’s pot luck which one you get. They read the essay and then they get a general impression, then they pluck a mark from the air. It’s like Russian roulette – you may shoot yourself and then get an A1. They have different criteria, they build up their own criteria.
- 51. There are criteria, but I find them really strange. There’s “writing coherently, making sure the argument that you present is backed up with evidence”.
- 54. Having ‘an eye for a dog’
- 55. The Art and Science of Evaluation Judging is both an art and a science: It is an art because the decisions with which a judge is constantly faced are very often based on considerations of an intangible nature that cannot be recognized intuitively. It is also a science because without a sound knowledge of a dog’s points and anatomy, a judge cannot make a proper assessment of it whether it is standing or in motion. Take them round please: the art of judging dogs (Horner, 1975).
- 56. Taking action: internalising goals and standards • Regular calibration exercises • Discussion and dialogue • Discipline specific criteria (no cut and paste) Lecturers • Rewrite/co-create criteria • Marking exercises • Discussing exemplars Lecturers and students • Enter secret garden - peer review • Engage in drafting processes • Self-reflection Students
- 58. References Arum R. and Roksa J. 2011. Academically Adrift: Limited Learning on College Campuses. University of Chicago Press. Barlow, A. and Jessop, T. 2016. “You can’t write a load of rubbish”: Why blogging works as formative assessment. Educational Development. 17(3), 12-15. SEDA. Boud, D. and Molloy, E. (2013) ‘Rethinking models of feedback for learning: The challenge of design’, Assessment & Evaluation in Higher Education, 38(6), pp. 698–712. Carr,. N. 2010. The Shallows. London. Atlantic Books Gibbs, G. & Simpson, C. (2004) Conditions under which assessment supports students' learning. Learning and Teaching in Higher Education. 1(1): 3-31. Harland, T., McLean, A., Wass, R., Miller, E. and Sim, K. N. (2014) ‘An assessment arms race and its fallout: High-stakes grading and the case for slow scholarship’, Assessment & Evaluation in Higher Education. Jessop, T. and Tomas, C. 2016 The implications of programme assessment on student learning. Assessment and Evaluation in Higher Education. Published online 2 August 2016. Jessop, T. and Maleckar, B. (2014). The Influence of disciplinary assessment patterns on student learning: a comparative study. Studies in Higher Education. Published Online 27 August 2014 http://www.tandfonline.com/doi/abs/10.1080/03075079.2014.943170 Jessop, T. , El Hakim, Y. and Gibbs, G. (2014) The whole is greater than the sum of its parts: a large-scale study of students’ learning in response to different assessment patterns. Assessment and Evaluation in Higher Education. 39(1) 73-88. Nicol, D. (2010) From monologue to dialogue: improving written feedback processes in mass higher education, Assessment & Evaluation in Higher Education, 35: 5, 501 – 517. O'Donovan, B , Price, M. and Rust, C. (2008) 'Developing student understanding of assessment standards: a nested hierarchy of approaches', Teaching in Higher Education, 13: 2, 205 -217. Sadler, D. R. (1989) ‘Formative assessment and the design of instructional systems’, Instructional Science, 18(2), pp. 119–144. doi: 10.1007/bf00117714.
Editor's Notes
- #2: Tansy
- #7: How do you measure soft stuff? 5 day cricket match versus 20/20
- #8: What started as a research methodology has become a way of thinking. David Nicol – changing the discourse, the way we think about assessment and feedback; not only technical, research, mapping, also shaping our thinking. Evidence, assessment principles. Habermas framework.
- #9: Academics operate in isolation from one another. Only see their part of the degree. Don’t see connections. Fragments into small tasks – hamster wheel. Curriculum design issue. The trouble is that students experience the whole elephant and it is often indigestible… Assessment is mainly sort of the topical knowledge and the topics never relate. We'll never do something again that we’ve already studied, like we learn something and then just move on (TESTA focus group data).
- #11: Feedback: all that effort, but what is the effect? Margaret Price. Feedback more important to teachers than students… But lots of projects and programmes do….
- #16: Hard to make connections, difficult to see the joins between assessments, much more assessment, much more assessment to accredit each little box. Multiplier effect. Less challenge, less integration. Lots of little neo-liberal tasks. The Assessment Arms Race.
- #17: Language of ‘covering material’ Should we be surprised?
- #19: Exeter economics/
- #29: Teach Less, learn more. Assess less, learn more.
- #49: Is anyone listening?
- #57: Students can increase their understanding of the language of assessment through their active engagement in: ‘observation, imitation, dialogue and practice’ (Rust, Price, and O’Donovan 2003, 152), Dialogue, clever strategies, social practice, relationship building, relinquishing power.