Assessing with confidence

Editor's Notes

• #2: Repeat of talk about 5 years ago – what has changed? Answer not much! Will review why it is a good assessment technique Not been able to persuade OU colleagues to use it Remains niche in rest of world – but may have found a good niche
• #3: This a confidence-based question Ask a question – often multiple choice, doesn’t have to be But also ask students to report their certainty Can give feedback as normal, mark as CBM.
• #4: Teachers & trainers need to know what students know and what they don’t know. But both students and teachers need to know whether students are certain about what they know and don’t know – if not, there could be problems. At worst, they could think they know but be wrong – and therefore make mistakes. If they are uncertain, they can’t make correct decisions.
• #5: Certainty or confidence? Subtlety in language – claim that what students are indicating and what CBM encourages is certainty, not confidence as character trait. In typical university settings, students are driven by marks! Show an example later which has taken a gaming approach Some examples: First, student who is tentative – if they get it correct they get some marks, if they get it wrong get nothing. Next, student who is both correct and certain – they get maximum marks. Finally, student who is overconfident – and wrong. They get penalty. So really important to judge certainty realistically – if you report high certainty, risk losing marks if you are wrong. If you report low certainty, you can’t score good marks.
• #6: Some examples: An easy question that everyone is certain about – average score is high A really difficult question on quantum physics – nearly everyone gets wrong, so score is close to zero. The tricky one – something that many university students ought to know, but quite a few get wrong, so average mark is negative.
• #7: Detail shows how students are answering Easy question: most answer with high confidence Difficult question: most answer with low confidence Tricky question: splits into those who know, and those who don’t.
• #8: Complicated graph! First note there is lots of data. This is for students doing practice tests for learning & revision; not for serious assessment. Means you get full range of poor to good results. Some students better than others – better students are to right of graph getting greater percentage correct. Weaker students are to left, getting fewer correct. If they get more correct, then expect score to be higher – that’s on vertical scale. But you can score best by setting certainty correctly, so someone who got say 60% correct overall would get higher mark if they set certainty/confidence sensibly for each question. Scores above the pale green line with corners show successfully judging confidence -- green line shows always setting confidence low, medium and high, but not adjusting for each individual question. Corners represent places where switch from low to medium to high confidence should occur, if student knows only how good they are overall. Most students, even for revision where marks don’t count, are setting confidence level sensibly Few students look like they are doing really badly – negative marks overall! But remember these are students exploring, not serious final assessment. Data from Tony Gardner-Medwin’s medical students
• #9: Another way of looking at same data which highlights the effect of certainty – most results are above the line, indicating students are aware of where their knowledge is reliable
• #10: Under exam conditions, then spread is very much reduced. Some low marks, but no negatives. Nearly all students are showing marks above the line – that is, they are correctly assessing their certainty for each question and so maximising their scores compared to answering with certainty set overall. So CBM is delivering accurate assessment of knowledge.
• #11: But CBM is not just about more accurate assessment. Could also have positives for student learning – better engagement with assessment helps students learn.
• #12: Students don’t like extra ‘stress’ imposed by setting certainty – but it is good for them!
• #13: Certainty or confidence as character trait? Can over/under-confidence be dissociated from knowledge in any case? Can argue that correct understanding of own knowledge is essential part of academic and professional practice. Is used in medical contexts because acknowledged that realistic judgement of certainty is essential to skills: ‘in decision-rich occupations such as medicine, mis-calibration of reliability is a serious handicap’ Gardner-Medwin (2014, p.6)
• #14: An open question asks a question but gives no clues. Here simple number is expected, but more generally a phrase, sentence, paragraph… A multiple-choice question gives options – choose the correct one. A confidence-marked question also asks student to say how certain they are of their answer.
• #15: MCQs are well established – objective, reliable, easy to implement But pedagogically not ideal – have some drawbacks. Open questions might be better tools for learning – but they are difficult to implement on a computer
• #16: This is a variant of CBM Starts with an open question. We don’t ask students to submit answer immediately – instead they have to set their confidence. Once set it is locked – can’t change. Now reveal options – actually multiple choice. Can give feedback as normal, mark as CBM.
• #17: Benefit of open CBM is -- an open question, so benefits for reflection -- retains benefits and avoids drawbacks of MCQ -- has some drawbacks – not always easy to implement, and needs further research on personality issues
• #18: Ipace features: -- bank of questions -- few questions each week pushed to students -- posed in CBM format -- repeated until question answered successfully more than once, then dropped from pool -- not used for summative assessment but for competency / mastery
• #19: Asked a question – have to set your confidence first. Confidence/certainty expressed in terms of a bet – how mauch are you prepared to stake that you will get this right? Virtual coins only!
• #20: Once certainty set (= stake bet), then select options
• #21: If incorrect, you lose your stake
• #22: If correct you win the bet – payback is 2 x stake
• #23: Leaderboards so competitive
• #24: Feedback on performance by topic and by confidence level
• #25: Badges as additional motivation
• #26: Fit to mastery: used in medical contexts because acknowledged that realistic judgement of certainty is essential to skills Gardner-Medwin (2014, but Bender?) indicates higher accuracy cf standard marking – students identify uncertain answers so reduces variance so predictive accuracy & reliability of exam improves. Complexity of setting up. Not much platform support – but now in Moodle and in Questionmark Perception. Dislike of negative marking. Difficulty of marking more complex question types: multiple- cf single-response questions, partial correct scores, differently weighted questions.