Harnessing the Power of Categories: Using Categories to Improve Exam Questions

Editor's Notes

• #3: There are many benefits to using categories. During the last webinar I discussed using the Categories area to physically build assessments. Today I will focus on their usefulness for reviewing and improving exam questions.
• #5: Questions by category are basically a subset or group of questions from the Folder area. Each institution’s categories are based on their specific needs. Here at UNMC we have the illustrated categories plus many more. Think of it as a way to filter or sort out certain questions you want to see or work with.
• #7: At UNMC we are guided by two things when determining if a question meets our formatting requirements. One is the NBME Writing Guide and the other is our own particular institution’s preferences.
• #8: First let me explain that NBME stands for the National Board of Medical Examiners. As you may remember from my introduction, I work with the first and second year medical students at our university. The NBME offers a variety of educational materials and instructional opportunities to help enhance the quality of test items used by educators and assessors in the healthcare professions around the world. The NBME's Item Writing Manual, Constructing Written Test Questions for the Basic and Clinical Sciences, is a resource for faculty members at schools of medicine, schools for healthcare professionals, those invested in the assessment of competence of healthcare providers, and others interested in learning how to write better quality test items. A link to the downloadable manual is listed. As just mentioned, this information is not just limited to use in the healthcare professions so I encourage you to take a look.
• #9: In addition to the writing guide manual, NBME offers several other item writing services with their tutorials, seminars and workshops.
• #10: Another major guiding force here at UNMC is the use of Bloom’s Taxonomy. This topic alone needs its own webinar, which guess what? Has already been done. You can find these webinars on the ExamSoft website and plenty of information on the web. Because of this, I will not get very deep into this topic. Suffice it to say that we try to use higher level taxonomic questions with our students to prepare them for their board exams and licensure.
• #11: I have listed a few examples here of types of questions that do not meet our formatting requirements. The first one – Not phrased as a question – is exactly that. These are actually question stems that end with a semicolon and continue on into the question foils to complete a sentence. All of the Above and None of the Above questions do not contain a simple answer or response and often cue the student to the correct answer. True/False questions require personal interpretation and may be confusing to students The last example is something specific for our students and institution.
• #12: Here is a list of categories we use for inappropriate formatting and a few other categories, some of which are more specific to our institution. There are two other important formatting issues I would like to mention: Remove Negative Phrasing – these are questions that contain the word “NOT” in the question stem or include All of the following EXCEPT: Two Answer Choice Allowed – these are questions that were determined after an exam to have had two possible correct answer.
• #13: Here is another major category, and sub categories, regarding the actual statistical performance of the exam questions. It can be broken up into two areas: the actual performance stats and whether or not it was removed from scoring.
• #14: The first thing we look at with our stats is the difficulty. This is the percentage of students that answered the question correctly. If the difficulty is below 0.50, then we look at the discriminator or point biserial. If it is below 0.20 the question is tagged with poor statistical properties, and usually removed from the exam. Negative discriminators also indicate a possible problem and are noted for review.
• #15: Questions that are removed from scoring after an exam are tagged with this information for the appropriate academic year.
• #16: Here is a screenshot of how we have our categories set up for poor statistics The last category is designed to be used by a course director if they need the question looked over by a content expert.
• #17: Unfortunately someone has to go through and tag all of the questions that need to be reviewed. Don’t worry, this process is not as bad as it sounds. This task was performed by myself and my colleague once we had imported all of our questions. I would highly recommend that you do this as questions are added to your database. We imported over 4,000 questions in our first year, so this was quite a project to go through all of the questions after that.
• #18: In my last webinar I talked about setting up a “construction zone” under the Categories area to build assessments. Using a similar method, we created an area for the faculty to review their questions. We set up the main category “2016 M1 Item Review” and then set up sub-categories based on the different courses. This allows the appropriate course director the ability to only see and review questions associated with their course.
• #19: Use the “Advanced Search” area located in the upper right area of your screen.
• #20: Search for questions with the category “Requires Review.” Scroll down and select done.
• #21: Select the course folder where all of the questions are located.
• #22: Select “All Questions” in the status field and then select “Search.”
• #23: You will now have a list of all of the questions from that course folder that “Requires Review.” Select all and add the review category tags to all of the question using the bulk add feature. Make sure to select “Update Categories” when you are all done.
• #24: All of this leads us back to the listing of questions, by course, in the 2016 Review category. This is the location for your faculty to access their questions. Once they click on the review category for their course, they will see all of the question they need to review. Based on this screenshot, if the Neuroscience Course Director click on his course review category, he would see the 129 questions located there. Most importantly, make sure to give the Course Director and/or Faculty a deadline to complete their review.
• #25: For those questions that are not revised or negotiated to remain the same, we add the word “ARCHIVE” at the front of the questions title. This indicates to us that the question should not be used on an assessment. By doing this, the question will still remain categorized in the Questions by Folder area, so the faculty can still see it, but it will not be used on the exam in its current state. Eventually the “ARCHIVE” questions will be removed from the folders, and then eventually deleted from the system.
• #26: Here is a diagram to help with the process for those that are more visual. The entire circle shows all of the questions in the ExamSoft database, located in the Questions by Folders area. Inside of the entire circle is a small subset of questions that need to be reviewed. Inside of the review circle are the course subsets. Remember, these are not separate sets of questions. They are just sorted out by categories to make them easier to review for the appropriate person.
• #27: And finally – a very important reminder.