Item Writing Guidelines
- 2. An item, also called an item set, is composed of four questions that measure the same discrete skill or concept at the same level of difficulty. We’ll be looking at item sets in another module. A question consists of a stem, a key, and three distractors. Let’s take a closer look at these individual parts. The Item
- 3. Creating a Question Stems, Keys, and Distractors:
- 4. • A stem can take the form of a question or a statement. • Stems are best written as complete sentences. They can also take the form of a statement with a blank at the end. • Be careful not to overdo fill-in-the-blank questions; a given item set should probably only contain one, if any. Composing Stems
- 5. Acceptable: A dash is a punctuation mark used to _____. Acceptable: Aristotle defines rhetoric as: Better: A dash is a punctuation mark with what function? What function does a dash have? How does Aristotle define rhetoric? Aristotle’s definition of rhetoric includes what? What does Aristotle’s definition of rhetoric include?
- 6. • Stems can also include a chart, table, or other graphic. This is especially useful in courses such as business or finance, but can fit with any subject, as with this question from a remedial reading course: Textbooks often use graphics to aid the reader's comprehension. Please examine the graphic. Which step would you take first in the process illustrated here?
- 7. • Stems should not contain negatives such as “except” or “not.” Not Acceptable: Which of the following is not an example of a service company? Better: Which of the following is an example of a service company?
- 8. The following is also a negative stem: Which of the following should be avoided when writing a topic sentence? Including supporting details (correct) Including a topic and an opinion Providing an accurate explanation of what will follow Including clear and easy to follow language This question asks the student to choose something that’s incorrect, with distractors that are “correct.” We want to avoid this style of question because of a concept called imprinting. We want our students to remember the correct answer. Think of it as positive reinforcement: students should be receiving validation that what they’ve learned and concluded is right. In our curriculum and our items, we want to model the correct answer for the students; negatives can inadvertently reinforce the wrong concept.
- 9. Composing Keys and Distractors • The second half of a question consists of a key and three distractors. • Distractors should be plausible, but incorrect. What kind of computer skills will an employer look for first in a job applicant’s resume? Microsoft Office World of Warcraft Pokémon Mario Brothers What kind of computer skills will an employer look for first in a job applicant’s resume? Microsoft Office QuickTime Microsoft Silverlight Google Not plausible Plausible, but incorrect
- 10. One of the biggest problems item writers can run into is creating distractors that are good— but not too good. In other words, the distractors have to be plausible without being logical answers. Take a look at the following items. Can you see how a student could argue for answering with one of the distractors? Ex. 1: Team Development and Motivation Tom is building a team. What characteristic elements best define the team he eventually builds? • A shared goal and an interdependent group of people (correct) • Independent team members and goal achievement • Diversity and perseverance • Working groups and customer service focus Ex. 2: Fundamental Reading Which of the following actions would you take before reading an unfamiliar textbook? • Preview the book and scan the chapter titles and sub- headings (correct) • Look up the author and determine her purpose for writing the book • Read the summary at the end of each chapter • Look up all the unfamiliar words in a dictionary When writing, try to look at the item from a student perspective. If you wouldn’t be able to answer the question on a test, the student certainly won’t be able to. Your course design team member will help you catch most distractor problems, but please do keep this issue in mind.
- 11. • Keys and distractors will appear in a randomly generated order on the test. Thus, answers such as “none of the above” and “all of the above” are not appropriate. • Multiple selection answers (for example, “Both a and b” or “Only b and c”) will also not work on the platform and should be avoided. More on Distractors
- 12. Structure • Keys and distractors should match one another in terms of length and grammatical structure. This prevents students from basing their choices (consciously or subconsciously) on minor structural differences • They should also be capitalized and do not have a period at the end There is one exception: In a course such as Fundamental Writing, which involves recognizing and correcting complete sentences and punctuation, keys and distractors can take periods.
- 13. When practicing active listening, what does it mean to paraphrase the speaker's message? • To restate what you think the speaker was trying to communicate • To write down exactly what the speaker is saying as he or she is talking • To make bullet points about what the speaker is communicating • To remember only the factual information that the speaker communicated When practicing active listening, what does it mean to paraphrase the speaker's message? • To restate what you think the speaker was trying to communicate • Writing down what the speaker says • To make bullet points about what the speaker is communicating and follow up with the speaker when he or she finishes • Remember factual information from the speaker’s message Look at the examples above. The key and distractors in the left-hand example do not match in terms of structure and length. A student may use these differences to eliminate or choose answers. The example on the right-hand side shows a corrected version of the question.
- 14. • Avoid giving clues in the stem that will lead the student to the right answer. In other words, don’t use the same words in both the stem and the key. 1. _______ refers to a kind of economic cooperation wherein member countries are free to independently determine trade policies with nonmember nations. a. Trade zone b. Export processing zone c. Free trade d. Port trade Better: What type of economic cooperation involves member countries independently determining trade policies with nonmember nations?
- 15. Most of the information for a question should be in the stem, rather than in the key or distractors. Avoid duplicating material in the answer choices, and include the information in the question stem instead. The European Union is an example of an economic union because: a. The union frames the economic policies of the member countries individually b. Some areas of the member countries have economic and other laws that are more free- market-oriented than the country's national laws c. The union frames common administrative policies for the member countries d. Member countries are free to independently determine trade policies with nonmember nations Better: What type of policies does the European Union frame for their member countries? a. Economic b. Free-market-oriented c. Common administrative d. Trade (Removed redundant information from distractors and added to stem)
- 16. • Stems should ideally be between two and three sentences long, and keys and distractors should consist of a word, phrase, or sentence. Students have a 90-minute time limit on their final exams, most of which will have 50 questions. If students are spending more than a minute reading each question, they will have trouble finishing the exam on time. Another consideration is student fatigue. Because the tests are relatively long, some students will begin to perform poorly as they become exhausted later in the test. Excessive numbers of long stems or distractors will contribute to fatigue and frustration levels. In some cases, a longer stem is acceptable. Take a look at the examples of overly lengthy and acceptable stems, keys, and distractors on the next slides. Length
- 17. Longer stem is acceptable: In this case, all of the information in the stem is necessary to understand the question itself (contrast this to the stem from Conflict Management on the next slide, which didn’t require all of the dialog and information to understand the question). These longer stems will be more common in courses such as business or marketing, which often use case studies and realistic scenarios. Principles of Marketing After conducting extensive research on tablets, John bought an iPad from the Apple store. After using it for a couple of weeks, he realized it does not have the functionality he assumed it would have. He is wondering if he should have obtained more information and made more comparisons than he did before purchasing his iPad. Which of the following is John suffering from? Post-purchase dissonance (correct) Product-evaluation psyche Evaluative-criteria psyche Post-purchase euphuism
- 18. Stem too long: This stem is long and convoluted. Imagine being in the middle of a timed test and coming across this - it could be stressful because you’d be uncertain of what the question was asking and what you should be paying attention to. One possible fix would be to cut some or all of the dialog component. Conflict Management Andrew was talking to Jordan and said, "That teacher has it in for me. She always ignores me in class and totally cheated me on that last assignment. I am not going to tolerate her anymore. I'm going to give her a chance to make this right and apologize, but if not, I will have her job." So, Andrew's initial goals were relational (being ignored) and content (being cheated). When he went into Mrs. Russell's office, he said, "I would like you to regrade my essay response on the last test. I really don't think you understood what I meant, Mrs. Russell". In response, she said, "Alright Andrew, I will look at it and return it to you in our next class." He smiled and left her office. Changing goals during a conflict is common. What goal was he smiling about when he left Mrs. Russell's office?
- 19. Answer and distractors too long: One sign that you can shorten your distractors is repetition. If something within the answers/distractors is being repeated, try moving that information to the stem. In the example, the first phrases (“Though the actors/actresses in pictures were occasionally identified in the trade press” and “studios resisted publicizing their stars in the popular press in a deliberate attempt”) could either be deleted or put in the stem. Introduction to Film In the early years of cinema (around 1900-1910), studios refrained from identifying their actors/actresses in the media. Why did they do this? A: Though the actors/actresses in pictures were occasionally identified in the trade press, studios resisted publicizing their stars in the popular press in a deliberate attempt to prevent a costly star system D: Though the actors/actresses in pictures were occasionally identified in the trade press, studios resisted publicizing their stars in the popular press in a deliberate attempt to communicate specific personas crafted for each major actor/actress D: Though the actors/actresses in pictures were occasionally identified in the trade press, studios resisted publicizing their stars in the popular press in a deliberate attempt to prevent fans from intruding on the lives of stars D: Though the actors/actresses in pictures were occasionally identified in the trade press, studios resisted publicizing their stars in the popular press because the studios felt the movie was the star and not those who appeared in it; everyone who worked on a film was equal
- 20. Appropriate Vocabulary and Language Other Considerations:
- 21. Many of our students are international or are returning for a degree after many years of not being in school. Keep this in mind when composing questions, and try to match the level of vocabulary to the level of the course (the wording in a 100 level course will look quite a bit different from that in a 600 level course). • Don’t use slang or idiomatic language, and be aware of cultural or socio- economic issues in your choice of words (for example, not everyone will know what foie gras is) • Stay away from jargon and terminology unique to the textbook(s) of the course • Avoid bias and stereotyping, even when you’re presenting something as incorrect information • Include only abbreviations that are commonly used in discussing the subject matter being tested (for example, m/s in Physics)
- 22. • Be creative: student engagement will impact performance • Incorporate job-related situations, especially in higher level courses • Give clues to a problem and ask for a best course of action • Avoid having an excessive number of long or involved questions; students have a time limit of 90 minutes on the final exam • When in doubt, ask your course design team member! Other Tips