How to develop completion and short answer tests?
- 1. Completion, Completion, Short-Answer, and Short-Answer, and True-False Items True-False Items Educational Assessment of Students Educational Assessment of Students Chapter 7 Chapter 7 Anthony J. Nitko Anthony J. Nitko (Brenda Stewart) (Brenda Stewart)
- 2. Three Fundamental Principles for Three Fundamental Principles for Crafting Assessments Crafting Assessments Focus each assessment task entirely on important Focus each assessment task entirely on important learning targets learning targets Craft each assessment task to elicit from students only Craft each assessment task to elicit from students only the knowledge and performance that are relevant to the the knowledge and performance that are relevant to the learning targets you are assessing learning targets you are assessing Craft each assessment task to neither prevent nor Craft each assessment task to neither prevent nor inhibit a student’s ability to demonstrate attainment of inhibit a student’s ability to demonstrate attainment of the leaning targets you are assessing the leaning targets you are assessing
- 3. Short-Answer Items Short-Answer Items Require a word, short phrase, number, or Require a word, short phrase, number, or symbol response. symbol response. Three types of short-answer items: Three types of short-answer items: Question – What is the capital of Texas? Question – What is the capital of Texas? Completion – The capital of Texas is _________. Completion – The capital of Texas is _________. Association – On the blank next to the state name, Association – On the blank next to the state name, write the abbreviation for that state. write the abbreviation for that state. Texas Texas TX TX California California CA CA
- 4. Usefulness of Short-Answer Items Usefulness of Short-Answer Items Abilities Assessed Abilities Assessed Lower-order thinking skills Lower-order thinking skills Recall Recall Comprehension of information Comprehension of information Higher-level abilities Higher-level abilities Make simple interpretations of data and applications of Make simple interpretations of data and applications of rules rules Ability to solve numerical problems in science and Ability to solve numerical problems in science and mathematics mathematics Ability to manipulate mathematical symbols and balance Ability to manipulate mathematical symbols and balance mathematical and chemical equations mathematical and chemical equations
- 5. Usefulness of Short-Answer Items Usefulness of Short-Answer Items Strengths Strengths Relatively easy to construct Relatively easy to construct Can be scored objectively Can be scored objectively Lowers the probability of getting the answer correct Lowers the probability of getting the answer correct by random guessing by random guessing Shortcomings Shortcomings Not free of subjectivity Not free of subjectivity Cannot anticipate all possible responses Cannot anticipate all possible responses Tends to lower the reliability of the obtained score Tends to lower the reliability of the obtained score
- 6. Crafting Short-Answer Items Crafting Short-Answer Items Importance of what is assessed Importance of what is assessed How does it match the test blueprint How does it match the test blueprint Question format is the preferred format for a Question format is the preferred format for a short-answer item. Focus the item on the short-answer item. Focus the item on the specific knowledge sought specific knowledge sought Word the items specifically and clearly Word the items specifically and clearly Put the blank near the end of the sentence Put the blank near the end of the sentence If possible do not copy statements verbatim If possible do not copy statements verbatim
- 7. Crafting Short-Answer Items Crafting Short-Answer Items A completion item should omit important words A completion item should omit important words and not trivial words and not trivial words Limit blanks to one or two Limit blanks to one or two Keep all blanks the same length Keep all blanks the same length Specify the precision you expect in your answer Specify the precision you expect in your answer Avoid irrelevant clues Avoid irrelevant clues
- 8. True-False Items True-False Items Consists of a statement or proposition the Consists of a statement or proposition the student may judge as true or false student may judge as true or false Six varieties: Six varieties: True-False True-False Yes-No Yes-No Right-Wrong Right-Wrong Correction Correction Multiple True-False Multiple True-False Yes-No with explanation Yes-No with explanation
- 9. Usefulness of True-False Items Usefulness of True-False Items Advantages Advantages Certain aspects of the subject matter readily lend themselves Certain aspects of the subject matter readily lend themselves to verbal propositions that can be judged true or false to verbal propositions that can be judged true or false Relatively easy to write Relatively easy to write Scored easily and objectively Scored easily and objectively Can cover a wide range of content within a relatively short Can cover a wide range of content within a relatively short period period Criticisms Criticisms Poorly constructed assess only specific, frequently trivial facts Poorly constructed assess only specific, frequently trivial facts Ambiguously worded Ambiguously worded Answered by random guessing Answered by random guessing Encourage students to study and accept only oversimplified Encourage students to study and accept only oversimplified statements of truth and factual details statements of truth and factual details
- 10. Usefulness of True-False Items Usefulness of True-False Items Generalizations in a subject area Generalizations in a subject area Comparisons among concepts Comparisons among concepts Casual or conditional propositions Casual or conditional propositions Relationships between two events, concepts, Relationships between two events, concepts, facts, or principles facts, or principles Explanations for why events or phenomena Explanations for why events or phenomena occurred occurred Instances or examples of a concept or principle Instances or examples of a concept or principle
- 11. Usefulness of True-False Items Usefulness of True-False Items Evidential statements Evidential statements Predictions about phenomena or events Predictions about phenomena or events Steps in a procedure of process Steps in a procedure of process Computations (or other kinds of results Computations (or other kinds of results obtained from applying a procedure) obtained from applying a procedure) Evaluations of events of phenomena Evaluations of events of phenomena
- 12. Validity of True-False Item Format Validity of True-False Item Format Essence of educational achievement is the Essence of educational achievement is the command of useful verbal knowledge command of useful verbal knowledge All verbal knowledge can be expressed in All verbal knowledge can be expressed in propositions propositions Proposition is any sentence that can be said to Proposition is any sentence that can be said to be true or false be true or false Particular area of knowledge is indicated by Particular area of knowledge is indicated by success in judging truth or falsity of propositions success in judging truth or falsity of propositions
- 13. Guessing on True-False Items: Guessing on True-False Items: An Ill Wind? An Ill Wind? Guessing – getting 50% correct Guessing – getting 50% correct Blind (complete random guessing) quite unlike Blind (complete random guessing) quite unlike informed guessing (based on partial knowledge) informed guessing (based on partial knowledge) Well-motivated guess blindly only on a small Well-motivated guess blindly only on a small percentage of questions on a test percentage of questions on a test Difficult to obtain a good score on a test by blind Difficult to obtain a good score on a test by blind guessing only guessing only A given true-false test has a high reliability A given true-false test has a high reliability coefficient – evidence blind guessing does not coefficient – evidence blind guessing does not seriously affect the scores seriously affect the scores
- 14. Guessing on True-False Items: Guessing on True-False Items: An Ill Wind? An Ill Wind? Random guessing – sometimes called blind Random guessing – sometimes called blind guessing guessing Partial knowledge – make an informed guess Partial knowledge – make an informed guess Higher than 50-50 chance of success Higher than 50-50 chance of success Measures of partial knowledge Measures of partial knowledge Full knowledge – 100% chance of answering Full knowledge – 100% chance of answering correctly correctly Percentages of correct answers go down with Percentages of correct answers go down with the number of blind guesses made the number of blind guesses made
- 15. Suggestions for Getting Started Suggestions for Getting Started Properly Properly Create pairs of items, one true and one false Create pairs of items, one true and one false Rephrase evaluative judgments into a Rephrase evaluative judgments into a comparative statement comparative statement Write false statements that reflect the actual Write false statements that reflect the actual misconceptions held by students misconceptions held by students Convert a multiple-choice item into two or more Convert a multiple-choice item into two or more true-false items true-false items
- 16. Suggestions for Improving Suggestions for Improving True-False Items True-False Items Importance of what is assessed Importance of what is assessed How does it match the test blueprint How does it match the test blueprint Assess important ideas, rather than trivia, Assess important ideas, rather than trivia, general knowledge, or common sense general knowledge, or common sense Make sure it is either definitely true or definitely Make sure it is either definitely true or definitely false false Use short statements when possible Use short statements when possible Use exact language Use exact language Use positive statements and avoid double Use positive statements and avoid double negatives negatives
- 17. Suggestions for Improving Suggestions for Improving True-False Items True-False Items Avoid copying sentence verbatim Avoid copying sentence verbatim True and false statements should have approximately True and false statements should have approximately the same number of words the same number of words Don’t present in a repetitive or easily learned pattern Don’t present in a repetitive or easily learned pattern Do not use verbal clues that give away the answer Do not use verbal clues that give away the answer Attribute the opinion in a statement to an appropriate Attribute the opinion in a statement to an appropriate source source Focus on one idea Focus on one idea
- 18. Crafting Multiple True-False Items Crafting Multiple True-False Items Looks like a multiple-choice item – followed by Looks like a multiple-choice item – followed by several alternatives several alternatives No single correct or best answer No single correct or best answer Response to each alternative is true or false Response to each alternative is true or false Each alternative is scored correct or incorrect Each alternative is scored correct or incorrect
- 19. Crafting Multiple True-False Items Crafting Multiple True-False Items Advantages Advantages Two or three multiple true-false responses in the Two or three multiple true-false responses in the time it takes to make one multiple-choice response time it takes to make one multiple-choice response Multiple true-false tests created from multiple-choice Multiple true-false tests created from multiple-choice items have higher reliability than original multiple- items have higher reliability than original multiple- choice choice Can assess same abilities as straight multiple-choice Can assess same abilities as straight multiple-choice items that are crafted to assess parallel content items that are crafted to assess parallel content
- 20. Crafting Multiple True-False Items Crafting Multiple True-False Items Advantages Advantages Students believe multiple true-false items do a better Students believe multiple true-false items do a better job of assessing knowledge than multiple-choice job of assessing knowledge than multiple-choice items items Students perceive them to be harder Students perceive them to be harder May be easier to write – not limited to one correct May be easier to write – not limited to one correct answer answer Limitations Limitations Shares some of same limitations as multiple-choice Shares some of same limitations as multiple-choice
- 21. Works Cited Works Cited Nitko, A. J., (2004). Educational Assessment of Nitko, A. J., (2004). Educational Assessment of Students, 4 Students, 4th th ed. Upper Saddle River, NJ: ed. Upper Saddle River, NJ: Pearson Education, Inc. Pearson Education, Inc.
Editor's Notes
- #2: First principle is a strong one. Learning trivial performance or minor points of content is a waste of time. Focus on important learning targets. Second principle. Instructors need a very clear idea of what the learning target is. Students will either have the knowledge or they may have to guess, which shows re-teaching is necessary. Third principle recognizes that imprecise wording in a question, may make an item so ambiguous that a student may answer wrong.
- #3: The question variety asks a direct question and a short answer is required. The completion variety requires a student to add one or more words to complete it. The association variety consists of a list of terms or a picture for which students have to recall numbers, labels, symbols, or other terms. It is also called identification variety.
- #5: A teacher may have to make judgments as to the correctness of what the students write. (Spelling, grammar, legibility) The capital of Texas is ________? Students can answer this in many different ways.
- #6: The author of Alice in Wonderland is _______. What is the pen name of the author of Alice in Wonderland? ________. Word your questions so that they require a single correct answer. Putting the blank at the end of the sentence will prevent the student from having to mentally rearrange the question. Verbatim encourages rote memorization.