Administering, analyzing, and improving the test or assessment
- 1. ADMINISTERING, ANALYZING, AND IMPROVING THE TEST OR ASSESSMENT NEMA GRACE B. MEDILLO
- 2. GOAL IN THIS CHAPTER: To provide suggestion on how to avoid common pitfalls in test assembly, administration, and scoring.
- 3. Start Assembling the Test Analysing the Test Debriefing Scoring the Test Process of Evaluating Classroom Achievement Administering the Test Nema Grace B. Medillo
- 4. ASSEMBLING THE TEST Packaging the Test Reproducing the Test
- 5. Packaging the Test Group together all items of similar format Arrange test items from easy to hard Space the items for easy reading Keep the items and options on the same page Position illustration near descriptions and above the item Check your answer Determine how the students record their answer Provide space for Name and Date Check test directions Proofread the test
- 6. Reproducing the Test Know the Photocopying Machine Specify Copying Instructions File Original Test
- 7. ADMINISTERING THE TEST Maintain a Positive Attitude Maximize Achievement Motivation Equalize Advantages Avoid Surprises Clarify the Rules Rotate Distribution
- 8. Remind Students to Check Their Copies Monitor Students Minimize Distractions Give Time Warnings Collect Test Uniformly
- 9. SCORING THE TEST Prepare the Answer Key Check the Answer Key Score Blindly Check Machine – Scored Answer Sheets Check Scoring Record Scores
- 10. ANALYZING THE TEST Quantitative Item Analysis Qualitative Item Analysis Item Analysis Modifications for the Criterion – Referenced Test Item Analysis Terminology
- 11. Item Analysis Terminology Quantitative Item Analysis • A numerical method for analyzing test items employing students response alternatives or options. Qualitative Item Analysis • A non – numerical method for analyzing test items not employing students responses, but considering test objectives, content validity, and technical item quality Key • Correct option in a multiple – choice item
- 12. Item Analysis Terminology Distractor • Incorrect option in a multiple – choice item Difficulty Index • Proportion of students who answered the item correctly. Discrimination Index D • Measure of the extent to which a test item discriminates or differentiates between students who do well on the overall test and those who do not do well on the overall test.
- 13. Discrimination Index (D) • Those who did well on the overall test chose the correct answer for a particular item more often than those who did poorly on the overall test. Positive Discrimination Index • Those who did poorly on the overall test chose the correct answer for a particular item more often than those who did well on the overall test. Negative Discrimination Index • Those who did well and those who did poorly on the overall test chose the correct answer for a particular item with equal frequency Zero Discriminanation Index
- 14. Quantitative Item Analysis Case Sample and Guide Questions Difficulty Level Discrimination Index Miskeying Guessing Ambiguity
- 15. Consider the case below Suppose your students chose the options to a four – alternative multiple – choice item. Let C as the correct answer. A B C* D 3 0 18 9 How does this information help us? Is the item too difficult/easy for the students? Are the distractors of the items effective? Item X
- 16. Guide questions in quantitative item analysis 1.What is the difficulty level? 2.What is the discrimination index? 3.Should this item be eliminated? 4.Should any distractor(s) be modified?
- 17. To compute the difficulty level of an item; A B C* D 3 0 18 9 What is the difficulty level of the item? Do you consider the item difficult or easy? Why? 𝒑 = 𝑵𝒖𝒎𝒃𝒆𝒓 𝒔𝒆𝒍𝒆𝒄𝒕𝒊𝒏𝒈 𝒄𝒐𝒓𝒓𝒆𝒄𝒕 𝒂𝒏𝒔𝒘𝒆𝒓 𝑻𝒐𝒕𝒂𝒍 𝑵𝒖𝒎𝒃𝒆𝒓 𝒕𝒂𝒌𝒊𝒏𝒈 𝒕𝒉𝒆 𝒕𝒆𝒔𝒕 Item X
- 18. 𝒑 = 𝑵𝒖𝒎𝒃𝒆𝒓 𝒔𝒆𝒍𝒆𝒄𝒕𝒊𝒏𝒈 𝒄𝒐𝒓𝒓𝒆𝒄𝒕 𝒂𝒏𝒔𝒘𝒆𝒓 𝑻𝒐𝒕𝒂𝒍 𝑵𝒖𝒎𝒃𝒆𝒓 𝒕𝒂𝒌𝒊𝒏𝒈 𝒕𝒉𝒆 𝒕𝒆𝒔𝒕 𝒑 = 𝟎. 𝟔𝟎 Solving the difficulty index 𝒑 = 𝟏𝟖 𝟑𝟎 Since the difficulty level of the item is 0. 60 (60%), the item is moderately difficult. A B C* D 3 0 18 9 Item X Note: If P level > 0.75, the item is considered relatively easy. If P level < 0. 25, the item is considered relatively difficult.
- 19. Discrimination Index Steps in determining Discrimination Index 1. Arrange the papers from highest to lowest score. 2. Separate the papers into an upper group and lower 3. For each item, count the number in the upper group and the number in the lower group that chose each alternatives
- 20. 4. Record your information for each item Example for item X ( Class Size = 30) Options A B C* D Upper 1 0 11 3 Lower 2 0 7 6
- 21. 5. Compute D, by plugging the appropriate numbers in the formula What is the discrimination index of item X? Is the discrimination index positive or negative? Which of the groups frequently get the item correctly? 𝑫 = 𝑵𝒖𝒎𝒃𝒆𝒓 𝒘𝒉𝒐 𝒈𝒐𝒕 𝒊𝒕𝒆𝒎 𝒄𝒐𝒓𝒓𝒆𝒄𝒕 𝒊𝒏 𝒖𝒑𝒑𝒆𝒓 𝒈𝒓𝒐𝒖𝒑 − 𝑵𝒖𝒎𝒃𝒆𝒓 𝒘𝒉𝒐 𝒈𝒐𝒕 𝒊𝒕𝒆𝒎 𝒄𝒐𝒓𝒓𝒆𝒄𝒕 𝒊𝒏 𝒍𝒐𝒘𝒆𝒓 𝒈𝒓𝒐𝒖𝒑 𝑵𝒖𝒎𝒃𝒆𝒓 𝒐𝒇𝒔𝒕𝒖𝒅𝒆𝒏𝒕𝒔 𝒊𝒏 𝒆𝒊𝒕𝒉𝒆𝒓 𝒈𝒓𝒐𝒖𝒑 (if group sizes are unequal, choose the higher number)
- 22. Plugging the data 𝑫 = 𝟏𝟏 − 𝟕 𝟏𝟓 = 𝟎. 𝟐𝟔𝟕 Since the discrimination index of item X is 0. 267, which is positive. More students who did well on the overall test answered the item correctly than students who did poorly on the overall test.
- 23. Implication Difficulty Level (p) = 0. 60 Discrimination Index (D) = 0.267 Should this item be eliminated? The item is considered a moderately difficult item that has positive (desirable) discrimination ability. NO
- 24. Should any distractor(s) be modified? Option B is ought to be modified or replaced. A B C* D 3 0 18 9 Item X YES (No one chose it)
- 26. Let’s look at the responses for another item Item Y ( Class size = 28) Options A* B C D Upper 4 1 5 4 Lower 1 7 3 3 Item Z ( Class size = 30) Options A B* C D Upper 3 4 3 5 Lower 0 10 2 3 1. What is the difficulty level? 2. What is the discrimination index? 3. Should this item be eliminated? 4. Should any distractor(s) be modified?
- 27. Item Y Difficulty Level = 0.18 Discrimination Index = 0. 214 Should this item be eliminated? Should any distractor(s) be eliminated? No, since it is positively discriminating. However, it is difficult item; only 18% of the class got it right. Yes; C and D attracted more students who did well on the test overall. Remember: Fewer students who do well on the test should choose each distractor than students who do poorly. More students who do well on the test should choose the correct answer than students who do poorly
- 28. Difficulty Level = 0.467 Discrimination Index = - 0.40 Should this item be eliminated? Should any distractor(s) be eliminated? Yes! The item is moderately difficult but it discriminate negatively. Since we already decided to eliminate the item, this is moot question. Item Z Remember: Testing is to discriminate between those students who know their stuff and those who do not.
- 29. MISKEYING Most students who did well on the test will likely select an option that is a distractor, rather than the option that is keyed.
- 30. Consider the miskeyed item Who was the first astronaut to set foot on the moon? a. John Glenn b. Scott Carpenter c. Neil Armstrong *d. Alan Sheppard Responses A B C D* Upper 1 1 9 2 Most students in the upper half of the class fail to select the keyed option. Remember, just as you are bound to make scoring errors, you are bound to miskey an item occasionally.
- 31. GUESSING Likely to occur when the item measures content that is, • Not covered in class or the text • So difficult that even the upper – half students have no idea what the correct answer is • So trivial that students are unable to choose from among the options provided.
- 32. The choice distribution suggest that guessing occurred: A B C* D Upper Half 4 3 3 3 Each alternative is about equally attractive to students in the upper half.
- 33. Ambiguity Among the upper group, one of the distractors is chosen with about the same frequency as the correct answer. The distribution suggests that an item is ambiguous: A B C D* Upper Half 7 0 1 7
- 34. QUALITATIVE ITEM ANALYSIS • Matching items and objectives • Editing poorly written items • Improving content of the validity of the test • Analyzing grammatical cues, specific determiners, double negatives, multiple defensible answers, and items that fail to match instructional objectives
- 35. ITEM ANALYSIS MODIFICATIONS FOR THE CRITERION – REFERENCED TEST Using Pre- and Posttest as Upper and Lower Group Comparing the Percentage Answering Each Item Correctly on Both Pre- and Posttest Determining the Percentage of Items Answered in the Expected Directions Limitations of Modifications
- 36. Using Pre- and Posttests as Upper and Lower Groups Pretest prior instructions Most of the test items answered incorrectly Results for Lower Group P level = 0. 30 or lower Post test after instructions Most items were answered correctly Results for Upper Group P = 0.70 or higher
- 37. Analyzing sample Data Example 1: Number of students choosing option (n = 25) Option At pretest (L) At posttest(U) A 9 1 B 7 1 C 3 2 D* (Key) 6 21
- 38. Steps 1. Compute p Levels for both tests. Pretest Posttest 𝑃 = 𝑁𝑜.𝑐ℎ𝑜𝑜𝑠𝑖𝑛𝑔 𝐶𝑜𝑟𝑟𝑒𝑐𝑡𝑙𝑦 𝑇𝑜𝑡𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 6 25 21 25 = 0.24 =0.84 It was an improvement from 24% to 84%.
- 39. 2. Determine the discrimination index (D) for the key 𝐷 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑝𝑜𝑠𝑡 −𝑁𝑢𝑚𝑏𝑒𝑟 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 (𝑝𝑟𝑒) 𝑁𝑢𝑚𝑏𝑒𝑟 𝑖𝑛 𝑒𝑖𝑡ℎ𝑒𝑟 𝑔𝑟𝑜𝑢𝑝 𝐷 = 21 −6 25 = 15 25 = 0.60 The key has a positive discrimination index.
- 40. 3. Determine whether each option separately discriminates negatively. Option A: 𝐷 = 1−9 25 = −8 25 = −0.32 Option B: 𝐷 = 1−7 25 = −6 25 = −0.24 Option C: 𝐷 = 2−3 25 = −1 25 = −0.04
- 41. Summary 1. There was sizeable increase in p value from pretest to posttest. 2. The D index for the key was positive. 3. The distractors are all discriminated negatively. If a criterion – reference test item manifests these features, it has passed our “test” and a probably is a good item with little or no need for modification. Contrast with, the test item failed. Rather than modify, it is probably more efficient to replace it with another.
- 42. Comparing the Percentage Answering Each Item Correctly on Both Pre- and Posttest If your test is sensitive to your objectives. What to do? Percentage Passing Posttest – Percentage of Passing Pretest The more positive the difference, the more the item tapping the content you are teaching.
- 43. Analyzing Sample Data Consider the following percentages for five test items: Item Percentage passing pretest Percentage passing posttest Difference 1 16 79 +63% 2 10 82 +72% 3* 75 75 0% 4 27 91 +64% 5* 67 53 -14% Item 3 may be eliminated. (Students already know the content it represents.) Item 5 may be revised. (Instruction was not related to the item or confused some students.)
- 44. Determining the Percentage of Items Answered in the Expected Directions for the Entire Test
- 45. Steps 1. Find the number of items each student failed on the pretest but passed on the posttest. Result of Mary Item Pretest Posttest 1* Incorrect Correct 2 Correct Correct 3* Incorrect Correct 4 Correct Incorrect 5 Incorrect Incorrect Do same things to other students.
- 46. 2. Add the counts and divide by the number of students. 18+15+22+20+13 5 = 88 5 = 17.6 Example Mary 18 Carlos 15 Sharon 22 Amanda 20 Charles 13
- 47. 3. Divide by number of test items. 17.6 25 = 0.70 4. Multiply by 100. 0.70 x 100 = 70% The greater the overall positive percentage of change, the more your test is likely to match your instruction and to be a content – valid test
- 48. Limitations of Modification Difficult Unit of instruction is brief From norm – referenced test to criterion – referenced test Time devote to instruction (pre – post)
- 49. DEBRIEFING Discuss Problem Items Listen to Students Reactions Avoid – on – the Spot Decisions Be Equitable with Changes Ask Students to Double – Check Ask Students to Identify Problems
- 50. PROCESS OF EVALUATING CLASSROOM ACHIEVEMENTS
- 51. THANK YOU… THANK YOU… THANK YOU… THANK YOU…THANK YOU… THANK YOU…
- 52. Shutting down