Fdu item analysis (1).ppt revised by dd
- 1. Item Analysis • A technique for the guidance and improvement of instruction. • The items analyzed must be valid measures of instructional objectives. • The items must be diagnostic in that the knowledge of which incorrect options students select are a clue to the nature of misunderstanding and are prescriptive of appropriate actions by the instructor.
- 2. Usefulness of Data • Instructors who construct their own test and examinations may greatly improve the effectiveness of test items and the validity of test scores if they select and rewrite their items on the basis of item performance data.
- 3. Difficulty Index • Used to determine the difficulty level of test items. • Asks instructors to calculate the proportion of students who answered the test item accurately. • Looking at alternative answers can also find out if there are answer choices that should be replaced.
- 4. Difficulty Index Example: The following table illustrates how many students selected each answer choice for questions 1 and 2 in a multiple choice test. For question 1 A was not a good distracter, no one selected the answer. We can compute the difficulty of an item by dividing the number of students who choose the correct answer (24) by the total number of students (30). Using this formula, the difficulty of question 1 (p) is equal to 24/30 or .80. Question A B C D #1 0 3 24* 3 #2 *= Correct answer. 12* 13 3 2
- 5. Item Difficulty • A rule-of-thumb is that if the item difficulty is more than .75 it is an easy item. • If the difficulty is below .23 it is a difficult item. • Given these parameters, this item could be regarded as moderately easy; many students (80%) got it correct. Question A B C D #1 0 3 24* 3 #2 *= Correct answer. 12* 13 3 2
- 6. Item Difficulty • In contrast question 2 is much more difficult (12/30=.40). • More students selected the incorrect answer (B) in question 2 than selected the correct answer (A). • This item should be carefully analyzed to ensure that B is an appropriate distractor. Question A B C D #1 0 3 24* 3 #2 *= Correct answer. 12* 13 3 2
- 7. Discrimination Index • How well an assessment differentiates between high and low scorers. • You should be able to expect that the high performing students would select the correct answer for each question more often than the low performing students. • If this is true then the assessment is said to have a positive discrimination index (between 0 and 1) indicating that students who received a high total score chose the correct answer for a specific item more often than the students who had a lower overall score. • However, if you find that more of the low performing students got a specific item correct then the item has a negative discrimination index (between -1 and 0).
- 8. Discrimination Index Student Total Score (%) Questions 1 2 3 Alan 90 1 0 1 Sam 90 1 0 1 Jill 80 0 0 1 Charles 80 1 0 1 Sonia 70 1 0 1 Roger 60 1 0 0 Clayton 60 1 0 1 Kelly 50 1 1 0 Justin 50 1 1 0 Cathy 40 0 1 0
- 9. Discrimination Index • This table displays the result of ten questions on a test. Note that the students are arranged with the top overall scorers at the top of the table. • The following steps can be followed to determine the Difficulty Index and the Discrimination Index.
- 10. Calculating Indices • 1. Arrange students with the highest overall scores at the top then count the number of students in the upper and lower group who got each item correct. For Question 1 there were 4 students in the top half who got it correct and 4 students in the bottom half. • 2. Determine the Difficulty Index by dividing the number who got it correct by the total number of students. For Question 1 this would be 8/10 or p=.80.
- 11. Calculating Indices • 3. Determine the Discrimination Index by subtracting the number of students in the lower group who got the item correct from the number of students in the upper group who got the item correct. Then divide by the number of students in each group (in this case there were 5 in each group). For Question 1 that means you would subtract 4 from 4, and divide by 5, which results in a Discrimination Index of 0. • These answers are given in the next table.
- 12. Calculating Indices Item # Correct (Upper Group) # Correct (Lower Group) Difficulty (p) Discrimination (D) Question 1 4 4 .80 0 Question 2 0 3 .30 -0.6 Question 3 5 1 .60 0.8
- 13. Calculating Indices • What does this table tell us? • Question 2 had a difficulty index of .30 meaning it was quite difficult, and a negative discrimination index of -0.6 meaning low performing students were more likely to get this item correct. This question should be carefully analyzed and probably deleted or changed. • Our best overall question is question 3 which had a moderate difficulty level (.60) and discriminated extremely well (0.8).
- 14. Cognitive Level • Consider what cognitive level the item is assessing. • The questions may be based on Bloom’s taxonomy, grouping those by level of difficulty. • In this way you can easily note which questions demand higher level thinking skills and may be too difficult, or do not discriminate well. Improve these questions and focus instructional strategies on higher level skills.
- 15. Biserial Correlation • The correlation between a student’s performance on an item (right or wrong) and his or her total score on the test. • Assumes the distribution of test scores is normal and there is a normal distribution underlying the right/wrong dichotomy. • Biserial correlation has the characteristic of having maximum values greater than unity. • There is no exact test for the statistical significance of the biserial correlation coefficient.
- 16. Biserial Correlation • Point biserial correlation is also a correlation between student performance on an item (right or wrong) and test score. • It assumes the test score distribution is normal and the division on item performance is a natural dichotomy. • The possible range of values for the point biserial correlation is +1 to -1.
- 17. Guidelines for Item Development • 1. Items that correlate less than .15 with total test score should be restructured. These items usually do not measure the same skill or ability as does the test on the whole or that they are confusing or misleading to students. A test is better (more reliable) the more homogeneous the items.
- 18. Guidelines for Item Development • 2. Distractors that are not chosen by any students should be replaced or eliminated. They do not contribute to the test’s ability to discriminate the good students from the poor students. Do not be concerned if each distractor is not chosen by the same number of students. The fact that a majority of students miss an item does not imply that the item should be changed, although such items should be double-checked for accuracy. Be suspicious about the correctness of any item in which a single distractor is chosen more often than all other options, including the answer, and especially if the distractor’s correlation with the total score is positive.
- 19. Guidelines for Item Development • 3. Items that virtually everyone gets right are useless for discriminating among students and should be replaced by more difficult items. This is particularly true if you adopt the traditional attitude toward letter grade assignments that letter grades more or less fit a predetermined distribution.