Laos Session 5: Analysing Test Items using Classical Test Theory (CTT)
Download as PPT, PDF2 likes1,004 views
The document outlines a workshop on analyzing tests and test items using Classical Test Theory (CTT), covering key concepts such as test mean, discrimination, reliability, and various methods of estimating item difficulty and reliability. It emphasizes the importance of test characteristics for effective item development and design, alongside practical examples like the Findlay index and the Guttman scale. Additionally, it discusses strategies to improve test reliability and validates the theoretical framework for evaluating different types of validity.
Laos Session 5: Analysing Test Items using Classical Test Theory (CTT)
- 1. Session 5: Analysing Tests and Test Items using Classical Test Theory (CTT) Professor Jim Tognolini
- 2. Analysing Tests and Test Items using Classical Test Theory (CTT) During this session we will •define some basic test level statistics using Classical Test Theory analyses: test mean, test discrimination and test reliability (Chronbach’s Alpha). •define some basic item level statistics from Classical test theory: item difficulty, item discrimination (Findlay Index and Point Biserial Correlation). Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 3. • Difficulty • Discrimination • Reliability • Validity Test characteristics to evaluate Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 4. Test difficulty Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 5. Test discrimination The ability of a test to discriminate between high- and low-achieving individuals is a function of the items that comprise the test. Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 6. Methods of estimating reliability Method Type of Reliability Procedure Test-Retest Stability Reliability Give the same test to the same group on different occasions with some time between tests. Equivalent Forms Equivalent Reliability Give two forms (parallel forms) of the test to the same group in close succession. Split-half Internal Consistency Give test once; split test in half (odd/even); get the correlation between the score; correct the correlation between halves using the Spearman-Brown formula. Coefficient Alpha Internal Consistency Give test once to a group and apply formula. Interrater Consistency of Ratings Get two or more raters to score the responses and calculate the correlation coefficient. Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 7. Split-halves method Reliability can also be estimated from a single administration of a test, either by correlating the two halves or by using the Kuder- Richardson Method. The Split-halves method requires the test to be split into halves which are most equivalent. To estimate the reliability of the full test the Spearman-Brown Adjustment is usually applied Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 8. Kuder-Richardson (KR-20 and KR-21) Method Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 9. Cronbach’s alpha method Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 10. 1. Test length In general the longer the test the higher the reliability (more adequate sampling) provided that the material that is added is identical in statistical and substantive properties 2.Homogeneity of group The more heterogeneous the group, the high the reliability. It can vary at different score levels, gender, location, etc. 3.Difficulty of items Tests that are too difficult or too hard provide results of low reliability. Generally set tests of item difficulty equal to 0.5. In general with tests that are required to discriminate, spread questions over the range in which the discrimination is required. Ways to improve reliability Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 11. 4. Objectivity The more objective the test (and marking scheme) the more reliable are the resulting test scores. 5.Retain Discriminating Items In general replace items with a low discrimination with those that highly discriminate. There comes a point where this practice raises the reliability to such a point that it lowers validity (attenuation paradox). 6.Increase Speededness of the Tests Highly speeded tests usually show higher reliability. Don’t use internal consistency estimates. Ways to improve reliability Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 12. Types of validity There are many different types of validity. Traditionally there are three main types: I. Content Validity (sometimes referred to as curricular or instructional validity) II. Criterion Related Validity (types include predictive and concurrent validity) III. Construct Validity IV. Face Validity Loevinger (1957) argued that “since predictive, concurrent and content validities are all essentially ad hoc, construct validity of the whole of validity from a scientific point of view” Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 13. Define some basic item level statistics from Classical Test Theory Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 14. Item difficulty Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 15. Item discrimination Methods for checking item discrimination include •The Findlay Index (FI) •The Point Biserial Correlation •The Biserial Correlation Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 16. The Findlay Index (FI) Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 17. The Findlay Index (FI) – An example Item NRU NRL NU FI Comment 1 9 2 10 0.7 Good item, better students do well 2 6 6 10 0.0 Weak item, does not discriminate 3 6 8 10 -0.2 Invalid item, weak students do better Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 18. The Findlay Index (FI) If the number of students in the top group is not equal to the number in the bottom group proportions must be used. where PRU = proportion of persons right in upper group PRL = proportion of persons right in lower group FI = PRU - PRL Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 19. Graphical display of the Findlay Index (FI) Calculate the proportion of the group getting the item correct and then plot this against the mean score for the particular group mean scores for each group. Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 20. Graphical display of the Findlay Index (FI) 0 0.2 0.4 0.6 0.8 1 L M U ProportionCorrect Score Group Item 2 Item 6.2 Item 7 Item 10.4 Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 21. The Findlay Index (FI) – An example Item Type SA SA SA SA SA E E E E E E E Total Item Number 1 2 3 4 5 6 7 8 9 10 11 12 Max Marks 1 1 1 1 1 3 2 2 3 4 3 6 28 Astha 1 1 0 0 1 3 0 1 3 1 3 4 18 Bosco 1 1 1 0 1 3 0 1 3 1 3 3 18 Chetan 1 1 1 1 1 3 0 2 1 2 3 5 21 Devika 1 1 1 0 1 3 0 2 1 1 2 3 16 Emily 1 1 1 1 1 3 0 1 3 4 2 3 21 Farhan 1 1 1 1 1 3 1 2 3 3 3 4 24 Gogi 1 1 1 0 1 0 0 1 0 0 0 1 6 Harshita 1 1 1 1 1 3 2 1 3 4 3 3 24 Indu 0 1 0 0 1 0 0 2 0 0 2 0 6 Jagat 1 1 1 1 1 2 1 1 3 2 3 5 22 TOTAL 9 10 8 5 10 23 4 14 20 18 24 31 176 Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 22. The Findlay Index (FI) – An example Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 23. The Findlay Index (FI) – An example Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 24. The Findlay Index (FI) – An example Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 25. The Findlay Index (FI) – An example Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 26. Guttman scale Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 27. Point-biserial correlation Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 28. The Guttman structure If person A scores better than person B on the test, then person A should have all the items correct that person B has, and in addition, some other items that are more difficult. Louis Guttman Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 29. The Guttman structure (cont.) 1 2 3 4 5 6 Total Score 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 1 0 0 0 0 2 1 1 1 0 0 0 3 1 1 1 1 0 0 4 1 1 1 1 1 0 5 1 1 1 1 1 1 6 Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 30. Reasons for not obtaining a strict Guttman pattern • The items do not go together as expected and the scores on the items should not be added. • The items are very close in difficulty and the persons are all close in ability. Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 31. Guttman scale Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 32. Individual reporting 3 11 2 15 14 9 8 1 7 4 13 12 5 10 6 Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016
- 33. Individual reporting 3 11 2 15 14 9 8 1 7 4 13 12 5 10 6 Capacity Development Workshop: Test and Item Development and Design, Laos, September 2016