Quality assurance on internal attributes of a good
Download as PPTX, PDF0 likes21 views
The document discusses the concept of reliability as it relates to measuring instruments and language tests. It defines reliability as the ability of a test to produce consistent and stable scores, regardless of who administers the test or when/where it is given. There are several approaches to estimating reliability, including test-retest, parallel forms, and internal consistency. Test-retest reliability involves giving the same test twice over time, while parallel forms uses two equivalent versions of a test. Internal consistency examines how well test items correlate with one another and measure the intended construct. A reliable test demonstrates consistency in scoring both between raters and across testing occasions.
Quality assurance on internal attributes of a good
- 1. QUALITY ASSURANCE ON INTERNAL ATTRIBUTES OF A GOOD MEASUREING LANGUAGE DEVICES (RELIABILITY) AMIRUL FAISAL RIZZA
- 3. English test tools / instruments to draw out evidence of the existance of English abilities
- 4. 1.Good instruments : 2.The hidden English abilities are guaranteed to be observable. 1.Bad instruments : 2.1. Damage measurements and evaluation. 3.2. Can not describe the real language ablities of the test takers.
- 5. 1. Reability 2. Validity 3. Practicallity/usability 4. Andeconomy Requirements for a good instrument
- 6. RELIABLE = STABLE = CONSISTENT Reliability • Reliable test is a test that can produce stable scores or consistent scores. • Test scores demonstate consistency or stability no matter who administers the test (Rater or Interrater). • The scores consistent no matter when or where the test is administrated.
- 7. Reliability – Observed Score is the data gathered by the researcher – True Score is the actual unknown values that correspond to the construct of interest – Error – Systematic Error is variations that results from constructs of disinterest – Unsystematic / Random Error is nonsystematic variations in the observed scores Observed Score = True Score + (Measurement) Error
- 8. Students Rater 1 Rater 2 A 8 8 B 8.6 8.6 C 9 9 D 8 8 E 9.4 9.4 Perfect Consistency Consistency between Raters or interraters
- 9. Students Rater 1 Rater 2 A 8.2 8 B 8.6 8.8 C 8.9 9 D 8 8.1 E 9.3 9.4 Consistent Test
- 10. Students Administrated on Wednesday Administrated on Friday A 7.8 8 B 8.6 8.3 C 9.1 9 D 8 8.2 E 9.3 9.4 Consistent across time
- 11. Inconsistency between raters or interraters Students Administrated on Wednesday Administrated on Friday A 3 8 B 8.6 2 C 3.1 9 D 8 8.2 E 9.3 5
- 12. Students Administrated on Wednesday Administrated on Friday A 4 8 B 8.6 2 C 5 9 D 6 8.2 E 8 8 Inconsistency across time
- 13. How do we determine whether a measurement is reliable? The principles of reliability estimation utilizing these APPROACHES: Test Retest Parrarel Forms Internal Consistency
- 14. TEST RETEST Uses the same test twice to the same group of subjects on different testing occasions. There is a repetation on the use of the same instrument and the invovements of the same subjects. The repetetion is done on different day.
- 15. TIME ACTIVITY PARTICIPANTS RESULT Wednesday, 3/10/18 Vocabulary test 50 students of SMA 1 Bangsal Result 1 Wednesday, 10/10/18 The same vocabulary test 50 students of SMA 1 Bangsal Result 2
- 16. Advantanges Disadvantages • We only need one set of a test. • Requires of testing occasions. • It is not easy to create a similar condition on different testing occasions. • Too close time for the test administration makes the test takers still remember the content of the test. • Far too long of the second test may affect the test takers’ performance. • Cause boredom, ailment and the like. Back
- 17. Parrarel Forms Requires two or more sets of tests. Each set of test is made equal in every aspect of the test with other test. Equal in : Test format Test lenght The level of difficulty Discrimination indexes used Time allocation Test content
- 18. SET A (administrated on Tuesday) SET B (administrated on Friday) Administered to a group of students Scores produced from completing Set A Scores produced from completing Set B Correlational analysis
- 19. Advantanges Disadvantages • Has more variations in sets of the tests. • Time consuming to make two more sets of the tests. • Not easy to keep the students’ motivation in doing the second test. Back
- 20. Internal Consistency Based on the logic that if the items in the test are highly correlated, the test is said to be reliable. Before develop a test, it should be built a theoritical ability that would be measured by the test. The items of the test should be constructed to measure a single ability (technically it is calaed as “construct”). Tests with higher internal consistency more accurately measure the intended construct of the test developers.
- 21. Vocabulary ability Synonym Indicator 1 Indicator 2 Antonym Indicator 1 Indicator 2 Item 1 Item 2 Item 3 Item 1 Item 2 Item 3 Item 4 Item 1 Item 2 Item 3 Item 1 Item 2 Item 3 Item 4 Concept level Dimension level Indicator level Test item level A tetst of vocabulary containing assembles & selected items Test level
- 22. Students item Total score A 1 1 0 1 0 1 1 0 1 0 6 B 1 0 1 0 1 0 1 1 1 0 6 C 0 1 1 1 1 1 1 1 1 1 9 D 1 0 1 0 0 0 1 1 1 0 4 E 1 0 0 0 1 1 1 1 1 1 7 Test takers’ scores (Hypothetical dichotoumous scoring on 10 items) F 1 1 0 1 1 1 1 1 1 1 9 Total score 5 3 3 3 4 4 6 5 6 3
- 23. Approaches to perform internal consistency Split half : split the scores based on the test achievement in the first half of the items and those on the second items. The split can be half of the total items or based on the odd or even numbers. Some drawbacks of split-half are: Inter-item estimation : the test scores are correlated with themselves within the same test. It is called as inter-item correlation. Obtained scores in each item are correlated with one another. Item 1 is correlated with item 2, 3, 4, 5, 6, 7, 8, 9, 10 or Item 2 is correlated with item 1, 3, 4, 5, 6, 7, 8, 9, 10. Examples :
- 24. Split-half Test takers’ identity Item Total (set A)1 2 3 4 5 A 1 1 0 1 0 3 B 0 1 1 0 1 3 C 1 1 1 1 1 5 D 1 1 0 0 0 2 E 1 0 1 1 0 3 F 1 0 1 0 1 3 TOTAL SCORE 5 4 4 3 3 1ST half
- 25. Split-half Test takers’ identity Item Total (set A)6 7 8 9 10 A 1 1 0 1 0 3 B 0 1 1 0 1 3 C 1 0 1 1 1 4 D 1 1 0 0 0 2 E 1 0 1 1 1 4 F 0 0 1 0 1 2 TOTAL SCORE 4 3 4 3 4 2nd half Back
- 26. Does not fully reflect the true value of reliablity of the test (Kline, 1993:11) Different split may cause different result of reliability (Cronbach, 1951) Test lenght affects the reliability of the test. The more items in the test, the reliable the test is (Wiersma and Jurs, 1990:163). DRAWBACKS OF SPLIT-HALF Back
- 27. Example of inter-item estimation There two or more raters evaluate students speaking skills. The scoring may be based on some several aspects.a statistical analysis may be used to analyze the data, usually uses t-test. A correlational analysis may be applied to examine the closeness of the scores got by the two rates.