What makes a good testA test is considered good” if the .docx
- 1. What makes a good test? A test is considered “good” if the following can be said about it: · The test measures what it claims to measure. For example, a test of mental ability does, in fact, measure mental ability and not some other characteristic. · The test measures what it claims to measure consistently or reliably. This means that, if a person were to take the test again, the person would get a similar test score. · The test is job-relevant. In other words, the test measures 1 or more characteristics that are important to the job. · By using the test, more effective decisions can be made about individuals. · The degree to which a test has these qualities is indicated by 2 technical properties: reliability and validity. Test Reliability Reliability refers to how consistently a test measures a characteristic. If a person takes the test again, will he or she get a similar test score or a much different score? A test that yields similar scores for a person who repeats the test is said to measure a characteristic reliably. How do we account for an individual who does not get exactly the same test score every time he or she takes the test? Some possible reasons are the following: · Test taker's temporary psychological or physical state. Test performance can be influenced by a person's psychological or physical state at the time of testing. For example, differing levels of anxiety, fatigue, or motivation may affect the
- 2. applicant's test results (unsystematic error). · Environmental factors. Differences in the testing environment, such as room temperature, lighting, noise, or even the test administrator can influence an individual's test performance (unsystematic error). · Test form. Many tests have more than 1 version or form. Items differ on each form, but each form is supposed to measure the same thing. Different forms of a test are known as parallel forms or alternateforms. These forms are designed to have similar measurement characteristics, but they contain different items. Because the forms are not exactly the same, a test taker might do better on 1 form than on another. · Multiple raters. In certain tests, scoring is determined by a rater’s judgments of the test taker’s performance or responses. Differences in training, experience, and frame of reference among raters can produce different test scores for the test taker. These factors are sources of chance or random measurement error in the assessment process. If there were no random errors of measurement, the individual would get the same test score, the individual's “true” score, each time. The degree to which test scores are unaffected by measurement errors is an indication of the reliability of the test. But, while psychometrics can give you a lot of this information, it is important to ask the client about how they experienced the process of taking the test. This will allow you to detect any potential unsystematic errors. When selecting an assessment, you want to remember that reliable assessment tools produce dependable, repeatable, and consistent information about people. In order to meaningfully interpret test scores and make useful decisions, you need reliable tools. This brings us to the next principle of assessment. Interpretation of Reliability Information from Test Manuals and Reviews
- 3. Test manuals and independent reviews of tests provide information on test reliability. The following discussion will help you interpret the reliability information about any test. The reliability of a test is indicated by the reliability coefficient. When you are looking at internal consistency, it is reported as Cronbach’s alpha. It is denoted by the letter “r,” and is expressed as a number ranging between 0 and 1.00 with r = 0 indicating no reliability and r = 1.00 indicating perfect reliability. Do not expect to find a test with perfect reliability. Generally, you will see the reliability of a test as a decimal, for example, r = .80 or r = .93. The greater the reliability coefficient value, the more repeatable or reliable the test scores. Table 1 serves as a general guideline for interpreting test reliability. However, do not select or reject a test solely based on the size of its reliability coefficient. To evaluate a test’s reliability, you must also consider the type of test, the type of reliability estimate used, and the context in which the test will be used. Table 1. General Guidelines for Interpreting Reliability Coefficients Reliability Coefficient Value Interpretation of Reliability .90 and up excellent .80 – .89 good .70 – .79 adequate below .70 may have limited applicability Types of Reliability Estimates
- 4. There are several types of reliability estimates, each influenced by different sources of measurement error. Test developers have the responsibility of reporting the reliability estimates that are relevant for a particular test. Before deciding to use a test, read the test manual and any independent reviews to determine if its reliability is acceptable. The acceptable level of reliability will differ depending on the type of test and the reliability estimate used. Test-retest reliability indicates the repeatability of test scores with the passage of time. This estimate also reflects the stability of the characteristic (or construct) being measured by the test. Some constructs are more stable than others. For example, an individual's reading ability is more stable over a particular period of time than that individual's anxiety level. Therefore, you would expect a higher test-retest reliability coefficient on a reading test than you would on a test that measures anxiety. For constructs that are expected to vary over time, an acceptable test-retest reliability coefficient may be lower than is suggested in Table 1. Alternate or parallel form reliability indicates how consistent test scores are likely to be if a person takes two or more forms of a test. A high parallel form reliability coefficient indicates that the different forms of the test are very similar which means that it makes virtually no difference which version of the test a person takes. On the other hand, a low parallel form reliability coefficient suggests that the different forms are probably not comparable; they may be measuring different things and therefore cannot be used interchangeably. Inter-rater reliability indicates how consistent test scores are likely to be if the test is scored by two or more raters. On some tests, raters evaluate responses to questions and determine the score. Differences in judgments among raters are likely to produce variations in test scores. A high inter-rater reliability
- 5. coefficient indicates that the judgment process is stable and the resulting scores are reliable. Inter-rater reliability coefficients are typically lower than other types of reliability estimates. However, it is possible to obtain higher levels of inter-rater reliabilities if raters are appropriately trained. Internal consistency reliability indicates the extent to which items on a test measure the same thing. A high internal consistency reliability coefficient for a test indicates that the items on the test are very similar to each other in content (homogeneous). It is important to note that the length of a test can affect internal consistency reliability. For example, a very lengthy test can spuriously inflate the reliability coefficient because the equation is based on the number of questions. (In other words, a test with fewer questions with a high r would technically be better than a similar test with double the questions). Tests that measure multiple characteristics are usually divided into distinct components. Manuals for such tests typically report a separate internal consistency reliability coefficient for each component in addition to one for the whole test. Test manuals and reviews report several kinds of internal consistency reliability estimates. Each type of estimate is appropriate under certain circumstances. The test manual should explain why a particular estimate is reported. (ie, internal consistency or split half). Standard Error of Measurement Test manuals report a statistic called the standard error of measurement (SEM). It gives the margin of error that you should expect in an individual test score because of imperfect reliability of the test. The SEM represents the degree of confidence that a person's "true" score lies within a particular range of scores. For example, an SEM of "2" indicates that a
- 6. test taker's "true" score probably lies within 2 points in either direction of the score he or she receives on the test. This means that if an individual receives a 91 on the test, there is a good chance that the person's "true" score lies somewhere between 89 and 93. The SEM is a useful measure of the accuracy of individual test scores. The smaller the SEM, the more accurate the measurement is. This is because it leaves little room for error. The SEM often is used to confirm or explain the reliability coefficients of the assessment. Final Thoughts: Evaluating Reliability Coefficients When evaluating the reliability coefficients of a test, it is important to review the explanations provided in the manual for the following: · Types of reliability used. The manual should indicate why a certain type of reliability coefficient was reported. The manual should also discuss sources of random measurement error that are relevant for the test. · How reliability studies were conducted. The manual should indicate the conditions under which the data were obtained, such as the length of time that passed between administrations of a test in a test-retest reliability study. In general, reliabilities tend to drop as the time between test administrations increases. · The characteristics of the sample group. The manual should indicate the important characteristics of the group used in gathering reliability information, such as education level, occupation, etc. This will allow you to compare the characteristics of the people you want to test with the sample group. If they are sufficiently similar, then the reported reliability estimates will probably hold true for your population as well. Test Validity
- 7. Validity is the most important issue in selecting a test. Validity refers to what characteristic the test measures and how well the test measures that characteristic. · Validity tells you if the characteristic being measured by a test is related to specific requirements. · Validity gives meaning to the test scores. Validity evidence indicates that there is linkage between test performance and job/behavior performance. It can tell you what you may conclude or predict about someone from his or her score on the test. If a test has been demonstrated to be a valid predictor of performance, you can conclude that persons scoring high on the test are more likely to perform well on the job/or what you’re predicting than persons who score low on the test, all else being equal. · Validity also describes the degree to which you can make specific conclusions or predictions about people based on their test scores. In other words, it indicates the usefulness of the test. It is important to understand the differences between reliability and validity. Validity will tell you how good a test is for a particular situation; reliability will tell you how trustworthy a score on that test will be. You cannot draw valid conclusions from a test score unless you are sure that the test is reliable. Even when a test is reliable, it may not be valid. You should be careful that any test you select is both reliable and valid for your situation. A test's validity is established in reference to a specific purpose; the test may not be valid for different purposes. For example, the test you use to make valid predictions about someone's technical proficiency on the job may not be valid for predicting his or her leadership skills or absenteeism rate. This leads to the next principle of assessment: what population was examined?
- 8. A test's validity is established in reference to specific groups. These groups are called the reference groups. The test may not be valid for different groups. For example, a test designed to predict the performance of managers in situations requiring problem solving may not allow you to make valid or meaningful predictions about the performance of clerical employees. If, for example, the kind of problem-solving ability required for the two positions is different, or the reading level of the test is not suitable for clerical applicants, the test results may be valid for managers, but not for clerical employees. Test developers have the responsibility of describing the reference groups used to develop the test. The manual should describe the groups for whom the test is valid, and the interpretation of scores for individuals belonging to each of these groups. You must determine if the test can be used appropriately with the particular type of people you want to test. This group of people is called your target population or target group. Your target group and the reference group do not have to match on all factors; they must be sufficiently similar so that the test will yield meaningful scores for your group. For example, a writing ability test developed for use with college seniors may be appropriate for measuring the writing ability of white-collar professionals or managers, even though these groups do not have identical characteristics. In determining the appropriateness of a test for your target groups, consider factors such as occupation, reading level, cultural differences, and language barriers. Recall that the Uniform Guidelines require assessment tools to have adequate supporting evidence for the conclusions you reach with them in the event adverse impact occurs. A valid personnel tool is one that measures an important characteristic of the job you are interested in. Using valid tools will, on
- 9. average, enable you to make better decisions. It is essential to only use tests that are valid for your intended use. Methods for Conducting Validation Studies · Criterion-related validation requires demonstration of a correlation or other statistical relationship between test performance and performance of what is being predicted. In other words, individuals who score high on the test tend to perform better than those who score low on the test. If the criterion is obtained at the same time the test is given, it is called concurrent validity; if the criterion is obtained at a later time, it is called predictive validity (i.e., marriage satisfaction after 10 yrs.). · Content-related validation requires a demonstration that the content of the test represents important related behaviors. In other words, test items should be relevant to and measure directly important requirements for a given construct (expert review) versus face validity. · Construct-related validation requires a demonstration that the test measures the construct or characteristic it claims to measure, and that this characteristic is important to successful performance. (empirical review/consensus) The 3 methods of validity-criterion-related, content, and construct-should be used to provide validation support depending on the situation. These three general methods often overlap, and, depending on the situation, one or more may be appropriate. The content validation method may be used when you want to determine if there is a relationship between behaviors measured by a test and behaviors involved in the job. For example, a typing test would be high validation support for a secretarial position, assuming much typing is required each day. If,
- 10. however, the job required only minimal typing, then the same test would have little content validity. Content validity does not apply to tests measuring learning ability or general problem- solving skills. Another method is construct validity. This method often pertains to tests that may measure abstract traits of a client. For example, construct validity may be used when a counselor wants to test the client’s aptitude. In this case, an aptitude is not an observable behavior, but a concept created to explain possible future behaviors. To demonstrate that the test possesses construct validation support, ". . . the counselor would need to show (1) that the test did indeed measure the desired trait and (2) that this trait corresponded to success on a given task. Professionally developed tests should come with reports on validity evidence, including detailed explanations of how validation studies were conducted. If you develop your own tests or procedures, you will need to conduct your own validation studies. As the test user, you have the ultimate responsibility for making sure that validity evidence exists for the conclusions you reach using the tests. This applies to all tests and procedures you use, whether they have been bought off-the-shelf, developed externally, or developed in-house. How to Interpret Validity Information from Test Manuals and Independent Reviews To determine if a particular test is valid for your intended use, consult the test manual and available independent reviews. The information below can help you interpret the validity evidence reported in these publications. In evaluating validity information, it is important to determine whether the test can be used in the specific way you intended,
- 11. and whether your target group is similar to the test reference group. Test manuals and reviews should describe · Available validation evidence supporting use of the test for specific purposes. The manual should include a thorough description of the procedures used in the validation studies and the results of those studies. · The possible valid uses of the test. The purposes for which the test can legitimately be used should be described, as well as the performance criteria that can validly be predicted. · The sample group(s) on which the test was developed. For example, was the test developed on a sample of high school graduates, managers, or people with graduate degrees? What was the racial, ethnic, age, and gender mix of the sample? · The group(s) for which the test may be used. The criterion-related validity of a test is measured by the validity coefficient. It is reported as a number between 0 and 1.00 that indicates the magnitude of the relationship, "r," between the test and a measure of performance (criterion). The larger the validity coefficient, the more confidence you can have in predictions made from the test scores. However, a single test can never fully predict outcome because of the many varied factors. Therefore, validity coefficients, unlike reliability coefficients, rarely exceed r = .40. As a general rule, the higher the validity coefficient the more beneficial it is to use the test. Validity coefficients of r =.21 to r =.35 are typical for a single test. Validities for selection systems that use multiple tests will probably be higher because you are using different tools to measure/predict different aspects of performance, where a single test is more likely to measure or predict fewer aspects of total performance. Table 2 serves as a general guideline for interpreting test validity for a single test. Table 1. General Guidelines for Interpreting Validity Coefficients
- 12. Validity Coefficient Value Interpretation of Validity above .35 very beneficial .21 - .35 likely to be useful .11 - .20 depends on circumstances below .11 unlikely to be useful Page 6 of 7