Logic based reasoning test paullin et al 2010
- 1. 1 Using a Logic-Based Measurement Approach to Measure Cognitive Ability Cheryl Paullin, Dan J. Putka, and Suzanne Tsacoumis, HumRRO Magda Colberg, Logos Corporation April 9, 2010 In C. Paullin (Chair), Cognitive ability testing: Exploring new models, methods, and statistical techniques. Symposium conducted at the 25th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.
- 2. 2 Logic-Based Measurement • Mid-80’s: Colberg & colleagues codified a taxonomy of logic rules that could be used to generate reasoning test items in mid-1980’s • 1990: LBR test used in selection process for 90+ professional and administrative occupations in U.S. federal gov’t • Today: – Several U.S. federal agencies (e.g., CBP, FBI) – At least one Fortune 500 company • Several conference papers – but not much in peer- reviewed journals • Promising approach -- could be used more widely
- 3. 3 LBR Item Police officers were led to believe that many weapons sold at a certain gun store were sold illegally. Upon investigating the lead, the officers learned that all of the weapons sold by the store that were made by Precision Arms were sold legally. They also found that none of the illegally sold weapons were .45 caliber. From the information given above, it can be validly concluded that A. all of the .45 caliber weapons were made by Precision Arms B. none of the.45 caliber weapons were made by Precision Arms C. some of the weapons made by Precision Arms were .45 caliber weapons D. all of the .45 caliber weapons that were sold were sold legally E. some of the weapons made by Precision Arms were sold illegally. From: Simpson, R.W. & Nester, M.A. (2002). The construct and content validity of logic-based tests of reasoning for personnel selection: The SIOP Conference Paper. In T.L. Hayes (Chair), The validity of logic-based measurement for selection and promotion decisions. Symposium conducted at the 17th Annual Conference of the Society for Industrial and Organizational Psychology, Toronto, Canada. Correct Answer: D
- 4. 4 LBR Alternate Testing Format Facts: Research demonstrates that for many computer-related crimes the severity of punishment is the strongest predictor of whether an individual becomes a repeat offender. In a study of 500 individuals convicted of hacking into corporate financial networks, it was determined that if offenders were sentenced to time in prison or fined large sums of money, they would tend not to repeat their crimes. Five years after the study, it was found that only 20% of the offenders in the study repeated their crimes. From the information given above, indicate whether each statement below is TRUE, FALSE, or INDETERMINABLE. Conclusion 1: According to the results of the study, offenders who are neither fined nor imprisoned are certain to become repeat offenders. Conclusion 2: All study participants who repeated their crimes during the five years that followed the study had been convicted of hacking into corporate financial networks. Conclusion 3: In the context of computer-related crimes, research has demonstrated that whether one becomes a repeat offender is determined entirely by the severity of punishment. Each passage followed by several conclusions; 3-choice answer for each conclusion. 5-8 conclusions based on each passage
- 5. 5 Simpler Format & Reasoning (Sienna Reasoning Test) A GATH resembles a SHET but is heavier. A SHET resembles a COUCH but is heavier. A MUNT resembles a LAMP but is heavier. Determine if each of the following statements is TRUE or FALSE. Conclusion 1. A GATH is heavier than a SHET. Conclusion 2. A COUCH is heavier than a GATH. Conclusion 3. A LAMP is heavier than a COUCH. Very simple passage, 2-choice answer about each conclusion. In this test, very common or nonsense words. NOTE: Sienna Reasoning Test also includes figural reasoning items
- 6. 6 What’s So Great about LBR Tests? • Items replicate logical thought processes required to perform job duties • Items involve application of reasoning skills • No subject knowledge required • No training in formal rules of logic required • Keyed answers can’t be disputed
- 7. 7 What’s so Great about LBR Tests (cont.)? • Good internal consistency reliability (~.80) • Strong criterion-related validity, higher than for many other measures of cognitive ability ~.60* for training perf, job knowledge, work sample ~.30* for supervisor ratings of job perf • Smaller subgroup effect sizes than for some traditional cognitive ability measures (personal communication, Colberg) * Meta-analysis based on 5-13 samples, N>2,500; values corrected for criterion unreliability
- 8. 8 Cautionary Notes • LBR tests often have a high reading load, so important to show that job requires a lot of reading • Native speakers earned higher scores than non- native in one large-scale study (still valid for all) • Persons from some cultures scored higher than persons from other cultures (still valid for all) – Confucian Asian and Nordic/Germanic GLOBE clusters highest, Anglo and Eastern Europe middle; Sub-Saharan Africa lowest
- 9. 9 Criterion-Related Validation Studies • U.S. Federal Agency • Jobs: – Special Agent (Study 1) – Intelligence Analyst (Study 2) • Design/Sample – Concurrent (incumbents) – N > 400 per study • Criterion measures – Job Knowledge Test (JKT) – Overall Performance Ratings Composite (supervisor)
- 10. 10 Criterion-Related Validation Studies (cont.) • Predictor measures – Developed by HumRRO specifically for this client • LBR Test • Situational Judgment Test (SJT) • Biodata/P-E Fit Scales – Vendor-provided instrument • Sienna Reasoning Test
- 11. 11 Predictor Psychometric Information Test Study 1 (Agent) Study 2 (Analyst) Logic-Based Reasoning (LBR) α = .79 α = .85 Siena Reasoning Test (SRT) -- α = .77 Situational Judgment Test (SJT) α = .60 α = .62 Correlations Between Predictors LBR Study 2 SRT SRT 0.41 -- SJT Study 2 .28 .23 N > 400 Do not appear to measure the exact same constructs
- 12. 12 Uncorrected Criterion-Related Validity Estimates JKT Overall Performance Ratings Composite LBR 1 (Agent) .39 .05 LBR 2 (Analyst) .60 .32 SRT (Analyst) .40 .25 SJT 1 (Agent) .18 .14 SJT 2 (Analyst) .28 .18 Corrected Criterion-Related Validity Estimates JKT Overall Performance Ratings Composite LBR 1 (Agent) .38 n.s. LBR 2 (Analyst) .69 .45 SRT (Analyst) .46 .35 SJT 1 (Agent) .24 .27 SJT 2 (Analyst) .32 .25 Note. N = 483 for Study 1; N = 359-461 for Study 2. All correlations except for the correlation between LBR 1 and Overall Performance are significant at the 0.05 level (1-tailed).
- 13. 13 Subgroup Effect Sizes Measure dC-African Amer dC-Hispanic dC-Asian dM-F dU40-40+ LBR 1 (Agent) 0.79 0.66 0.28 0.22 -- LBR 2 (Analyst) 1.00 0.58 0.76 0.03 0.42 SRT 2 (Analyst) 0.38 0.56 0.18 0.03 0.74 SJT 1 (Agent) ~.0.79 ~0.05 ~0.18 ~0.08 -- SJT 2 (Analyst) 0.75 0.27 0.09 -0.02 0.24 Notes. Study 1: Caucasian n = 609, African-American n = 43, Hispanic n = 72, Asian n = 59, Male n = 626, Female n = 159. Study 2: Caucasian n = 338-350, African-American n = 36-44, Hispanic n = 26-28, Asian n = 31-33, Male n = 209-220, Female n = 237-250, Under 40 n = 353-367, 40+ n = 88-99. Caution: Small sample sizes for several race subgroups
- 14. 14 How do Results Compare with Prior LBR Research? • Validity: – Study 1 lower – Study 2 higher • Effect Sizes: – Study 1 consistent – Study 2 larger for some Corrected Criterion-Related Validity Estimates JKT Overall Performance Rtg Compos Meta-Analysis (Hayes & Reilly, 2002) .58 .27 LBR 1 (Agent) .38 n.s. LBR 2 (Analyst) .69 .45 Measure dC-African Amer dC-Hispanic dC-Asian dM-F dU40-40+ Harris, Callas, Busciglio, 2002 0.69 0.59 -- 0.14 -- LBR 1 (Agent) 0.79 0.66 .028 0.22 -- LBR 2 (Analyst) 1.00 0.58 0.76 0.03 0.42
- 15. 15 Difference in LBR Performance Across Studies • Job differences? – Analyst job revolves around reading vast amounts of material and using it to reach logical conclusions (validity) • Criterion differences? – Supervisors may have placed relatively more weight on applied reasoning skills in evaluations of Analysts than of Agents (validity for predicting perf) • Sample differences? – Study 1 sample more directly screened on cognitive ability, more range restriction (validity)
- 16. 16 Difference in LBR Performance Across Studies (cont.) • Test difficulty differences? – No appreciable difference in mean performance (confounded with sample characteristics) • Test format differences? – Study 2 LBR format is more grammatically complex • May increase race subgroup differences • May also increase validity
- 17. 17 LBR Test vs Siena Reasoning Test • Validity: LBR higher in direct comparison • Effect Size: LBR larger for some; smaller for others • Difficulty: LBR more difficult – as least as configured in these studies • Incremental Validity when adding SRT to LBR Test: • ΔR2 = .025 for JKT • ΔR2 = .012 for overall perf Measure dC-African Amer dC-Hispanic dC-Asian dM-F dU40-40+ LBR 2 (Analyst) 1.00 0.58 0.76 0.03 0.42 SRT 2 (Analyst) 0.38 0.56 0.18 0.03 0.74 JKT Overall Performance Ratings Composite LBR 2 (Analyst) .69 .45 SRT (Analyst) .46 .35 Test Mean % Correct SD LBR 2 60% 17 % points SRT 75% 14 % points
- 18. 18 Summary Points • LBR tests – Reliable, valid way to measure applied reasoning skills – As or more valid than other cognitive ability measures – Same or smaller subgroup differences as other cognitive ability measures – Best for jobs with a high verbal/reading load and that require a high level of reading and reasoning skills • LBR and SRT are viable choices for measuring cognitive ability for employee selection
- 19. 19 References Colberg, M. (1984). Towards a taxonomy of verbal tests based on logic. Educational and Psychological Measurement, 44, 113-120. Colberg, M. (1985). Logic-based measurement of verbal reasoning: A key to increased validity and economy. Personnel Psychology, 38, 347-359. Goldstein, H.W., Scherbaum, C.A., & Yusko, K.P. (2009). Revisiting g: Intelligence, adverse impact, and personnel selection. In J. Outtz’ (Ed.) Adverse Impact: Implications for Organizational Staffing and High Stakes Selection. Routledge Academic: New York, NY. Hayes, T.L. (Chair) (2002). The validity of logic-based measurement for selection and promotion decisions. Symposium conducted at the 17th annual conference of the Society for Industrial and Organizational Psychology, Toronto, Canada. Includes these papers: Harris, P.A., Callen, N.F., Busciglio, H. Transportability of the logic-based measurement approach for law enforcement selection with the U.S. Customs Service. Hayes, T. L. & Reilly, S. M. The criterion-related validity of logic-based measurement tests: The SIOP Conference paper. Leaman, J.A., & Gast, I.F. Content validation of a logic-based assessment of thinking skills. Simpson, R.W., & Nester, M.A. The construct and content validity of logic-based tests of reasoning for personnel selection. Simpson, R., Nester, M.A., & Palmer, E. (2007). The validity of logic-based tests. Presented at the annual conference of the International Public Management Association- Assessment Council, St. Louis, MO. Tsacoumis, S., Putka, D.J., & Colberg, M. (2007). A cross-cultural look at items of logic- based reasoning. In A.S. Boyce & R.E. Gibby (Chairs), Global cognitive ability testing: Psychometric issues and applicant reactions. Symposium conducted at the 22nd annual conference of the Society for Industrial and Organizational Psychology, New York, NY.