Validity & reliability of screening & diagnostic tests
- 1. VALIDITYAND RELIABILITY OF DIAGNOSTIC AND SCREENING TESTS DR TANVEER REHMAN SCIENTIST – B (MEDICAL) PUBLIC HEALTH DIVISION
- 2. Why assess the quality of diagnostic tests? • Infectious disease modelling • Patient care • Secondary prevention programs • Primary prevention policies 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 2
- 3. Tests with Dichotomous Results 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 3
- 4. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 4
- 5. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 5
- 6. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 6
- 7. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 7 • “Gold standard” • Which is more dangerous – FN or FP?
- 8. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 8
- 9. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 9 • FN: critical cut- off treatment point missed • FP: expensive, burden to health system, stigma
- 10. 10/2/2022 10
- 11. Validity of Screening Tests Ability to distinguish between who has a disease and who does not I. Sensitivity of a test – ability to identify correctly those who have the disease: if high - “screening test” II. Specificity of the test – ability to identify correctly those who do not have the disease: if high - “confirmatory test” III. Characteristics of the test: not dependent on the prevalence 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 11
- 12. Example 10/2/2022 12 DrTanveer Rehman, Sc-B, ICMR-RMRCBB
- 13. Tests of Continuous Variables 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 13
- 14. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 14
- 15. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 15
- 16. Choosing cut-offs: Receiver Operating Characteristic (ROC) curve 10/2/2022 16 WHY
- 17. Use of Multiple Tests 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 17
- 18. Sequential (Two-stage) Testing • Test A followed by Test B • Those screened positive (A) are recalled for further testing (B) • Usually 2nd test (B): better validity (more expensive & invasive) • A person diagnosed positive (net/ final) if positive in both the tests • A loss in net sensitivity & a gain in net specificity 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 18
- 19. Simultaneous Testing • Test A & Test B at same time • All will undergo both A & B • Usually 2 screening tests employed • A person positive (net/ final) if positive in any of the two test • A gain in net sensitivity & a loss in net specificity 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 19
- 20. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 20
- 21. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 21
- 23. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 23 Clinical perspective
- 24. Predictive Value of a Test • The Positive Predictive Value (PPV) – the proportion of patients who test positive, actually having the disease in question • The Negative Predictive Value (NPV) – the proportion of patients who test negative, actually NOT having the disease in question. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 24
- 25. Example 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 25
- 26. Relationship between Positive Predictive Value and Disease Prevalence 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 26
- 27. Relationship between Positive Predictive Value and Disease Prevalence 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 27
- 28. Relationship between Positive Predictive Value and Disease Prevalence 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 28 Similarly, increasing prevalence will lead to decreasing NPV
- 29. Reliability (Repeatability) of Tests • Ability of a measurement to give the same result or a similar result with repeated measurement • Intrasubject variation (variation within individual subjects), • Intra-observer variation (variation in the reading of test results by the same reader) 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 29
- 30. Percent Agreement 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 30 OBSERVATION NO. 1 OBSERVATION NO. 2
- 31. Percent Agreement OR Diagnostic accuracy 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 31 OBSERVATION NO. 1 OBSERVATION NO. 2
- 32. Kappa Statistic 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 32 WHY NEEDED ??
- 33. Kappa Statistic 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 33
- 34. Kappa Statistic 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 34
- 35. Kappa Statistic 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 35
- 36. Kappa Statistic 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 36 LANDIS & KOCH
- 37. 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 37
- 38. SUMMARY 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 38 VARIABLE INDEX VALIDITY RELIABILITY Categorical Sensitivity /specificity Yes No Youden’s J statistic Yes ? Percent agreement/ Kappa statistic ?* Yes Continuous Scatter plot ? Yes Intraclass correlation coefficient (ICC) ? Yes Bland-Altman plot Yes Yes
- 39. For intrepid scholars • Confidence interval of sensitivity, specificity • Percent positive agreement (PPA) • Chamberlain’s PPA • Weighted kappa • How kappa dependent on prevalence? • Derivation of formulae for net sensitivity & specificity • ICC • Bland-Altman plot 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 39
- 40. THANK YOU 10/2/2022 DrTanveer Rehman, Sc-B, ICMR-RMRCBB 40
Editor's Notes
- #39: Systematic differences (bias) between readings & outliers