Clinical Prediction Rules
Download as PPT, PDF6 likes2,814 views
This document discusses clinical prediction rules (CPRs), which are decision tools used by clinicians to predict outcomes. It covers the development, validation, and functions of CPRs. Specifically, it outlines 8 standards for developing a CPR, including clearly defining outcomes and predictors, ensuring reliability of predictors, having an adequate sample size, and accurately measuring a CPR's performance. An example CPR for ankle fractures is used to illustrate the development process. The document emphasizes the importance of prospectively validating CPRs in new populations before implementation, to assess their accuracy outside the initial study.
Clinical Prediction Rules
- 1. Clinical Prediction Rules Jen-Hsiang Chuang, MD, MS, PhD Centers for Disease Control Taiwan jhchuang@cdc.gov.tw 1
- 2. Clinical Prediction Rules (CPRs) • Synonym: clinical decision rules • Definition: decision-making tools for clinicians including 3 or more variables – Provide the probability of an outcome – Suggest a diagnostic or therapeutic course of action Laupacis A, et al. Clinical prediction rules. JAMA 1997;277:488-494. 2
- 3. Clinical Prediction Rules Vs. Clinical Practice Guidelines • Clinical prediction rules – Derived from original research involving many patients and mathematical analysis • Clinical practice guidelines – Consensus among experts – GOBSAT (Good Old Boys Sat At Table) (Miller J, et al. Lancet 2000;355:82-3) – But can include CPRs 3
- 4. Functions of CPRs • CPRs help clinicians cope with uncertainty and improve efficiency – Cope with uncertainty • Community-acquired pneumonia (Fine MJ, et al. NEJM 1997;336:243-250) – Improve efficiency • Ottawa Ankle Rules for the use of radiography (Stiell IG, et al. Ann Emerg Med 1992;21:384-90) Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 4
- 5. Prototype of a CPR for Predicting Death Predictor variables Score Age > 75 yr 6 Severe pain 10 Emergency 5 Total points 0-21 Interpretation of the score High risk: > 6 points (30% deaths) -> aggressive Tx Low risk: ≤ 6 points (3% deaths) -> conservative Tx Wasson JH, et al. Clinical prediction rules. NEJM 1985;313:793-9. 5
- 6. Three Stages in the Evaluation of a CPR 1. Development of a CPR 2. Prospective validation of a CPR 3. Impact analysis of a CPR McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 6
- 7. So What? • Q: “Give me the reasons why I need to stay here to listen your presentation?” • A: a medical informatician may play two roles – Reader role – Developer role 7
- 8. Development of a CPR 8
- 9. Checklist of Standards for Development of a CPR 1. Definition of outcome 2. Definition of predictor variables 3. Reliability of predictor variables 4. Selection of subjects 5. Sample size 6. Mathematical techniques 7. Sensibility of CPR 8. Accuracy Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 9
- 10. 1. Definition of Outcome • Clearly defined and clinically important – Explicit criteria for diagnosis – Biologic better than behavioral outcome • Blind assessment of outcome – More important for a “soft” outcome – Less important for a “hard” outcome 10
- 11. 2. Definition of Predictor Variables • Clearly defined – Best: collected prospectively, specifically – Less good: collected prospectively as part of another study – Worst: collected from retrospective review of records • Blind assessment of predictor variables 11
- 12. 3. Reliability of Predictor Variables • Only reliable variables be included – Intraobserver reliability – Interobserver reliability • Measurement of reliability – Dichotomous or nominal data: κ – Ordinal data: weighted κ – Continuous data: intraclass correlation coefficient http://www.dmi.columbia.edu/homepages/chuangj/kappa/ 12
- 13. 4. Selection of Subjects • Patient characteristics stated – Inclusion and exclusion criteria – Method of selection – Clinical and demographic characteristics • Study site described – Type of institution (primary, secondary, tertiary) – Setting (clinic, ER, hospital ward) – Teaching or non-teaching 13
- 14. 5. Sample Size • Overfitting problem – Too few outcome events per predictor variable • Appropriate sample size – Rule of thumb: at least 10 outcome events per independent variable – e.g., 3 findings to predict death => at least 30 patients died 14
- 15. 6. Mathematical Techniques • Mathematical methods adequately described and justified – Multivariate analysis • Logistic regression • Discriminant analysis – Machine learning • Recursive partitioning (including decision tree learning) • Neural networks – Survival analysis (survival data only) • Cox's Proportional Hazard Model 15
- 16. Multivariate Analysis • General model • Logistic regression – Where P is probability of outcome; G is log odds of outcome • Discriminant analysis – Compute cutoff (C) – Assign patient to class 1 if G < C; otherwise assign patient to class 2 nn xbxbxbbG ++++= 22110 P P G − = 1 ln 16
- 17. Logistic Regression Vs. Discriminant Analysis • Logistic regression is much more popular than discriminant analysis (Concato et, al. 1993) – Logistic regression • Binary outcome • Estimate individual risk and odds ratios – Discriminant analysis • Categorical outcome • Optimal performance requires many predictor variables as continuous data • Famous application: diagnosis alcoholism 17
- 18. Recursive Partitioning • Principle – Build an empirical tree diagram by repetitively splitting patient population into smaller and smaller categories Yes No 4 5Employed 2 1 Yes 3 No Age>30 18
- 19. Recursive Partitioning Vs. Multivariate Analysis • Recursive partitioning provides a simpler classification rule • Recursive partitioning may identify nonlinear relationships with outcome event • Recursive partitioning need greater sample size • Logistic regression can estimate individual risk and odds ratios Cook EF, et al. J Chron Dis 1984;37:721-31. 19
- 20. Cross SS, et al. Introduction to neural network. Lancet 1995;346:1075-9. Neural Networks 20
- 21. Clinical Applications of Neural Networks • Diagnosis – AMI, Appendicitis, back pain, dementia, STD • Imaging – Radiographs, PET, NMR, perfusion scans • Analysis of wave forms – ECGs, EEGs • Outcome prediction – Recovery from surgery, cancer, liver transplantation • Identification of pathological specimens • Genomics Baxt WG. Application of ANN to clinical medicine. Lancet 1995;346:1135-8. 21
- 22. Kattan MW, et al. ANN for medical classification decisions. Arch Pathol Lab Med 1995;119:672-7. Advantages of Neural Networks •Multiple partitioning •Nonlinear partitioning 22
- 23. Disadvantages of Neural Networks • Slow to train • “Black-boxes” Kattan MW, et al. ANN for medical classification decisions. Arch Pathol Lab Med 1995;119:672-7.23
- 24. 7. Sensibility of CPR • “Sensibility” – clinically reasonable, easy to use, course of action described – judgment • Clinically reasonable – Content validity • Easy to use – Length of time needed to apply – Simplicity of interpretation • Course of action described 24
- 25. 8. Accuracy of CPR • Rationale • Measurement of accuracy – 2x2 table with sensitivity, specificity, with respective 95% CIs – Receiver operator characteristic (ROC) curves • Statistical validation – Cross-validation: Training set vs. test set Wasson JH, et al. Clinical prediction rules. NEJM 1985;313:793-9. 25
- 26. Classification Performance of a CPR Predicted Outcome Actual Outcome Disease No Disease Disease 74 244 No Disease 0 247 Sensitivity (95% CI): 1.0 (0.95-1.0) Specificity (95% CI): 0.50 (0.46-0.55) Stiell IG, et al. Implementation of the Ottawa Ankle Rules. JAMA 1994;271:827-32. 26
- 27. Kuo HS, Chuang JH, Tang GJ, et al. Chin Med J (Taipei) 1999;62:673-681. 27
- 28. Example: Development of a CPR Ottawa Ankle Rules Stiell IG, et al. Ann Emerg Med 1992;21:384-90 28
- 30. The Need for an Ankle Rule • Blunt ankle trauma – One of the most common injuries in ER – Less than 15% of patients have fractures – Physicians used to order radiography for all ankle injury patients – 85% negative for fracture – $500 M annually in North America – No widely accepted guideline Stiell IG, et al. Implementation of the Ottawa Ankle Rules. JAMA 1994;271:827-32. 30
- 31. Study Design • Objective – Develop CPR with 100% sensitivity • Design – Prospective survey of ED patients over 5 months • Patient population: – Setting: Two university hospital EDs in Ottawa – Inclusion: All acute blunt injuries of ankle – Exclusion: < 18 y/o, pregnant, referral, etc • Data collection – 32 clinical variables collected by 21 trained physicians before radiography – 100 patients examined by a 2nd physician 31
- 32. Study Design (Cont.) • Measurements of outcomes – Radiography interpreted by a radiologist blinded to the contents of data collection sheets • No fracture or insignificant fracture • Clinically significant fracture • Data analysis – Variables found to be both strongly associated with a significant fracture (P < 0.05) and reliable (κ > 0.6) were analyzed by logistic regression and recursive partitioning 32
- 33. Results • 70 (10.2%) significant malleolar fractures in 689 ankle injury patients • Univariate analysis: 17 variables were significantly associated with fractures • 9 non-reliable variables were further eliminated • Logistic regression: Sen: 1.0, Spe: .29 • Recursive partitioning: Sen: 1.0, Spe: .40 33
- 34. 689 561128 70# 39# Yes No 31# 67 Yes 12# 494 No 19# 21118# Yes 283 No 1# 35 248 0# NoYes 1# 441 High Risk 248 Low Risk A B C D LEGEND # Fracture A Unable to bear weight immediately and in ED B Age 55 or greater C Bone tenderness B4 or B5 D Bone tenderness B8 or B9 Recursive Partitioning of 689 Cases 34
- 35. Prospective Validation of a CPR 35
- 36. Problems of CPRs With Statistical Validation Only • Many statistically derived rules fail to perform well when tested in a new population – Overfitting or instability in the original derived model – Differences in prevalence of disease – Differences in severity of cases – Differences in how the CPR is applied Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 36
- 37. Prospective Validation of a CPR • Validation – Its repeated application leads to the same results • Types of validation – Narrow validation: application of rule in a similar setting and population – Broad validation: application of rule in multiple clinical settings with varying prevalence and outcomes of disease McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 37
- 38. Development of a Clinical Prediction Rule McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 38
- 39. Methodological Standards for Validation of a CPR • Unbiased, wide spectrum patient population • Blinded assessment of outcomes and predictor variables • Careful follow-up of predicted normal patients • Training for correctly applying rules McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 39
- 40. Results of Validation Studies of Ottawa Ankle Rules Markert RJ, et al. Am J Emerg Med 1998;16:564-7. Country (Year) # of Subjects Sensitivity (%) Specificity (%) CA (1993) 1032 100 39 US (1994) 71 100 19 NZ (1994) 350 93 11 US (1994) 631 100 19 US (1995) 422 95 16 40
- 41. Impact Analysis of a CPR 41
- 42. Reasons for No Impact of an Accurate CPR • Clinician’s intuition may be as good as the CPR • Calculations involved may be cumbersome • Practical barriers to acting on the results of CPR – Medical liability risk – Patient demand factor McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 42
- 43. Methodological Standards for Impact Analysis of a CPR • Study design – Cluster-based randomized control trial – Before-after study • Effect on use – e.g., ordering of radiography • Accuracy of rule • Acceptability of physicians & patients Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 43
- 44. Impact Analysis of Ottawa Ankle Rules in France • Randomized 5 EDs to use or not use CPR • 2 in intervention group (906 patients) – Meeting, pocket cards, posters, and data collection forms • 3 in usual care group (1005 patients) – data collection forms only • Results: (unit of analysis was physician) – ordering of radiography: I: 79%; C: 99% (P=.03) – I: 3/112 missed fractures (incomplete data forms: 2, rule interpretation error by physician: 1) Auleley GR, et al. JAMA 1997;277:1935-9. 44
- 45. Summary • Development of an effective prediction rule is a long, rigorous, and expensive process • Properly developed and validated prediction rules can influence clinical practice 45
- 46. Performance Evaluation • Discrimination – Ability of a prediction model to separate those who experience events from those who do not – Area under a ROC curve (c statistic) • Calibration – Measures how closely predicted outcomes agree with actual outcomes – Hosmer-Lemeshow goodness-of-fit test 46
- 48. 48
- 49. Thank You! 49