Make clinical prediction models great again
- 1. Make clinical prediction models Great Again Ben Van Calster Department of Development and Regeneration, KU Leuven (B) Department of Biomedical Data Sciences, LUMC (NL) Research Ethics Committee, University Hospitals Leuven (B) Epi-Centre, KU Leuven (B) Istanbul, February 29th 2020
- 2. Contents • What are we talking about? • Developing models: 1. What do you want, when, and how? 2. Do not ignore information 3. Mind overfitting • Externally validating models: 1. Calibration is essential 2. Expect heterogeneity • What about machine learning? 2
- 3. What are we talking about? 3
- 4. 4 What are we talking about Development Validation Machine learning
- 5. To explain • Study (strength of) independent associations with the outcome, e.g. to find risk factors 5 What are we talking about Development Validation Machine learning Kempenaers et al. Injury 2018;49;2269-75.
- 6. To predict • Obtain a system that gives risk estimates of the outcome • Aim is the use in NEW patients: it should work ‘tomorrow’, not now 6 What are we talking about Development Validation Machine learning Edlinger et al. BMJ Open 2017;7;e014467.
- 7. To predict 7 What are we talking about Development Validation Machine learning cvriskcalculator.com
- 8. Developing clinical risk prediction models 8
- 9. 1. What do you want? • What specific outcome should be predicted? Is there a clinical need? • When during the clinical workflow should the prediction be made? • Which predictors are available at that time point? • What is the purpose? E.g. which treatment decision should it support? • What is the quality of the data? 9Cronin & Vickers. Urology 2010;76:1298-1301 What are we talking about Development Validation Machine learning
- 10. Mistaking the objective… 10 Riley. Nature 2019;572:27-9. What are we talking about Development Validation Machine learning
- 11. Example 11Hernandez-Suarez et al. JACC Cardiovasc Interv 2019;12:1328-38. What are we talking about Development Validation Machine learning
- 12. Example (contd) 12 The model also uses postoperative information. David J Cohen, MD: “The model can’t be run properly until you know about both the presence and the absence of those complications, but you don’t know about the absence of a complication until the patient has left the hospital.” https://www.tctmd.com/news/machine-learning-helps-predict-hospital-mortality-post-tavr-skepticism-abounds What are we talking about Development Validation Machine learning
- 13. 2. Do not ignore information a. Continuous variables should not be dichotomized, only decisions based on a prediction model should! 13Butts & Ng. In Lance & Vandenberg. Routledge 2009. What are we talking about Development Validation Machine learning
- 14. 2. Do not ignore information b. Use available knowledge, do not always ask the data! 14Good & Hardin. Wiley 2006. “Perhaps the most serious source of error lies in letting statistical procedures make decisions for you” “Don’t be too quick to turn on the computer. Bypassing the brain to compute by reflex is a sure recipe for disaster” What are we talking about Development Validation Machine learning
- 15. 15Rajkomar et al. Npj Digit Med 2018;1:18. Will the hype of “machine learning” make us bypass our brain once more? What are we talking about Development Validation Machine learning
- 16. But what if you don’t know at all? 16Good & Hardin. Wiley 2006. If you have no knowledge on what variables could be good predictors (and what variables not), are you ready to make a good prediction model? What are we talking about Development Validation Machine learning
- 17. 3. Mind overfitting You think of buying a Porsche. But if you do not want to pay for it, you may get this. The same applies for developing risk models. 17 What are we talking about Development Validation Machine learning
- 18. Our currency is sample size The more complicated (or ‘fancy’) the modeling strategy, the more you have to pay with sample size. Preferably good data (no counterfeit money!) Match sample size to a sensible modeling strategy, or vice versa Further recommendations to avoid overfitting: - Avoid data driven variable selection where you can: you have to pay! - Be careful with interactions: you have to pay (and often get little back)! - Do not use train-test split: you’re burning your money! 18 What are we talking about Development Validation Machine learning
- 19. Flexible algorithms are data hungry 19http://www.portlandsports.com/hot-dog-eating-champ-kobayashi-hits-psu/ What are we talking about Development Validation Machine learning
- 20. Externally validating clinical risk prediction models 20
- 21. 1. Calibration is essential Key elements: discrimination between patients with and without the event calibration (correctness) of risk estimates 21 DISCRIMINATION When it rained, was the estimated chance of rain higher (on average)? CALIBRATION For days with 80% estimated chance of rain, did it rain on 8 out of 10 days? What are we talking about Development Validation Machine learning
- 22. Assess calibration! Management decisions are influenced by the magnitude of the estimated risk of an outcome of interest. If this estimation is systematically off, decisions are ill-informed. 22 What are we talking about Development Validation Machine learning
- 23. 2. Expect heterogeneity 23Cronin & Vickers. Urology 2010;76:1298-1301 What are we talking about Development Validation Machine learning
- 24. Performance will depend on location Expect heterogeneity across hospitals, regions, countries One external validation study does not tell you much about the model! 24Pennells et al. Am J Epidemiol 2014;179:621-632. Van Calster et al, submitted. What are we talking about Development Validation Machine learning
- 25. Performance will depend on time Care changes, populations change, so will model performance 25Davis et al. JAMIA 2017;24:1052-61. What are we talking about Development Validation Machine learning
- 26. Model updating? 26Riley et al. BMJ 2016;353:i3140. Snell et al. J Clin Epidemiol 2016;69:40-50. Every hospital its own model that is kept up-to-date: Realistic or utopic? What are we talking about Development Validation Machine learning Before After
- 28. Reason for popularity 28 Claim: “Typical machine learning algorithms are highly flexible, so will uncover associations we could not find before, And hence lead to better predictions and management decisions” → One of the master keys, with guaranteed success! What are we talking about Development Validation Machine learning
- 29. Machine Learning: success guaranteed? 29Christodoulou et al. J Clin Epidemiol 2019;110:12-22. What are we talking about Development Validation Machine learning
- 30. Traditional Statistics vs Machine Learning 30Christodoulou et al. J Clin Epidemiol 2019;110:12-22. What are we talking about Development Validation Machine learning
- 31. Poor modeling and unclear reporting 31 What was done about missing data? 45% fully unclear, 100% poor or unclear How were continuous predictors modeled? 20% unclear, 25% categorized How were hyperparameters tuned? 66% unclear, 19% tuned with information How was performance validated? 68% unclear or biased approach Was calibration of risk estimates studied? 79% not at all, HL test common Prognosis: time horizon often ignored completely Christodoulou et al. J Clin Epidemiol 2019;110:12-22. What are we talking about Development Validation Machine learning
- 32. Concerns for predictive analytics 32 Poor study design and modeling strategy Do we need machine learning? Get design and methodology right first. Flexible algorithms and complicated modeling strategies are data hungry Large datasets often have poor quality There is large heterogeneity between settings and studies Populations change over time, using a model further changes it Reporting is often problematic What are we talking about Development Validation Machine learning