Clinical Prediction Models and the Multiverse of Madness

Richard D. Riley, Alexander Pate, Paula Dhiman, Lucinda Archer, Glen Martin, Gary S. Collins

Research output: Contribution to journalArticlepeer-review

Abstract

Background
Each year thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.
Main body
We discuss how the creation of a prediction model (e.g., using regression or machine learning methods) is dependent on the sample and size of data used to develop it - were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created there exists a multiverse of other potential models for that sample size and, crucially, an individual’s predicted value (e.g. estimated risk)s may vary greatly across this multiverse. The more an individual’s prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.
Conclusions
Instability is concerning as an individual’s predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.
Original languageEnglish
JournalBMC Medicine
Publication statusAccepted/In press - 5 Dec 2023

Keywords

  • Clinical prediction model
  • instability
  • variance
  • risk prediction
  • bootstrapping
  • mean absolute prediction error (MAPE)

Fingerprint

Dive into the research topics of 'Clinical Prediction Models and the Multiverse of Madness'. Together they form a unique fingerprint.

Cite this