Projects per year
Abstract
Introduction
There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation.
Methods
We studied pseudovalues based on the AalenJohansen estimator, binary logistic regression with inverse probability of censoring weights (BLRIPCW), and multinomial logistic regression with inverse probability of censoring weights (MLRIPCW). The MLRIPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records.
Results
The pseudovalue, BLRIPCW and MLRIPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in lowdensity regions of predicted transition probability.
Conclusions
We recommend implementing either the pseudovalue or BLRIPCW approaches to produce a calibration curve, combined with the MLRIPCW approach to produce a calibration scatter plot. The methods have been incorporated into the ‘calibmsm’ R package available on CRAN.
There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation.
Methods
We studied pseudovalues based on the AalenJohansen estimator, binary logistic regression with inverse probability of censoring weights (BLRIPCW), and multinomial logistic regression with inverse probability of censoring weights (MLRIPCW). The MLRIPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records.
Results
The pseudovalue, BLRIPCW and MLRIPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in lowdensity regions of predicted transition probability.
Conclusions
We recommend implementing either the pseudovalue or BLRIPCW approaches to produce a calibration curve, combined with the MLRIPCW approach to produce a calibration scatter plot. The methods have been incorporated into the ‘calibmsm’ R package available on CRAN.
Original language  English 

Journal  Statistics in medicine 
DOIs  
Publication status  Published  8 May 2024 
Keywords
 calibration
 risk prediction
 clinical prediction
 model validation
 multistate model
Fingerprint
Dive into the research topics of 'Calibration plots for multistate risk predictions models: an overview and simulation comparing novel approaches'. Together they form a unique fingerprint.Projects
 1 Finished

HOD2: Toward Holistic Approaches to Clinical Prediction of MultiMorbidity: A Dynamic Synergy of InterConnected Risk Models.
Martin, G., Peek, N., Sergeant, J. & Van Staa, T.
1/05/20 → 30/04/23
Project: Research