Free
Correspondence  |   October 2016
Prediction Model for In-hospital Mortality Should Accurately Predict the Risks of Patients Who Are Truly at Risk
Author Notes
  • Vanderbilt University Medical Center, Nashville, Tennessee (T.H.K). teus.kappen@vanderbilt.edu
  • (Accepted for publication June 16, 2016.)
    (Accepted for publication June 16, 2016.)×
Article Information
Correspondence
Correspondence   |   October 2016
Prediction Model for In-hospital Mortality Should Accurately Predict the Risks of Patients Who Are Truly at Risk
Anesthesiology 10 2016, Vol.125, 815-816. doi:10.1097/ALN.0000000000001269
Anesthesiology 10 2016, Vol.125, 815-816. doi:10.1097/ALN.0000000000001269
To the Editor:
With great interest, we read the article by Le Manach et al.1  The article presents a prediction model for postoperative in-hospital mortality with very good discriminative abilities (C statistic of 0.93 in a validation cohort). However, we contend the conclusion that the predictive model is well calibrated.
A prediction model should first and foremost provide accurate predicted probabilities. When validating a prediction model, it is essential to answer the question whether predicted probabilities correspond to observed probabilities, especially for patients who may have a clinically relevant risk of the predicted outcome.
The reported calibration plot (fig. 2 in the article1 ) seems to show a well-calibrated model. However, the calibration plot is truncated at a predicted probability of 0.10, and the plot shows only 9 out of 10 deciles. Patients with the highest risks seem to have been omitted from the reported calibration plot. Figure 3 of the article1  shows the observed mortality in the validation cohort for a wider range of risk scores. Supplemental Digital Content 3 reports the predicted probabilities for all Preoperative Score to Predict Postoperative Mortality (POSPOM) scores. If we overlay the predicted probabilities for all POSPOM scores onto figure 3 of the article,1  we observe that the prediction model greatly overestimates the in-hospital mortality risk in the high-risk patients (fig. 1). Although the high-risk patients form only a small group, they are in fact the patients for whom the prediction model is most clinically relevant. We would not want to be the physician who communicates a 62% risk of death to a patient (POSPOM value of 40) while the actual risk is 23%.
Fig. 1.
Distribution of the Preoperative Score to Predict Postoperative Mortality (POSPOM) values in the validation cohort (n = 2,789,932) in relation to the observed in-hospital mortality rate (solid line) at each POSPOM value. Gray line is the predicted probability according to Supplemental Digital Content 3 of the article by Le Manach et al.1 
Distribution of the Preoperative Score to Predict Postoperative Mortality (POSPOM) values in the validation cohort (n = 2,789,932) in relation to the observed in-hospital mortality rate (solid line) at each POSPOM value. Gray line is the predicted probability according to Supplemental Digital Content 3 of the article by Le Manach et al.1
Fig. 1.
Distribution of the Preoperative Score to Predict Postoperative Mortality (POSPOM) values in the validation cohort (n = 2,789,932) in relation to the observed in-hospital mortality rate (solid line) at each POSPOM value. Gray line is the predicted probability according to Supplemental Digital Content 3 of the article by Le Manach et al.1 
×
From the results presented in the article, it is impossible to deduce what the reason is for the discrepancy in the results. There are several inconsistencies in the reporting of the predicted probabilities. The text of the article mentions a predicted probability of 5.65% for a POSPOM value of 30, whereas the Supplemental Digital Content reports a predicted probability 7.403%. There is no way to reconstruct the predicted probability as the model’s intercept is not reported. It is possible that the predicted probabilities in Supplemental Digital Content 3 are wrong, which then could explain the overestimation of the in-hospital mortality risk. Even then, it would still be a necessity to report a continuous calibration plot of the model for the entire range of predicted probabilities, with a histogram within that calibration plot.
We believe that the prediction model may be of great value to both physicians and patients, but only after the overestimation of the high-risk patients has been addressed.
Competing Interests
The authors declare no competing interests.
Teus H. Kappen, M.D., Ph.D., Jonathan P. Wanderer, M.D., M.Phil., Linda M. Peelen, Ph.D., Karel G. M. Moons, Ph.D., Jesse M. Ehrenfeld, M.D., M.P.H. Vanderbilt University Medical Center, Nashville, Tennessee (T.H.K). teus.kappen@vanderbilt.edu
Reference
Reference
Le Manach, Y, Collins, G, Rodseth, R, Le Bihan-Benjamin, C, Biccard, B, Riou, B, Devereaux, PJ, Landais, P Preoperative score to predict postoperative mortality (POSPOM): Derivation and validation.. Anesthesiology. (2016). 124 570–9 [Article] [PubMed]
Fig. 1.
Distribution of the Preoperative Score to Predict Postoperative Mortality (POSPOM) values in the validation cohort (n = 2,789,932) in relation to the observed in-hospital mortality rate (solid line) at each POSPOM value. Gray line is the predicted probability according to Supplemental Digital Content 3 of the article by Le Manach et al.1 
Distribution of the Preoperative Score to Predict Postoperative Mortality (POSPOM) values in the validation cohort (n = 2,789,932) in relation to the observed in-hospital mortality rate (solid line) at each POSPOM value. Gray line is the predicted probability according to Supplemental Digital Content 3 of the article by Le Manach et al.1
Fig. 1.
Distribution of the Preoperative Score to Predict Postoperative Mortality (POSPOM) values in the validation cohort (n = 2,789,932) in relation to the observed in-hospital mortality rate (solid line) at each POSPOM value. Gray line is the predicted probability according to Supplemental Digital Content 3 of the article by Le Manach et al.1 
×