Free
Correspondence  |   May 2016
In Reply
Author Notes
  • Vanderbilt University Medical Center, Nashville, Tennessee (M.A.T.). maxim.terekhov@vanderbilt.edu
  • (Accepted for publication January 28, 2016.)
    (Accepted for publication January 28, 2016.)×
Article Information
Correspondence
Correspondence   |   May 2016
In Reply
Anesthesiology 5 2016, Vol.124, 1196-1197. doi:10.1097/ALN.0000000000001067
Anesthesiology 5 2016, Vol.124, 1196-1197. doi:10.1097/ALN.0000000000001067
We thank Dr. Hyder for his interest in our recent article published in Anesthesiology, “Preoperative Surgical Risk Predictions Are Not Meaningfully Improved by Including the Surgical Apgar Score: An Analysis of the Risk Quantification Index and Present-On-Admission Risk Models.”1 
As suggested by Dr. Hyder, we performed additional analyses using an alternative sampling interval for vital signs and added a calculation of risk reclassification to better test the clinical utility of the Surgical Apgar Score (SAS) when combined with preoperative risk stratification models.
A sampling method for slowest heart rate (HR) and lowest mean arterial pressure (MAP) was established before initiating data analyses. The method was based on “windows” or intervals of data and was established as follows: 10-min nonoverlapping windows, with windows beginning at the time of incision (0 to 10 min, 11 to 20 min, 21 to 30 min, etc.). Within each window, a median value was determined. Median values for HR and MAP were the basis for the original SAS investigations, and median values were chosen for this investigation. Estimated blood loss as recorded by the in-room anesthesia provider was calculated for the entire case.2 
We also added a calculation of risk reclassification to better test the clinical utility of the SAS. The use of a reclassification measure may be applied to provide a more clinically meaningful assessment of change in risk prediction. A concept of categorizing patients into high- and low-risk groups is clinically intuitive and actionable, as we treat high-risk patients differently, such as with admission to the intensive care unit. Traditionally, risk prediction models have been evaluated using the area under the receiver operating characteristic curve, along with model calibration, Brier score, information criteria, etc., but this can be an insensitive measure for model comparison in a healthcare setting, providing little direct clinical relevance. Since its description in 2006, much interest has been generated in reclassification, which assesses the ability of new models to more accurately classify individuals into higher or lower risk strata. This has led to new methods of evaluating and comparing risk prediction models, including the reclassification calibration test and the net reclassification index (NRI). Pencina et al.3  developed the NRI and the integrated discrimination improvement (fig. 1).
Fig. 1.
Reclassification tables. If the larger model (which includes the Surgical Apgar Score) on average assigns a higher risk class to cases and a lower risk class to noncases than the small model (no Surgical Apgar Score), then net reclassification index is positive.
Reclassification tables. If the larger model (which includes the Surgical Apgar Score) on average assigns a higher risk class to cases and a lower risk class to noncases than the small model (no Surgical Apgar Score), then net reclassification index is positive.
Fig. 1.
Reclassification tables. If the larger model (which includes the Surgical Apgar Score) on average assigns a higher risk class to cases and a lower risk class to noncases than the small model (no Surgical Apgar Score), then net reclassification index is positive.
×
After performing analyses using alternative sampling for vital signs and calculating risk reclassification, the Risk Quantification Index and present-on-admission preoperative risk models were not meaningfully improved by adding intraoperative risk using the SAS, as determined by the NRI value of 0.02 (P = 0.10). These analyses supported the original findings: adding the SAS did not substantively improve predictions. In addition to the estimated blood loss, lowest HR, and lowest MAP, other dynamic clinical parameters from the patient’s intraoperative course may need to be combined with procedural risk estimate models to improve risk stratification.
Acknowledgments
This work was funded, in part, by the Department of Anesthesiology, Vanderbilt University, Nashville, Tennessee, and the Foundation for Anesthesia Education and Research and Anesthesia Quality Institute Health Services Research Mentored Research Training Grant, Schaumburg, Illinois (to Dr. Wanderer).
Competing Interests
The authors declare no competing interests.
Maxim A. Terekhov, M.S., Jesse M. Ehrenfeld, M.D., M.P.H., Jonathan P. Wanderer, M.D., M.Phil. Vanderbilt University Medical Center, Nashville, Tennessee (M.A.T.). maxim.terekhov@vanderbilt.edu
References
Terekhov, MA, Ehrenfeld, JM, Wanderer, JP Preoperative surgical risk predictions are not meaningfully improved by including the Surgical Apgar Score: An analysis of the risk quantification index and present-on-admission risk models.. Anesthesiology. (2015). 123 1059–66 [Article] [PubMed]
Hyder, JA, Kor, DJ, Cima, RR, Subramanian, A How to improve the performance of intraoperative risk models: An example with vital signs using the surgical apgar score.. Anesth Analg. (2013). 117 1338–46 [Article] [PubMed]
Pencina, MJ, D’Agostino, RBSr, D’Agostino, RBJr, Vasan, RS Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond.. Stat Med. (2008). 27 157–72; discussion 207–12 [Article] [PubMed]
Fig. 1.
Reclassification tables. If the larger model (which includes the Surgical Apgar Score) on average assigns a higher risk class to cases and a lower risk class to noncases than the small model (no Surgical Apgar Score), then net reclassification index is positive.
Reclassification tables. If the larger model (which includes the Surgical Apgar Score) on average assigns a higher risk class to cases and a lower risk class to noncases than the small model (no Surgical Apgar Score), then net reclassification index is positive.
Fig. 1.
Reclassification tables. If the larger model (which includes the Surgical Apgar Score) on average assigns a higher risk class to cases and a lower risk class to noncases than the small model (no Surgical Apgar Score), then net reclassification index is positive.
×