Correspondence  |   May 2011
Bias and Precision Statistics: Should We Still Adhere to the 30% Benchmark for Cardiac Output Monitor Validation Studies?
Author Notes
  • Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, New Territories, Hong Kong.
Article Information
Correspondence   |   May 2011
Bias and Precision Statistics: Should We Still Adhere to the 30% Benchmark for Cardiac Output Monitor Validation Studies?
Anesthesiology 5 2011, Vol.114, 1245. doi:10.1097/ALN.0b013e318215e1e5
Anesthesiology 5 2011, Vol.114, 1245. doi:10.1097/ALN.0b013e318215e1e5
To the Editor:
It was with great interest that I read Peyton and Chong's meta-analysis of papers comparing minimally invasive cardiac output monitors to thermodilution,1 particularly as much of the discussion centered around a much-cited paper by me and my late brother Julian Critchley.2 Our 1999 paper made a number of important contributions to assessing cardiac output monitoring2 : (1) it provided guidelines on how the results from validation studies should be presented; (2) it defined a standard statistical variable by which performance could be compared, the percentage error or limits of agreement; (3) it established that the error of the reference method, usually thermodilution, was also an important factor in validation studies; and (4) it set limits of acceptance of ± 30% based on a precision for the reference method of ± 20%. Peyton and Chong should be congratulated on providing a thorough and comprehensive meta-analysis of cardiac output validation studies published during the last decade.1 However, I would like to take issue with two of their conclusions.
Peyton and Chong make reference to the accuracy of thermodilution cardiac output.1 I would agree that the true precision of the thermodilution method in many of these validation studies is unknown. Cecconi et al.  have suggested that the precision of the reference method should be known before using Bland and Altman analysis in validation studies.3 However, this is not easily done, and simply calculating the coefficient of variability from a series of readings is not the answer.4 The precision of the thermodilution can vary quite considerably depending of the type of catheter and measurement system, as we have shown in a recent in vitro  investigation.5 However, I would have to take issue with Peyton and Chong in their assertion that precision is significantly worse during in vivo  or clinical testing. I agree that cardiac output varies with respiration and other physiologic effects, but comparative measurements are made simultaneously, and the mean of several readings is used to average out these background physiologic effects.
Peyton and Chong also suggest that our ± 30% limits of agreement are set too low and a more realistic benchmark would be ± 45%, which they base on reference and test precision errors of ± 30%, reset from our ± 20%. Good news for the manufacturers. However, our choice of a ± 20% error for the reference readings was not just based on data from papers by Stetz et al.  and Mackenzie et al.,  6,7 as Peyton and Chong suggest. In Bland and Altman's 1986 paper,8 the decision as to what were acceptable limits of agreement was left to the judgment of the clinician, and for cardiac output measurement, most authors at the time were suggesting a precision of less than 1 l/min, which for a mean cardiac output of 5 l/min was 20%.9,10 Furthermore, increasing the limits ± 45% would give a false impression that many monitors are measuring cardiac output reliably, which simply is not true. Therefore, I would still recommend using ± 20% for our criteria.
Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, New Territories, Hong Kong.
Peyton PJ, Chong SW: Minimally invasive measurement of cardiac output during surgery and critical care: A meta-analysis of accuracy and precision. Anesthesiology 2010; 113:1220–35Peyton, PJ Chong, SW
Critchley LA, Critchley JA: A meta-analysis of studies using bias and precision statistics to compare cardiac output measurement techniques. J Clin Monit Comput 1999; 15:85–91Critchley, LA Critchley, JA
Cecconi M, Rhodes A, Poloniecki J, Della Rocca G, Grounds RM: Bench-to-bedside review: The importance of the precision of the reference technique in method comparison studies–with specific reference to the measurement of cardiac output. Crit Care 2009; 13:201Cecconi, M Rhodes, A Poloniecki, J Della Rocca, G Grounds, RM
Critchley LA, Lee A, Ho AM: A critical review of the ability of continuous cardiac output monitors to measure trends in cardiac output. Anesth Analg 2010; 111:1180–92Critchley, LA Lee, A Ho, AM
Yang XX, Critchley LA, Joynt GM: Determination of the precision error of the pulmonary artery thermodilution catheter using an in vitro  continuous flow test rig. Anesth Analg 2011; 112:70–7Yang, XX Critchley, LA Joynt, GM
Stetz CW, Miller RG, Kelly GE, Raffin TA: Reliability of the thermodilution method in the determination of cardiac output in clinical practice. Am Rev Respir Dis 1982; 126:1001–4Stetz, CW Miller, RG Kelly, GE Raffin, TA
Mackenzie JD, Haites NE, Rawles JM: Method of assessing the reproducibility of blood flow measurement: Factors influencing the performance of thermodilution cardiac output computers. Br Heart J 1986; 55:14–24Mackenzie, JD Haites, NE Rawles, JM
Bland JM, Altman DG: Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1(8476):307–10Bland, JM Altman, DG
Wong DH, Tremper KK, Stemmer EA, O'Connor D, Wilbur S, Zaccari J, Reeves C, Weidoff P, Trujillo RJ: Noninvasive cardiac output: Simultaneous comparison of two different methods with thermodilution. Anesthesiology 1990; 72:784–92Wong, DH Tremper, KK Stemmer, EA O'Connor, D Wilbur, S Zaccari, J Reeves, C Weidoff, P Trujillo, RJ
LaMantia KR, O'Connor T, Barash PG: Comparing methods of measurement: An alternative approach. Anesthesiology 1990; 72:781–3LaMantia, KR O'Connor, T Barash, PG