Free
Correspondence  |   June 2006
Better Methods for Meta-analysis Available
Author Notes
  • University of Utah, Salt Lake City, Utah.
Article Information
Correspondence
Correspondence   |   June 2006
Better Methods for Meta-analysis Available
Anesthesiology 6 2006, Vol.104, 1350. doi:
Anesthesiology 6 2006, Vol.104, 1350. doi:
To the Editor:—
Recently, Wu et al.  1 published a systematic review comparing the treatment of postoperative pain by intravenous patient-controlled analgesia (PCA) versus  epidural analgesia—either patient-controlled epidural analgesia (PCEA) or continuous infusion epidural analgesia (CEI), usually with a local anesthetic–opioid mixture. The literature search discovered 50 articles. Using visual analog scale measurements of pain as the primary outcome, the meta-analysis used a fixed effect analysis of variance (ANOVA) to compare the treatment groups—PCA versus  PCEA and CEI. The average visual analog scale values for all data (mean ± SD) were 3.2 ± 1.6 versus  2.1 ± 1.3; this difference favoring PCEA and CEI was declared statistically significant at P  < 0.001.
The authors specially limited their literature search to the English language, but some non-English research reports otherwise qualifying for inclusion were incidentally identified. The authors report that inclusion of five such studies would not have changed the meta-analytic results. Current recommendations for performance of systematic reviews specifically discourage exclusion—without good reason—of publications in languages other than English.2 Empirical research has shown that under some circumstances, trials not published in English demonstrated statistically significant results less often3; other research on language bias in systematic reviews concluded that exclusion of non-English language trials had shown unpredictable consequences on the summary statistics of a meta-analysis.4 The authors report no reasons for limiting their literature search to the English language. They should reconsider their exclusion of trials in other languages.
The numbers of patients reported in the 50 trials were 1,625 (PCEA and CEI) and 1,583 (PCA), whereas the numbers of observations included in the ANOVA of overall data were 7,744 (PCEA and CEI) and 7,666 (PCA). This difference in the number of observations versus  the number of patients is the consequence of including visual analog scale scores obtained at multiple times in each patient. These multiple observations in each patient are not considered independent variables. The inclusion of multiple observations has been denoted as a “unit of analysis” error.5 The likely consequence of a unit of analysis error is a spurious precision in the calculation of SD. The authors should restrict their meta-analysis to the observations obtained independently; this can be done simply by dropping all analyses using “overall” and “all” data in table 2.1 
The authors chose ANOVA as the statistical method for comparing PCEA and CEI versus  PCA. ANOVA is a method for hypothesis testing. The main emphasis in a systematic review is the effect measure. The effect measure is a single number that contrasts the treatments; statistically, this is parameter estimation, not hypothesis testing. Because the visual analog scale score may be considered approximately a continuous variable, the relevant effect measure is the difference in mean values—also known as the weighted mean difference.6 By contrast, the ANOVA results presented show the mean values for each treatment group. The meta-analysis should be redone using the weighted mean difference effect measure. The calculation of a summary effect measure is accompanied by statistical tests of heterogeneity. The identification of heterogeneity may encourage the use of a different statistical model, the random effects model. The fixed effect ANOVA reported in this study does not allow identification of heterogeneity among the 50 studies. Finally, with the use of an effect measure, the results should be presented in a Forest plot.7 This allows the inspection of the effect measure for each individual trial as well as the summary value of the effect measure; heterogeneity may become more easily recognizable. The Forest plot also makes clear the statistical significance of both the individual studies and the summary effect measure. For a weighted mean difference effect measure, if the lower and upper boundaries of the 95% confidence interval do not bound the zero value, the estimate of the effect measure is declared to be statistically significant.
Although this systematic review was not created for the Cochrane Collaboration, the methods for systematic reviews and meta-analysis presented in their publications provide a rigorous guide for this research.8 The authors should reconsider several of their experimental methods that may have produced biased inferences. It is possible and to be desired that the conclusions of a revised systematic review will be unchanged from the current version.
University of Utah, Salt Lake City, Utah.
References
Wu C, Cohen S, Richman J, Rowlingson A, Courpas G, Cheung K, Lin E, Liu S: Efficacy of postoperative patient-controlled and continuous infusion epidural analgesia versus  intravenous patient-controlled analgesia with opioids: A meta-analysis. Anesthesiology 2005; 103:1079–88Wu, C Cohen, S Richman, J Rowlingson, A Courpas, G Cheung, K Lin, E Liu, S
Locating and selecting studies, Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. Edited by Higgins J, Green S. Chichester, United Kingdom, John Wiley & Sons, 2005, section 5
Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G: Language bias in randomised controlled trials published in English and German. Lancet 1997; 350:326–9Egger, M Zellweger-Zahner, T Schneider, M Junker, C Lengeler, C Antes, G
Juni P, Holenstein F, Sterne J, Bartlett C, Egger M: Direction and impact of language bias in meta-analyses of controlled trials: Empirical study. Int J Epidemiol 2002; 31:115–23Juni, P Holenstein, F Sterne, J Bartlett, C Egger, M
Repeated observations on participants, Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. Edited by Higgins J, Green S. Chichester, United Kingdom, John Wiley & Sons, 2005, section 8.3.3
Effect measures for continuous outcomes, Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. Edited by Higgins J, Green S. Chichester, United Kingdom, John Wiley & Sons, 2005, section 8.2.2
Presenting, illustrating, and tabulating results, Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. Edited by Higgins J, Green S. Chichester, United Kingdom, John Wiley & Sons, 2005, section 8.9
Higgins J, Green S: Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. The Cochrane Library, Issue 3, 2005. Chichester, United Kingdom, John Wiley & Sons, 2005Higgins, J Green, S Chichester, United Kingdom John Wiley & Sons