Free
Correspondence  |   July 2013
Anesthetic Exposure and Academic Achievement: Missing the Forest for the Trees
Author Affiliations
  • Joel B. Gunter, M.D.
    Children’s Hospital Medical Center, University of Cincinnati, Cincinnati, Ohio. joel.gunter@cchmc.org
Article Information
Correspondence
Correspondence   |   July 2013
Anesthetic Exposure and Academic Achievement: Missing the Forest for the Trees
Anesthesiology 07 2013, Vol.119, 233-234. doi:10.1097/ALN.0b013e318297a659
Anesthesiology 07 2013, Vol.119, 233-234. doi:10.1097/ALN.0b013e318297a659
To the Editor:
I read with interest the recent report from Block et al.,1  examining the relation between anesthetic exposure in the first year of life and academic achievement, but have concerns that their analysis inappropriately emphasizes the post hoc observation of an excess of subjects with extremely low achievement scores rather than overall academic achievements. The authors do not explicitly state their Null (Alternative) Hypotheses; however, based on the title of their report and its Introduction, I suspect that they were of the form “the distribution of achievement test scores in children exposed to anesthesia before age 1 yr does not (does) differ from that seen in children not so exposed.” The analysis presented by the authors would be appropriate only to the far more specific Null (Alternative) Hypotheses that “children exposed to anesthesia before 1 yr of age are not (are) more likely to score in the lowest fifth percentile on tests of academic achievement.” The authors’ analysis unnecessarily discards almost all the informations available from the distribution of scores.
When considering distributions of outcomes, a more appropriate analysis would be the chi-square goodness-of-fit test. Based on a minimum cell expected value of 5, we can divide the authors’ data from figure 1 into 10 deciles, each with an expected value of 5.8 (table 1).
Table 1.
Frequency of Academic Achievement Score by Decile
Frequency of Academic Achievement Score by Decile×
Frequency of Academic Achievement Score by Decile
Table 1.
Frequency of Academic Achievement Score by Decile
Frequency of Academic Achievement Score by Decile×
×
For this distribution, chi-square test is equal to 8.55 with 9 degrees of freedom (df; P = 0.48). This should be interpreted as meaning that the probability of randomly drawing a sample of 58 subjects from a uniformly distributed population, at least this extreme, is 48%. In other words, the distribution of the achievement scores for the authors’ subjects is entirely consistent with a randomly selected sample from the population of all children taking the test. A similar analysis of subjects’ percentile ranks converted to z-scores yields the even less significant result of chi-square test of 2.16 for df = 5 (P = 0.83).
An analogy may be helpful. We are given an icosahedral (12-sided) die and wish to determine if it is fair. The die is rolled 60 times so that the expected value for each possible outcome is five; the results are recorded in a table similar to that in the preceding paragraph (table 2). It is tempting to interpret the excess number of 4’s rolled as evidence that the die is unfair; the result is certainly statistically “significant” for the binomial outcome “4” versus “not 4” (the 95% CI for the observed frequency for each result based on 60 rolls of a fair 12-sided die is 0–9, which does not include the observed value of 10). This conclusion would be incorrect. There was no a priori reason to suspect that the die would roll more 4’s than any other number. Because we had neither specified nor predicted the outcome of interest, we must be prepared to analyze every possible outcome, including an appropriate adjustment to the test-wise level of α to maintain an experiment-wise level of α of 0.05. Because this adjustment can result in an extremely conservative test-wise α, we instead look at the overall distribution of rolls; based on this (chi-square test = 9.60; df = 11; P = 0.57), we conclude that there is no evidence based on this experiment that the die is unfair. Of course, we may wish to repeat the experiment with the hypothesis that the die favors 4’s, especially given the relative deficit of rolls on the side opposite the 4 (the 9).
Table 2.
Frequency of Outcomes for 60 Rolls of a 12-sided Die
Frequency of Outcomes for 60 Rolls of a 12-sided Die×
Frequency of Outcomes for 60 Rolls of a 12-sided Die
Table 2.
Frequency of Outcomes for 60 Rolls of a 12-sided Die
Frequency of Outcomes for 60 Rolls of a 12-sided Die×
×
In a similar way, it seems obvious that the excess number of subjects observed in the lowest fifth percentile must mean that the academic achievement was lower in the group exposed to anesthesia in the first year of life. However, there are other deciles whose deviation from the expected is more extreme (50–60 and 60–70%); we are not free to pick and choose which extreme deviations to treat as random chance and which to analyze as meaningful unless we have included a specific deviation in the Null and Alternative Hypotheses. There are many ways that exposure to anesthetics might impact academic achievement. Exposure might decrease each subject’s achievement by a fairly consistent amount relative to that which would have been seen absent exposure, in effect shifting the entire normal curve to the left. Although this would indeed result in an excess of subjects in the lowest fifth percentile, it would also result in a relative deficit in the higher percentiles. This is not seen in the authors’ results; indeed, two subjects scored at the 100th percentile, far above the expected 0.58 subjects. Alternatively, anesthetic exposure might result in disparate impact on the subjects with the highest potential achievement, resulting in a dearth of subjects with high scores, but no major alteration in the distribution of low scores; again, this is not seen. Examination of the authors’ figure 1 shows that their results are almost entirely dependent on the presence of an unexpectedly large number of subjects (five) scoring at the second percentile; indeed, if we remove one of these subjects by excluding the four subjects who had additional surgical procedures beyond those in the inclusion criteria (authors’ figs. 2 and 3 and related discussion; see next paragraph), the resulting distribution falls nicely along the line of identity with the expected uniform distribution (chi-square test = 8.96; df = 9; P = 0.44) and the mean percentile score of the remaining 54 subjects is 49.8. As per the authors' discussion, their results seem most consistent with a mechanism that leads to minimal or no impact in the vast majority of subjects and severe, pervasive neurocognitive impairment in a very small subset of subjects.
Finally, some observations about their figures 2 and 3: it seems that the significance of the linear relations reported is entirely dependent on the presence of subjects who either had multiple exposures to anesthesia (fig. 2) or are extreme outliers with respect to duration of surgery (fig. 3). It seems unlikely that this subject, who scored at the second percentile and whose anesthetic lasted 4 h, had only a circumcision, herniorrhaphy, or pyloromyotomy; exclusion of this subject from figure 3 leaves a simple vertical cluster without any obvious trend. It would be interesting to see whether the relation between academic achievement and duration of surgery or anesthesia remains significant if the analysis is restricted to the 45 subjects who had only circumcision, herniorrhaphy, or pyloromyotomy. If nothing else, listing of the additional surgical procedures would permit the reader to decide whether or not the nine subjects who had additional surgical procedures were representative of the entire sample.
I salute the authors for attempting to address this complex and timely issue, but ask that they consider reanalyzing their data in light of my concerns. In particular, I would welcome publication of the details of the distribution of scores in their larger data sets, including analysis of goodness-of-fit as outlined above. If the larger data sets also include excess numbers of subjects with the very lowest scores, but are otherwise indistinguishable from the expected uniform distribution, then further, in-depth review of the medical and school records of the most impaired subjects might reveal common attributes that either increased their joint probability of poor academic achievement and the need for surgical intervention in the first year of life or their susceptibility to any neurocognitive impairment related to anesthetic or surgical exposure. Given the potentially incendiary impact of research on this topic, it is critical that results be carefully analyzed and thoughtfully presented. I urge the authors to consider repeating their experiment with the more specific Null and Alternative Hypotheses presented above to see whether their results are reproducible. Until then, their results only support a conclusion that the distribution of academic achievement scores in otherwise neurologically normal children with a single exposure to anesthesia in the first year of life for minor, peripheral surgery is completely consistent with that seen in the population at large. I find this conclusion reassuring.
Joel B. Gunter, M.D., Children’s Hospital Medical Center, University of Cincinnati, Cincinnati, Ohio. joel.gunter@cchmc.org
Reference
Reference
Block, RI, Thomas, JJ, Bayman, EO, Choi, JY, Kimble, KK, Todd, MM Are anesthesia and surgery during infancy associated with altered academic performance during childhood?. Anesthesiology. (2012). 117 494–503 [Article] [PubMed]
Table 1.
Frequency of Academic Achievement Score by Decile
Frequency of Academic Achievement Score by Decile×
Frequency of Academic Achievement Score by Decile
Table 1.
Frequency of Academic Achievement Score by Decile
Frequency of Academic Achievement Score by Decile×
×
Table 2.
Frequency of Outcomes for 60 Rolls of a 12-sided Die
Frequency of Outcomes for 60 Rolls of a 12-sided Die×
Frequency of Outcomes for 60 Rolls of a 12-sided Die
Table 2.
Frequency of Outcomes for 60 Rolls of a 12-sided Die
Frequency of Outcomes for 60 Rolls of a 12-sided Die×
×