Free
Perioperative Medicine  |   June 2015
Monitoring Obstetric Anesthesia Safety across Hospitals through Multilevel Modeling
Author Notes
  • From the Department of Anesthesiology, Columbia University College of Physicians and Surgeons, New York, New York (J.G., G.L.); Département d’Anesthésie-Réanimation, APHP, Hôpital Bichat-Claude Bernard, Paris, France (J.G.); INSERM, UMR 1137, IAME, Paris, France (J.G.); and Department of Epidemiology, Columbia University Mailman School of Public Health, New York, New York (G.L.).
  • Submitted for publication June 18, 2014. Accepted for publication January 30, 2015.
    Submitted for publication June 18, 2014. Accepted for publication January 30, 2015.×
  • Address correspondence to Dr. Li: Center for Health Policy and Outcomes in Anesthesia and Critical Care, Department of Anesthesiology, Columbia University College of Physicians and Surgeons, 622 West 168th Street, Room 529, PH5-505, New York, New York 10032. gl2240@cumc.columbia.edu. Information on purchasing reprints may be found at www.anesthesiology.org or on the masthead page at the beginning of this issue. Anesthesiology’s articles are made freely accessible to all readers, for personal use only, 6 months from the cover date of the issue.
Article Information
Perioperative Medicine / Clinical Science / Obstetric Anesthesia / Regional Anesthesia / Quality Improvement
Perioperative Medicine   |   June 2015
Monitoring Obstetric Anesthesia Safety across Hospitals through Multilevel Modeling
Anesthesiology 6 2015, Vol.122, 1268-1279. doi:10.1097/ALN.0000000000000617
Anesthesiology 6 2015, Vol.122, 1268-1279. doi:10.1097/ALN.0000000000000617
Abstract

Background:: The rate of anesthesia-related adverse events (ARAEs) is recommended for monitoring patient safety across hospitals. To ensure comparability, it is adjusted for patients’ characteristics with logistic models (i.e., risk adjustment). The rate adjusted for patient-level characteristics and hospital affiliation through multilevel modeling is suggested as a better metric. This study aims to assess a multilevel model-based rate of ARAEs.

Methods:: Data were obtained from the State Inpatient Database for New York 2008–2011. Discharge records for labor and delivery and ARAEs were identified with International Classification of Diseases, Ninth Revision, Clinical Modification codes. The rate of ARAEs for each hospital during 2008–2009 was calculated using both the multilevel and the logistic modeling approaches. Performance of the two methods was assessed with (1) interhospital variability measured by the SD of the rates; (2) reclassification of hospitals; and (3) prediction of hospital performance in 2010–2011. Rankability of each hospital was assessed with the multilevel model.

Results:: The study involved 466,442 discharge records in 2008–2009 from 144 hospitals. The overall observed rate of ARAEs in 2008–2009 was 4.62 per 1,000 discharges [95% CI, 4.43 to 4.82]. Compared with risk adjustment, multilevel modeling decreased SD of ARAE rates from 4.7 to 1.3 across hospitals, reduced the proportion of hospitals classified as good performers from 18% to 10%, and performed similarly well in predicting future ARAE rates. Twenty-six hospitals (18%) were nonrankable due to inadequate reliability.

Conclusion:: The multilevel modeling approach could be used as an alternative to risk adjustment in monitoring obstetric anesthesia safety across hospitals.

Abstract

In an analysis of nearly 500,000 labor and delivery records from 144 hospitals in New York, multilevel modeling substantially improved the reliability in the estimated rates of obstetric anesthesia-related adverse events across hospitals compared with the traditional risk-adjustment method.

What We Already Know about This Topic
  • Comparison of patient safety indicators across hospitals is usually based on the risk-adjustment method through logistic regression modeling

  • Multilevel modeling that takes into account both patient- and hospital-level characteristics is suggested to be a more precise method for this comparison

  • Although it is adopted by the American College of Surgeons for hospital ranking, multilevel modeling has received little attention in anesthesia

What This Article Tells Us That Is New
  • In an analysis of nearly 500,000 labor and delivery records from 144 hospitals in New York, multilevel modeling substantially improved the reliability in the estimated rates of obstetric anesthesia-related adverse events across hospitals compared with the traditional risk-adjustment method

ON the top of the list of the 27 patient safety indicators (PSIs) issued by the Agency for Healthcare Research and Quality (AHRQ) in 2002 was the rate of complications of anesthesia or PSI-01.*01 1  Based on routinely collected administrative data, PSI-01 was designed for reporting and monitoring anesthesia safety across hospitals and for identifying safety concerns and targeting areas for safety improvement. Despite a long-standing culture of safety and safety indicators in anesthesia, application of PSI-01 in anesthesia research and practice has remained scarce.2–5  PSI-01 has lagged behind other PSIs, with some of them being publicly reported in the annual National Healthcare Quality and National Healthcare Disparities Reports and routinely calculated with hospitals information technology systems.†02 
The conventional approach for calculating the rate of adverse events and making it comparable across hospitals is risk adjustment.6  Risk adjustment takes into consideration differences in characteristics of patients (case-mix) and types of procedures (procedure-mix). It is based on logistic regression models that express the relationship between the binary outcome (i.e., the patient did or did not have an adverse event) and a set of predictors describing the case- and procedure-mixes. Recent research indicates that, to produce a more precise estimate that takes into consideration correlations of patients within hospitals (clustering), the rate of adverse events should be further adjusted for the hospital identifier (i.e., patients’ hospital affiliation) through multilevel modeling.7–9  In surgery, adjustment based on multilevel models has been demonstrated to increase the precision of the estimated rate of adverse outcomes and to significantly change ranking of hospitals that may also change the priority targets for safety measures.7,9  Furthermore, multilevel models quantify the level of confidence one can have in the estimated rate of adverse events for each hospital with rankability.10,11  Rankability identifies hospitals that should not be included in league tables or be identified as nonrankable in league tables. Finally, multilevel model–based adjustment may provide a better prediction for future patients than risk adjustment.12–14  Multilevel-based adjustment is now adopted by the American College of Surgeons in monitoring adverse outcomes after surgery across hospitals but has received little attention in anesthesia.15,16 
Each year, over 50 million surgical procedures are performed in the United States; of them, about 8% are related to labor and delivery.‡03  The median cost of anesthesia-related adverse events (ARAEs) in obstetrics is nearly twice compared with other anesthesia specialties.17,18  Despite the decrease in anesthesia-related mortality and severe morbidity during the last two decades, the cost of obstetric anesthesia-related complications has not significantly decreased.18–21  Currently, ARAEs occur in about one out of every 200 parturients.2  This figure may be increasing owing to the increased rate of cesarean section and parturients’ request for analgesics during labor.22,23  This study aims therefore to develop and assess a multilevel model–based rate of ARAEs in labor and delivery using administrative data for monitoring obstetric anesthesia safety across hospitals.
Materials and Methods
The study protocol was reviewed by the Institutional Review Board of Columbia University Medical Center and was granted exemption under the Code of Federal Regulations, Title 45, Part 46 (not human subjects research). The study adheres to the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement.24 
Study Sample
The study sample consisted of all women admitted for labor and delivery in the State of New York between January 1, 2008, and December 31, 2011. Data for years 2008–2009 were used to develop the logistic and multilevel models and data for years 2010–2011 to assess predictive ability of both models. Hospital discharge record data for these women collected in the de-identified New York State Inpatient Database were analyzed. State Inpatient Databases (SIDs) are part of the Healthcare Cost and Utilization Project sponsored by the AHRQ. SIDs capture all inpatient discharges from nonfederal acute care community hospitals in participating states since 1988. Nonfederal community hospitals account for 85% of U.S. hospitals. For each discharge, the SID includes patients’ demographic, economic, and outcome characteristics, one hospital identifier, and up to 15 procedural and 25 diagnostic codes defined in the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM). Discharges with neonatal or maternal diagnoses and procedures identified with the neonatal-maternal code provided by the SID and female sex were first selected. Then, discharges indicating labor and delivery were identified with a combination of ICD-9-CM diagnosis and procedure codes developed by Kuklina et al.25  (appendix 1). However, Diagnosis-Related Group codes were not used in this study since they changed during the 4-yr study period. In addition, discharges were excluded if the hospital identifier was missing or if delivery took place in a hospital with less than 2 deliveries/year.
Outcome Measure
ARAEs were identified with a combination of ICD-9-CM diagnosis and procedure codes developed by Cheesman and colleagues2,3  (appendix 2). We also analyze the subgroup of ARAEs related to neuraxial anesthesia and local anesthetics (appendix 2). This subgroup of complications was selected owing to both its high incidence and preventability.2,26 
Patient and Hospital Variables
The following demographic and delivery characteristics were recorded directly from the SID: age, admission for delivery during weekend, and admission type for delivery (elective or nonelective). Other patient- and procedure-related risk factors for ARAEs were identified with ICD-9-CM diagnosis and procedure codes (appendix 3).
Consequences of Multilevel Model–based Adjustment and Reporting
Consequences of multilevel model–based adjustment were assessed with (1) the interhospital variability of the estimated rate of ARAEs; (2) the reclassification of hospitals based on their outlier status; and (3) the ability of the multilevel model developed with the 2008–2009 data to predict future hospital performance in 2010–2011. In addition, the confidence in the point estimate for each hospital or hospital rankability was estimated with the multilevel model.
Since the rate of ARAEs may have been influenced by coding practice at each hospital, the relationship between the reporting index of ICD-9-CM codes for each hospital and the multilevel model–based rate of ARAEs was assessed. For each hospital, the reporting index was defined as the ratio of the sum of ICD-9-CM diagnosis (including E-codes) and procedure codes recorded for each discharge to the number of discharges.27 
Statistical Analysis
Results are expressed as mean ± 1 SD or number (%). When indicated, 95% CI was calculated.
The statistical analysis was performed with R version 3.0.2 (R Foundation for Statistical Computing, Austria) and specific packages for the multilevel model (lme4 and arm).
Development of the Logistic and Multilevel Models
A three-stage approach was used to develop the logistic and multilevel models using data for years 2008 and 2009. The first stage was a logistic regression model specifying each component of the case- and procedure-mixes. The second stage was a multilevel model including the hospital level only (“empty model”). The third stage was a multilevel model specifying each component of the case- and procedure-mixes as the first-level variables and the hospital identifier as the second-level variable. At each of the three stages, goodness of fit was assessed with the Akaike information criterion, discrimination with the c-index, and calibration with the Hosmer–Lemeshow test. A lower Akaike information criterion indicates a better fit with a difference greater than 6 indicating a strong difference.28  For the logistic model, univariate comparisons between discharges with and without ARAEs were made using unpaired Wilcoxon test for quantitative variables and chi-square test or Fisher exact test for qualitative variables. Unadjusted odds ratios were calculated with univariate logistic regression. Variables with a P value less than 0.2 in the univariate analysis were entered in the logistic model with a backward selection using the entire dataset for 2008–2009. For the multilevel model, hospital affiliation was treated as a random-effect predictor. It corresponded to the hospital identifier with the assumption of a normally distributed hospital intercept and a constant slope.
Interhospital Variability of the Estimated Rate of ARAEs
The risk-adjusted and multilevel model–based rates of ARAEs for each hospital were calculated as the ratio of the observed to the expected (O/E) rate multiplied by the observed rate in the study sample. The expected rate was the mean of the individual probabilities of experiencing an ARAE in that hospital. For the risk-adjusted rate, probabilities were calculated with a logistic regression model including the case-mix and the procedure-mix as fixed-effect predictors. For the multilevel model–based rate, probabilities were calculated with a multilevel model including the case-mix and the procedure-mix as fixed-effect predictors (first level) and the hospital identifier as a random-effect predictor (second level).
The extent to which multilevel model–based adjustment reduced interhospital variability was assessed by the comparison of the SDs and skewnesses of the grand mean of the rates (i.e., the mean of the hospitals in the sample study).
Rankability
Rankability of each hospital was calculated with the following formula: where σ2 indicates the variance.8,10,11  The between-hospital variance corresponds to the variance of the random effect in the multilevel model. It is sometimes described as the “signal” since it corresponds to the difference between hospitals beyond chance. The within-hospital variance corresponds to the variance of the random effect for each hospital. It is sometimes described as the “statistical noise” since it corresponds to the within-hospital uncertainty. Rankability ranges from 0 to 1. Rankability greater than 0.7 is considered as good and greater than 0.9 as excellent. As indicated by the formula, rankability depends not only on the difference between hospitals and the measurement or sampling error but also on the hospital volume.
Definition of Hospitals’ Outlier Status and Reclassification
Hospitals were divided into three groups based on their outlier status with risk and multilevel model–based adjustment methods and reclassification tables built. Hospital outliers were defined according to the American College of Surgeons’ National Surgical Quality Improvement Program criteria.13,16 
For risk adjustment, outlier definition used the hospital O/E ratio. O/E ratio was calculated as the ratio of the observed to the expected rate in the hospital as described in the section “Interhospital Variability of the Estimated Rate of ARAEs,” but without including the constant term observed rate in the study sample. Definitions of outliers were as follows: high outlier or bad performer if O/E was greater than 1 with its 95% CI not including 1, low outlier or good performer if O/E was less than 1 with its 95% CI not including 1, and as expected or average performer if the 95% CI of O/E included 1. The lower and upper limits of the 95% CI of O/E was calculated as LL (or UL)/E, where LL (or UL) was the lower (or upper) limit of the CI of a Poisson distribution for the observed number of cases in the hospital and E the expected numbers of ARAEs in the hospital.
For multilevel model–based adjustment, outlier definition used the hospital odds ratio calculated directly from the multilevel model as the exponential of the random effect for each hospital estimated in the multilevel model. The definitions of high outlier, low outlier, or as expected were identical to the ones used for risk adjustment. The 95% CI of the hospital odds ratio was calculated as ±1.96 standard error, where the standard error was estimated in the multilevel model.
Prediction of Future Hospital Performance
The prediction of future performance for hospitals present both in 2008–2009 and 2010–2011 was based on hospital outlier status based on risk adjustment and multilevel model–based adjustment in 2008–2009. It was assessed in two ways: (1) the adjusted odds ratio of ARAEs for the high- and average-outlier status relative to the low-outlier status and (2) the proportion of between-hospital variance in ARAE rates in 2010–2011 explained by hospital outlier status in 2008–2009.29,30  To calculate the adjusted odds ratio, two logistic regression models were developed for patients admitted in 2010–2011 with the occurrence of ARAEs as the dependent variable and the previously identified patient- and procedure-level risk factors in 2010–2011 and the 2008–2009 outlier status as independent variables. The first model used the outlier status based on risk adjustment and the second model used the outlier status based on multilevel model–based adjustment. If the adjusted odds ratio was significantly greater than 1 for the high- and average-outlier status, then past hospital performance did predict future hospital performance. To calculate the proportion of between-hospital variance in ARAE rates in 2010–2011 explained by hospital outlier status in 2008–2009, three multilevel models were developed with the occurrence of ARAEs as the dependent variable. The first model used patient- and procedure-level risk factors previously identified as the first-level variables (fixed effect) and the hospital identifier as the second-level variable (random effect) for the years 2010–2011. In the two other models, the hospital outlier status in 2008–2009 based on either the logistic or the multilevel model was added to the set of the first-level variables. The proportion of variation in subsequent ARAE rates explained by hospital outlier status was calculated by the percent reduction in the between-hospital variance between the multilevel model without the hospital outlier status and the multilevel model with the hospital outlier status.
Reporting Index and Multilevel Model–based Rate
For each hospital, the association between the reporting index of ICD-9-CM codes and the multilevel model-based rate of ARAEs was assessed with the Pearson correlation coefficient. Comparison of the reporting index across hospitals was based on the Kruskal–Wallis test.
Sensitivity Analysis
A sensitivity analysis was performed using the subset of hospitals with a rankability greater than 0.7 regarding performance of the multilevel model, between-hospital variability in ARAEs rate, and prediction of future hospital performance.
Results
During the years 2008–2009, 466,442 discharges in 144 hospitals met the inclusion and exclusion criteria for labor and delivery and were included in the analysis (fig. 1). At least one ARAE was recorded in 2,156 discharges, yielding an observed rate in the study sample of 4.62 per 1,000 discharges (95% CI, 4.43 to 4.82). At least one ARAE related to neuraxial anesthesia and local anesthetics was recorded in 1,746 discharges (3.74/1,000; 95% CI, 3.57 to 3.92) (appendix 2).
Fig. 1.
Selection of the study sample. NEOMAT = neonatal-maternal code; SID = State Inpatient Database.
Selection of the study sample. NEOMAT = neonatal-maternal code; SID = State Inpatient Database.
Fig. 1.
Selection of the study sample. NEOMAT = neonatal-maternal code; SID = State Inpatient Database.
×
Development of the Logistic and Multilevel Models
Seven risk factors for ARAEs were identified in the logistic model using data for years 2008–2009: age, obesity, pulmonary hypertension, cardiac valvular disease, asthma, cesarean delivery, and postpartum hemorrhage (tables 1 and 2). The c-index of the model was 0.60 (0.58–0.61), and the Hosmer–Lemeshow test P value was 0.28 (appendix 4). The results from univariate analysis, multivariate logistic regression, and multilevel modeling for ARAEs related to neuraxial anesthesia and local anesthetics are presented respectively in appendices 5-7.
Table 1.
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009×
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Table 1.
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009×
×
Table 2.
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009×
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Table 2.
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009×
×
Interhospital Variability of the Estimated Rate of ARAEs
The grand mean of the risk-adjusted ARAE rate for the 144 hospitals was 5.29 per 1,000 deliveries, whereas the grand mean of the multilevel model–based rate was 4.38 per 1,000 deliveries. Compared with risk adjustment, multilevel model–based adjustment reduced both the SD from 4.68 to 1.35 and the skewness from 2.08 to −1.05 of the distribution of the estimated rates across the 144 hospitals. Multilevel-based adjustment tended to shrink estimated individual hospital ARAE rates toward the grand mean of all hospitals. The magnitude of shrinkage increased as the hospital volume of deliveries decreased (fig. 2).
Fig. 2.
(A) Relationship between the number of deliveries and the risk-adjusted rate of any type of anesthesia-related adverse events (ARAEs). The dashed horizontal lines represent the grand mean or the mean of ARAEs rates across the 144 hospitals in the study sample. The filled red points indicate hospitals with a rankability ≤0.7 or nonrankable hospitals. (B) Relationship between the number of deliveries and the multilevel model-based rate of any type of ARAEs. Adjustment with multilevel model tends to shrink estimated individual hospital ARAE rates toward the grand mean.
(A) Relationship between the number of deliveries and the risk-adjusted rate of any type of anesthesia-related adverse events (ARAEs). The dashed horizontal lines represent the grand mean or the mean of ARAEs rates across the 144 hospitals in the study sample. The filled red points indicate hospitals with a rankability ≤0.7 or nonrankable hospitals. (B) Relationship between the number of deliveries and the multilevel model-based rate of any type of ARAEs. Adjustment with multilevel model tends to shrink estimated individual hospital ARAE rates toward the grand mean.
Fig. 2.
(A) Relationship between the number of deliveries and the risk-adjusted rate of any type of anesthesia-related adverse events (ARAEs). The dashed horizontal lines represent the grand mean or the mean of ARAEs rates across the 144 hospitals in the study sample. The filled red points indicate hospitals with a rankability ≤0.7 or nonrankable hospitals. (B) Relationship between the number of deliveries and the multilevel model-based rate of any type of ARAEs. Adjustment with multilevel model tends to shrink estimated individual hospital ARAE rates toward the grand mean.
×
Rankability
The mean rankability of ARAE rates for the 144 hospitals was 0.81 ± 0.11. One hundred eighteen hospitals (81.9%) had a rankability greater than 0.7. The rankability increased with the hospital volume of deliveries (fig. 3). The mean volume of deliveries for the 26 hospitals with reliability less than or equal to 0.7 was 254.
Fig. 3.
Relationship between the number of deliveries and the rankability for any type of anesthesia-related adverse events. The filled red points indicate hospitals with rankability ≤0.7. The mean volume of delivery for the 26 hospitals with rankability ≤0.7 is 254.
Relationship between the number of deliveries and the rankability for any type of anesthesia-related adverse events. The filled red points indicate hospitals with rankability ≤0.7. The mean volume of delivery for the 26 hospitals with rankability ≤0.7 is 254.
Fig. 3.
Relationship between the number of deliveries and the rankability for any type of anesthesia-related adverse events. The filled red points indicate hospitals with rankability ≤0.7. The mean volume of delivery for the 26 hospitals with rankability ≤0.7 is 254.
×
Reclassification of Hospitals Based on Outlier Status
Eleven of the 26 low-outlier hospitals (42.3%) identified with risk adjustment were reclassified as as-expected outlier with multilevel model–based adjustment (table 3). Six of the 93 as-expected outlier hospitals (6.4%) identified with risk adjustment were reclassified as high outlier with multilevel model–based adjustment. With multilevel model–based adjustment, the proportion of low outliers decreased from 18.0% to 10.4% and the proportion of high outliers increased from 17.4% to 21.6%.
Table 3.
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment×
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment
Table 3.
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment×
×
Prediction of Future Hospital Performance
One hundred thirty-nine hospitals were present during both the 2008–2009 and 2020–2011 periods in the SID. During the 2010–2011 period, 453,617 discharges were analyzed and at least one ARAE of any type was recorded in 2,133 discharges, yielding an observed rate of 4.70 per 1,000 discharges (95% CI, 4.50 to 4.90). At least one ARAE related to neuraxial anesthesia and local anesthetics was recorded in 1,755 discharges (3.87/1,000; 95% CI, 3.69 to 4.05).
The adjusted odds ratios of experiencing an ARAE for a patient admitted during 2010–2011 in a high and average outlier status relative to a low outlier based on the logistic model in 2008–2009 were 1.84 (95% CI, 1.63 to 2.08) and 3.34 (95% CI, 2.93 to 3.81), respectively (table 4). The outlier status explained 47.1% of the between-hospital variance in the subsequent ARAEs rates. The adjusted odds ratio and proportion of variance explained were similar for outlier status based on the multilevel model.
Table 4.
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009×
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009
Table 4.
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009×
×
Reporting Index and Multilevel Model–based Rate of ARAEs
The mean reporting index of ICD-9-CM codes for hospitals was 6.7 ± 3.0 codes per discharge, with a significant difference across hospitals (P < 0.0001). No significant association was observed between the reporting index and the multilevel model–based rate of any type of ARAEs (r = 0.11; 95% CI, −0.06 to 0.27) and between the reporting index and the multilevel model–based rate of neuraxial and local anesthetics of ARAEs (r = 0.11; 95% CI, −0.05 to 0.27).
Sensitivity Analysis
Diagnostic statistics of the logistic regression model and the multilevel model changed little after excluding the 26 hospitals with rankability less than or equal to 0.7. As expected, excluding the 26 hospitals with rankability less than or equal to 0.7 decreased the interhospital variability and increased the proportion of between-hospital variance explained by hospital outlier status in both the logistic regression model and the multilevel model.
Discussion
Results of this study indicate that compared with risk adjustment, multilevel model–based adjustment considerably reduces the between-hospital variability in ARAE rates, leads to reclassification of hospitals, and identifies nonrankable hospitals. The predictive validity of the two adjustment methods, however, is similar.
The most striking result of the multilevel model–based adjustment was the reduction in the between-hospital variability in ARAE rates. This phenomenon is also known as shrinkage toward the grand mean. When the number of deliveries is low in a given hospital, the estimated rate of adverse events can be very unreliable. Multilevel models combine the limited information from the particular hospital with the information from all hospitals in the study sample to produce a more robust estimate of the rate in this particular hospital. Multilevel-based adjustment with the multilevel model tends therefore to shrink estimated individual hospital ARAE rates toward the grand mean, which is the mean of ARAE rates across all hospitals in the study sample.16  The magnitude of shrinkage increases as the hospital volume of deliveries decreases. One limitation of shrinkage, which is also the source of controversies about the multilevel model–based adjustment method, is that low-volume hospitals can be credited with average performance. However, the level of confidence in the shrunk estimate for each hospital and the ability to compare this hospital with other hospitals can be assessed with the rankability of each hospital.31,32  A rankability greater than 0.7 is considered as good and is suggested as the threshold to include one hospital in league table.8,11  In the current study, 26 hospitals (18.1%) had rankability less than 0.7 and corresponded to low-volume hospitals. These hospitals should therefore not be compared with other hospitals or should be identified as nonrankable in league tables. However, all the hospitals in the study sample had a reliability greater than 0.5, which is sometimes used to define a “fair” rankability. This is very different from surgery where rankability is lower (i.e., less than 0.5) owing to a high number of very low case volume hospitals, raising concern about the validity of reporting and comparing surgical outcomes across hospitals.7,8,10,31,32  In addition to the reduction in the between-hospital variability, multilevel based–adjustment resulted in a significant reclassification in hospital outliers ranking. More specifically, it increased the proportion of bad performers, which may allow a more efficient targeting of hospitals that may benefit the most from further investigation.
Compared with logistic regression models, multilevel models have been suggested to improve the prediction for individual hospitals of the rate of adverse outcomes in a subsequent time period based on models developed using data from an earlier time period.12  Using different metrics to assess future prediction (median absolute difference, root median square error, and percentage of hospitals whose predicted ARAE rates are within 95% CIs of the observed rates), the improvement in prediction was reported in mortality after uncommon surgical procedures.13,14  The improvement was less for mortality after more common surgical procedures or for mortality in trauma patients. With metrics to assess performance identical to the ones used in the current study (adjusted odds ratio, proportion of between-hospital variance explained by hospital outlier status), no significant improvement in prediction of mortality among trauma patients was observed with multilevel models compared with logistic model.30  In the current study, the risk of experiencing an ARAE for a patient admitted during 2010–2011 in a high and average outlier status was 1.8 and 3.3 times the risk in a low outlier status. The estimated odds ratios associated with outlier status were similar between logistic regression and multilevel-based models. Our results are consistent with previous reports and suggest that multilevel modeling does not seem to improve future prediction of ARAE rates within hospitals compared with logistic modeling.13,14,30  However, we do not think the performance of predictive validity within the same individual hospitals over time is a diagnostic statistic directly relevant to the purpose of our study. In essence, hospital ranking on anesthesia safety is the comparison of performance across hospitals at a given time point (i.e., a cross-sectional comparison) rather than the forecast of future performance within the same hospitals. In addition, changes can be observed over time such as the number of hospitals included or individual hospital performance, making the multilevel developed on a previous time period no longer valid for a next time period. In other words, comparison of hospitals should probably be based on a regularly updated multilevel model that takes into consideration these possible changes.
The definitions and coding practice of adverse events at different hospitals may raise concerns about the use of indicators based on administrative data for routine surveillance. First, as previously reported in the literature, the definition of adverse events is complex and somewhat subjective compared with other clear-cut outcomes such as death.33  However, the marked decrease in anesthesia-related mortality and severe morbidity in obstetric anesthesia over the last 20 yr precludes the use of most severe outcomes to assess anesthesia safety at the hospital level.20,21  Second, adverse events may be recorded inconsistently across hospitals. The lack of association between the reporting index and the rate of ARAEs suggests that the pattern of coding may have little influence on the validity of our study results. These concerns should not be viewed as a limitation of this type of indicators based on administrative data but rather considered as an incentive to improve medical record documentation and accuracy of coding. The alternative options to administrative data are prospective registries such as the National Anesthesia Outcomes Registry or the Society for Obstetric Anesthesia and Perinatalogy Serious Complication Repository.26,34,35 . They may ensure a more homogeneous definition of ARAEs and a more consistent recording across hospitals. However, relying on these data systems usually poses a significant delay between data collection, analysis, risk identification, and development of interventions, making it difficult to implement timely safety improvement measures. Moreover, the rarity of adverse events in obstetric anesthesia with an incidence rate of 5/1,000 may preclude their comprehensive capture and sufficient statistical power if the volume of the gathered data is not large enough, as recently illustrated with the Society for Obstetric Anesthesia and Perinatology's Serious Complication Repository project.26  Finally, creating and maintaining quality registries requires a significant amount of financial resources, which may threaten the long-term viability of these specialty data systems.36  Assessing and monitoring anesthesia safety must consider the tradeoff between the perceived higher credibility of prospectively gathered clinical data and the low-cost and readily available administrative data. In that sense, the administrative-data approach and the prospective registry–data approach should be viewed as complementary means to the same end.
This study has several limitations. First, it was conducted in New York and included only 144 hospitals. The number of community hospitals in the United States is about 5,000, and the analysis performed on this limited sample may not be generalizable to all the hospitals in the United States. Second, obstetric patients are usually healthy with little comorbidity. The results may therefore not apply to different patient populations and anesthetic specialties, such as cardiac or vascular anesthesia where the weight of the case- and procedure-mixes is probably higher. Third, the definition of ARAEs was based on a combination of ICD-9-CM codes, and the ICD-10-CM is expected to be introduced soon. However, Li et al.3  demonstrated that ARAEs can also be identified with ICD-10-CM codes. Fourth, the AHRQ definitions of PSIs and the definition of ARAEs used in this study are very heterogeneous and nonspecific as they include complications of varying severity. Use of a more specific indicator, such as the one for ARAEs related to neuraxial anesthesia and local anesthetics, may allow identification of frequent and preventable adverse events that may benefit the most from safety measures.26 
In conclusion, the multilevel modeling approach allows us to assess the rankability of the study hospitals while providing similarly accurate estimate of the risk of obstetric ARAEs as the conventional risk-adjustment method. Therefore, the multilevel modeling approach could serve as a practical alternative to the risk-adjustment method in monitoring obstetric anesthesia safety across hospitals.
Acknowledgments
The authors thank Joanne Brady, Ph.D., and Barbara H. Lang, M.P.H., from the Department of Anesthesiology, Columbia University College of Physicians and Surgeons, New York, New York, for their invaluable data and administrative assistance.
Support was provided solely from institutional and/or departmental sources.
Competing Interests
The authors declare no competing interests.
*2008 Patient Safety Indicators (PSI) Composite Measure Workgroup Final Report, Agency for Healthcare Research and Quality. Available at: http://www.qualityindicators.ahrq.gov/Downloads/Modules/PSI/PSI_Composite_Development.pdf. Accessed October 20, 2014.
2008 Patient Safety Indicators (PSI) Composite Measure Workgroup Final Report, Agency for Healthcare Research and Quality. Available at: http://www.qualityindicators.ahrq.gov/Downloads/Modules/PSI/PSI_Composite_Development.pdf. Accessed October 20, 2014.×
2012 National Healthcare Quality Report, Agency for Healthcare Research and Quality. Available at: http://www.ahrq.gov/research/findings/nhqrdr/nhqr12/. Accessed October 20, 2014.
2012 National Healthcare Quality Report, Agency for Healthcare Research and Quality. Available at: http://www.ahrq.gov/research/findings/nhqrdr/nhqr12/. Accessed October 20, 2014.×
Health United States 2013, Centers for Disease Control and Prevention. Available at: http://www.cdc.gov/nchs/data/hus/hus13.pdf. Accessed October 20, 2014.
Health United States 2013, Centers for Disease Control and Prevention. Available at: http://www.cdc.gov/nchs/data/hus/hus13.pdf. Accessed October 20, 2014.×
References
McDonald, KM, Romano, PS, Geppert, J, Davies, SM, Duncan, BW, Shojania, KG, Hansen, A Measures of Patient Safety Based on Hospital Administrative Data—The Patient Safety Indicators. (2002). Rockville, MD Agency for Healthcare Research and Quality [PubMed] [PubMed]
Cheesman, K, Brady, JE, Flood, P, Li, G Epidemiology of anesthesia-related complications in labor and delivery, New York State, 2002-2005.. Anesth Analg. (2009). 109 1174–81 [Article] [PubMed]
Li, G, Warner, M, Lang, BH, Huang, L, Sun, LS Epidemiology of anesthesia-related mortality in the United States, 1999-2005.. Anesthesiology. (2009). 110 759–65 [Article] [PubMed]
Haller, G, Stoelwinder, J, Myles, PS, McNeil, J Quality and safety indicators in anesthesia: A systematic review.. Anesthesiology. (2009). 110 1158–75 [Article] [PubMed]
El Haj Ibrahim, S, Fridman, M, Korst, LM, Gregory, KD Anesthesia complications as a childbirth patient safety indicator.. Anesth Analg. (2014). 119 911–7 [Article] [PubMed]
Shahian, DM, Normand, SL Comparison of “risk-adjusted” hospital outcomes.. Circulation. (2008). 117 1955–63 [Article] [PubMed]
Dimick, JB, Ghaferi, AA, Osborne, NH, Ko, CY, Hall, BL Reliability adjustment for reporting hospital outcomes with surgery.. Ann Surg. (2012). 255 703–7 [Article] [PubMed]
Lawson, EH, Ko, CY, Adams, JL, Chow, WB, Hall, BL Reliability of evaluating hospital quality by colorectal surgical site infection type.. Ann Surg. (2013). 258 994–1000 [Article] [PubMed]
Osborne, NH, Ko, CY, Upchurch, GRJr, Dimick, JB The impact of adjusting for reliability on hospital quality rankings in vascular surgery.. J Vasc Surg. (2011). 53 1–5 [Article] [PubMed]
van Dishoeck, AM, Koek, MB, Steyerberg, EW, van Benthem, BH, Vos, MC, Lingsma, HF Use of surgical-site infection rates to rank hospital performance across several types of surgery.. Br J Surg. (2013). 100 628–36; discussion 637 [Article] [PubMed]
Adams, JL, Mehrotra, A, Thomas, JW, McGlynn, EA Physician cost profiling–Reliability and risk of misclassification.. N Engl J Med. (2010). 362 1014–21 [Article] [PubMed]
Dimick, JB, Staiger, DO, Birkmeyer, JD Ranking hospitals on surgical mortality: The importance of reliability adjustment.. Health Serv Res. (2010). 45 1614–29 [Article] [PubMed]
Clark, DE, Hannan, EL, Wu, C Predicting risk-adjusted mortality for trauma patients: Logistic versus multilevel logistic models.. J Am Coll Surg. (2010). 211 224–31 [Article] [PubMed]
Hannan, EL, Wu, C, DeLong, ER, Raudenbush, SW Predicting risk-adjusted mortality for CABG surgery: Logistic versus hierarchical logistic models.. Med Care. (2005). 43 726–35 [Article] [PubMed]
Glaser, D, Hastings, RH An introduction to multilevel modeling for anesthesiologists.. Anesth Analg. (2011). 113 877–87 [PubMed]
Cohen, ME, Ko, CY, Bilimoria, KY, Zhou, L, Huffman, K, Wang, X, Liu, Y, Kraemer, K, Meng, X, Merkow, R, Chow, W, Matel, B, Richards, K, Hart, AJ, Dimick, JB, Hall, BL Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: Patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus.. J Am Coll Surg. (2013). 217 336–46.e1 [Article] [PubMed]
Chadwick, HS, Posner, K, Caplan, RA, Ward, RJ, Cheney, FW A comparison of obstetric and nonobstetric anesthesia malpractice claims.. Anesthesiology. (1991). 74 242–9 [Article] [PubMed]
Davies, JM, Posner, KL, Lee, LA, Cheney, FW, Domino, KB Liability associated with obstetric anesthesia: A closed claims analysis.. Anesthesiology. (2009). 110 131–9 [Article] [PubMed]
Butwick, A What’s new in obstetric anesthesia in 2011? Reducing maternal adverse outcomes and improving obstetric anesthesia quality of care.. Anesth Analg. (2012). 115 1137–45 [Article] [PubMed]
Hawkins, JL, Chang, J, Palmer, SK, Gibbs, CP, Callaghan, WM Anesthesia-related maternal mortality in the United States: 1979-2002.. Obstet Gynecol. (2011). 117 69–74 [Article] [PubMed]
Callaghan, WM, Creanga, AA, Kuklina, EV Severe maternal morbidity among delivery and postpartum hospitalizations in the United States.. Obstet Gynecol. (2012). 120 1029–36 [PubMed]
Ducloy-Bouthors, AS, Prunet, C, Tourrès, J, Chassard, D, Benhamou, D, Blondel, B [Medical care organization in analgesia, anaesthesia and intensive care in maternity units: Results from the National Perinatal Surveys in 2003 and 2010].. Ann Fr Anesth Reanim. (2013). 32 18–24 [Article] [PubMed]
Martin, JA, Hamilton, BE, Ventura, SJ, Osterman, MJ, Mathews, TJ Births: Final data for 2011.. Natl Vital Stat Rep. (2013). 62 1–70
von Elm, E, Altman, DG, Egger, M, Pocock, SJ, Gøtzsche, PC, Vandenbroucke, JP STROBE Initiative, The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies.. Lancet. (2007). 370 1453–7 [Article] [PubMed]
Kuklina, EV, Whiteman, MK, Hillis, SD, Jamieson, DJ, Meikle, SF, Posner, SF, Marchbanks, PA An enhanced method for identifying obstetric deliveries: Implications for estimating maternal morbidity.. Matern Child Health J. (2008). 12 469–77 [Article] [PubMed]
D’Angelo, R, Smiley, RM, Riley, ET, Segal, S Serious complications related to obstetric anesthesia: The serious complication repository project of the Society for Obstetric Anesthesia and Perinatology.. Anesthesiology. (2014). 120 1505–12 [Article] [PubMed]
Iezzoni, LI, Daley, J, Heeren, T, Foley, SM, Hughes, JS, Fisher, ES, Duncan, CC, Coffman, GA Using administrative data to screen hospitals for high complication rates.. Inquiry. (1994). 31 40–55 [PubMed]
Kass, RE, Raftery, AE Bayes factors.. J Am Stat Assoc. (1995). 90 773–95 [Article]
Birkmeyer, JD, Dimick, JB, Staiger, DO Operative mortality and procedure volume as predictors of subsequent hospital performance.. Ann Surg. (2006). 243 411–7 [Article] [PubMed]
Glance, LG, Mukamel, DB, Osler, TM, Dick, AW Ranking trauma center quality: Can past performance predict future performance?. Ann Surg. (2014). 259 682–6 [Article] [PubMed]
van Dishoeck, AM, Lingsma, HF, Mackenbach, JP, Steyerberg, EW Random variation and rankability of hospitals using outcome indicators.. BMJ Qual Saf. (2011). 20 869–74 [Article] [PubMed]
van Dishoeck, AM, Looman, CW, van der Wilden-van Lier, EC, Mackenbach, JP, Steyerberg, EW Displaying random variation in comparing hospital performance.. BMJ Qual Saf. (2011). 20 651–7 [Article] [PubMed]
Silber, JH, Rosenbaum, PR, Schwartz, JS, Ross, RN, Williams, SV Evaluation of the complication rate as a measure of quality of care in coronary artery bypass graft surgery.. JAMA. (1995). 274 317–23 [Article] [PubMed]
Hannenberg, AA, Warner, MA The registry imperative.. Anesthesiology. (2009). 111 687–9 [Article] [PubMed]
Fleischut, PM, Mazumdar, M, Memtsoudis, SG Perioperative database research: Possibilities and pitfalls.. Br J Anaesth. (2013). 111 532–4 [Article] [PubMed]
Lagasse, RS National performance data registries: Preparing for the perfect storm.. Anesthesiology. (2012). 117 449–50 [Article] [PubMed]
International Classification of Diseases, Ninth Revision, Clinical Modification Codes for Identifying Labor and Delivery-related Discharges
International Classification of Diseases, Ninth Revision, Clinical Modification Codes for Identifying Anesthesia-related Adverse Events and Number (%) Recorded for the Years 2008–2009 in New York
International Classification of Diseases, Ninth Revision, Clinical Modification Codes to Define Patient- and Procedure-related Risk Factors for Anesthesia-related Adverse Events
Performance of the Logistic and Multilevel Models for Anesthesia-related Adverse Events, New York, 2008-2009
Univariate Analysis of Risk Factors for Adverse Events Related to Neuraxial Anesthesia and Local Anesthetics, New York, 2008-2009
Multivariate Analysis of Risk Factors for Adverse Events Related to Neuraxial Anesthesia and Local Anesthetics, New York, 2008-2009
Performance of the Logistic and Multilevel Models for Adverse Events Related to Neuraxial Anesthesia and Local Anesthetics, New York, 2008-2009
Fig. 1.
Selection of the study sample. NEOMAT = neonatal-maternal code; SID = State Inpatient Database.
Selection of the study sample. NEOMAT = neonatal-maternal code; SID = State Inpatient Database.
Fig. 1.
Selection of the study sample. NEOMAT = neonatal-maternal code; SID = State Inpatient Database.
×
Fig. 2.
(A) Relationship between the number of deliveries and the risk-adjusted rate of any type of anesthesia-related adverse events (ARAEs). The dashed horizontal lines represent the grand mean or the mean of ARAEs rates across the 144 hospitals in the study sample. The filled red points indicate hospitals with a rankability ≤0.7 or nonrankable hospitals. (B) Relationship between the number of deliveries and the multilevel model-based rate of any type of ARAEs. Adjustment with multilevel model tends to shrink estimated individual hospital ARAE rates toward the grand mean.
(A) Relationship between the number of deliveries and the risk-adjusted rate of any type of anesthesia-related adverse events (ARAEs). The dashed horizontal lines represent the grand mean or the mean of ARAEs rates across the 144 hospitals in the study sample. The filled red points indicate hospitals with a rankability ≤0.7 or nonrankable hospitals. (B) Relationship between the number of deliveries and the multilevel model-based rate of any type of ARAEs. Adjustment with multilevel model tends to shrink estimated individual hospital ARAE rates toward the grand mean.
Fig. 2.
(A) Relationship between the number of deliveries and the risk-adjusted rate of any type of anesthesia-related adverse events (ARAEs). The dashed horizontal lines represent the grand mean or the mean of ARAEs rates across the 144 hospitals in the study sample. The filled red points indicate hospitals with a rankability ≤0.7 or nonrankable hospitals. (B) Relationship between the number of deliveries and the multilevel model-based rate of any type of ARAEs. Adjustment with multilevel model tends to shrink estimated individual hospital ARAE rates toward the grand mean.
×
Fig. 3.
Relationship between the number of deliveries and the rankability for any type of anesthesia-related adverse events. The filled red points indicate hospitals with rankability ≤0.7. The mean volume of delivery for the 26 hospitals with rankability ≤0.7 is 254.
Relationship between the number of deliveries and the rankability for any type of anesthesia-related adverse events. The filled red points indicate hospitals with rankability ≤0.7. The mean volume of delivery for the 26 hospitals with rankability ≤0.7 is 254.
Fig. 3.
Relationship between the number of deliveries and the rankability for any type of anesthesia-related adverse events. The filled red points indicate hospitals with rankability ≤0.7. The mean volume of delivery for the 26 hospitals with rankability ≤0.7 is 254.
×
Table 1.
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009×
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Table 1.
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Univariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009×
×
Table 2.
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009×
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Table 2.
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009
Multivariate Analysis of Risk Factors for Anesthesia-related Adverse Events, New York, 2008-2009×
×
Table 3.
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment×
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment
Table 3.
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment
Hospital Reclassification Table Based on Outlier Status for Any Type of Anesthesia-related Adverse Events with Risk Adjustment and Multilevel Model–based Adjustment×
×
Table 4.
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009×
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009
Table 4.
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009
Adjusted Odds Ratio of ARAEs for the High- and Average-outlier Status Relative to the Low-outlier Status in 2010–2011 Based on Hospital Outliers Status in 2008–2009 and Proportion of Between-hospital Variance in ARAE Rates in 2010–2011 Explained by Hospital Outlier Status in 2008–2009×
×