Defining the Intrinsic Cardiac Risks of Operations to Improve Preoperative Cardiac Risk Assessments

Jason B. Liu; Yaoming Liu; Mark E. Cohen; Clifford Y. Ko; Bobbie J. Sweitzer

doi:10.1097/ALN.0000000000002024

Free

Perioperative Medicine | February 2018

Defining the Intrinsic Cardiac Risks of Operations to Improve Preoperative Cardiac Risk Assessments

Jason B. Liu, M.D., M.S.; Yaoming Liu, Ph.D.; Mark E. Cohen, Ph.D.; Clifford Y. Ko, M.D., M.S., M.S.H.S., F.A.C.S.; Bobbie J. Sweitzer, M.D., F.A.C.P.

Author Notes

From the American College of Surgeons, Chicago, Illinois (J.B.L., Y.L., M.E.C., C.Y.K.); the Department of Surgery, University of Chicago Medicine, Chicago, Illinois (J.B.L.); the Department of Surgery, University of California Los Angeles David Geffen School of Medicine, Veterans Administration Greater Los Angeles Healthcare System, Los Angeles, California (C.Y.K.); and the Department of Anesthesiology, Northwestern University, Chicago, Illinois (B.J.S.).
This article is featured in “This Month in Anesthesiology,” page 1A.
This article is featured in “This Month in Anesthesiology,” page 1A.×
Corresponding articles on pages 244 and 247.
Corresponding articles on pages 244 and 247.×
Supplemental Digital Content is available for this article. Direct URL citations appear in the printed text and are available in both the HTML and PDF versions of this article. Links to the digital files are provided in the HTML text of this article on the Journal’s Web site (www.anesthesiology.org).
Supplemental Digital Content is available for this article. Direct URL citations appear in the printed text and are available in both the HTML and PDF versions of this article. Links to the digital files are provided in the HTML text of this article on the Journal’s Web site (www.anesthesiology.org).×
Submitted for publication February 22, 2017. Accepted for publication July 19, 2017.
Submitted for publication February 22, 2017. Accepted for publication July 19, 2017.×
Address correspondence to Dr. Liu: American College of Surgeons, 633 N. St. Clair, 22nd Floor, Chicago, Illinois 60611. jliu@facs.org. This article may be accessed for personal use at no charge through the Journal Web site, www.anesthesiology.org.

Article Information

Anesthesiology 2 2018, Vol.128, 283-292. doi:10.1097/ALN.0000000000002024

Abstract

Background: Current preoperative cardiac risk stratification practices group operations into broad categories, which might inadequately consider the intrinsic cardiac risks of individual operations. We sought to define the intrinsic cardiac risks of individual operations and to demonstrate how grouping operations might lead to imprecise estimates of perioperative cardiac risk.

Methods: Elective operations (based on Common Procedural Terminology codes) performed from January 1, 2010 to December 31, 2015 at hospitals participating in the American College of Surgeons National Surgical Quality Improvement Program were studied. A composite measure of perioperative adverse cardiac events was defined as either cardiac arrest requiring cardiopulmonary resuscitation or acute myocardial infarction. Operations’ intrinsic cardiac risks were derived from mixed-effects models while controlling for patient mix. Resultant risks were sorted into low-, intermediate-, and high-risk categories, and the most commonly performed operations within each category were identified. Intrinsic operative risks were also examined using a representative grouping of operations to portray within-group variation.

Results: Sixty-six low, 30 intermediate, and 106 high intrinsic cardiac risk operations were identified. Excisional breast biopsy had the lowest intrinsic cardiac risk (overall rate, 0.01%; odds ratio, 0.11; 95% CI, 0.02 to 0.25) relative to the average, whereas aorto-bifemoral bypass grafting had the highest (overall rate, 4.1%; odds ratio, 6.61; 95% CI, 5.54 to 7.90). There was wide variation in the intrinsic cardiac risks of operations within the representative grouping (median odds ratio, 1.40; interquartile range, 0.88 to 2.17).

Conclusions: A continuum of intrinsic cardiac risk exists among operations. Grouping operations into broad categories inadequately accounts for the intrinsic cardiac risk of individual operations.

What We Already Know about This Topic

The intrinsic risk of cardiac adverse events after surgery has historically been attributed to broad categories of surgeries based upon anatomical region (e.g., intraperitoneal) or surgical service (e.g., plastic surgery)
Detailed procedure-specific risks, independent of underlying patient comorbidities, have not been robustly analyzed or reported

What This Article Tells Us That Is New

An analysis of 3 million surgeries in the American College of Surgeons National Surgical Quality Improvement Program registry demonstrated a broad range of procedure-specific cardiac adverse event risk for 200 commonly performed procedures
These data may advance our patient-specific risk/benefit analyses and medical decision-making

MORE than 26.8 million operations are performed annually in the United States.¹ Of these, approximately 1.5% of patients will die during the subsequent 30 days, most often due to perioperative cardiac complications.² Accurate preoperative risk stratification is paramount to mitigate the risk of perioperative cardiac complications.^3–5 It allows for appropriate preoperative medical optimization, timely cardiac-specific interventions, and guidance regarding perioperative management. Most important, it provides patients the opportunity to make a truly informed decision regarding surgical treatment.^6,7

Primary care physicians, internists, hospitalists, cardiologists, and anesthesiologists play an integral role in preoperative cardiac risk assessments.⁵ Currently, these assessments rely on clinical risk indices, such as the Revised Cardiac Risk Index (RCRI), and clinical practice guidelines to inform decision-making.^7–10 These strategies typically group procedures into broadly defined anatomical categories for the sake of simplicity and to facilitate ease of use. However, these broad categories can potentially underestimate the true risk contributed by any one operation. Additionally, risk assessments such as the RCRI consider patient factors more than the risk of the operation itself. This approach may overestimate risk and result in unnecessary consultations, unnecessary costs, delays in surgery, and even harm from further interventions.^5,11,12

Operations themselves carry risks for adverse outcomes beyond the influence of patient comorbidities for myriad reasons, such as amount of blood loss, fluid shifts, cytokine release from tissue injury, inflammation, and other acute pathophysiologic changes. Accordingly, surgeons and anesthesiologists recognize that different operations carry intrinsically different risks of complication, but this gestalt can be difficult to formally share with those performing preoperative cardiac risk assessments.¹³ For instance, the RCRI labels all intraperitoneal, intrathoracic, and suprainguinal vascular operations as high-risk.⁹ However, a laparoscopic cholecystectomy, gastric bypass for morbid obesity, and pancreatoduodenectomy (all intraperitoneal) appear to have different intrinsic risks for perioperative adverse cardiac events. Classifying the risk of operations based on anatomic location or clinical impression may inadequately inform risk assessment for an individual procedure. Lack of granularity may therefore result in misleading predictions of risk and affect clinical decision-making.

To improve upon current preoperative cardiac risk assessment strategies and to facilitate interdisciplinary communication, the objectives of this study were to define the intrinsic risks of operations for perioperative adverse cardiac events (PACEs), and to demonstrate how grouping operations into broad categories rather than considering operations individually might be insufficient for preoperative cardiac risk assessments.

Materials and Methods

Data Source and Study Population

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) is a prospective, peer-controlled, internally validated registry that quantifies 30-day risk-adjusted surgical outcomes to benchmark all voluntarily participating hospitals.^14–16 Currently, more than 700 hospitals in the United States and internationally participate. The strength of the program is its use of highly standardized and robust clinical data collected by dedicated, trained, and regularly audited surgical clinical reviewers.¹⁷ Specifics have been reported elsewhere.^15,18–20 Briefly, reviewers collect patient demographics, operative details, and postoperative outcomes from the clinical record, through communication with other involved physicians, or directly from patients up to 30 days from the index operation regardless of discharge status. Operations are identified by Common Procedural Terminology (CPT) codes. Certain operations are not included: ophthalmologic procedures, obstetrical procedures, primary trauma operations, and transplantations. While inpatient and outpatient operations are included, minor procedures in freestanding surgery centers or in offices are not included.

This study included elective, noncardiac operations from ACS NSQIP Essentials data between January 1, 2010, and December 31, 2015. CPT codes with less than 25 occurrences were then excluded to create the analytic data set.²¹ Submitted data comply with the Health Insurance Portability and Accountability Act and are subsequently deidentified for analyses. The Chesapeake Institutional Review Board (Columbia, Maryland) deemed this study exempt from oversight because it analyzed preexisting deidentified data.

Outcome

The primary outcome was any PACE, defined as either cardiac arrest requiring cardiopulmonary resuscitation or acute myocardial infarction (MI).^7,10 Cardiac arrest was defined as any cardiac rhythm requiring initiation of basic or advanced cardiac life support. Patients who experienced simultaneous loss of consciousness and implantable cardioverter defibrillation were included. Acute MI was defined as evidence of acute MI on electrocardiogram (i.e., ST segment elevation greater than 1 mm in two or more contiguous leads, new left bundle branch blow, new Q-wave in two or more contiguous leads) or new troponin elevation greater than three times the upper limit of the reference range with clinical correlation with myocardial ischemia. Furthermore, troponin elevation due to myocardial supply-and-demand mismatch is included. Specifically, MI types 1 to 3 are included.^22,23 Death was excluded as an endpoint because death in the ACS NSQIP database is all-cause and not cardiac-specific.

Covariates

Patient characteristics included in this study were identical to those included in the ACS NSQIP Surgical Risk Calculator^18,21 and represent those typically known at the time of preoperative cardiac risk assessment: age (less than 65, 65 to 74, 75 to 84, greater than or equal to 85 yr), sex (male/female), American Society of Anesthesiologists (ASA) physical status (I, II, III, IV, or V), body mass index (underweight, normal, overweight, class I, class II, class III), ascites (yes/no), bleeding disorder (yes/no), disseminated cancer (yes/no), diabetes (none, oral, insulin-dependent), dyspnea (none, with moderate exertion, at rest), functional status (independent, partially dependent, totally dependent), history of chronic obstructive pulmonary disease (yes/no), history of congestive heart failure (yes/no), hypertension requiring medication (yes/no), renal failure (yes/no), sepsis (no, systemic inflammatory response syndrome, sepsis/septic shock), weight loss (yes/no), smoking status (yes/no), and steroid use (yes/no).

Statistical Analyses

To obtain intrinsic risks for PACE for individual operations, a mixed-effects logistic model was constructed from the analytic data set using all patient characteristics as fixed effects and with the CPT code as a random intercept (i.e., random effect).²⁴ Akin to hospital profiling, this approach allows an empirical-Bayes shrinkage adjustment of the CPT effect from the risk-adjustment model to rank CPT codes relative to the overall average.¹⁵ Odds ratios (ORs) with 95% CIs were then generated for each CPT code (i.e., random effect); ORs greater than 1.0 represent higher than average risk for PACE, whereas values less than 1.0 represent lower than average risk for PACE.

Each CPT code’s OR was then ranked in ascending order and divided into thirds to form “low,” “intermediate,” and “high” intrinsic cardiac risk categories. Three categories were chosen to align with convention.^25,26 Because of the large number of CPT codes, we report those CPT codes that accounted for 80% of the operations, or the most commonly performed operations, in each risk category for practicality.

ORs for patient-level characteristics (i.e., fixed effects) were also generated. The stability of these ORs was assessed using a 50:50 split-sample approach, representing derivation and validation cohorts. Model parameters obtained from the derivation cohort were applied to the validation cohort, and the c-statistics and Brier scores were visually compared between cohorts to assess stability. The c-statistic and Brier score were also calculated for the overall analytic data set.

The stability of each CPT code’s OR derived from the overall analytic data set was assessed by calculating statistical reliability, which reflects sample size, event rate, and variance within each CPT code, and conceptually represents the “signal-to-noise” ratio.^27–30 Because hierarchical modeling “shrinks” the estimates of CPT codes with low sample sizes toward the overall average, operations might be inappropriately ranked. For example, “intermediate”-risk CPT codes may be misclassified as “intermediate” because of low sample sizes. To assess the effect of sample size, event rate, and variance within each CPT code, we calculated the statistical reliability for each CPT code’s OR following previously published methodologies.^27–31 Reliability values greater than 0.7 were considered “good” and those greater than 0.9 were considered “excellent.”^32,33

Because CPT code ORs are relative to the statistically estimated average operation and can be difficult to interpret, we estimated the risk for PACE of one hypothetical patient undergoing every reported operation. This hypothetical “base case” patient was a 67-year-old white female with hypertension, diabetes requiring oral therapy, and a body mass index of 32 (class I obesity), who is functionally independent, does not smoke, and is of ASA physical class 2. We calculated her estimated risk for PACE for every reported CPT code.

Finally, we selected thoracic surgery as a representative example of a broad category to demonstrate how variable intrinsic operative cardiac risk is within a single category. A box plot of CPT code ORs was constructed for all operations within the thoracic surgery specialty. ORs and 95% CIs for individual operations within the thoracic surgery specialty were also plotted. This visually depicted the variability of the intrinsic cardiac risks captured within the broad category example of thoracic surgery. All statistical analyses were two-sided, set at the 0.05 level for significance, and performed using SAS v9.4 (SAS Institute Inc., USA).

Results

Overall, there were 3,247,537 operations encompassing 1,880 unique CPT codes. The overall rate of PACE was 0.5% (n = 16,050). Characteristics of patients with PACE compared to those without can be found in table 1, along with their respective ORs on multivariable analysis. As expected, older age, male sex, higher ASA class, and presence of comorbidities had greater odds of PACE as compared to their respective reference categories. For the derivation cohort, the c-statistic was 0.876 and Brier score was 0.0048; for the validation cohort, the c-statistic was 0.866 and Brier score was 0.0048. The overall analytic data set had a c-statistic of 0.874 and Brier score of 0.005. Figure 1 ranks all 1,880 CPT codes on a caterpillar plot based upon each operation’s OR for PACE relative to the average CPT code.

Table 1.

Patient Characteristics by Perioperative Adverse Cardiac Event Status

Full Size | Slide (.ppt)

Fig. 1.

Caterpillar plot demonstrating all 1,880 operations ranked by intrinsic cardiac risk. Horizontal reference line represents odds ratio of 1.0. Values greater than 1.0 represent higher than average risk for perioperative adverse cardiac events, whereas values less than 1.0 represent lower than average risk for perioperative adverse cardiac events. Vertical reference lines represent cut points used to classify operations as low, intermediate, and high intrinsic cardiac risks. 95% CIs are omitted for clarity.

Full Size | Slide (.ppt)

There were 66 low-, 30 intermediate-, and 106 high-risk operations that accounted for the majority of the operations (80% of operations in each risk category) performed, and the descriptions of these are reported in ascending order of intrinsic cardiac risk in supplemental table 1, Supplemental Digital Content (http://links.lww.com/ALN/B593). Examples are shown in table 2. The median OR for these operations was 1.20 (interquartile range [IQR], 0.62 to 1.78). Excisional breast biopsy was the operation with the lowest intrinsic cardiac risk of PACE (OR, 0.11; 95% CI, 0.02 to 0.25) relative to the average, whereas aorto-bifemoral bypass grafting had the highest (OR, 6.61; 95% CI, 5.54 to 7.90). For a different perspective, if our hypothetical patient was to undergo an excisional breast biopsy, she would have an estimated 0.03% risk for PACE, as compared to her risk of 1.43% if she had aorto-bifemoral bypass grafting—a nearly 48-fold greater risk.

Table 2.

Selected Examples of Low, Intermediate, and High Intrinsic Cardiac Risk Operations

Full Size | Slide (.ppt)

Other examples of high-risk operations included cystectomy with ileal conduit (OR, 3.76; 95% CI, 3.02 to 4.68), pylorus-preserving pancreatoduodenectomy (OR, 4.70; 95% CI, 4.00 to 5.53), and Ivor-Lewis esophagectomy (OR, 4.62; 95% CI, 3.56 to 5.99), whereas examples of low-risk operations included laparoscopic appendectomy (OR, 0.45; 95% CI, 0.33 to 0.62), sleeve gastrectomy (OR, 0.57; 95% CI, 0.45 to 0.73), and cholecystectomy (OR, 0.62; 95% CI, 0.53 to 0.72). Notably, these examples are all intraperitoneal operations and vary in their intrinsic cardiac risk profiles. Furthermore, the surgical approach also changed the intrinsic cardiac risk of the operation. For example, if our hypothetical patient was to undergo a laparoscopic cholecystectomy, she would have a 0.14% risk for PACE, whereas if she had an open cholecystectomy, she would have a 0.34% risk—two and a half–fold greater.

Statistical reliability of the most commonly performed operations was excellent: overall median reliability was 0.96 (IQR, 0.88 to 0.99). The median reliability of the 66 low-risk operations was 1.00 (IQR, 0.99 to 1.00), the 30 intermediate-risk 0.96 (IQR, 0.94 to 0.99), and the 106 high-risk 0.89 (IQR, 0.81 to 0.95).

To illustrate how variable risk is within a broad category, figure 2A shows a box plot of the ORs for operations within the thoracic surgery specialty. The median OR was 1.40 (IQR, 0.88 to 2.17). For a more granular perspective, figure 2B demonstrates the variability in intrinsic cardiac risk for each of the CPT codes within the thoracic surgery specialty.

Fig. 2.

(A and B) Variability in intrinsic cardiac risk when using thoracic surgery as an example of a broad operation category. (A) shows the box plot depicting the variability in odds ratio (OR) point estimates. Diamond represents the mean OR (1.77). Median OR, 1.40 (interquartile range, 0.88 to 2.17). (B) shows the ORs and 95% CI for individual operations (i.e., Common Procedural Terminology [CPT] codes) within the thoracic surgery specialty. Note that the x-axis is on the log scale.

Full Size | Slide (.ppt)

Discussion

Using high-quality clinical data from more than 3.2 million patients, we empirically derived the intrinsic risk of individual operations for PACE. Broad groupings of operations, as has been conventionally done,^9,10 can be misleading because they do not consider the continuum of intrinsic cardiac risk that exists for each operation. Current practices of preoperative cardiac risk assessments might be improved by recognizing the intrinsic cardiac risk of each operation when performing preoperative cardiac risk assessments.

Accurately predicting patients’ perioperative cardiac risk is paramount to mitigating PACE.^3,6 Overestimation of risk may be as problematic as underestimation. Patients may forgo beneficial procedures; limited resources, such as intensive care, may be misappropriated; or providers may not refer patients for surgical care. Accurate estimation of risk allows patients to make informed decisions,¹³ and providers can appropriately plan care, such as deciding between an endovascular versus an open procedure, or between a free muscle and tissue transfer flap versus tissue expanders for breast reconstruction.

A significant contributor to a patient’s risk is the intrinsic risk the operation itself carries for PACE. However, defining this intrinsic risk has been mostly left to expert opinion.¹³ Operations have been traditionally grouped into categories based upon anatomy, physiology, or surgical specialty for simplicity and ease of use.^{9,10,26,34–36} For example, the 2014 American College of Cardiology/American Heart Association Guidelines provide “plastic surgery” as an example of low risk operations.⁷ However, surgeons and anesthesiologists recognize that a free flap requiring microvascular anastomoses with an operative duration potentially exceeding 12 h likely carries greater intrinsic cardiac risk as compared to the placement of breast tissue expanders for breast reconstruction, although both are “plastic surgery.” Indeed, if our hypothetical patient was to undergo a free flap with microvascular anastomosis, she would experience a 13-fold greater risk for PACE as compared to if she was to undergo placement of breast tissue expanders (0.52% vs. 0.04%, respectively).

Instead of grouping operations into anatomic or physiologic categories, our data-driven approach derives the intrinsic cardiac risk of each operation. For instance, Gupta et al.¹⁰ grouped foregut and hepatopancreatobiliary (FG/HPB) operations together. However, the FG/HPB category contains operations each with its own intrinsic cardiac risk, as we report. Without acknowledging this continuum of intrinsic cardiac risks, grouping all FG/HPB operations together implies that they all have the same risk (i.e., given equal weight in calculations). Using the thoracic surgery specialty as an example, the intrinsic cardiac risk of individual operations within the thoracic cavity varies (fig. 2). Without incorporating individual operations’ intrinsic cardiac risks into the preoperative cardiac risk stratification calculus, inaccurate assessments might be made.

Our study sorted individual operations based upon their intrinsic cardiac risks into three risk groups. However, a continuum of intrinsic cardiac risk exists based upon our framework (fig. 1) and simple classification into these three categories seem inappropriate. It is precisely because a continuum exists that we advocate for the adoption of sophisticated risk calculators that can accommodate such granularity of individual operations and their intrinsic cardiac risks. Computer-based risk calculators can handle this complex continuum of intrinsic cardiac risk, an improvement over relying on conventional broad groupings of operations.^37–41 Indeed, the 2014 ACC/AHA Guidelines have begun to recognize and cautiously encourage the use of calculators.⁷ For those that do not wish to use a calculator, our study provides a comprehensive listing of the most commonly performed operations.

For rarely performed operations, procedure-specific risk estimates are generally less accurate (i.e., stable), partly because of low sample sizes.^27–31 By using a mixed-effects model that employs empirical Bayes-like shrinkage, CPT codes with small samples are “shrunk” toward the population average. Reliability was calculated to estimate the confidence of our procedure-specific risk estimates for the CPT codes reported, which overall was “excellent.”^28–33 However, for uncommonly performed operations, procedure-specific risk derived this way might have decreased levels of reliability and should be carefully interpreted.^27–29,31 Computer-based calculators relying on procedure-specific cardiac risk might be subject to limitations in procedure sample size; however, issues of small CPT code sample size are likely rare in the current analysis of more than 3.2 million patients, representing the most robust and powered data available to address these issues. Furthermore, shrunken estimates might represent the best available estimate in the absence of better information, and certainly represent an improvement over currently utilized tools to assess cardiac risk.^9,10 Further research is needed to determine the procedure-specific sample size acceptable for reliable estimates of procedure-specific cardiac risk, and to identify CPT codes associated with similar but more common operations that could be informative when the primary CPT code is rare.

This study is not the first to suggest that consideration of an operation’s intrinsic risk could improve risk stratification strategies. Noordzij et al.⁴² utilized registry data from The Netherlands and studied postoperative all-cause mortality in 3.7 million patients. Although they grouped operations into 36 bins “according to generally accepted medical theory and practice,” they concluded, as we do, that predictions of postoperative outcomes can be improved if the intrinsic risks of the operations are given more consideration in the preoperative risk assessment calculus. Estimates of risk will always be inaccurate and misleading if the intrinsic risk of an operation for an outcome is not appropriately considered.

Operations may carry intrinsic cardiac risk, independent of patient characteristics, because PACE has different etiologies from nonoperative cardiac events. Myocardial infarction is more likely to occur because of supply-and-demand mismatches (type 2 MI) rather than thrombosis (type 1 MI). Type 1 MI causes less than 5% of postoperative troponin increase.⁴³ Many more patients will suffer myocardial injury after noncardiac surgery than overt infarction.⁴⁴ These episodes are secondary to demand ischemia, which can be caused by pain, anemia, fluid shifts, inflammation, and the stress responses commensurate with an operation’s intrinsic risk.^4,45 The inherent metabolic demands and levels of stress induced by surgery are likely explanations as to why certain operations or approaches (e.g., laparoscopic vs. open) to operations are associated with varying risk of PACE.⁴⁶

The results of this study must be interpreted considering several limitations. First, as a retrospective cohort study, unmeasured confounding is always probable. Second, our results could not be compared to de facto low-risk operations, such as cataract surgery,⁴⁷ since cataract surgeries are not available in the ACS NSQIP data. Third, the ACS NSQIP stopped collecting additional cardiac-specific risk factors other than the ones we noted in the Materials and Methods section because they did not improve the statistical models for quality improvement purposes.³⁵ The absence of cardiac-specific variables, such as history of MI or stroke, hyperlipidemia requiring statin therapy, or whether a patient underwent preoperative cardiac testing, however, should not be overlooked. Our estimated intrinsic cardiac risks might therefore be underestimations in certain populations with greater cardiac disease burden (e.g., patients undergoing vascular operations). Our results must also be interpreted considering secular trends in preoperative cardiac risk evaluations. Vascular surgical procedures were highly scrutinized before the 2014 ACC/AHA Guidelines, which might again underestimate our risk estimates. Many operations with high intrinsic cardiac risk were, however, noted in our results to be vascular procedures, and our results should be interpreted as relative to the statistically estimated average procedure. Fourth, our data were unable to discern cardiac-specific from all-cause mortality, and our results might represent underestimations. However, it should be noted that the RCRI does not include cardiac-specific death.⁹ Last, the focus of ACS NSQIP hospitals on quality may be inconsistently present in non–ACS NSQIP hospitals, and this might impact the generalizability of our findings.

Our approach may challenge providers who have relied on intuition or expert opinion based on experience of estimating risk by looking at both patient factors and the planned procedure. Disassociating patient factors from the operation can thus contradict clinical intuition. The risk categories shown in this study represent the intrinsic risk of an operation for PACE without considering patient factors. But it is precisely because of the complex interplay between the intrinsic cardiac risk of each operation and patient characteristics that we recommend the use of risk calculators that can efficiently determine the most accurate predictions of cardiac risk. Notwithstanding, clinicians should be prepared to interpret these predictions in a clinical context.

In conclusion, conventional methods for preoperative cardiac risk stratification may be inadequate because they do not consider the intrinsic cardiac risk each operation itself carries. The ever-increasing use of technology in health care will allow us to adopt the use of sophisticated risk calculators at the bedside and during a clinical encounter rather than relying on simplistic risk scores. Our comprehensive lists of operations demonstrating their individual risks for PACE can serve as a reference source for routine preoperative medical consultations. Cardiologists, primary care physicians, and advanced practice providers who provide preoperative medical consultations could easily reference the intrinsic cardiac risks of operations to deliver high-quality perioperative care to surgical patients and improve multidisciplinary communication with surgeons and anesthesiologists.

Acknowledgments

The authors would like to thank Karl Y. Bilimoria, M.D., M.S. (Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago, Illinois) for providing editorial guidance during manuscript preparation.

Research Support

No sources of financial support contributed to the design and conduct of the study; analysis and interpretation of the data; and preparation, review, or approval of the manuscript. Supported by a research fellowship from the University of Chicago Medicine, under the auspices of Jeffrey B. Matthews, M.D. (Chicago, Illinois; to Dr. Liu), and by the American College of Surgeons, Clinical Scholars in Residence Program (Chicago, Illinois; to Dr. Liu).

Competing Interests

The authors declare no competing interests.

References

Agency for Healthcare Research and Quality: Overview of the State Ambulatory Surgery and Services Databases (SASD). Healthcare Cost and Utilization Project (HCUP): 2014 Rockville, Maryland, Agency for Healthcare Research and Quality.

Semel, ME, Lipsitz, SR, Funk, LM, Bader, AM, Weiser, TG, Gawande, AA . Rates and patterns of death after surgery in the United States, 1996 and 2006. Surgery 2012; 151:171–82 [Article] [PubMed]

Devereaux, PJ, Sessler, DI . Cardiac complications in patients undergoing major noncardiac surgery. N Engl J Med 2015; 373:2258–69 [Article] [PubMed]

Devereaux, PJ, Goldman, L, Cook, DJ, Gilbert, K, Leslie, K, Guyatt, GH . Perioperative cardiac events in patients undergoing noncardiac surgery: A review of the magnitude of the problem, the pathophysiology of the events and methods to estimate and communicate risk. CMAJ 2005; 173:627–34 [Article] [PubMed]

Thilen, SR, Wijeysundera, DN, Treggiari, MM . Preoperative consultations. Anesthesiol Clin 2016; 34:17–33 [Article] [PubMed]

Biccard, BM, Rodseth, RN . What evidence is there for intraoperative predictors of perioperative cardiac outcomes? A systematic review. Perioper Med (Lond) 2013; 2:14 [Article] [PubMed]

Fleisher, LA, Fleischmann, KE, Auerbach, AD, Barnason, SA, Beckman, JA, Bozkurt, B, Davila-Roman, VG, Gerhard-Herman, MD, Holly, TA, Kane, GC, Marine, JE, Nelson, MT, Spencer, CC, Thompson, A, Ting, HH, Uretsky, BF, Wijeysundera, DN ; American College of Cardiology; American Heart Association: 2014 ACC/AHA guideline on perioperative cardiovascular evaluation and management of patients undergoing noncardiac surgery: A report of the American College of Cardiology/American Heart Association Task Force on practice guidelines. J Am Coll Cardiol 2014; 64:e77–137 [Article] [PubMed]

Mangano, DT, Goldman, L . Preoperative assessment of patients with known or suspected coronary disease. N Engl J Med 1995; 333:1750–6 [Article] [PubMed]

Lee, TH, Marcantonio, ER, Mangione, CM, Thomas, EJ, Polanczyk, CA, Cook, EF, Sugarbaker, DJ, Donaldson, MC, Poss, R, Ho, KK, Ludwig, LE, Pedan, A, Goldman, L . Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation 1999; 100:1043–9 [Article] [PubMed]

Gupta, PK, Gupta, H, Sundaram, A, Kaushik, M, Fang, X, Miller, WJ, Esterbrooks, DJ, Hunter, CB, Pipinos, II, Johanning, JM, Lynch, TG, Forse, RA, Mohiuddin, SM, Mooss, AN . Development and validation of a risk calculator for prediction of cardiac risk after surgery. Circulation 2011; 124:381–7 [Article] [PubMed]

Beattie, WS, Abdelnaem, E, Wijeysundera, DN, Buckley, DN . A meta-analytic comparison of preoperative stress echocardiography and nuclear scintigraphy imaging. Anesth Analg 2006; 102:8–16 [Article] [PubMed]

McFalls, EO, Ward, HB, Moritz, TE, Goldman, S, Krupski, WC, Littooy, F, Pierpont, G, Santilli, S, Rapp, J, Hattler, B, Shunk, K, Jaenicke, C, Thottapurathu, L, Ellis, N, Reda, DJ, Henderson, WG . Coronary-artery revascularization before elective major vascular surgery. N Engl J Med 2004; 351:2795–804 [Article] [PubMed]

Schwarze, ML, Barnato, AE, Rathouz, PJ, Zhao, Q, Neuman, HB, Winslow, ER, Kennedy, GD, Hu, YY, Dodgion, CM, Kwok, AC, Greenberg, CC . Development of a list of high-risk operations for patients 65 years and older. JAMA Surg 2015; 150:325–31 [Article] [PubMed]

Khuri, SF, Henderson, WG, Daley, J, Jonasson, O, Jones, RS, Campbell, DAJr, Fink, AS, Mentzer, RMJr, Neumayer, L, Hammermeister, K, Mosca, C, Healey, N ; Principal Investigators of the Patient Safety in Surgery Study: Successful implementation of the Department of Veterans Affairs’ National Surgical Quality Improvement Program in the private sector: The Patient Safety in Surgery study. Ann Surg 2008; 248:329–36 [Article] [PubMed]

Cohen, ME, Ko, CY, Bilimoria, KY, Zhou, L, Huffman, K, Wang, X, Liu, Y, Kraemer, K, Meng, X, Merkow, R, Chow, W, Matel, B, Richards, K, Hart, AJ, Dimick, JB, Hall, BL . Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: Patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J Am Coll Surg 2013; 217:336–46.e1 [Article] [PubMed]

Khuri, SF, Daley, J, Henderson, W, Hur, K, Demakis, J, Aust, JB, Chong, V, Fabri, PJ, Gibbs, JO, Grover, F, Hammermeister, K, Irvin, GIII, McDonald, G, Passaro, EJr, Phillips, L, Scamman, F, Spencer, J, Stremple, JF . The Department of Veterans Affairs’ NSQIP: The first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998; 228:491–507 [Article] [PubMed]

Shiloach, M, Frencher, SKJr, Steeger, JE, Rowell, KS, Bartzokis, K, Tomeh, MG, Richards, KE, Ko, CY, Hall, BL . Toward robust information: Data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg 2010; 210:6–16 [Article] [PubMed]

Liu, Y, Cohen, ME, Hall, BL, Ko, CY, Bilimoria, KY . Evaluation and enhancement of calibration in the American College of Surgeons NSQIP Surgical Risk Calculator. J Am Coll Surg 2016; 223:231–9 [Article] [PubMed]

Ko, CY, Hall, BL, Hart, AJ, Cohen, ME, Hoyt, DB . The American College of Surgeons National Surgical Quality Improvement Program: Achieving better and safer surgery. Jt Comm J Qual Patient Saf 2015; 41:199–204 [Article] [PubMed]

Maggard-Gibbons, M . The use of report cards and outcome measurements to improve the safety of surgical care: The American College of Surgeons National Surgical Quality Improvement Program. BMJ Qual Saf 2014; 23:589–99 [Article] [PubMed]

Bilimoria, KY, Liu, Y, Paruch, JL, Zhou, L, Kmiecik, TE, Ko, CY, Cohen, ME . Development and evaluation of the universal ACS NSQIP surgical risk calculator: A decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 2013; 217:833–42.e13 [Article] [PubMed]

Thygesen, K, Alpert, JS, White, HD, Jaffe, AS, Apple, FS, Galvani, M, Katus, HA, Newby, LK, Ravkilde, J, Chaitman, B, Clemmensen, PM, Dellborg, M, Hod, H, Porela, P, Underwood, R, Bax, JJ, Beller, GA, Bonow, R, Van der Wall, EE, Bassand, JP, Wijns, W, Ferguson, TB, Steg, PG, Uretsky, BF, Williams, DO, Armstrong, PW, Antman, EM, Fox, KA, Hamm, CW, Ohman, EM, Simoons, ML, Poole-Wilson, PA, Gurfinkel, EP, Lopez-Sendon, JL, Pais, P, Mendis, S, Zhu, JR, Wallentin, LC, Fernández-Avilés, F, Fox, KM, Parkhomenko, AN, Priori, SG, Tendera, M, Voipio-Pulkki, LM, Vahanian, A, Camm, AJ, De Caterina, R, Dean, V, Dickstein, K, Filippatos, G, Funck-Brentano, C, Hellemans, I, Kristensen, SD, McGregor, K, Sechtem, U, Silber, S, Tendera, M, Widimsky, P, Zamorano, JL, Morais, J, Brener, S, Harrington, R, Morrow, D, Lim, M, Martinez-Rios, MA, Steinhubl, S, Levine, GN, Gibler, WB, Goff, D, Tubaro, M, Dudek, D, Al-Attar, N ; Joint ESC/ACCF/AHA/WHF Task Force for the Redefinition of Myocardial Infarction: Universal definition of myocardial infarction. Circulation 2007; 116:2634–53 [Article] [PubMed]

Thygesen, K, Alpert, JS, Jaffe, AS, Simoons, ML, Chaitman, BR, White, HD, Thygesen, K, Alpert, JS, White, HD, Jaffe, AS, Katus, HA, Apple, FS, Lindahl, B, Morrow, DA, Chaitman, BR, Clemmensen, PM, Johanson, P, Hod, H, Underwood, R, Bax, JJ, Bonow, JJ, Pinto, F, Gibbons, RJ, Fox, KA, Atar, D, Newby, LK, Galvani, M, Hamm, CW, Uretsky, BF, Steg, PG, Wijns, W, Bassand, JP, Menasche, P, Ravkilde, J, Ohman, EM, Antman, EM, Wallentin, LC, Armstrong, PW, Simoons, ML, Januzzi, JL, Nieminen, MS, Gheorghiade, M, Filippatos, G, Luepker, RV, Fortmann, SP, Rosamond, WD, Levy, D, Wood, D, Smith, SC, Hu, D, Lopez-Sendon, JL, Robertson, RM, Weaver, D, Tendera, M, Bove, AA, Parkhomenko, AN, Vasilieva, EJ, Mendis, S, Bax, JJ, Baumgartner, H, Ceconi, C, Dean, V, Deaton, C, Fagard, R, Funck-Brentano, C, Hasdai, D, Hoes, A, Kirchhof, P, Knuuti, J, Kolh, P, McDonagh, T, Moulin, C, Popescu, BA, Reiner, Z, Sechtem, U, Sirnes, PA, Tendera, M, Torbicki, A, Vahanian, A, Windecker, S, Morais, J, Aguiar, C, Almahmeed, W, Arnar, DO, Barili, F, Bloch, KD, Bolger, AF, Botker, HE, Bozkurt, B, Bugiardini, R, Cannon, C, de Lemos, J, Eberli, FR, Escobar, E, Hlatky, M, James, S, Kern, KB, Moliterno, DJ, Mueller, C, Neskovic, AN, Pieske, BM, Schulman, SP, Storey, RF, Taubert, KA, Vranckx, P, Wagner, DR ; Joint ESC/ACCF/AHA/WHF Task Force for Universal Definition of Myocardial Infarction; Authors/Task Force Members Chairpersons; Biomarker Subcommittee; ECG Subcommittee; Imaging Subcommittee; Classification Subcommittee; Intervention Subcommittee; Trials and Registries Subcommittee; ESC Committee for Practice Guidelines (CPG); Document Reviewers: Third universal definition of myocardial infarction. J Am Coll Cardiol 2012; 60:1581–98 [Article] [PubMed]

Vittinghoff, E . Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models, 20122nd edition. New York, Springer.

Boersma, E, Kertai, MD, Schouten, O, Bax, JJ, Noordzij, P, Steyerberg, EW, Schinkel, AF, van Santen, M, Simoons, ML, Thomson, IR, Klein, J, van Urk, H, Poldermans, D . Perioperative cardiovascular mortality in noncardiac surgery: Validation of the Lee cardiac risk index. Am J Med 2005; 118:1134–41 [Article] [PubMed]

Jørgensen, ME, Torp-Pedersen, C, Gislason, GH, Jensen, PF, Berger, SM, Christiansen, CB, Overgaard, C, Schmiegelow, MD, Andersson, C . Time elapsed after ischemic stroke and risk of adverse cardiovascular events and mortality following elective noncardiac surgery. JAMA 2014; 312:269–77 [Article] [PubMed]

Adams, JL, Mehrotra, A, Thomas, JW, McGlynn, EA . Physician cost profiling–reliability and risk of misclassification. N Engl J Med 2010; 362:1014–21 [Article] [PubMed]

Hall, BL, Huffman, KM, Hamilton, BH, Paruch, JL, Zhou, L, Richards, KE, Cohen, ME, Ko, CY . Profiling individual surgeon performance using information from a high-quality clinical registry: Opportunities and limitations. J Am Coll Surg 2015; 221:901–13 [Article] [PubMed]

Hofer, TP, Hayward, RA, Greenfield, S, Wagner, EH, Kaplan, SH, Manning, WG . The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease. JAMA 1999; 281:2098–105 [Article] [PubMed]

Huffman, KM, Cohen, ME, Ko, CY, Hall, BL . A comprehensive evaluation of statistical reliability in ACS NSQIP profiling models. Ann Surg 2015; 261:1108–13 [Article] [PubMed]

Liu, JB, Huffman, KM, Palis, BE, Shulman, LN, Winchester, DP, Ko, CY, Hall, BL . Reliability of the American College of Surgeons Commission on Cancer’s Quality of Care Measures for Hospital and Surgeon Profiling. J Am Coll Surg 2017; 224: 180–90.e8 [Article] [PubMed]

Wakeam, E, Hyder, JA . Reliability of Reliability Adjustment for Quality Improvement and Value-Based Payment. Anesthesiology 2016; 124:16–8 [Article] [PubMed]

Guglielminotti, J, Li, G . Monitoring obstetric anesthesia safety across hospitals through multilevel modeling. Anesthesiology 2015; 122:1268–79 [Article] [PubMed]

Sutton, R, Bann, S, Brooks, M, Sarin, S . The Surgical Risk Scale as an improved tool for risk-adjusted analysis in comparative surgical audit. Br J Surg 2002; 89:763–8 [Article] [PubMed]

Davenport, DL, Ferraris, VA, Hosokawa, P, Henderson, WG, Khuri, SF, Mentzer, RMJr . Multivariable predictors of postoperative cardiac adverse events after general and vascular surgery: Results from the Patient Safety in Surgery study. J Am Coll Surg 2007; 204:1199–210 [Article] [PubMed]

Kumar, R, McKinney, WP, Raj, G, Heudebert, GR, Heller, HJ, Koetting, M, McIntire, DD . Adverse cardiac events after surgery: Assessing risk in a veteran population. J Gen Intern Med 2001; 16:507–18 [Article] [PubMed]

Rudin, RS, Schneider, EC, Predmore, Z, Gidengil, CA . Knowledge gaps inhibit health IT development for coordinating complex patients’ care. Am J Manag Care 2016; 22:e317–22 [PubMed]

Goldzweig, CL, Towfigh, A, Maglione, M, Shekelle, PG . Costs and benefits of health information technology: New trends from the literature. Health Aff (Millwood) 2009; 28:w282–93 [Article] [PubMed]

Main, C, Moxham, T, Wyatt, JC, Kay, J, Anderson, R, Stein, K . Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: Systematic reviews of the effects and cost-effectiveness of systems. Health Technol Assess 2010; 14:1–227 [PubMed]

Dziadzko, MA, Gajic, O, Pickering, BW, Herasevich, V . Clinical calculators in hospital medicine: Availability, classification, and needs. Comput Methods Programs Biomed 2016; 133:1–6 [Article] [PubMed]

Sacks, GD, Dawes, AJ, Ettner, SL, Brook, RH, Fox, CR, Russell, MM, Ko, CY, Maggard-Gibbons, M . Impact of a risk calculator on risk perception and surgical decision making: A randomized trial. Ann Surg 2016; 264:889–95 [Article] [PubMed]

Noordzij, PG, Poldermans, D, Schouten, O, Bax, JJ, Schreiner, FA, Boersma, E . Postoperative mortality in The Netherlands: A population-based analysis of surgery-specific risk in adults. Anesthesiology 2010; 112:1105–15 [Article] [PubMed]

van Waes, JA, Nathoe, HM, de Graaff, JC, Kemperman, H, de Borst, GJ, Peelen, LM, van Klei, WA ; Cardiac Health After Surgery (CHASE) Investigators: Myocardial injury after noncardiac surgery and its association with short-term mortality. Circulation 2013; 127:2264–71 [Article] [PubMed]

Botto, F, Alonso-Coello, P, Chan, MT, Villar, JC, Xavier, D, Srinathan, S, Guyatt, G, Cruz, P, Graham, M, Wang, CY, Berwanger, O, Pearse, RM, Biccard, BM, Abraham, V, Malaga, G, Hillis, GS, Rodseth, RN, Cook, D, Polanczyk, CA, Szczeklik, W, Sessler, DI, Sheth, T, Ackland, GL, Leuwer, M, Garg, AX, Lemanach, Y, Pettit, S, Heels-Ansdell, D, Luratibuse, G, Walsh, M, Sapsford, R, Schünemann, HJ, Kurz, A, Thomas, S, Mrkobrada, M, Thabane, L, Gerstein, H, Paniagua, P, Nagele, P, Raina, P, Yusuf, S, Devereaux, PJ, Devereaux, PJ, Sessler, DI, Walsh, M, Guyatt, G, McQueen, MJ, Bhandari, M, Cook, D, Bosch, J, Buckley, N, Yusuf, S, Chow, CK, Hillis, GS, Halliwell, R, Li, S, Lee, VW, Mooney, J, Polanczyk, CA, Furtado, MV, Berwanger, O, Suzumura, E, Santucci, E, Leite, K, Santo, JA, Jardim, CA, Cavalcanti, AB, Guimaraes, HP, Jacka, MJ, Graham, M, McAlister, F, McMurtry, S, Townsend, D, Pannu, N, Bagshaw, S, Bessissow, A, Bhandari, M, Duceppe, E, Eikelboom, J, Ganame, J, Hankinson, J, Hill, S, Jolly, S, Lamy, A, Ling, E, Magloire, P, Pare, G, Reddy, D, Szalay, D, Tittley, J, Weitz, J, Whitlock, R, Darvish-Kazim, S, Debeer, J, Kavsak, P, Kearon, C, Mizera, R, O’Donnell, M, McQueen, M, Pinthus, J, Ribas, S, Simunovic, M, Tandon, V, Vanhelder, T, Winemaker, M, Gerstein, H, McDonald, S, O’Bryne, P, Patel, A, Paul, J, Punthakee, Z, Raymer, K, Salehian, O, Spencer, F, Walter, S, Worster, A, Adili, A, Clase, C, Cook, D, Crowther, M, Douketis, J, Gangji, A, Jackson, P, Lim, W, Lovrics, P, Mazzadi, S, Orovan, W, Rudkowski, J, Soth, M, Tiboni, M, Acedillo, R, Garg, A, Hildebrand, A, Lam, N, Macneil, D, Mrkobrada, M, Roshanov, PS, Srinathan, SK, Ramsey, C, John, PS, Thorlacius, L, Siddiqui, FS, Grocott, HP, McKay, A, Lee, TW, Amadeo, R, Funk, D, McDonald, H, Zacharias, J, Villar, JC, Cortés, OL, Chaparro, MS, Vásquez, S, Castañeda, A, Ferreira, S, Coriat, P, Monneret, D, Goarin, JP, Esteve, CI, Royer, C, Daas, G, Chan, MT, Choi, GY, Gin, T, Lit, LC, Xavier, D, Sigamani, A, Faruqui, A, Dhanpal, R, Almeida, S, Cherian, J, Furruqh, S, Abraham, V, Afzal, L, George, P, Mala, S, Schünemann, H, Muti, P, Vizza, E, Wang, CY, Ong, GS, Mansor, M, Tan, AS, Shariffuddin, II, Vasanthan, V, Hashim, NH, Undok, AW, Ki, U, Lai, HY, Ahmad, WA, Razack, AH, Malaga, G, Valderrama-Victoria, V, Loza-Herrera, JD, De Los Angeles Lazo, M, Rotta-Rotta, A, Szczeklik, W, Sokolowska, B, Musial, J, Gorka, J, Iwaszczuk, P, Kozka, M, Chwala, M, Raczek, M, Mrowiecki, T, Kaczmarek, B, Biccard, B, Cassimjee, H, Gopalan, D, Kisten, T, Mugabi, A, Naidoo, P, Naidoo, R, Rodseth, R, Skinner, D, Torborg, A, Paniagua, P, Urrutia, G, Maestre, ML, Santaló, M, Gonzalez, R, Font, A, Martínez, C, Pelaez, X, De Antonio, M, Villamor, JM, García, JA, Ferré, MJ, Popova, E, Alonso-Coello, P, Garutti, I, Cruz, P, Fernández, C, Palencia, M, Díaz, S, Del Castillo, T, Varela, A, de Miguel, A, Muñoz, M, Piñeiro, P, Cusati, G, Del Barrio, M, Membrillo, MJ, Orozco, D, Reyes, F, Sapsford, RJ, Barth, J, Scott, J, Hall, A, Howell, S, Lobley, M, Woods, J, Howard, S, Fletcher, J, Dewhirst, N, Williams, C, Rushton, A, Welters, I, Leuwer, M, Pearse, R, Ackland, G, Khan, A, Niebrzegowska, E, Benton, S, Wragg, A, Archbold, A, Smith, A, McAlees, E, Ramballi, C, Macdonald, N, Januszewska, M, Stephens, R, Reyes, A, Paredes, LG, Sultan, P, Cain, D, Whittle, J, Del Arroyo, AG, Sessler, DI, Kurz, A, Sun, Z, Finnegan, PS, Egan, C, Honar, H, Shahinyan, A, Panjasawatwong, K, Fu, AY, Wang, S, Reineks, E, Nagele, P, Blood, J, Kalin, M, Gibson, D, Wildes, T ; Vascular events In noncardiac Surgery patIents cOhort evaluatioN (VISION) Writing Group, on behalf of The Vascular events In noncardiac Surgery patIents cOhort evaluatioN (VISION) Investigators: Myocardial injury after noncardiac surgery: A large, international, prospective cohort study establishing diagnostic criteria, characteristics, predictors, and 30-day outcomes. Anesthesiology 2014; 120:564–78 [Article] [PubMed]

Kamel, H, Johnston, SC, Kirkham, JC, Turner, CG, Kizer, JR, Devereux, RB, Iadecola, C . Association between major perioperative hemorrhage and stroke or Q-wave myocardial infarction. Circulation 2012; 126:207–12 [Article] [PubMed]

Desborough, JP . The stress response to trauma and surgery. Br J Anaesth 2000; 85:109–17 [Article] [PubMed]

Chen, CL, Lin, GA, Bardach, NS, Clay, TH, Boscardin, WJ, Gelb, AW, Maze, M, Gropper, MA, Dudley, RA . Preoperative medical testing in Medicare patients undergoing cataract surgery. N Engl J Med 2015; 372:1530–8 [Article] [PubMed]

Fig. 1.