Education  |   April 2018
Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes
Author Notes
  • From the Department of Anesthesiology, Perioperative and Pain Medicine, Boston Children’s Hospital and Harvard Medical School, Boston, Massachusetts (R.H.B.); the Department of Surgery, Emory University School of Medicine, Atlanta, Georgia (S.L.M.-W.); the Foundation for Advancement of International Medical Education and Research, Philadelphia, Pennsylvania (J.R.B.); the Center for Medical Simulation, Charlestown, Massachusetts (J.B.C., R.H.B.); and the Department of Anesthesia, Critical Care and Pain Medicine (J.B.C.), the Department of Surgery and Massachusetts General Hospital Learning Laboratory (E.R.P.), Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts.
  • Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
  • Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
  • Children’s Hospital Boston, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Center for Medical Simulation, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
  • Newton-Wellesley Hospital, Newton, Massachusetts
  • Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
  • Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
  • Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
  • Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
  • Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
  • Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Corresponding article on page 707.
    Corresponding article on page 707.×
  • Submitted for publication November 17, 2016. Accepted for publication October 26, 2017.
    Submitted for publication November 17, 2016. Accepted for publication October 26, 2017.×
  • *Members of the Harvard Assessment of Anesthesia Resident Performance Research Group are listed in appendix 1.
    Members of the Harvard Assessment of Anesthesia Resident Performance Research Group are listed in appendix 1.×
  • Address correspondence to Dr. Blum: Department of Anesthesiology, Pain and Perioperative Medicine, Boston Children’s Hospital, 300 Longwood Avenue, Boston, Massachusetts 02115. richard.blum@childrens.harvard.edu. Information on purchasing reprints may be found at www.anesthesiology.org or on the masthead page at the beginning of this issue. Anesthesiology’s articles are made freely accessible to all readers, for personal use only, 6 months from the cover date of the issue.
Article Information
Education / Education / CPD
Education   |   April 2018
Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes
Anesthesiology 4 2018, Vol.128, 821-831. doi:10.1097/ALN.0000000000002091
Anesthesiology 4 2018, Vol.128, 821-831. doi:10.1097/ALN.0000000000002091
Abstract

Background: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment.

Methods: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail.

Results: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room.

Conclusions: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.