Free
Education  |   September 2014
This Is Not a Test!: Misconceptions Surrounding the Maintenance of Certification in Anesthesiology Simulation Course
Author Affiliations & Notes
  • Matthew B. Weinger, M.D.
    From the Vanderbilt University School of Medicine and VA Tennessee Valley Healthcare Systems, Nashville, Tennessee (M.B.W.); Cooper Medical School of Rowan University, Camden, New Jersey (A.R.B.); University of California, Los Angeles, Los Angeles, California (R.H.S.); and VA Palo Alto and Stanford University, Stanford, California (D.M.G.).
  • This article is featured in “This Month in Anesthesiology,” page 3A.
    This article is featured in “This Month in Anesthesiology,” page 3A.×
  • Submitted for publication January 30, 2014. Accepted for publication April 9, 2014.
    Submitted for publication January 30, 2014. Accepted for publication April 9, 2014.×
  • Address correspondence to Dr. Weinger: Center for Research and Innovation in Systems Safety, 1211 21st Avenue South, Medical Arts Building, Suite 732, Nashville, Tennessee 37212. matt.weinger@vanderbilt.edu. Information on purchasing reprints may be found at www.anesthesiology.org or on the masthead page at the beginning of this issue. Anesthesiology’s articles are made freely accessible to all readers, for personal use only, 6 months from the cover date of the issue.
Article Information
Education / Clinical Concepts and Commentary / Patient Safety
Education   |   September 2014
This Is Not a Test!: Misconceptions Surrounding the Maintenance of Certification in Anesthesiology Simulation Course
Anesthesiology 09 2014, Vol.121, 655-659. doi:10.1097/ALN.0000000000000303
Anesthesiology 09 2014, Vol.121, 655-659. doi:10.1097/ALN.0000000000000303

Simulation-based training courses for maintenance of certification in anesthesiology have been very well received and have led to widespread reports of meaningful practice improvement.

THE Maintenance of Certification in Anesthesiology (MOCA®) Simulation Course*01  is an important element of Part IV (Practice Performance Assessment and Improvement) of the American Board of Anesthesiology’s MOCA program.†02  These Courses are offered at endorsed programs that form the American Society of Anesthesiologists’ (ASA) Simulation Education Network. Although the MOCA Simulation Course has been described previously,1  discussions with ASA members suggest that misunderstandings remain about several aspects of the MOCA Simulation Course (the “Course”) and its place in the overall MOCA program. We would like to clarify the nature and conduct of the MOCA Simulation Courses vis-à-vis the goal of honing the skills of board-certified anesthesiologists (“Anesthesiologists”).
Although the American Board of Anesthesiology sets the requirement for the MOCA program, the ASA’s Simulation Editorial Board (SEB) is responsible for overseeing the content and conduct of the simulation experiences. The SEB has established core Course requirements but provides latitude to the endorsed centers to “do what they do best” and to determine their own course scheduling and fees. The Course is an interactive experience designed to stimulate participants to create and subsequently engage in meaningful practice improvement activities. It is a 6- to 8-h immersive learning experience, held in an ASA-endorsed simulation center, that focuses on the management of challenging clinical events. The Course must address both the medical and technical skills of managing acute perioperative situations as well as the nontechnical skills of dynamic decision making and team management. A Course goal is to help participants identify possible system issues and approaches to improve patient care in their individual practices. Every participant is the primary anesthesiologist in at least one simulation scenario. During a scenario, they work with other participants as well as with role-playing instructors or staff as a clinical management team. Each scenario is followed by a detailed instructor-facilitated debriefing where participants reflect on what transpired and articulate lessons to improve their own practices. To achieve endorsement, among other criteria, a center must describe its various Course policies (e.g., confidentiality and cancelation) and provide evidence that its instructors can conduct simulations and debriefings of experienced clinicians with skill and sensitivity.
We emphasize that the MOCA Simulation Course is NOT A TEST. There are no individual or team scores or performance evaluations. Debriefing discussions address practice improvement, focusing on what lessons can be drawn from the scenario, and how they can be applied to actual patient care. The Course provides an opportunity for each participant to reflect on their own performance, and that of their peers, with constructive feedback from the instructors and other course participants.
The MOCA Simulation Course culminates in the creation of practice improvement plans by participants, to be implemented in the following 90 days. A goal of the American Board of Anesthesiology is for the learning that takes place during the Course to translate into subsequent practice improvement and behavior change. During the course, the instructors facilitate discussions on potential practice improvement activities stemming from the scenario experiences (e.g., “I am going to get my operating room team to practice managing an airway fire”).
Completion of the Course and the subsequent practice improvement activities qualify the participant for MOCA Part IV credit. Course participation may also qualify for Continuing Medical Education (CME) credit although this is site dependent. Currently, the Course is required once in each 10-yr MOCA cycle.
Conducting robust Courses for experienced anesthesiologists, with appropriate simulations and debriefings, is a complex process requiring extensive effort by faculty instructors, simulation specialists, pedagogy experts, and administrative personnel. Significant preparations are required before and during each Course.
Is MOCA Simulation Useful?
The MOCA Simulation Courses began in 2010. The SEB monitors Courses offered at endorsed sites to assure their value and relevance to ASA members. In addition to analyzing participants’ deidentified course evaluations and practice improvement plans, the SEB sometimes sends observers to Courses.
Experiences from the first 583 MOCA participants were evaluated in 2011 by the SEB yielding a peer-reviewed publication.1  The evaluations, practice improvement plans, and testimonials from MOCA Course participants have been overwhelmingly positive and represent powerful demonstrations of meaningful practice improvement at participants’ home institutions. As of early 2014, more than 2,700 anesthesiologists have completed a Course. More than 97% rated it as “relevant to my practice.” Ninety-six percent reported that the “course was a positive learning experience.” Almost all Course participants reported that, “what I learned will change my practice,” and that they would recommend the Course to their colleagues. Importantly, 94% of Course participants reported that they changed something in their practice. Few other CME activities lead to such robust practice reflection and change.2,3  A preliminary qualitative analysis of submitted deidentified practice improvement plans and accomplishments suggests deliberate efforts of many Course participants to conduct activities with real potential to impact the entire perioperative team.4 
Is MOCA Simulation Necessary for Anesthesiologists?
Our specialty was the pioneer in patient safety. Although we have made great strides during the past 3 decades, we must admit that patients are still harmed, even in seemingly simple, “routine” cases. Ample data show that approximately 30% of anesthetics contain “nonroutine events”5  and that a subset of these events represent significant physiological disturbances that, if not treated adequately, could lead to patient harm.6  Despite many advances in our care, the low but meaningful incidence of perioperative physiological disturbances7  has not changed appreciably since the 1980s.8  Although serious adverse events are uncommon, the ASA’s Closed Claims Project, the Anesthesia Quality Institute’s Anesthesia Incident Reporting System, and individual experiences include many examples of what should have been the “routine” anesthetics on healthy people that end tragically. The Anesthesia Patient Safety Foundation was formed in 1985 with the mission that “no patient shall be harmed from anesthesia,” a mission that clearly remains incompletely fulfilled.
For anesthesia, as for all human endeavors that strive to be “high reliability,” simulation training to hone skills and strategies is an important part of maximizing patient safety. Along with pioneering patient safety, anesthesiologists introduced simulation into medicine to address both technical and nontechnical aspects of decision making and patient care. This work led directly to the incorporation of various types of simulation into virtually all healthcare domains.
But Are Not Anesthesiologists Already Good Enough to Manage All Challenging Events?
The literature shows that even highly experienced anesthesiologists may still fail to optimally respond to challenging clinical situations.9,10  Murray et al.11  found that Anesthesiologists performed less than 20% of the indicated key actions during hyperkalemia and malignant hyperthermia scenarios. Currently, a team of 18 anesthesiology simulation experts have created four standardized high-fidelity simulation scenarios and delivered them to 300 anesthesiologists who volunteered to participate in a research study (funded by the Agency for Healthcare Research and Quality) grafted onto their MOCA Simulation Courses. State-of-the-art performance assessment metrics are being applied to evaluate performance of the volunteer individuals and teams (whose identity is kept confidential via code numbers). Domain experts have begun to review the video recordings and a germane preliminary finding is that indeed, as per previous research on smaller cohorts, the performance of anesthesiologists is quite variable. Although the vast majority of videos reveal skilled clinical practice, we have observed many simulation performances demonstrating room for improvement. Such suboptimal performances include both technical and nontechnical deficits and are seen in all four of the standardized scenarios. Only a small number of videos show exemplary performance proving that, as expected, no one is perfect. Similar patterns of performance during critical event management have been found for other highly skilled personnel in other arenas of high intrinsic risk, such as airline pilots and nuclear power plant operators.12,13  In anesthesiology, as in these other areas, when human life is at risk, exemplary performance is the goal and middling performance may not be good enough.
Could these findings just be a “simulation artifact,” that performance in the simulator does not reflect performance in the “real world”? Even though high-fidelity simulation has high face validity, it is not identical to actual practice. Nonetheless, 94% of Course participants rated the simulations as “realistic.” In a recent study, Weller et al.14  found that communication patterns were similar in actual and in simulated routine cases, thereby lending support for the contextual “validity of the simulation environment and its value in teamwork training.” To extend such findings to all components of decision making, communication, and clinical management during unusual but critical events in the simulated versus real world may be logistically impossible because the events most needing the acute interventional skills of the anesthesiologist will be uncommon.15  However, the types of performance gaps observed in MOCA Courses appear similar to events reported to the Anesthesia Incident Reporting System.
To date, the balance of evidence supports our assertion that simulation training enhances physicians’ clinical performance (and reduces complications) during actual patient care.16–19  Simulation training has been shown to improve outcome and decrease costs after central venous catheter insertion in medical intensive care units.20,21  With regard to nontechnical (i.e., behavioral or teamwork) skills, standardized patient-based simulations have improved the quality of actual handovers between anesthesia providers and recovery room nurses.22  Mannequin-based simulations targeting crisis resource management have been conducted for more than 2 decades and have become important components of anesthesiology resident training. The impact of in-person high-fidelity simulations conducted and debriefed by expert instructors is profound. For those who have not observed or taken part in such activities, the value of the experience is difficult to appreciate.
Issues about the MOCA Simulation Process
The 39 ASA-endorsed centers that offer Courses have worked hard to deliver high-quality education and practice improvement experiences to anesthesiologists. The process requires an individual to spend most of a day in the Course and, in some cases, significant travel is required. Endorsed centers are located in 21 states and more than 75% of anesthesiologists enrolled in MOCA live in those states. Most centers are located in urban areas proximate to the largest concentration of anesthesiologists in their state. New centers are continuing to apply for endorsement and the network is growing. However, for some individuals, attending a Course may be logistically inconvenient. Course tuition, although relatively high, is commensurate with its high intensity, the substantial infrastructure and preparations required, and the high instructor-to-participant ratio (no less than 1-to-5, and often greater than 2-to-5). Nonetheless, despite the cost and inconvenience, the more than 2,700 Course participants to date have been overwhelmingly positive about their experience.
Those on the SEB and in the simulation education community need to better inform our colleagues about the Course to allay concerns and anxiety about the experience. In fact, the SEB is quite sensitive about these issues. Several of the authors of this editorial, and many MOCA Simulation instructors, are themselves enrolled in the MOCA program and have already taken, or will need to take, a Course. However, the facts articulated earlier about there being no assessment, no evaluation, and no test do not seem to have been fully appreciated by many anesthesiologists. It is understandable that those who have never experienced simulation training may be concerned about “performing” in front of peers. The SEB requires endorsed programs to demonstrate their ability to conduct MOCA Simulation training in a way that addresses such concerns. The resoundingly positive responses by course participants to date show that these efforts have been successful.
Will One Simulation Course Every 10 Yr Make a Difference?
The frequency of simulation training is not the only determinant of its impact on anesthesia practice.23  The Course is designed to hone many general skills that are applicable to every case, and certainly to every challenging case. The lessons of the Course and of the practice improvement efforts are not limited only to the few scenarios run in a day, but should have lasting value. With any course or training intervention, there is a danger that skills learned will erode24  and, in a perfect world, simulation training should probably occur more regularly than once per decade. Judging from the experiences of other high-consequence industries, there is likely to be further benefit to more frequent simulation exercises. In aviation, simulation training occurs at least annually, and nuclear power operator teams typically spend an astounding 1 week in every five in simulation training. Although anesthesiology and other healthcare domains may never approach this level of commitment to proactive training of experienced professionals, to their credit some hospitals and anesthesia practices conduct their own regular simulation training. For now, the MOCA Simulation Course, attended once every 10 yr, is a credible step to providing anesthesiologists with a significant opportunity to improve their practices and hence it is an important component of the overall MOCA program.
The goal of MOCA overall, the Courses, and other patient safety initiatives is to save as many patients’ lives, hearts, and brains as possible. There are already anecdotes that these Courses have helped anesthesiologists to serve their patients and have even saved lives. Every life counts. There is a saying so profound that it appears in varying forms in both the Muslim Quran and the Hebrew Talmud, that “Whoever saves a life, it is as if he has saved all of mankind.” The public looks to board certification of clinicians as the mark of excellence. The MOCA program, and perhaps especially the simulation component, can reassure the public that anesthesiologists are taking seriously their responsibility to continually hone their skills to offer the care expected from those who are certified.
Simulation is one of the many innovations contributed by anesthesiologists that are used throughout health care every day to provide safer patient care. It is a willingness to do what is right even when it is not popular, even when it is difficult, even when it requires real effort from each of us, that demonstrates to all our healthcare colleagues, to our patients, and to the entire world, the leadership of anesthesiologists.
Acknowledgments
The authors thank the review and comment on this article by Jeff Cooper, Ph.D. (Department of Anesthesiology and Critical Care, Massachusetts General Hospital, Boston, Massachusetts), and Deborah Culley, M.D. (Department of Anesthesiology, Brigham and Women’s Hospital/Harvard Medical School, Boston, Massachusetts). The authors thank the other members of the Simulation Performance Study Group: Arna Banerjee, M.D. (Department of Anesthesiology, Vanderbilt University, Nashville, Tennessee), John (Jack) Boulet, Ph.D. (Foundation for the Advancement of International Medical Education and Research, Philadelphia, Pennsylvania), William McIvor, M.D. (The Peter M. Winter Institute for Simulation Education and Research, University of Pittsburgh, Pittsburgh, Pennsylvania), Robert L. Nadelberg, M.D. (Department of Anesthesia, Critical Care, and Pain Medicine, Massachusetts General Hospital, Boston, Massachusetts), Christine S. Park, M.D. (Department of Anesthesiology, Northwestern University, Chicago, Illinois), Eric Porterfield, M.S. (Department of Anesthesiology, Vanderbilt University, Nashville, Tennessee), Elizabeth (Lisa) Sinz, M.D. (Department of Anesthesiology, Pennsylvania State University, Hershey, Pennsylvania), Jason M. Slagle, Ph.D. (Center for Perioperative Research in Quality, Vanderbilt University, Nashville, Tennessee), and Laurence C. Torsher, M.D. (Department of Anesthesiology, Mayo Clinic, Rochester, Minnesota).
Dr. Weinger’s effort was supported by the Department of Veterans Affairs Tennessee Valley Healthcare Systems’ Geriatric Research Education and Clinical Center (Nashville, Tennessee), by grant R18 HS020415 from the Agency for Healthcare Research and Quality (Rockville, Maryland), and by institutional resources of the Vanderbilt University School of Medicine, Nashville, Tennessee. Dr. Burden’s effort was supported by grant R18 HS020415 from the Agency for Healthcare Research and Quality (Rockville, Maryland), by the Cooper Medical School of Rowan University (Camden, New Jersey), and by Cooper Health System (Camden, New Jersey). Dr. Steadman’s effort was supported, in part, by grant R18 HS020415 from the Agency for Healthcare Research and Quality (Rockville, Maryland). Dr. Gaba’s effort was supported, in part, by grant R18 HS020415 from the Agency for Healthcare Research and Quality (Rockville, Maryland).
Competing Interests
The opinions or assertions contained in this article are the private views of the authors and are not to be construed as the official views or positions of the American Society of Anesthesiologists, the American Board of Anesthesiology, or the Department of Veterans Affairs. All of the authors are members of the American Society of Anesthesiologists Simulation Education Editorial Board (Park Ridge, Illinois). Drs. Weinger and Gaba are members of the Executive Committee of the Anesthesia Patient Safety Foundation (Indianapolis, Indiana) and have previously received grant funding from them for simulation-related patient safety research. All of the authors are MOCA simulation course instructors at their respective institutions and receive market-based compensation or other academic recognition for the time they spend teaching these courses.
*Available at: http://education.asahq.org/sim/faq. Accessed March 5, 2014.
Available at: http://education.asahq.org/sim/faq. Accessed March 5, 2014.×
Available at: http://www.theaba.org/Home/anesthesiology_maintenance. Accessed March 6, 2014.
Available at: http://www.theaba.org/Home/anesthesiology_maintenance. Accessed March 6, 2014.×
References
McIvor, W, Burden, A, Weinger, MB, Steadman, R Simulation for maintenance of certification in anesthesiology: The first two years.. J Contin Educ Health Prof. (2012). 32 236–42 [Article] [PubMed]
Domino, FJ, Chopra, S, Seligman, M, Sullivan, K, Quirk, ME The impact on medical practice of commitments to change following CME lectures: A randomized controlled trial.. Med Teach. (2011). 33 e495–500 [Article] [PubMed]
Mazmanian, PE, Johnson, RE, Zhang, A, Boothby, J, Yeatts, EJ Effects of a signature on rates of change: A randomized controlled trial involving continuing education and the commitment-to-change model.. Acad Med. (2001). 76 642–6 [Article] [PubMed]
Steadman, RS, Burden, AR, Huang, YM, Cooper, JBPractice improvements implemented based on participation in a simulation program for maintenance of certification in anesthesiology. Presented at: Annual Meeting of the American Society of Anesthesiologists. (October 14, 2013). San Francisco, CA#LBC09
Oken, A, Rasmussen, MD, Slagle, JM, Jain, S, Kuykendall, T, Ordonez, N, Weinger, MB A facilitated survey instrument captures significantly more anesthesia events than does traditional voluntary event reporting.. Anesthesiology. (2007). 107 909–22 [Article] [PubMed]
Walsh, M, Devereaux, PJ, Garg, AX, Kurz, A, Turan, A, Rodseth, RN, Cywinski, J, Thabane, L, Sessler, DI Relationship between intraoperative mean arterial pressure and clinical outcomes after noncardiac surgery: Toward an empirical definition of hypotension.. Anesthesiology. (2013). 119 507–15 [Article] [PubMed]
Slagle, JM, Anders, S, Porterfield, E, Arnold, A, Calderwood, C, Weinger, MBSignificant physiological disturbances associated with non-routine event containing and routine anesthesia cases.. J Patient Saf. (2014). [Epub ahead of print]
Forrest, JB, Cahalan, MK, Rehder, K, Goldsmith, CH, Levy, WJ, Strunin, L, Bota, W, Boucek, CD, Cucchiara, RF, Dhamee, S Multicenter study of general anesthesia. II. Results.. Anesthesiology. (1990). 72 262–8 [Article] [PubMed]
Schwid, HA, O’Donnell, D Anesthesiologists’ management of simulated critical incidents.. Anesthesiology. (1992). 76 495–501 [Article] [PubMed]
DeAnda, A, Gaba, DM Role of experience in the response to simulated critical incidents.. Anesth Analg. (1991). 72 308–15 [Article] [PubMed]
Murray, DJ, Boulet, JR, Avidan, M, Kras, JF, Henrichs, B, Woodhouse, J, Evers, AS Performance of residents and anesthesiologists in a simulation-based skill assessment.. Anesthesiology. (2007). 107 705–13 [Article] [PubMed]
Gaba, DM Structural and organizational issues in patient safety: A comparison of health care to other high-hazard industries.. Calif Manage Rev. (2001). 43 83–102 [Article]
Gaba, DM Weinger, MB, Halbert, BP, Logan, MK The thorniest issues in healthcare. Risk and Reliability in Healthcare and Nuclear Power: Learning from Each Other. (2013). Arlington Association for the Advancement of Medical Instrumentation pp 11–5
Weller, J, Henderson, R, Webster, CS, Shulruf, B, Torrie, J, Davies, E, Henderson, K, Frampton, C, Merry, AF Comparison of anesthesiologists’ communication patterns in real and simulated cases.. Anesthesiology. (2014). 120 1–7 [Article] [PubMed]
Gaba, DM The pharmaceutical analogy for simulation: A policy perspective.. Simul Healthc. (2010). 5 5–7 [Article] [PubMed]
Bruppacher, HR, Alam, SK, LeBlanc, VR, Latter, D, Naik, VN, Savoldelli, GL, Mazer, CD, Kurrek, MM, Joo, HS Simulation-based training improves physicians’ performance in patient care in high-stakes clinical setting of cardiac surgery.. Anesthesiology. (2010). 112 985–92 [Article] [PubMed]
Barsuk, JH, McGaghie, WC, Cohen, ER, O’Leary, KJ, Wayne, DB Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit.. Crit Care Med. (2009). 37 2697–701 [Article] [PubMed]
Wayne, DB, Didwania, A, Feinglass, J, Fudala, MJ, Barsuk, JH, McGaghie, WC Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: A case-control study.. Chest. (2008). 133 56–61 [Article] [PubMed]
Thomas, EJ, Taggart, B, Crandell, S, Lasky, RE, Williams, AL, Love, LJ, Sexton, JB, Tyson, JE, Helmreich, RL Teaching teamwork during the Neonatal Resuscitation Program: A randomized trial.. J Perinatol. (2007). 27 409–14 [Article] [PubMed]
Cohen, ER, Feinglass, J, Barsuk, JH, Barnard, C, O’Donnell, A, McGaghie, WC, Wayne, DB Cost savings from reduced catheter-related bloodstream infection after simulation-based education for residents in a medical intensive care unit.. Simul Healthc. (2010). 5 98–102 [Article] [PubMed]
Burden, AR, Torjman, MC, Dy, GE, Jaffe, JD, Littman, JJ, Nawar, F, Rajaram, SS, Schorr, C, Staman, GW, Reboli, AC Prevention of central venous catheter-related bloodstream infections: Is it time to add simulation training to the prevention bundle?. J Clin Anesth. (2012). 24 555–60 [Article] [PubMed]
Weinger, MB, Slagle, JM, Kuntz, A, Banerjee, A, Schildcrout, J, Mercaldo, N, France, D, Speroff, T, Bills, J, Walston, K and the PACU Handover Improvement Team, Improving actual handover behavior with a simulation-based training intervention.. Proc Hum Factors Ergon Soc. (2010). 54 957–61 [Article]
Weinger, MB The pharmacology of simulation: A conceptual framework to inform progress in simulation research.. Simul Healthc. (2010). 5 8–15 [Article] [PubMed]
Kurrek, MM, Devitt, JH, Cohen, M Cardiac arrest in the OR: How are our ACLS skills?. Can J Anaesth. (1998). 45 130–2 [Article] [PubMed]