Education  |   October 2019
Learners and Luddites in the Twenty-first Century: Bringing Evidence-based Education to Anesthesiology
Author Notes
  • From the Department of Anesthesiology, University of North Carolina School of Medicine, Chapel Hill, North Carolina (S.M.M., R.S.I., F.C.); the Department of Anesthesiology, University of Kentucky College of Medicine, Lexington, Kentucky (R.M.S.); Beth Israel Deaconess Medical Center, Department of Anesthesia, Critical Care and Pain Medicine, Harvard Medical School, Boston, Massachusetts, (J.D.M.); and Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee (M.D.M.).
  • This article is featured in “This Month in Anesthesiology,” page 1A.
    This article is featured in “This Month in Anesthesiology,” page 1A.×
  • Submitted for publication September 4, 2018. Accepted for publication May 6, 2019.
    Submitted for publication September 4, 2018. Accepted for publication May 6, 2019.×
  • Address correspondence to Dr. Martinelli: N2198 UNC Hospitals #7010, Chapel Hill, North Carolina 27599. Information on purchasing reprints may be found at or on the masthead page at the beginning of this issue. Anesthesiology’s articles are made freely accessible to all readers, for personal use only, 6 months from the cover date of the issue.
Article Information
Education / Review Article / Education / CPD / Ethics / Medicolegal Issues
Education   |   October 2019
Learners and Luddites in the Twenty-first Century: Bringing Evidence-based Education to Anesthesiology
Anesthesiology 10 2019, Vol.131, 908-928. doi:
Anesthesiology 10 2019, Vol.131, 908-928. doi:

Anesthesiologists are both teachers and learners and alternate between these roles throughout their careers. However, few anesthesiologists have formal training in the methodologies and theories of education. Many anesthesiology educators often teach as they were taught and may not be taking advantage of current evidence in education to guide and optimize the way they teach and learn. This review describes the most up-to-date evidence in education for teaching knowledge, procedural skills, and professionalism. Methods such as active learning, spaced learning, interleaving, retrieval practice, e-learning, experiential learning, and the use of cognitive aids will be described. We made an effort to illustrate the best available evidence supporting educational practices while recognizing the inherent challenges in medical education research. Similar to implementing evidence in clinical practice in an attempt to improve patient outcomes, implementing an evidence-based approach to anesthesiology education may improve learning outcomes.

Anesthesiology has a long history of developing and implementing innovations in education, and most anesthesiologists are educators in some capacity. Anesthesiologists teach fellows, residents, and medical students. They also educate patients, peers, and colleagues in anesthesiology and other disciplines. Despite the pervasive nature of teaching and learning in this specialty, few anesthesiologists have formal training in how to effectively educate. Many anesthesiologists often teach the way they were taught and learn the way they learned in the past. Applying current evidence in education is likely to change the way many teach and learn.1 
Accordingly, in this review, we discuss current evidence in education, provide recommendations of how the evidence might be applied to teaching and learning in anesthesiology, and suggest areas of future educational research. More specifically, educators across multiple disciplines have moved away from passive, lecture-based presentations to more active learning through techniques such as the flipped classroom, peer teaching, and educational games. In order to improve actual knowledge retention, as opposed to mass learning or blocking, evidence suggests that the more effective techniques may be spaced learning, interleaving, and retrieval practice. To combat noncompliance due to knowledge and memory errors in clinical crises, cognitive aids have proven to be a critical component of anesthesiology training. E-learning provides platforms to enable flexible utilization of these knowledge acquisition, retention, and assistance techniques on a daily basis to optimize learning. In addition to the acquisition and retention of knowledge, practicing anesthesiology also entails manual dexterity and workflow management. Thus, the techniques, theory, and metrics for teaching procedural skills are of the utmost importance. Additionally, teaching and assessing clinical performance and professionalism through role modeling and role play are becoming a more recognized need in our field. Finally, experiential learning through simulation and objective structured clinical examinations is becoming more prominent in the form of both education and summative assessment.
Although this review is organized based on the teaching of (1) knowledge, (2) skills, and (3) professionalism, we recognize there is substantial overlap between these topics. This review is meant to be used as a primer on educational methods in anesthesiology for any clinician who aspires to be a better teacher. It is written in a module-based format so that the reader can easily navigate through the review and integrate the material. Figure 1 provides a tool that can be used to guide the instructional needs of your learners. For those interested in a specific topic, the references included in this review may serve as a more detailed resource.
Fig. 1.
Bringing novel educational methods to your learners. This review suggests many novel ways in which educational research and innovation can make a difference to your learners. Each blue circle contains different goals for your learning environment. Ways in which one may utilize novel educational interventions to reach these goals are described in the corresponding boxes.
Bringing novel educational methods to your learners. This review suggests many novel ways in which educational research and innovation can make a difference to your learners. Each blue circle contains different goals for your learning environment. Ways in which one may utilize novel educational interventions to reach these goals are described in the corresponding boxes.
Fig. 1.
Bringing novel educational methods to your learners. This review suggests many novel ways in which educational research and innovation can make a difference to your learners. Each blue circle contains different goals for your learning environment. Ways in which one may utilize novel educational interventions to reach these goals are described in the corresponding boxes.
Teaching Knowledge
One goal in anesthesiology education is to help learners acquire, retain, recall, and apply knowledge (table 1). The volume of medical knowledge continues to rapidly expand. Unfortunately, without repetition, knowledge is forgotten quickly after learning. Implementation of the evidence from cognitive science and healthcare education research offers the promise of making lessons “stickier” with improved knowledge retention.2–7 
Table 1.
Educational Methodologies Based on Domains of Utilization
Educational Methodologies Based on Domains of Utilization×
Educational Methodologies Based on Domains of Utilization
Table 1.
Educational Methodologies Based on Domains of Utilization
Educational Methodologies Based on Domains of Utilization×
Active Learning
Medical education is frequently delivered passively through teacher-centered delivery of lectures. Learners memorize facts, recite them for examinations, and then often forget the majority of the information. Active learning describes an approach to learning, not a singular educational method.8,9  This approach involves learner-centered education in which learners apply knowledge and solve problems in the classroom. Although active learning has been demonstrated to be effective in many areas of science education,8,10,11  more work needs to be done to determine its benefit in graduate medical education.
There is a variety of active learning methodologies, which all share some basic principles. First, control shifts from teacher to learner as the teacher takes on the role of a facilitator or coach, and the students assume ownership for their education. Another key component is that learners actively participate during classroom time by applying knowledge they are learning through higher-order thinking and problem solving. One advanced model of active learning is the Master Adaptive Learner. In this model, learners work actively through an iterative cycle of planning, learning, assessing, and adjusting as they master concepts or skills (fig. 2).12  This process is heavily reliant on application of critical thinking and reflection in each phase. The goal of all active learning approaches is to increase knowledge retention so that learners will be better equipped to apply that knowledge to solve new problems they encounter in the future. In addition, the teacher is better able to recognize and correct knowledge gaps as the learners are working through problems.
Fig. 2.
The Master Adaptive Learning model is an advanced model of active learning. The learner works through an iterative cycle of planning, learning, assessing, and adjusting as they master skills or concepts. The skills needed to be a successful learner are listed in each cog below the respective phase headings within the learner. The interactivity of the cogs represent that the learner must engage in all of these areas to develop successfully through a continuous and dynamic process. Reprinted with permission from Cutrer et al. (2017).12 
The Master Adaptive Learning model is an advanced model of active learning. The learner works through an iterative cycle of planning, learning, assessing, and adjusting as they master skills or concepts. The skills needed to be a successful learner are listed in each cog below the respective phase headings within the learner. The interactivity of the cogs represent that the learner must engage in all of these areas to develop successfully through a continuous and dynamic process. Reprinted with permission from Cutrer et al. (2017).12
Fig. 2.
The Master Adaptive Learning model is an advanced model of active learning. The learner works through an iterative cycle of planning, learning, assessing, and adjusting as they master skills or concepts. The skills needed to be a successful learner are listed in each cog below the respective phase headings within the learner. The interactivity of the cogs represent that the learner must engage in all of these areas to develop successfully through a continuous and dynamic process. Reprinted with permission from Cutrer et al. (2017).12 
Flipped Classroom.
The flipped classroom is one educational method that utilizes active learning techniques. Traditionally, classroom time was used for teacher-centered lecture-based learning in which the teacher passively transferred factual information to learners. Learners then applied that information to solve problems independently on their own time as homework. Flipped classroom reverses this process. Learners obtain foundational knowledge before coming to class through “homework.” Although multiple formats can be utilized (reading assignments, electronic modules, videocasts, etc.), a brief, 15 to 20 min video is the most common format for the “homework” component. (Table 2 provides some examples of educational resources that can be used for preclass homework for anesthesiology learners.) Learners then spend in-class time using that foundational knowledge for problem solving through active learning.
Table 2.
Sampling of Educational Tools Specific to Anesthesiology Learners
Sampling of Educational Tools Specific to Anesthesiology Learners×
Sampling of Educational Tools Specific to Anesthesiology Learners
Table 2.
Sampling of Educational Tools Specific to Anesthesiology Learners
Sampling of Educational Tools Specific to Anesthesiology Learners×
There are many proposed benefits of flipped classroom. It allows flexibility in the time, place, and pace in which learners obtain foundational knowledge. It promotes teamwork, which also helps hold learners accountable to each other for being prepared for in-class sessions. By observing how learners solve problems, facilitators can identify and correct knowledge gaps in real time.13 
There have been a number of studies assessing flipped classroom in healthcare education. Utilization of flipped classroom has led to knowledge gain in pharmacy education.14–16  A recent systematic review of flipped classroom in medical education found that learners preferred flipped classroom, but there were mixed results regarding the benefit of flipped classroom in knowledge and skill gain. Most studies in the review were performed with medical student learners.17  Since this review, there have been several additional studies published in graduate medical education, again with mixed results. One study showed that within a group of internal medicine residents learning about quality improvement, there was a preference for flipped classroom, and it was found to increase knowledge acquisition.18  However, another study with emergency medicine residents utilizing flipped classroom was equivocal with respect to knowledge acquisition.19  A large multiinstitutional study with residents preparing for the American Board of Anesthesiology (ABA; Raleigh, North Carolina) BASIC Exam found that anesthesiology residents preferred flipped classroom to traditional lectures. Although there was not a significant improvement in immediate knowledge gain, testing done 4 months after learning demonstrated a significant improvement in knowledge retention in flipped classroom group compared to traditional lectures.20 
Much of the work assessing the benefit of flipped classroom utilized multiple-choice tests for knowledge assessment. Most multiple-choice questions assess the “knows” level of Miller’s pyramid, in which the learner demonstrates knowledge of factual information. Some may reach the “knows how” level, in which the learner demonstrates knowledge of factual information and also how to interpret and apply that information (fig. 3).21,22  As one of the goals of flipped classroom is to foster better problem-solving skills and improve knowledge application, multiple-choice questions may not be the best assessment tool.23  Gillispie et al. demonstrated a stronger benefit of flipped classroom in an obstetrics and gynecology medical student clerkship when assessing with an Objective Structured Clinical Exam compared to multiple-choice questions.24  Future work assessing the use of flipped classroom for anesthesiology residents may therefore benefit from testing with Objective Structured Clinical Exams in addition to multiple-choice questions.
Fig. 3.
Miller’s Pyramid of Assessment. The face of the pyramid demonstrates varying levels of assessment, which progress in their ability to demonstrate competence moving from the base to the peak. The desired goal is for learners to reach the “DOES” level. The side of the pyramid breaks down the educational domains in which our learners must gain competence. Adapted from Mehay (2012).22 
Miller’s Pyramid of Assessment. The face of the pyramid demonstrates varying levels of assessment, which progress in their ability to demonstrate competence moving from the base to the peak. The desired goal is for learners to reach the “DOES” level. The side of the pyramid breaks down the educational domains in which our learners must gain competence. Adapted from Mehay (2012).22
Fig. 3.
Miller’s Pyramid of Assessment. The face of the pyramid demonstrates varying levels of assessment, which progress in their ability to demonstrate competence moving from the base to the peak. The desired goal is for learners to reach the “DOES” level. The side of the pyramid breaks down the educational domains in which our learners must gain competence. Adapted from Mehay (2012).22 
Peer Teaching.
Peer teaching is a learning environment in which learners are taught by their peers. Technically, the learners and teachers in this scenario should be at the same level of training. Similarly, near-peer teaching occurs when learners are taught by other learners who have more experience or education (e.g., residents teaching medical students). These near-peer teachers have less experience and knowledge than faculty and typically have little to no formal education in teaching methodology.
Peer teaching and near-peer teaching have many suggested benefits. First, in this era of competing priorities and time limitations, the teaching burden is shifted away from faculty. Second, peer teachers often benefit both from their own mastery of the content and gaining early exposure to teaching opportunities. This method also fosters teamwork skills that may enable peer teachers and learners to more easily identify their strengths and weaknesses.25  In addition, faculty may succumb to the “curse of knowledge” in which they assume that learners have a stronger knowledge base than they actually do. As peers may be better equipped to put themselves in the position of the learner, they may have an advantage in teaching a new concept.
Peer teaching can be utilized in a large group learning environment, facilitated by a faculty member.26  The faculty poses a question, case, or problem to the learners. This can be done with an audience response system or through open-ended questions. Learners are given a short amount of time to think through the answer individually. They then organize into pairs, or small groups if this method is modified to “think-group-share,” to discuss their answers and further problem-solve. The entire group of learners then reconvenes to review the possible solutions to the problem under the direction of the faculty leader.
Peer teaching has a long history of utilization in anesthesiology with hands-on experience in the perioperative environments such as the operating room. For instance, senior residents often mentor and teach junior residents, while both senior and junior residents teach medical students.
Although, to our knowledge, there is no recent work on peer teaching in graduate medical education, several studies have investigated this method with medical students. A review of undergraduate medical education found some evidence that peer teachers gained greater mastery of subject matter. Although learning gains were not always demonstrated, neither was there evidence of detrimental effects to the peer teachers’ learning.25  Gregory et al. demonstrated that both preparation for teaching and the actual act of teaching increased peer teachers’ knowledge gain and retention in medical students who were peer teachers compared to those who did not peer teach.27  Peer teachers felt they benefited through the gained mentoring experience, engagement with and learning from learners, and an increased motivation to study.28  In addition, it was shown that peer teaching was as effective as faculty-led learning. This was thought to be due to the cognitive and social congruencies that peer teachers shared with their learners.25 
Educational Games.
Educational games are gaining popularity in the medical education community. These can be categorized as simulation, virtual environments, social and cooperative play, or alternative reality.29  For the purpose of this review, we will only discuss educational games that fit into the social and cooperative play category, as these are a form of active learning. Educational games have preset rules and include an element of competition. A variety of board games and TV gameshows have been adapted for medical education including Jeopardy, Monopoly, Hollywood Squares, and Snakes and Ladders.
Proposed benefits of educational games include the stimulation of higher levels of thinking, a fun and exciting educational environment that reduces stress and anxiety, and an increase in knowledge retention.29  However, a Cochrane review was unable to determine a knowledge benefit from game-based learning and concluded that more studies are needed.30  There have been a few studies done with medical students and one in continuing medical education that have not shown an increase in knowledge gain from game-based learning.31–33  However, participants in all of these studies preferred game-based learning.31–33  There is one report in graduate medical education literature on the use of an adaptation of the game show Jeopardy for teaching general surgery residents. Although there was demonstration of knowledge acquisition and retention compared to pretest knowledge, there was no comparison group utilized in this study.34 
Optimizing Knowledge Retention
Three evidence-based methods of teaching and learning that facilitate long-term retention of knowledge include spaced learning, interleaving, and retrieval practice. Although the benefits of these strategies have mainly focused on improved knowledge acquisition and retention, evidence is growing that they also may be applied to attainment of psychomotor skills.35,36 
Spaced Learning.
Repetition is a successful strategy for learning. When repetitive study is spaced out over an extended period of time, it is referred to as “distributed” or “spaced.” The “spacing effect” describes the finding that when an equal amount of cumulative study or practice time is spaced out between sessions with an intervening time gap, knowledge acquisition and long-term knowledge retention are better than if sessions of the same material are massed together. An initially short, and then expanding, interstudy time interval may be optimal. A longer interstudy interval is generally associated with improved long-term retention, but the time between studying should not be so long that the learner forgets what they previously learned.37–39  Spaced education has been used to improve knowledge retention in healthcare education including graduate medical education.40–45 
Many learners utilize massed learning or “cramming” and have experienced the short-term benefit of passing their examinations. Massing “works” when the information to be learned is not difficult and testing is in the near future. However, the more time spent in a single session, the less effective that time becomes, and massed learning leads to poor knowledge retention.
Examples of using spaced learning in anesthesiology include (1) distributing rotations throughout a multiyear curriculum rather than massing them together (e.g., pediatric anesthesia, critical care medicine, and cardiac anesthesia), (2) teaching a topic (e.g., neonatal resuscitation, mechanical ventilation management, or cardiac physiology) in shorter sessions and distributing teaching sessions over time, (3) utilizing mobile devices to deliver spaced repetition of information to learners (e.g., emails, text messages, and questions), and (4) encouraging learners to start studying early and often, and utilizing initially short time intervals between repetitions of a specific topic and then expanding the time intervals between repetitions.
“Blocking,” which is typically preferred by learners, refers to the repetitive studying of information in one topic or category completely before going on to another, or practicing one skill at a time toward mastery before practicing another. Interleaving is a process where students mix, or interleave, the study or practice of several related concepts or skills. An example would be when learners are tasked to understand and retain knowledge about anesthetic pharmacokinetics in infants, adults, and the elderly. If applying the interleaving technique, they might go back and forth (i.e., utilizing comparison and contrast) between studying about different anesthetics in each of the age groups. This is opposed to a blocking technique in which they would limit their reading and studying of pharmacokinetics to one age group until obtaining mastery of the concepts before going on to another age group. A learner utilizing interleaving to help master the skill of transesophageal echocardiography would mix up their study and skill practice. They might work on acquiring images on the simulator and on patients while also reading about physics and hemodynamic calculations. This would be opposed to blocking, in which they focus only on image acquisition during their first rotation and then later learn about physics and hemodynamic calculations on their second rotation. Medical school curricula are now frequently mixed or interleaved. Separate but related topics are interleaved together such as cardiac anatomy, cardiac physiology, and cardiac pharmacology rather than presenting all anatomy topics until completion, then physiology, and finally pharmacology in a similar blocked format.
Content understanding and memory/retention for information learned through interleaving are better than through massed practice.46,47  However, interleaving feels slower and less efficient to the learner. Massing is often preferred over interleaving, even when the learner is confronted with evidence from personal performance measures that demonstrate better retention and ability to transfer knowledge to other contexts with interleaving.48,49 
Interleaving the study of separate but related topics can help the learner see similarities and differences between ideas or concepts, creating connections between related parts. It also takes advantage of the spacing effect by distributing learning. Cognitive psychology has provided evidence for the benefits of mixing up categories of topics during learning, and interleaving has been used to improve learning in mathematics, radiology, and neuroanatomy.47,50,51 
The optimal timing and patterns for interleaving may vary between learners, and determination of such requires further investigation. While there is an educational advantage to switching between categories of information, switching too often or spending too little time on any one idea may be detrimental. Interleaving appears to be most beneficial when the category discrimination is difficult; that is, differences between presented materials (cases, disease pathophysiology, and images) are difficult to distinguish from each other, and interleaving provides the opportunity for observing differences by comparison and contrast.49,52 
An example of interleaving in clinical practice is a month-long rotation in the Clinical Anesthesia 3 year of training called “Transition to Practice” in which a resident is not on a specific subspecialty for an extended period of time such as pediatrics for 1 month or cardiac for 1 month, but instead the subspecialty case type and the learning experiences vary day to day and possibly patient to patient. The resident may one day be doing predominantly pediatrics, the next day doing predominantly ambulatory, the next thoracic, etc. This format mixes up the study of related concepts and skills.
Retrieval Practice.
Cognitive science suggests that we take information (audio, visual) into our limited short-term memory, encode that information into our “unlimited” long-term memory, and then retrieve it from long-term memory when needed. The act of retrieving a memory changes it, making it easier to retrieve again later, and increases long-term retention.53,54  One example of this is the long-term retention of phone numbers that you regularly recall from memory as opposed to those you look up and enter from a contact list. Retrieval practice is the act of calling information to mind that has previously been encoded or learned rather than rereading or rehearing it.
Test-enhanced learning refers to the use of retrieval practice through testing as a way to enhance retention of knowledge.55,56  The “testing effect” refers to the use of retrieval as a powerful learning tool. A direct benefit of testing is that recalling previously learned information enhances the ability to recall information in the future. When information is recalled from memory during low- or no-stakes testing by doing questions and receiving feedback, retention is better than if the material is simply restudied over and over again.57,58 
The benefit of retrieval practice in health professions education has recently been reviewed.57  The first studies assessing the positive effect of formative assessment on learning outcomes in medical education were published within the last decade, and most studies since that time have compared retrieval practice with the passive learning strategy of restudy.58,59  However, the benefits of testing relative to active learning strategies, such as concept mapping and case-based learning, have been demonstrated.60,61  In a recent systematic review, retrieval practice demonstrated consistent and strong effects across learners in various health professions and for various learner outcomes. Moreover, the effectiveness of retrieval practice extended beyond knowledge assessed by examinations to clinical applications.57 
Ideally, tests used to enhance learning should be composed of questions related to the topic previously studied, repeated, and spaced over time; should utilize questions that require production of information rather than just recognition; and should include feedback with correct responses and rationale.46,57  Tests that require generation of information (e.g., short answer) as opposed to recognizing and selecting a response from information provided (e.g., multiple choice) and that require more effortful recall appear to have a greater benefit on knowledge retention. Repeated testing that is spaced over time allows the learner to take advantage of feedback and practice to correct errors.62  Feedback enhances the benefits of testing; although testing alone improves retention, feedback should be provided to help the learner close the gap between what they desire to learn and what they actually learned. Interestingly, delayed feedback may be more beneficial than immediate feedback.63  When the test taker is conditioned by instantaneous feedback to expect the certainty of continual correction, it may reduce the likelihood they will engage in effortful recall.63  The efficacy of repeated testing seems to be greater the more complicated the subject being studied. It is apparent that repeated testing promotes learning in ways that restudying cannot.64,65  Moreover, there is a small but growing body of evidence that those who learn foundational information using retrieval practice are more likely to transfer their knowledge to novel situations than those who learned the foundational information by repeatedly studying it.64,66,67 
Posttests are commonly administered immediately after learning sessions such as with online education or active learning didactics. However, recall is relatively easy immediately after study, and the level of performance immediately after learning is not a good indicator of future retention. Electronic flash cards with questions, question-generating applications, questions delivered to learners via mobile devices, and utilization of question banks that include feedback are examples of ways to utilize formative retrieval practice via testing. The ABA Maintenance of Certification in Anesthesiology Minute is an example of a program that uses both retrieval practice and spaced learning via quizzing (table 2). These principles might be applied beyond using test questions to enhance learning and knowledge retention to other forms of retrieval practice such as oral examinations, peer teaching, and even psychomotor skills such as cardiopulmonary resuscitation.68 
After initial encoding of information, if a learner preparing for the ABA In-Training Examination used questions from a question bank to encourage retrieval of knowledge from memory and chose to mix questions from various topics rather than answering all questions on a specific topic before moving on to another, this would be an example of interleaving and retrieval practice.
Cognitive Aids
In 1924, an article was published that called for routine training on, and use of, cognitive aids during intraoperative emergencies.69  In recent years, multiple institutions and national organizations have promoted the use of cognitive aids for perioperative emergencies, leading to the development and publication of several emergency manuals.70  One article defined cognitive aids as “tools created to guide users while they are performing a task, or group of tasks, with the goal of reducing errors and omissions and increasing the speed and fluidity of performance.”71  In the broadest sense, anything that acts as a memory aid for the clinician could be considered a cognitive aid. Specifically, within the domain of anesthesia, critical care, and perioperative medicine, the focus of research has centered around checklists.71–73  The characteristics of an optimal cognitive aid are accurate and reliable clinical content, a user-friendly design that includes human factors assessment, and iterative assessment and improvement of the aid itself using simulation or clinical use.72,74  With the advent of the smartphone also came the question of the best format for a cognitive aid: paper versus electronic.72,73  Taken together, there can be a complex taxonomy of cognitive aid and end user, which can range from “static parallel checklists” where one operator reads and performs all tasks on the checklist (e.g., anesthesia machine checkout) to dynamic clinical checklists in which one or more team members uses a tool that involves branching logic and clinical decision-making (e.g., management of pulseless arrest and return of spontaneous circulation).71,75–78 
The research concerning the effect of cognitive aids within anesthesia, critical care, and perioperative care has been extensive. Numerous prospective randomized controlled trials have evaluated the effect of cognitive aids on the technical performance of individuals or teams. Most trials concerning perioperative and in-hospital emergencies have demonstrated a positive effect for completing actions considered to be best practice, or standard of care, for a given situation.76–82  However, some have noted that the cognitive aid can inhibit performance by being a distraction,83  which is most commonly the case when cognitive aids are used without appropriate education on the tool prior to use.84  Beyond crisis situations, a number of trials have shown improved adherence to accepted care guidelines when using cognitive aids in simulated nonacute perioperative situations, such as applying guidelines for preoperative cardiac evaluation, assessment of patients on anticoagulants for regional anesthesia, and compatibility in blood transfusions.85–87  Furthermore, improvement in technical performance and nontechnical skills (including teamwork and communication) have been shown to be improved with the use of checklists in the operating room.88  In summary, evidence has resulted in numerous statements calling for increased training and implementation of cognitive aids in health care.73,89,90 
Much remains to be done, however, to include routine education on the use of cognitive aids in most training curricula. For instance, while simulation experiences are included as a program requirement, training on routine use of cognitive aids is not included in the Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements or the Anesthesiology Milestones,91,92  even though the science shows clear benefit. Recent research has shown that brief operating room–based team simulations can increase familiarity with and likelihood to use cognitive aids.93  Furthermore, recent data have demonstrated that targeted education and implementation efforts, including structured use during simulation, are highly associated with self-reported use of cognitive aids in actual crisis situations.94,95  This is particularly important because studies have shown that strict adherence to practice guidelines and completion of all checklist items are associated with improved clinical outcomes.96,97  It is encouraging that national societies and major journals are publishing cognitive aids and recommending their routine use for managing critical events.
Nonetheless, more is needed. In order to have a truly ideal cognitive aid, not only must the tool itself have the three qualities listed above, but also there must be repeated familiarization and practice with the cognitive aid within the team environment, and the aid must be accessible within the context of education and clinical practice.72,74  Cognitive aids should be utilized when testing learners on scenarios such as malignant hyperthermia, anaphylaxis, or cardiac arrest. Based on current evidence, it is recommended that routine education be provided on the use of cognitive aids and that cognitive aids be referred to during the management of a life-threatening event, whether simulated or real. Given the current state of the evidence, it is incumbent upon policy makers, program directors, and faculty educators to make routine education on, and use of, cognitive aids a reality.98 
E-learning is defined as learning utilizing electronic technologies to access internet or intranet-based resources outside of a traditional classroom.99,100  In practice, e-learning includes any digital content taught and distributed physically or online, such as web-based tutorials, virtual patients, and discussion boards.99,101 
The advantages of e-learning in medical education are clear. E-learning offers greater flexibility in terms of time and accessibility.99,100,102,103  As long as the learning outcomes of e-learning are comparable to other methods, the flexibility e-learning provides makes it particularly appealing to graduate medical education, as it can help relieve faculty constraints and allocate limited resources more efficiently.102 
In their meta-analysis on the effects of e-learning in the health professions, Cook et al. found that e-learning is associated with large positive effects compared to no intervention on all learning outcomes, including knowledge, skills, behaviors, and patient effects. Compared with traditional noninternet instructional formats, e-learning has positive effects on all educational outcomes as well, although the effects are heterogeneous and marginal.99  More recent studies in primary care and surgical education also reported predominantly positive learner perceptions and improvement in knowledge and skills as a result of engagement in e-learning compared to traditional methods.100,103  Evidence of the effects of e-learning on long-term knowledge retention are scarce and yet to be validated.100,102 
Despite the general positive impacts on e-learning found in the literature, it is important to understand how and when to use e-learning to leverage its educational value.104  Access to digital content alone is not sufficient for in-depth learning.102  The technology should be used in an appropriate context, targeted to the goal of the curriculum, adapted to the needs of the specific audience, and adhering to principles of educational psychology.102,104,105  Good educators should play active roles as content curators and technology adaptors to evaluate and select the appropriate content materials and technology tools.106  Low learner participation rate also appears to be an issue in some e-learning courses, which points to the importance of incorporating compliance measures into implementation.102  In order to maximize the benefits of e-learning, including increased engagement and self-directedness, focused and immediate feedback, repeated opportunities for improvement, and clear emphasis on areas in need of improvement should be provided to learners.102  The incorporation of spaced education and blending self-paced e-learning with in-class instruction have also proven to be successful in improving learning in medical education.100,102  (Table 2 contains examples of e-learning environments utilized for anesthesiology education.)
Teaching Procedural Skills
Technical skills related to the practice of anesthesiology are numerous and require competence with a wide range of techniques and equipment (table 1). The ACGME Milestones92  are focused on performance of arterial lines, central lines, and regional anesthetics (central and neuraxial). Point-of-care ultrasound and transesophageal echocardiography (TEE) have also emerged as valuable skills in the armamentarium of practicing anesthesiologists. Both the ACGME and ABA have recognized the importance of training residents in these skill sets and have set forward expectations for skills acquisition (ACGME) and subsequent verification of basic working knowledge in these areas (ABA). In addition to TEE and transthoracic echocardiography, lung ultrasound, and imaging of the inferior vena cava including the Focused Assessment with Sonography for Trauma and Focused Assessed Transthoracic Exam are growing in popularity.107,108 
Teaching procedural skills in anesthesiology is challenging due to the complex interplay of knowledge, manual dexterity, and workflow elements, all of which need to be learned and applied in concert. Despite the synergy of training in all these areas simultaneously, most training programs focus solely on manual dexterity. Additionally, skills application is episodic in nature, resulting in loss of facility and requiring reacquisition of skills through repeated training.109  Furthermore, verification of proficiency relies on the assumption that participation in a fixed number of procedures will result in uniform learner proficiency; best evidence suggests that this approach is fundamentally flawed.110,111  Finally, there are few validated instruments for procedural skills assessment; most are simple checklists. There is a need for validated instruments that compare the performance of the trainee not only against a list of steps, but also against objective target metrics of experts in order to verify competence.112–115 
Techniques Utilized
Direct Observation.
The most common approach to teaching procedural skills involves direct observation of skills by senior physicians, which is best facilitated by feedback (table 1). There are a variety of suggested approaches to providing feedback, including real-time, delayed, or video-based techniques.116–118  While immediate feedback to improve an element of performance may be provided effectively during the procedure, summative feedback (feedback on overall performance) should be provided later to avoid distraction and increasing cognitive load on the learner.119,120  Checklists of required elements can also be incorporated into review sessions to enhance the feedback quality to learners.121,122 
Partial task training has become a mainstay of procedural training (table 1). It is highly effective as a teaching modality and allows for low-stakes training of invasive procedures.123–126  Task training may be done with either high- or low-fidelity models.126,127  Low-fidelity models are useful for novices,128,129  but higher-fidelity models are often required to engage experienced learners and to assess communication skills and procedural team dynamics.130  Hybrid approaches, incorporating both live actors and procedural trainers, have also been successful in teaching communication and procedures simultaneously.131  Task-trainer–based learning is most effective when supervised by an experienced educator who can provide feedback to improve performance.121,122  Limitations of task trainers can include availability of adequate resources to train learners simultaneously, expense, and availability of teaching staff. Incorporation of small group teaching, use of low-fidelity models where appropriate, inexpensive three-dimensional printing of models,126,132  and peer teaching133  can ameliorate some of these issues. Virtual reality and augmented reality, where additional visual cues are overlaid on the physical environment to enhance learning, are both popular and proving increasingly effective as a technique for teaching procedural skills.134–140 
Live models and virtual reality simulators represent the dominant methods of teaching TEE and point-of-care ultrasound, supplemented by live patient encounters. Live models can help with teaching image acquisition skills for transthoracic echocardiography and lung ultrasound but are typically limited to healthy volunteers and thus cannot demonstrate complicating medical issues or show abnormal anatomy. They also require set teaching times, based upon live models’ availability, and therefore result in limited exposures for learners. In contrast, simulators can be made available on flexible schedules and programed to show abnormal findings.141,142 
Theory-grounded Approaches to Skills Training
Mastery Learning with Deliberate Practice.
Mastery learning entails achievement of a high set standard for all learners regardless of the time and effort required to achieve these levels.143  If learners do not achieve these goals, they must retrain until they do so without limitations of time on training; standards are typically set by experts with consideration for best evidence and data.144  Deliberate practice is the pursuit of techniques known to augment performance above rote repetition. It is characterized by a desire of the learner to focus on the task and build on existing knowledge to formulate discrete goals requiring little instruction. Immediate feedback and performance data should also be provided as part of this process.145  Deliberate practice and reflection with mentorship allow for more rapid and comprehensive acquisition of skills and knowledge. Mastery learning and deliberate practice are synergistic approaches and represent the ideal framework to apply to simulation-based technical skills teaching.143  Mastery learning is superior to conventional techniques in terms of the degree of technical skill attained and compliance with required steps to ensure safety and reduce complications.146–148  A recent best evidence review of five basic anesthesiology procedures (provision of general anesthesia, airway management, spinal anesthesia, epidural anesthesia, and central line placement) suggests enhanced patient outcomes and decreased complications with mastery-based learning, but mixed outcomes in terms of long-term retention and skill transfer.149  Mastery learning concepts are also being applied to point-of-care ultrasound training in anesthesiology, which will be discussed in detail later.111 
Workflow and Communication Skills.
Workflow and communication skills have been increasingly recognized as important to ensure that learners can practically apply skills in the clinical environment. Traditionally taught through the “hands-on” approach in the clinical environment, they should be included as part of teaching in the simulated environment in order to ensure consistent and safe skills performance.150,151 
Cognitive Load and Task Complexity.
Cognitive load theory suggests that, as working memory has limited capacity, more complex tasks take up increased amounts of that working memory. The more working memory required to complete a task, the more difficult it becomes to learn or accomplish at a consistently high level.120  Learner, patient, and supervisor features are associated with different types of cognitive load during procedural skills training.118–120,152,153  Optimizing factors impacting cognitive load (such as reducing fatigue and enhancing supervision and feedback) can improve initial learning.118  Repetitive simulation-based learning can reduce cognitive load and improve performance.152,153 
Making sure that skills are transferrable to the clinical realm requires different strategies than initial skill training. While skills are initially learned better by reducing variability and providing feedback, they also need to be practiced in situations with higher variability.119  Thus, a curriculum with graduated complexity is advisable.142,153 
Spaced Learning.
The spacing effect applies not only to knowledge retention but also to psychomotor skill development, as skills predictably decay over time if not reinforced. Students will lose technical skills between 6 and 12 weeks after training without further practice.109  It is unclear what training interval will keep trainees from losing skills, but more frequent training results in more rapid skills acquisition.154  Distributed practice over time has been used to improve learning of surgical and resuscitative skills.35,155,156  Both knowledge and skill acquisition are dependent on memory consolidation, and spacing allows this to occur.157  The repetitive practice of several separate but related skills, as opposed to practicing one skill at a time until “mastery,” may also improve the learning of psychomotor skills.158 
Retrieval Practice.
The principle of retrieval practice can be applied to psychomotor skills and procedures that are retrieved from memory.159  However, it is important to emphasize that retrieval practice is based on the initial encoding or learning of the information prior to testing. Repeated testing such as an Objective Structured Clinical Exam may produce more favorable skills retention than repeated practice.36  In general, the type of retrieval that is practiced during learning should be a good match for the way in which the information will need to be retrieved and used in the future whether that is a written test, oral examination, mannequin simulation scenario, or standardized patient encounter (Objective Structured Clinical Exam).
Metrics for TEE and Point-of-Care Ultrasound
TEE and point-of-care ultrasound represent excellent examples of how innovative learning programs and metrics should be rigorously applied and assessed to gauge progress toward mastery. For live model sessions, metrics are typically limited to checklists of task completion, assessment of quality of images obtained, or time spent acquiring images. While useful, they do not convey the entire range of skills required. Simulation-based metrics include the total path length taken to achieve the image, numbers of starts and stops, and latency/processing delay before beginning probe motion.160  Multiple studies in TEE and transthoracic echocardiography training have demonstrated similar patterns of progression across a range of learner cohorts and disciplines.112,113,161  The growing ability to compare images of learners to those of experts quantitatively enhances the objectivity of image quality assessment and eliminates the need for human raters.115 
Integrated teaching and testing paradigms have been essential to the assessment of skills, knowledge, and workflow for point-of-care ultrasound and TEE.107,142,161–163  Most incorporate multimodal elements of live workshops, simulation, online flipped classroom, and classroom-based learning. Testing takes the form of written exams, checklists, and analysis of simulator based motion data.142,160–162  Similar to other areas, Objective Structured Clinical Exam–type exams focused on demonstrating skills, workflow, interpretation of images, and diagnoses are proving useful to assess ability of learners to apply these skills.163,164 
The key question of transferability to the clinical setting for TEE and point-of-care ultrasound skills has been addressed via several approaches. Assessment of image quality and exam time in live patients at the end of a simulator-based course showed moderate quality images for trainees completing the course.112,113  Another approach involved course participants completing exams overread by experts, which suggested improvement in patient-level outcomes following the course.165  A more recent approach integrated a novel informatics-based solution to compare basic reports generated by trainees to elements of more complex reports generated by experts and showed excellent agreement following course completion.114  While more work needs to be done, these studies suggest that skills learned outside the clinical arena can be transferred to point-of-care ultrasound or TEE in live patients.
Teaching Professionalism
The ACGME requires programs to provide instruction and assessment in professionalism, which is more challenging than imparting knowledge and evaluating technical and clinical skills.166–168  One major educational challenge is the lack of universal agreement on the definition of, and criteria for, medical professionalism.167,169,170  Professionalism can be conceptualized differently based on virtues, personal attributes, behaviors, and identity formation processes.171–174  Some see professionalism as a unidimensional entity while others view it as a construct that encompasses everything a doctor needs to fulfill the job.171  Some think that professionalism is a commitment to lifelong self-directed learning for improvement, demonstrating compassion and respect to patients, yet some frame professionalism under a legal and regulatory model.171,172  It is a concept in flux.175  Professionalism is essentially unteachable from the perspective of those who define professionalism as an attitude.167,168  The mainstay for teaching and assessing medical professionalism has focused on professionalism as competencies that can be observed and described by behaviors, traits, and roles.167,172,174  According to the ACGME milestone framework, professionalism competencies involve strong interpersonal and communication skills as well as commitment to, and care of, others and oneself, which are separated from the medical knowledge competency.176  Nonetheless, some scholars believe that specialty expertise, including medical knowledge and clinical skills, is a key component of professionalism and that it is artificial to differentiate between medical knowledge and professionalism.168 
Another challenge in teaching professionalism is the difficulty in standardizing a formal curriculum, as much of professionalism is learned through “hidden curricula,” the informal curricula that cultivate alternative norms, values, attitudes, and domains of learning.169,170,172,177  Sometimes formal and hidden curricula contradict each other and create confusion, misunderstanding, and even cynicism in learners.172  Diversity in the perception of professionalism due to generational and cultural differences also add to the challenge in the design, implementation, and evaluation of a curriculum.172  In their 2013 systematic review, Birden et al. identified over 200 papers describing methods to teach professionalism, but few studies reported evaluation data.167  The most commonly adopted strategies in teaching professionalism were role modeling and guided personal reflections.167 
Role Play
Role play, a form of active learning, can be used to teach communication and professionalism. In role play, the learners act as themselves or a scripted character, typically a patient or clinician. The amount of detail involved in the script that the learner follows can vary. The role can be entirely scripted, improvised based on a brief description, or somewhere in between.178  Using the concept broadly, role play can be viewed as a form of simulation in which the learners play themselves in a scenario with a standardized actor. However, for our purposes, the definition will be more specific to the replacement of standardized actors, allowing learners to experience the perspectives of other team members (e.g., surgeon, nurse, patient) while simultaneously decreasing the resource utilization of the learning experience by reducing the need for standardized actors.
To our knowledge, there are no data supporting role play in graduate medical education. It is suggested that role play can increase learners’ awareness of their usage of body language, teach empathy, and provide insight into personal communication skills, fears, and emotions.179  A study done with medical students comparing the use of standardized patients to the use of learners playing the role of the patient found that both methods improved learners’ communication skills, but learners felt they had a better understanding of the patient perspective when they had the opportunity to play the patient themselves.180 
There may be some stigma associated with the idea of role play in medical education. One study showed that prior to participating in a role play exercise, only 77% of medical students thought it would provide insight into their, and others’, behaviors and allow them the opportunity to practice skills. The remaining students expressed feelings of anxiety, fear, intimidation, and embarrassment. However, after participating in role play, 99% of these students thought it was helpful in allowing them to practice their skills in a safe environment with immediate feedback.178  In order to provide a nonthreatening environment, it is critical to clarify the learning objectives in advance both to define ground rules and to allow for adequate time for preparation and role play.181  It is also recommended to actively include all learners in the role play exercise. This can be done by breaking into groups of three in which one person is the doctor, one the patient or other health professional, and the third an observer whose role is to provide feedback.182 
The assessment of professionalism as a complex, multidimensional construct entails a combination of tools171  (table 1). In their systematic review, Wilkinson et al. identified nine categories of assessment tools that have been used separately or jointly, including (1) observed clinical encounters, (2) collated views of coworkers, (3) records of incidents of unprofessionalism, (4) critical incident reports, (5) simulation, (6) paper-based tests, (7) patients’ opinions, (8) global views of supervisors, and (9) self-administrated rating scales.171  More recent techniques, such as the use of simulated data and situational judgment tests, may also be useful to assess and teach professionalism, especially if the concept is defined by observable behaviors.183,184 
There is recognition of the importance of professionalism teaching within anesthesiology, but research has been limited.185,186  Some programs have focused on curricular development, while others aimed to promote enhanced feedback to reinforce professional behavior.185,187,188  Further research in graduate medical education is needed in this domain.
Experiential Learning: Knowledge, Skills, and Professionalism
Anesthesiologists have been using mannequin-based simulators in nonpatient care settings to enhance knowledge and skills of learners for decades.189,190  Since simulated clinical experience is a core requirement for anesthesiology residency training programs, many anesthesiologists are familiar with the benefits of learning novel concepts, or solidifying previously learned skills, in a low-stakes environment like a simulation center.91  Traditionally, residency programs focused their simulation-based educational efforts on the core competency areas of medical knowledge, patient care, and procedural skills, by teaching basic anesthesia principles, responses to anesthesia-related emergencies (e.g., crisis resource management), and “trouble-shooting” common anesthetic issues.191  However, simulation-based Objective Structured Clinical Exams are uniquely designed to provide both formative and summative assessment of all core competency areas including professionalism, interpersonal and communication skills, and practice-based learning and improvement.
An Objective Structured Clinical Exam consists of a series of brief stations or clinical scenarios, where a variety of technical and nontechnical tasks are completed. Examiners either complete an objective checklist of behaviors or provide a global rating score to evaluate examinees based upon observing their performances within each individual station. With evaluations from a variety of skill areas, completed by different examiners at each station, Objective Structured Clinical Exams provide a comprehensive evaluation of an examinee.192  The use of Objective Structured Clinical Exams in “high-stakes” situations (e.g., board certification) for anesthesiologists has been occurring for several years. In the United Kingdom, the Royal College of Anaesthetists (London) delivers an Objective Structured Clinical Exam of 17 brief stations where examinees are evaluated in a variety of skill areas such as history taking, physical examinations, x-ray film interpretation, electrocardiogram interpretation, resuscitation, and safety checks of anesthesia equipment.193  Alternatively, the Israeli Board of Anesthesiology (Jerusalem) delivers an Objective Structured Clinical Exam for board certification where examinees are exposed to five longer mannequin-based scenarios focusing on trauma and crisis management, resuscitation, mechanical ventilation, and regional anesthesia.194  The ABA recently implemented an Objective Structured Clinical Exam as part of the APPLIED examination.195  The ABA recognized that the written and oral examinations assessed lower levels of competence and they were not assessing higher level competencies required in clinical practice such as communication, professionalism, and certain technical skills.196 
High-stakes implications and summative assessments are not the only use of an Objective Structured Clinical Exam. Rather, an Objective Structured Clinical Exam can provide formative feedback to learners, educators, and an educational program as a whole (e.g., residency program).197  For instance, an anesthesiology department may decide to implement a new teaching module into their educational program on an emerging topic relevant to clinical practice (e.g., point-of-care ultrasound). In order for the department to evaluate the effectiveness of implementing the program, an Objective Structured Clinical Exam could be administered to assess examinees’ performance of the skills taught in the program. The result from the Objective Structured Clinical Exam would then give feedback to learners regarding their personal deficits, educators on areas to focus their training, and the department regarding resource utilization and overall efficacy of the training program. Objective Structured Clinical Exams may also play a role in evaluating ACGME subcompetency milestone achievement.92  Evidence is emerging that a resident’s performance in an Objective Structured Clinical Exam is associated with both their experience level and performance on standard clinical evaluations.198  As a result, an Objective Structured Clinical Exam could be used as an alternative option to assess competence on certain milestones that are either hard to capture (e.g., “Patient Care 5 – Crisis Management” when there is a moral imperative for the most competent physicians to intervene in the crisis) or those in which an opportunity for assessment is infrequent or unpredictable (e.g., “Professionalism 2 – Honesty, Integrity, and Ethical Behavior” when an opportunity to evaluate a resident addressing an ethical issue may not arise).
As part of the ABA Maintenance of Certification in Anesthesiology process, board-certified anesthesiologists have been participating in simulation courses to satisfy a portion of Part IV of Maintenance of Certification in Anesthesiology 2.0. During Maintenance of Certification in Anesthesiology simulation courses, anesthesiologists participate in high-fidelity scenarios focusing on management of difficult patient care situations, with emphasis placed on teamwork and communication.199  The course allows anesthesiologists to reflect on their experiences and subsequently propose practice improvement plans. Follow-up evaluations from the simulation course demonstrate that greater than 90% of participants successfully implement some or all of the planned practice improvements.200  One drawback of the Maintenance of Certification in Anesthesiology simulation program is the logistical challenge presented to anesthesiologists regarding the need to take time from their practice to travel to simulation centers. However, in the summer of 2017, the American Society of Anesthesiologists and Canadian Aviation Electronics Healthcare released a novel suite of screen-based high-fidelity simulation scenarios known as Anesthesia SimSTAT (table 2). The simulation scenarios are computer-based and set in a gaming environment with the same goal of improving performance in managing anesthesia-related emergencies. The scenarios allow the anesthesiologist to interact with realistic diagnostic and monitoring equipment in a virtual three-dimensional operating room. Upon completion of the scenarios, anesthesiologists are provided with data-based feedback on their performance, identification of strengths, and opportunities for improvement.
Simulation is also being used to address burnout, which is a significant area of concern for anesthesiologists.201  However, instead of using simulation to train anesthesiologists regarding burnout, nonmedical persons involved in the social support of anesthesia providers are being educated on the tasks, expectations, and stressors anesthesiologists are exposed to in an attempt to enhance understanding.202  The simulation exercises consist of hands-on activities (e.g., use of a task trainer to learn how to perform a procedure such as an epidural) and observation and debriefing of a high-fidelity simulation of a critical event in the operating room. As a result of these experiences, anesthesiologists may have an improved ability to communicate with their social support persons regarding their daily activities, which may lead to improved social relatedness and subsequently enhanced wellness of providers.
Evidence-based Education in Anesthesiology: Challenges and Future
Many educational theories and methods have been studied outside of graduate medical education. However, there are major differences between students outside of graduate medical education and anesthesiology residents.11  Most students utilize a classroom-based learning environment in which they are expected to prepare ahead of time. They often have dedicated study time, and their primary role is that of a learner with minimal other responsibilities. Our residents learn on the job while taking care of patients, which is their primary role. Study time is limited due to clinical duties. Additionally, whereas most students are evaluated through frequent formal testing, our resident learners undergo less frequent, but higher-stakes, examinations.203 
Although there is great need for additional educational research in graduate medical education, it is inherently challenging. Although it is often critiqued in the same way that one might critically assess basic science research, it is intrinsically different. Sample sizes are often small, due to the size of residency programs. It is difficult to control for confounding variables in a learning environment, and there may be disagreement on which variables are important.11  Additionally, learner randomization may be unethical or impractical.204  Our definitive goal is to improve residents’ application of knowledge, problem solving, and ultimately patient care;12  but it can be difficult to assess these endpoints. It is often logistically easiest to utilize instruments comprised of multiple-choice exams; these questions will not be successful in assessing the higher levels of Miller’s pyramid21  or application of adaptive learning principles.12 
Multiinstitutional educational studies in anesthesiology are needed to help develop and evaluate evidence that will optimize learning of knowledge, skills, and attitudes/professionalism in this specialty in the future.
Although there are many ways to educate our anesthesiology learners, we have presented techniques with variable levels of demonstrated efficacy in medical education (fig. 1). Education is an ever-evolving field. Anesthesiologists must continue to develop and implement innovative approaches for their residents’ education. It is vitally important to assess and validate educational methodologies in order to provide evidence-based best practices for teaching knowledge, skills, and professionalism to anesthesiology learners. Figure 1 provides suggestions on how to incorporate some of these novel educational methods at your institution.
Research Support
Support was provided solely from departmental sources.
Competing Interests
The authors declare no competing interests.
Schell, RM, Bowe, EA, Fragneto, RY. Education in anesthesiology should be evidenced-based. Anesth Analg. 2008; 106:1587–8. discussion 1588 [Article] [PubMed]
Desy, J, Busche, K, Cusano, R, Veale, P, Coderre, S, McLaughlin, K . How teachers can help learners build storage and retrieval strength. Med Teach. 2018; 40:407–13 [Article] [PubMed]
Roediger, HL, Pyc, MA . Inexpensive techniques to improve education: Applying cognitive psychology to enhance educational practice. J Appl Res Mem Cogn. 2012; 1:242–8 [Article]
Van Hoof, TJ, Doyle, TJ . Learning science as a potential new source of understanding and improvement for continuing education and continuing professional development. Med Teach. 2018 1–6
Weidman, J, Baker, K . The cognitive science of learning: Concepts and strategies for the educator and learner. Anesth Analg. 2015; 121:1586–99 [Article] [PubMed]
Yeh, DD, Park, YS . Improving learning efficiency of factual knowledge in medical education. J Surg Educ. 2015; 72:882–9 [Article] [PubMed]
Prober, CG, Heath, C . Lecture halls without lectures - A proposal for medical education. N Engl J Med. 2012; 366:1657–9 [Article] [PubMed]
Gleason, BL, Peeters, MJ, Resman-Targoff, BH, Karr, S, McBane, S, Kelley, K, Thomas, T, Denetclaw, TH . An active-learning strategies primer for achieving ability-based educational outcomes. Am J Pharm Educ. 2011; 75:186 [Article] [PubMed]
Wilke, RR . The effect of active learning on student characteristics in a human physiology course for nonmajors. Adv Physiol Educ. 2003; 27:207–23 [Article] [PubMed]
Freeman, S, Eddy, SL, McDonough, M, Smith, MK, Okoroafor, N, Jordt, H, Wenderoth, MP . Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A. 2014; 111:8410–5 [Article] [PubMed]
Michael, J . Where’s the evidence that active learning works? Adv Physiol Educ. 2006; 30:159–67 [Article] [PubMed]
Cutrer, WB, Miller, B, Pusic, MV, Mejicano, G, Mangrulkar, RS, Gruppen, LD, Hawkins, RE, Skochelak, SE, Moore, DEJr . Fostering the development of master adaptive learners: A conceptual model to guide skill acquisition in medical education. Acad Med. 2017; 92:70–5 [Article] [PubMed]
Sait, MS, Siddiqui, Z, Ashraf, Y . Advances in medical education and practice: Student perceptions of the flipped classroom. Adv Med Educ Pract. 2017; 8:317–20 [Article] [PubMed]
Wong, TH, Ip, EJ, Lopes, I, Rajagopalan, V . Pharmacy students’ performance and perceptions in a flipped teaching pilot on cardiac arrhythmias. Am J Pharm Educ. 2014; 78:185 [Article] [PubMed]
McLaughlin, JE, Rhoney, DH . Comparison of an interactive e-learning preparatory tool and a conventional downloadable handout used within a flipped neurologic pharmacotherapy lecture. Curr Pharm Teach Learn. 2015; 7:12–9 [Article]
Pierce, R, Fox, J . Vodcasts and active-learning exercises in a “flipped classroom” model of a renal pharmacotherapy module. Am J Pharm Educ. 2012; 76:196 [Article] [PubMed]
Chen, F, Lui, AM, Martinelli, SM . A systematic review of the effectiveness of flipped classrooms in medical education. Med Educ. 2017; 51:585–97 [Article] [PubMed]
Bonnes, SL, Ratelle, JT, Halvorsen, AJ, Carter, KJ, Hafdahl, LT, Wang, AT, Mandrekar, JN, Oxentenko, AS, Beckman, TJ, Wittich, CM . Flipping the quality improvement classroom in residency education. Acad Med. 2017; 92:101–7 [Article] [PubMed]
Riddell, J, Jhun, P, Fung, CC, Comes, J, Sawtelle, S, Tabatabai, R, Joseph, D, Shoenberger, J, Chen, E, Fee, C, Swadron, SP . Does the flipped classroom improve learning in graduate medical education? J Grad Med Educ. 2017; 9:491–6 [Article] [PubMed]
Martinelli, SM, Chen, F, DiLorenzo, AN, Mayer, DC, Fairbanks, S, Moran, K, Ku, C, Mitchell, JD, Bowe, EA, Royal, KD, Hendrickse, A, VanDyke, K, Trawicki, MC, Rankin, D, Guldan, GJ, Hand, W, Gallagher, C, Jacob, Z, Zvara, DA, McEvoy, MD, Schell, RM . Results of a flipped classroom teaching approach in anesthesiology residents. J Grad Med Educ. 2017; 9:485–90 [Article] [PubMed]
Miller, GE . The assessment of clinical skills/competence/performance. Acad Med. 1990; 65(9 suppl):S63–7 [Article] [PubMed]
Mehay, R . The essential handbook for GP training and education. 2012. London: Radcliffe Publishing
Morton, DA, Colbert-Getz, JM . Measuring the impact of the flipped anatomy classroom: The importance of categorizing an assessment by Bloom’s taxonomy. Anat Sci Educ. 2017; 10:170–5 [Article] [PubMed]
Gillispie, V . Using the flipped classroom to bridge the gap to Generation Y. Ochsner J. 2016; 16:32–6 [PubMed]
Benè, KL, Bergus, G . When learners become teachers: A review of peer teaching in medical student education. Fam Med. 2014; 46:783–7 [PubMed]
Crouch, CH, Mazur, E . Peer instruction: Ten years of experience and results. Am J Phys. 2001; 69:970–7 [Article]
Gregory, A, Walker, I, McLaughlin, K, Peets, AD . Both preparing to teach and teaching positively impact learning outcomes for peer teachers. Med Teach. 2011; 33:e417–22 [Article] [PubMed]
Lin, JA, Farrow, N, Lindeman, BM, Lidor, AO . Impact of near-peer teaching rounds on student satisfaction in the basic surgical clerkship. Am J Surg. 2017; 213:1163–5 [Article] [PubMed]
Akl, EA, Pretorius, RW, Sackett, K, Erdley, WS, Bhoopathi, PS, Alfarah, Z, Schünemann, HJ . The effect of educational games on medical students’ learning outcomes: A systematic review: BEME Guide No 14. Med Teach. 2010; 32:16–27 [Article] [PubMed]
Akl, EA, Kairouz, VF, Sackett, KM, Erdley, WS, Mustafa, RA, Fiander, M, Gabriel, C, Schunemann, H . Educational games for health professionals. Cochrane Database Syst Rev. 2013 CD006411
Shiroma, PR, Massa, AA, Alarcon, RD . Using game format to teach psychopharmacology to medical students. Med Teach. 2011; 33:156–60 [Article] [PubMed]
O’Leary, S, Diepenhorst, L, Churley-Strom, R, Magrane, D . Educational games in an obstetrics and gynecology core curriculum. Am J Obstet Gynecol. 2005; 193:1848–51 [Article] [PubMed]
Telner, D, Bujas-Bobanovic, M, Chan, D, Chester, B, Marlow, B, Meuser, J, Rothman, A, Harvey, B . Game-based versus traditional case-based learning: Comparing effectiveness in stroke continuing medical education. Can Fam Physician. 2010; 56:e345–51 [PubMed]
Webb, TP, Simpson, D, Denson, S, Duthie, EJr . Gaming used as an informal instructional technique: Effects on learner knowledge and satisfaction. J Surg Educ. 2012; 69:330–4 [Article] [PubMed]
Spruit, EN, Band, GPH, van der Heijden, KB, Hamming, JF . The effects of spacing, naps, and fatigue on the acquisition and retention of laparoscopic skills. J Surg Educ. 2017; 74:530–8 [Article] [PubMed]
Sennhenn-Kirchner, S, Goerlich, Y, Kirchner, B, Notbohm, M, Schiekirka, S, Simmenroth, A, Raupach, T . The effect of repeated testing vs repeated practice on skills learning in undergraduate dental education. Eur J Dent Educ. 2018; 22:e42–7 [Article] [PubMed]
Cepeda, NJ, Pashler, H, Vul, E, Wixted, JT, Rohrer, D . Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychol Bull. 2006; 132:354–80 [Article] [PubMed]
Karpicke, JD, Bauernschmidt, A . Spaced retrieval: Absolute spacing enhances learning regardless of relative spacing. J Exp Psychol Learn Mem Cogn. 2011; 37:1250–7 [Article] [PubMed]
Moulton, CA, Dubrowski, A, Macrae, H, Graham, B, Grober, E, Reznick, R . Teaching surgical skills: What kind of practice makes perfect?: A randomized, controlled trial. Ann Surg. 2006; 244:400–9 [PubMed]
Blazek, MC, Dantz, B, Wright, MC, Fiedorowicz, JG . Spaced learning using emails to integrate psychiatry into general medical curriculum: Keep psychiatry in mind. Med Teach. 2016; 38:1049–55 [Article] [PubMed]
Dobson, JL, Perez, J, Linderholm, T . Distributed retrieval practice promotes superior recall of anatomy information. Anat Sci Educ. 2017; 10:339–47 [Article] [PubMed]
Kerfoot, BP, Baker, HE, Koch, MO, Connelly, D, Joseph, DB, Ritchey, ML . Randomized, controlled trial of spaced education to urology residents in the United States and Canada. J Urol. 2007; 177:1481–7 [Article] [PubMed]
Kerfoot, BP, Brotschi, E . Online spaced education to teach urology to medical students: A multi-institutional randomized trial. Am J Surg. 2009; 197:89–95 [Article] [PubMed]
Kerfoot, BP, DeWolf, WC, Masser, BA, Church, PA, Federman, DD . Spaced education improves the retention of clinical knowledge by medical students: A randomised controlled trial. Med Educ. 2007; 41:23–31 [Article] [PubMed]
Smeds, MR, Thrush, CR, Mizell, JS, Berry, KS, Bentley, FR . Mobile spaced education for surgery rotation improves National Board of Medical Examiners scores. J Surg Res. 2016; 201:99–104 [Article] [PubMed]
Brown, PC, Roediger, HLI, McDaniel, MA . Make It Stick: The Science of Successful Learning. 2014. Cambridge (MA): Belknap Press of Harvard University Press
Rohrer, D, Taylor, K . The shuffling of mathematics problems improves learning. Instr Sci. 2007; 35:481–98 [Article]
Kang, SHK, Pashler, H . Learning painting styles: Spacing is advantageous when it promotes discriminative contrast. Appl Cogn Psychol. 2012; 26:97–103 [Article]
Zulkiply, N, Burt, JS . The exemplar interleaving effect in inductive learning: Moderation by the difficulty of category discriminations. Mem Cognit. 2013; 41:16–27 [Article] [PubMed]
Rozenshtein, A, Pearson, GD, Yan, SX, Liu, AZ, Toy, D . Effect of massed versus interleaved teaching method on performance of students in radiology. J Am Coll Radiol. 2016; 13:979–84 [Article] [PubMed]
Pani, JR, Chariker, JH, Naaz, F . Computer-based learning: Interleaving whole and sectional representation of neuroanatomy. Anat Sci Educ. 2013; 6:11–8 [Article] [PubMed]
Birnbaum, MS, Kornell, N, Bjork, EL, Bjork, RA . Why interleaving enhances inductive learning: The roles of discrimination and retrieval. Mem Cognit. 2013; 41:392–402 [Article] [PubMed]
Karpicke, JD, Roediger, HL3rd . The critical importance of retrieval for learning. Science. 2008; 319:966–8 [Article] [PubMed]
Roediger, HL3rd, Butler, AC . The critical role of retrieval practice in long-term retention. Trends Cogn Sci. 2011; 15:20–7 [Article] [PubMed]
Larsen, DP, Butler, AC, Roediger, HL3rd . Test-enhanced learning in medical education. Med Educ. 2008; 42:959–66 [Article] [PubMed]
Phelps, RP . The effect of testing on student achievement, 1910–2010. Int J Test. 2012; 12:21–43 [Article]
Green, ML, Moeller, JJ, Spak, JM . Test-enhanced learning in health professions education: A systematic review: BEME Guide No. 48. Med Teach. 2018; 40:337–50 [Article] [PubMed]
Larsen, DP, Butler, AC, Roediger, HL3rd . Repeated testing improves long-term retention relative to repeated study: A randomised controlled trial. Med Educ. 2009; 43:1174–81 [Article] [PubMed]
Rowland, CA . The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychol Bull. 2014; 140:1432–63 [Article] [PubMed]
Karpicke, JD, Blunt, JR . Retrieval practice produces more learning than elaborative studying with concept mapping. Science. 2011; 331:772–5 [Article] [PubMed]
Raupach, T, Andresen, JC, Meyer, K, Strobel, L, Koziolek, M, Jung, W, Brown, J, Anders, S . Test-enhanced learning of clinical reasoning: A crossover randomised trial. Med Educ. 2016; 50:711–20 [Article] [PubMed]
Larsen, DP . Picking the right dose: The challenges of applying spaced testing to education. J Grad Med Educ. 2014; 6:349–50 [Article] [PubMed]
Butler, AC, Roediger, HL3rd . Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Mem Cognit. 2008; 36:604–16 [Article] [PubMed]
Butler, AC . Repeated testing produces superior transfer of learning relative to repeated studying. J Exp Psychol Learn Mem Cogn. 2010; 36:1118–33 [Article] [PubMed]
Halamish, V, Bjork, RA . When does testing enhance retention? A distribution-based interpretation of retrieval as a memory modifier. J Exp Psychol Learn Mem Cogn. 2011; 37:801–12 [Article] [PubMed]
Carpenter, SK . Testing enhances the transfer of learning. Curr Dir Psychol Sci. 2012; 21:279–83 [Article]
van Eersel, GG, Verkoeijen, PP, Povilenaite, M, Rikers, R . The testing effect and far transfer: The role of exposure to key information. Front Psychol. 2016; 7:1977 [Article] [PubMed]
Kromann, CB, Jensen, ML, Ringsted, C . The effect of testing on skills learning. Med Educ. 2009; 43:21–7 [Article] [PubMed]
Babcock, WW . Resuscitation during anesthesia. Anesth Analg. 1924; 3:208–13 [Article]
Emergency Manuals Implementation Collaborative. Available at: Accessed February 1, 2019.
Marshall, S . The use of cognitive aids during emergencies in anesthesia: A review of the literature. Anesth Analg. 2013; 117:1162–71 [Article] [PubMed]
Wen, LY, Howard, SK . Value of expert systems, quick reference guides and other cognitive aids. Curr Opin Anaesthesiol. 2014; 27:643–8 [Article] [PubMed]
Webster, CS . Checklists, cognitive aids, and the future of patient safety. Br J Anaesth. 2017; 119:178–81 [Article] [PubMed]
Goldhaber-Fiebert, SN, Howard, SK . Implementing emergency manuals: Can cognitive aids help translate best practices for patient care during acute events? Anesth Analg. 2013; 117:1149–61 [Article] [PubMed]
De Bie, AJR, Nan, S, Vermeulen, LRE, Van Gorp, PME, Bouwman, RA, Bindels, AJGH, Korsten, HHM . Intelligent dynamic clinical checklists improved checklist compliance in the intensive care unit. Br J Anaesth. 2017; 119:231–8 [Article] [PubMed]
McEvoy, MD, Hand, WR, Stoll, WD, Furse, CM, Nietert, PJ . Adherence to guidelines for the management of local anesthetic systemic toxicity is improved by an electronic decision support tool and designated “Reader.” Reg Anesth Pain Med. 2014; 39:299–305 [Article] [PubMed]
Field, LC, McEvoy, MD, Smalley, JC, Clark, CA, McEvoy, MB, Rieke, H, Nietert, PJ, Furse, CM . Use of an electronic decision support tool improves management of simulated in-hospital cardiac arrest. Resuscitation. 2014; 85:138–42 [Article] [PubMed]
Burden, AR, Carr, ZJ, Staman, GW, Littman, JJ, Torjman, MC . Does every code need a “reader?” Improvement of rare event management with a cognitive aid “reader” during a simulated emergency: A pilot study. Simul Healthc. 2012; 7:1–9 [Article] [PubMed]
Arriaga, AF, Bader, AM, Wong, JM, Lipsitz, SR, Berry, WR, Ziewacz, JE, Hepner, DL, Boorman, DJ, Pozner, CN, Smink, DS, Gawande, AA . Simulation-based trial of surgical-crisis checklists. N Engl J Med. 2013; 368:246–53 [Article] [PubMed]
Lipps, J, Meyers, L, Winfield, S, Durda, M, Yildiz, V, Kushelev, M . Physiologically triggered digital cognitive aid facilitates crisis management in a simulated operating room: A randomized controlled study. Simul Healthc. 2017; 12:370–6 [PubMed]
Low, D, Clark, N, Soar, J, Padkin, A, Stoneham, A, Perkins, GD, Nolan, J . A randomised control trial to determine if use of the iResus© application on a smart phone improves the performance of an advanced life support provider in a simulated medical emergency. Anaesthesia. 2011; 66:255–62 [Article] [PubMed]
Lelaidier, R, Balança, B, Boet, S, Faure, A, Lilot, M, Lecomte, F, Lehot, JJ, Rimmelé, T, Cejka, JC . Use of a hand-held digital cognitive aid in simulated crises: The MAX randomized controlled trial. Br J Anaesth. 2017; 119:1015–21 [Article] [PubMed]
Nelson McMillan, K, Rosen, MA, Shilkofski, NA, Bradshaw, JH, Saliski, M, Hunt, EA . Cognitive aids do not prompt initiation of cardiopulmonary resuscitation in simulated pediatric cardiopulmonary arrests. Simul Healthc. 2018; 13:41–6 [Article] [PubMed]
Nelson, KL, Shilkofski, NA, Haggerty, JA, Saliski, M, Hunt, EA . The use of cognitive AIDS during simulated pediatric cardiopulmonary arrests. Simul Healthc. 2008; 3:138–45 [Article] [PubMed]
McEvoy, MD, Hand, WR, Stiegler, MP, DiLorenzo, AN, Ehrenfeld, JM, Moran, KR, Lekowski, R, Nunnally, ME, Manning, EL, Shi, Y, Shotwell, MS, Gupta, RK, Corey, JM, Schell, RM . A smartphone-based decision support tool improves test performance concerning application of the guidelines for managing regional anesthesia in the patient receiving antithrombotic or thrombolytic therapy. Anesthesiology. 2016; 124:186–98 [Article] [PubMed]
Hand, WR, Bridges, KH, Stiegler, MP, Schell, RM, DiLorenzo, AN, Ehrenfeld, JM, Nietert, PJ, McEvoy, MD . Effect of a cognitive aid on adherence to perioperative assessment and management guidelines for the cardiac evaluation of noncardiac surgical patients. Anesthesiology. 2014; 120:1339–49, quiz 134953 [Article] [PubMed]
MacDougall, N, Dong, F, Broussard, L, Comunale, ME . Preventing mistransfusions: An evaluation of institutional knowledge and a response. Anesth Analg. 2018; 126:247–51 [Article] [PubMed]
Russ, S, Rout, S, Sevdalis, N, Moorthy, K, Darzi, A, Vincent, C . Do safety checklists improve teamwork and communication in the operating room? A systematic review. Ann Surg. 2013; 258:856–71 [Article] [PubMed]
Augoustides, JG, Atkins, J, Kofke, WA . Much ado about checklists: Who says I need them and who moved my cheese? Anesth Analg. 2013; 117:1037–8 [Article] [PubMed]
Webster, CS . Cognitive aids, checklists and mental models. Anaesthesia. 2017; 72:1041–2 [Article] [PubMed]
ACGME Program Requirements for Graduate Medical Education in Anesthesiology 2018 edition. Accreditation Council for Graduate Medical Education. Available at: Accessed May 30, 2018
The Anesthesiology Milestone Project, The Accreditation Council for Graduate Medical Education and The American Board of Anesthesiology 2015. Available at: Accessed May 30, 2018.
Goldhaber-Fiebert, SN, Lei, V, Nandagopal, K, Bereknyei, S . Emergency manual implementation: Can brief simulation-based or staff trainings increase familiarity and planned clinical use? Jt Comm J Qual Patient Saf. 2015; 41:212–20 [Article] [PubMed]
Alidina, S, Goldhaber-Fiebert, SN, Hannenberg, AA, Hepner, DL, Singer, SJ, Neville, BA, Sachetta, JR, Lipsitz, SR, Berry, WR . Factors associated with the use of cognitive aids in operating room crises: A cross-sectional study of US hospitals and ambulatory surgical centers. Implement Sci. 2018; 13:50 [Article] [PubMed]
Huang, J, Wu, J, Dai, C, Zhang, X, Ju, H, Chen, Y, Zhang, C, Ye, F, Tan, Y, Zong, Y, Liu, T . Use of emergency manuals during actual critical events in China: A multi-institutional study. Simul Healthc. 2018; 13:253–60 [Article] [PubMed]
McEvoy, MD, Field, LC, Moore, HE, Smalley, JC, Nietert, PJ, Scarbrough, SH . The effect of adherence to ACLS protocols on survival of event in the setting of in-hospital cardiac arrest. Resuscitation. 2014; 85:82–7 [Article] [PubMed]
van Klei, WA, Hoff, RG, van Aarnhem, EE, Simmermacher, RK, Regli, LP, Kappen, TH, van Wolfswinkel, L, Kalkman, CJ, Buhre, WF, Peelen, LM . Effects of the introduction of the WHO “Surgical Safety Checklist” on in-hospital mortality: A cohort study. Ann Surg. 2012; 255:44–9 [Article] [PubMed]
Gaba, DM . Perioperative cognitive aids in anesthesia: What, who, how, and why bother? Anesth Analg. 2013; 117:1033–6 [Article] [PubMed]
Cook, DA, Levinson, AJ, Garside, S, Dupras, DM, Erwin, PJ, Montori, VM . Internet-based learning in the health professions: A meta-analysis. JAMA. 2008; 300:1181–96 [Article] [PubMed]
Tarpada, SP, Hsueh, WD, Gibber, MJ . Resident and student education in otolaryngology: A 10-year update on e-learning. Laryngoscope. 2017; 127:E219–24 [Article] [PubMed]
De Leeuw, RA, Westerman, M, Nelson, E, Ket, JC, Scheele, F . Quality specifications in postgraduate medical e-learning: An integrative literature review leading to a postgraduate medical e-learning model. BMC Med Educ. 2016; 16:168 [Article] [PubMed]
Maertens, H, Madani, A, Landry, T, Vermassen, F, Van Herzeele, I, Aggarwal, R . Systematic review of e-learning for surgical training. Br J Surg. 2016; 103:1428–37 [Article] [PubMed]
Reeves, S, Fletcher, S, McLoughlin, C, Yim, A, Patel, KD . Interprofessional online learning for primary healthcare: Findings from a scoping review. BMJ Open. 2017; 7:e016872 [Article] [PubMed]
Cook, DA . Where are we with web-based learning in medical education? Med Teach. 2006; 28:594–8 [Article] [PubMed]
Taveira-Gomes, T, Ferreira, P, Taveira-Gomes, I, Severo, M, Ferreira, MA . What are we looking for in computer-based learning interventions in medical education? A systematic review. J Med Internet Res. 2016; 18:e204 [Article] [PubMed]
Simpson, D, Marcdante, K, Souza, KH, Anderson, A, Holmboe, E . Job roles of the 2025 medical educator. J Grad Med Educ. 2018; 10:243–6 [Article] [PubMed]
Mitchell, JD, Matyal, R . Bowe, EA, Schell, RM, Dilorenzo, AN . Teaching Transesophageal Echocardiography, Education in Anesthesia: How to Deliver the Best Learning Experience. 20181st edition. Cambridge: Cambridge University Press 121–130
McCormick, TJ, Miller, EC, Chen, R, Naik, VN . Acquiring and maintaining point-of-care ultrasound (POCUS) competence for anesthesiologists. Can J Anaesth. 2018; 65:427–36 [Article] [PubMed]
Fisher, J, Viscusi, R, Ratesic, A, Johnstone, C, Kelley, R, Tegethoff, AM, Bates, J, Situ-Lacasse, EH, Adamas-Rappaport, WJ, Amini, R . Clinical skills temporal degradation assessment in undergraduate medical education. J Adv Med Educ Prof. 2018; 6:1–5 [PubMed]
Barsuk, JH, Cohen, ER, Feinglass, J, McGaghie, WC, Wayne, DB . Residents’ procedural experience does not ensure competence: A research synthesis. J Grad Med Educ. 2017; 9:201–8 [Article] [PubMed]
Jensen, JK, Dyre, L, Jørgensen, ME, Andreasen, LA, Tolsgaard, MG . Simulation-based point-of-care ultrasound training: A matter of competency rather than volume. Acta Anaesthesiol Scand. 2018; 62:811–9 [Article] [PubMed]
Matyal, R, Mitchell, JD, Hess, PE, Chaudary, B, Bose, R, Jainandunsing, JS, Wong, V, Mahmood, F . Simulator-based transesophageal echocardiographic training with motion analysis: A curriculum-based approach. Anesthesiology. 2014; 121:389–99 [Article] [PubMed]
Matyal, R, Montealegre-Gallegos, M, Mitchell, JD, Kim, H, Bergman, R, Hawthorne, KM, O’Halloran, D, Wong, V, Hess, PE, Mahmood, F . Manual skill acquisition during transesophageal echocardiography simulator training of cardiology fellows: A kinematic assessment. J Cardiothorac Vasc Anesth. 2015; 29:1504–10 [Article] [PubMed]
Bick, JS, Wanderer, JP, Myler, CS, Shaw, AD, McEvoy, MD . Standard setting for clinical performance of basic perioperative transesophageal echocardiography: Moving beyond the written test. Anesthesiology. 2017; 126:718–28 [Article] [PubMed]
Matyal, R, Mahmood, F, Knio, ZO, Jones, S, Yeh, L, Amir, R, Bose, R, Mitchell, JD . Evaluation of the quality of transesophageal echocardiography images and verification of proficiency. Echo Res Pract. 2018; 5:89–95 [Article] [PubMed]
Rammell, J, Matthan, J, Gray, M, Bookless, LR, Nesbitt, CI, Rodham, P, Moss, J, Stansby, G, Phillips, AW . Asynchronous unsupervised video-enhanced feedback as effective as direct expert feedback in the long-term retention of practical clinical skills: Randomised trial comparing 2 feedback methods in a cohort of novice medical students. J Surg Educ. 2018; 75:1463–70 [Article] [PubMed]
Phillips, AW, Matthan, J, Bookless, LR, Whitehead, IJ, Madhavan, A, Rodham, P, Porter, ALR, Nesbitt, CI, Stansby, G . Individualised expert feedback is not essential for improving basic clinical skills performance in novice learners: A randomized trial. J Surg Educ. 2017; 74:612–20 [Article] [PubMed]
Sewell, JL, Boscardin, CK, Young, JQ, Ten Cate, O, O’Sullivan, PS . Learner, patient, and supervisor features are associated with different types of cognitive load during procedural skills training: Implications for teaching and instructional design. Acad Med. 2017; 92:1622–31 [Article] [PubMed]
Van Merrienboer, JJG, Kester, L, Paas, F . Teaching complex rather than simple tasks: Balancing intrinsic and germane load to enhance transfer of learning. Appl Cogn Psychol. 2006; 20:343–52 [Article]
van Merriënboer, JJ, Sweller, J . Cognitive load theory in health professional education: Design principles and strategies. Med Educ. 2010; 44:85–93 [Article] [PubMed]
Issenberg, SB, McGaghie, WC, Petrusa, ER, Lee Gordon, D, Scalese, RJ . Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review. Med Teach. 2005; 27:10–28 [Article] [PubMed]
Motola, I, Devine, LA, Chung, HS, Sullivan, JE, Issenberg, SB . Simulation in healthcare education: A best evidence practical guide. AMEE Guide No. 82. Med Teach. 2013; 35:e1511–30 [Article] [PubMed]
Ziv, A, Wolpe, PR, Small, SD, Glick, S . Simulation-based medical education: An ethical imperative. Acad Med. 2003; 78:783–8 [Article] [PubMed]
Madenci, AL, Solis, CV, de Moya, MA . Central venous access by trainees: A systematic review and meta-analysis of the use of simulation to improve success rate on patients. Simul Healthc. 2014; 9:7–14 [Article] [PubMed]
Komasawa, N, Berg, BW . Simulation-based airway management training for anesthesiologists - A brief review of its essential role in skills training for clinical competency. J Educ Perioper Med. 2017; 19:E612 [PubMed]
Bortman, J, Baribeau, Y, Jeganathan, J, Amador, Y, Mahmood, F, Shnider, M, Ahmed, M, Hess, P, Matyal, R . Improving clinical proficiency using a 3-dimensionally printed and patient-specific thoracic spine model as a haptic task trainer. Reg Anesth Pain Med. 2018; 43:819–24 [PubMed]
Nagendran, M, Toon, CD, Davidson, BR, Gurusamy, KS . Laparoscopic surgical box model training for surgical trainees with no prior laparoscopic experience. Cochrane Database Syst Rev. 2014 CD010479
Raj, D, Williamson, RM, Young, D, Russell, D . A simple epidural simulator: A blinded study assessing the ‘feel’ of loss of resistance in four fruits. Eur J Anaesthesiol. 2013; 30:405–8 [Article] [PubMed]
Nachshon, A, Mitchell, JD, Mueller, A, Banner-Goodspeed, VM, McSparron, JI . Expert evaluation of a chicken tissue-based model for teaching ultrasound-guided central venous catheter insertion. J Educ Perioper Med. 2017; 19:E503 [PubMed]
Sadideen, H, Hamaoui, K, Saadeddin, M, Kneebone, R . Simulators and the simulation environment: Getting the balance right in simulation-based surgical education. Int J Surg. 2012; 10:458–62 [Article] [PubMed]
Goolsby, CA, Goodwin, TL, Vest, RM . Hybrid simulation improves medical student procedural confidence during EM clerkship. Mil Med. 2014; 179:1223–7 [Article] [PubMed]
Mashari, A, Montealegre-Gallegos, M, Jeganathan, J, Yeh, L, Qua Hiansen, J, Meineri, M, Mahmood, F, Matyal, R . Low-cost three-dimensional printed phantom for neuraxial anesthesia training: Development and comparison to a commercial model. PLoS One. 2018; 13:e0191664 [Article] [PubMed]
Huang, CC, Hsu, HC, Yang, LY, Chen, CH, Yang, YY, Chang, CC, Chuang, CL, Lee, WS, Lee, FY, Hwang, SJ . Peer-assisted learning model enhances clinical clerk’s procedural skills. J Chin Med Assoc. 2018; 81:747–53 [Article] [PubMed]
Piromchai, P, Avery, A, Laopaiboon, M, Kennedy, G, O’Leary, S . Virtual reality training for improving the skills needed for performing surgery of the ear, nose or throat. Cochrane Database Syst Rev. 2015 CD010198
Nagendran, M, Gurusamy, KS, Aggarwal, R, Loizidou, M, Davidson, BR . Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. 2013 CD006575
Yoganathan, S, Finch, DA, Parkin, E, Pollard, J . 360° virtual reality video for the acquisition of knot tying skills: A randomised controlled trial. Int J Surg. 2018; 54(pt A):24–7 [Article] [PubMed]
Huang, CY, Thomas, JB, Alismail, A, Cohen, A, Almutairi, W, Daher, NS, Terry, MH, Tan, LD . The use of augmented reality glasses in central line simulation: “See one, simulate many, do one competently, and teach everyone”. Adv Med Educ Pract. 2018; 9:357–63 [Article] [PubMed]
Izard, SG, Juanes, JA, García Peñalvo, FJ, Estella, JMG, Ledesma, MJS, Ruisoto, P . Virtual reality as an educational and training tool for medicine. J Med Syst. 2018; 42:50 [Article] [PubMed]
Rochlen, LR, Levine, R, Tait, AR . First-person point-of-view-augmented reality for central line insertion training: A usability and feasibility study. Simul Healthc. 2017; 12:57–62 [PubMed]
Mahmood, F, Mahmood, E, Dorfman, RG, Mitchell, J, Mahmood, FU, Jones, SB, Matyal, R . Augmented reality and ultrasound education: Initial experience. J Cardiothorac Vasc Anesth. 2018; 32:1363–7 [Article] [PubMed]
Bose, RR, Matyal, R, Warraich, HJ, Summers, J, Subramaniam, B, Mitchell, J, Panzica, PJ, Shahul, S, Mahmood, F . Utility of a transesophageal echocardiographic simulator as a teaching tool. J Cardiothorac Vasc Anesth. 2011; 25:212–5 [Article] [PubMed]
Mitchell, JD, Mahmood, F, Bose, R, Hess, PE, Wong, V, Matyal, R . Novel, multimodal approach for basic transesophageal echocardiographic teaching. J Cardiothorac Vasc Anesth. 2014; 28:800–9 [Article] [PubMed]
McGaghie, WC, Barsuk, JH, Wayne, DB . AM last page: Mastery learning with deliberate practice in medical education. Acad Med. 2015; 90:1575 [Article] [PubMed]
Yudkowsky, R, Park, YS, Lineberry, M, Knox, A, Ritter, EM . Setting mastery learning standards. Acad Med. 2015; 90:1495–500 [Article] [PubMed]
Ericsson, KA, Krampe, RT, Teschromer, C . The role of deliberate practice in the acquisition of expert performance. Psychol Rev. 1993; 100:363–406 [Article]
McGaghie, WC, Issenberg, SB, Barsuk, JH, Wayne, DB . A critical review of simulation-based mastery learning with translational outcomes. Med Educ. 2014; 48:375–85 [Article] [PubMed]
McGaghie, WC, Harris, IB . Learning theory foundations of simulation-based mastery learning. Simul Healthc. 2018; 13(3S suppl 1):15–20 [Article]
Griswold-Theodorson, S, Ponnuru, S, Dong, C, Szyld, D, Reed, T, McGaghie, WC . Beyond the simulation laboratory: A realist synthesis review of clinical outcomes of simulation-based mastery learning. Acad Med. 2015; 90:1553–60 [Article] [PubMed]
Bisgaard, CH, Rubak, SLM, Rodt, SA, Petersen, JAK, Musaeus, P . The effects of graduate competency-based education and mastery learning on patient care and return on investment: A narrative review of basic anesthetic procedures. BMC Med Educ. 2018; 18:154 [Article] [PubMed]
Alsaad, AA, Bhide, VY, Moss, JLJr, Silvers, SM, Johnson, MM, Maniaci, MJ . Central line proficiency test outcomes after simulation training versus traditional training to competence. Ann Am Thorac Soc. 2017; 14:550–4 [Article] [PubMed]
Cox, T, Seymour, N, Stefanidis, D . Moving the needle: Simulation’s impact on patient outcomes. Surg Clin North Am. 2015; 95:827–38 [Article] [PubMed]
Andersen, SAW, Konge, L, Sorensen, MS . The effect of distributed virtual reality simulation training on cognitive load during subsequent dissection training. Med Teach. 2018; 40:1–6 [Article] [PubMed]
Haji, FA, Cheung, JJ, Woods, N, Regehr, G, de Ribaupierre, S, Dubrowski, A . Thrive or overload? The effect of task complexity on novices’ simulation-based learning. Med Educ. 2016; 50:955–68 [Article] [PubMed]
Stefanidis, D, Walters, KC, Mostafavi, A, Heniford, BT . What is the ideal interval between training sessions during proficiency-based laparoscopic simulator training? Am J Surg. 2009; 197:126–9 [Article] [PubMed]
Andersen, SA, Konge, L, Cayé-Thomasen, P, Sørensen, MS . Learning curves of virtual mastoidectomy in distributed and massed practice. JAMA Otolaryngol Head Neck Surg. 2015; 141:913–8 [PubMed]
Patocka, C, Khan, F, Dubrovsky, AS, Brody, D, Bank, I, Bhanji, F . Pediatric resuscitation training-instruction all at once or spaced over time? Resuscitation. 2015; 88:6–11 [Article] [PubMed]
Shea, CH, Lai, Q, Black, C, Park, JH . Spacing practice sessions across days benefits the learning of motor skills. Hum Mov Sci. 2000; 19:737–60 [Article]
Kantak, SS, Sullivan, KJ, Fisher, BE, Knowlton, BJ, Winstein, CJ . Neural substrates of motor memory consolidation depend on practice structure. Nat Neurosci. 2010; 13:923–5 [Article] [PubMed]
Kromann, CB, Bohnstedt, C, Jensen, ML, Ringsted, C . The testing effect on skills learning might last 6 months. Adv Health Sci Educ Theory Pract. 2010; 15:395–401 [Article] [PubMed]
Shakil, O, Mahmood, B, Matyal, R, Jainandunsing, JS, Mitchell, J, Mahmood, F . Simulation training in echocardiography: The evolution of metrics. J Cardiothorac Vasc Anesth. 2013; 27:1034–40 [Article] [PubMed]
Mitchell, JD, Montealegre-Gallegos, M, Mahmood, F, Owais, K, Wong, V, Ferla, B, Chowdhury, S, Nachshon, A, Doshi, R, Matyal, R . Multimodal perioperative ultrasound course for interns allows for enhanced acquisition and retention of skills and knowledge. A A Case Rep. 2015; 5:119–23 [Article] [PubMed]
Ramsingh, D, Alexander, B, Le, K, Williams, W, Canales, C, Cannesson, M . Comparison of the didactic lecture with the simulation/model approach for the teaching of a novel perioperative ultrasound curriculum to anesthesiology residents. J Clin Anesth. 2014; 26:443–54 [Article] [PubMed]
Rebel, A, DiLorenzo, AN, Fragneto, RY, Dority, JS, Rose, G, Nguyen, D, Hassan, ZU, Schell, RM . A competitive objective structured clinical examination event to generate an objective assessment of anesthesiology resident skills development. A A Case Rep. 2016; 6:313–9 [Article] [PubMed]
Mitchell, JD, Amir, R, Montealegre-Gallegos, M, Mahmood, F, Shnider, M, Mashari, A, Yeh, L, Bose, R, Wong, V, Hess, P, Amador, Y, Jeganathan, J, Jones, SB, Matyal, R . Summative objective structured clinical examination assessment at the end of anesthesia residency for perioperative ultrasound. Anesth Analg. 2018; 126:2065–8 [Article] [PubMed]
Ramsingh, D, Rinehart, J, Kain, Z, Strom, S, Canales, C, Alexander, B, Capatina, A, Ma, M, Le, KV, Cannesson, M . Impact assessment of perioperative point-of-care ultrasound training on anesthesiology residents. Anesthesiology. 2015; 123:670–82 [Article] [PubMed]
Wali, E, Pinto, JM, Cappaert, M, Lambrix, M, Blood, AD, Blair, EA, Small, SD . Teaching professionalism in graduate medical education: What is the role of simulation? Surgery. 2016; 160:552–64 [Article] [PubMed]
Birden, H, Glass, N, Wilson, I, Harrison, M, Usherwood, T, Nass, D . Teaching professionalism in medical education: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 25. Med Teach. 2013; 35:e1252–66 [Article] [PubMed]
Lockman, JL, Schwartz, AJ, Cronholm, PF . Working to define professionalism in pediatric anesthesiology: A qualitative study of domains of the expert pediatric anesthesiologist as valued by interdisciplinary stakeholders. Paediatr Anaesth. 2017; 27:137–46 [Article] [PubMed]
Wynia, MK, Papadakis, MA, Sullivan, WM, Hafferty, FW . More than a list of values and desired behaviors: A foundational understanding of medical professionalism. Acad Med. 2014; 89:712–4 [Article] [PubMed]
Hafferty, FW . Professionalism–The next wave. N Engl J Med. 2006; 355:2151–2 [Article] [PubMed]
Wilkinson, TJ, Wade, WB, Knock, LD . A blueprint to assess professionalism: Results of a systematic review. Acad Med. 2009; 84:551–8 [Article] [PubMed]
Kelly, AM, Mullan, PB . Designing a curriculum for professionalism and ethics within radiology: Identifying challenges and expectations. Acad Radiol. 2018; 25:610–8 [Article] [PubMed]
Chestnut, DH . On the road to professionalism. Anesthesiology. 2017; 126:780–6 [Article] [PubMed]
Irby, DM, Hamstra, SJ . Parting the clouds: Three professionalism frameworks in medical education. Acad Med. 2016; 91:1606–11 [Article] [PubMed]
Bryden, P, Ginsburg, S, Kurabi, B, Ahmed, N . Professing professionalism: Are we our own worst enemy? Faculty members’ experiences of teaching and evaluating professionalism in medical education at one school. Acad Med. 2010; 85:1025–34 [Article] [PubMed]
Stern, DT, Papadakis, M . The developing physician–Becoming a professional. N Engl J Med. 2006; 355:1794–9 [Article] [PubMed]
Cruess, RL . Teaching professionalism: Theory, principles, and practices. Clin Orthop Relat Res. 2006; 449:177–85 [PubMed]
Nestel, D, Tierney, T . Role-play for medical students learning about communication: Guidelines for maximising benefits. BMC Med Educ. 2007; 7:3 [Article] [PubMed]
Baile, WF, Blatner, A . Teaching communication skills: Using action methods to enhance role-play in problem-based learning. Simul Healthc. 2014; 9:220–7 [Article] [PubMed]
Bosse, HM, Schultz, JH, Nickel, M, Lutz, T, Möltner, A, Jünger, J, Huwendiek, S, Nikendei, C . The effect of using standardized patients or peer role play on ratings of undergraduate communication training: A randomized controlled trial. Patient Educ Couns. 2012; 87:300–6 [Article] [PubMed]
Joyner, B, Young, L . Teaching medical students using role play: Twelve tips for successful role plays. Med Teach. 2006; 28:225–9 [Article] [PubMed]
Stobbs, N . Role-play without humiliation: Is it possible? Clin Teach. 2015; 12:128–30 [Article] [PubMed]
Dong, T, Kelly, W, Hays, M, Berman, NB, Durning, SJ . An investigation of professionalism reflected by student comments on formative virtual patient encounters. BMC Med Educ. 2017; 17:3 [Article] [PubMed]
Goss, BD, Ryan, AT, Waring, J, Judd, T, Chiavaroli, NG, O’Brien, RC, Trumble, SC, McColl, GJ . Beyond selection: The use of situational judgement tests in the teaching and assessment of professionalism. Acad Med. 2017; 92:780–4 [Article] [PubMed]
Riveros, R, Kimatian, S, Castro, P, Dhumak, V, Honar, H, Mascha, EJ, Sessler, DI . Multisource feedback in professionalism for anesthesia residents. J Clin Anesth. 2016; 34:32–40 [Article] [PubMed]
Gaiser, RR . The teaching of professionalism during residency: Why it is failing and a suggestion to improve its success. Anesth Analg. 2009; 108:948–54 [Article] [PubMed]
Dorotta, I, Staszak, J, Takla, A, Tetzlaff, JE . Teaching and evaluating professionalism for anesthesiology residents. J Clin Anesth. 2006; 18:148–60 [Article] [PubMed]
Mitchell, JD, Ku, C, Diachun, CAB, DiLorenzo, A, Lee, DE, Karan, S, Wong, V, Schell, RM, Brzezinski, M, Jones, SB . Enhancing feedback on professionalism and communication skills in anesthesia residency programs. Anesth Analg. 2017; 125:620–31 [Article] [PubMed]
Denson, JS, Abrahamson, S . A computer-controlled patient simulator. JAMA. 1969; 208:504–8 [Article] [PubMed]
Gaba, DM . The future vision of simulation in health care. Qual Saf Health Care. 2004; 13suppl 1 i2–10 [Article] [PubMed]
Isaak, RS, Chen, F, Arora, H, Martinelli, SM, Zvara, DA, Stiegler, MP . A descriptive survey of anesthesiology residency simulation programs: How are programs preparing residents for the New American Board of Anesthesiology APPLIED Certification Examination? Anesth Analg. 2017; 125:991–8 [Article] [PubMed]
Hastie, MJ, Spellman, JL, Pagano, PP, Hastie, J, Egan, BJ . Designing and implementing the objective structured clinical examination in anesthesiology. Anesthesiology. 2014; 120:196–203 [Article] [PubMed]
McIndoe, A . High stakes simulation in anaesthesia. Continuing Educ Anaesth Crit Care Pain. 2012; 12:268–73 [Article]
Ben-Menachem, E, Ezri, T, Ziv, A, Sidi, A, Brill, S, Berkenstadt, H . Objective Structured Clinical Examination-based assessment of regional anesthesia skills: The Israeli National Board Examination in Anesthesiology experience. Anesth Analg. 2011; 112:242–5 [Article] [PubMed]
ABA News March 2013, The American Board of Anesthesiology 2013. Available at: Accessed May 28, 2018.
Rathmell, JP, Lien, C, Harman, A . Objective structured clinical examination and board certification in anesthesiology. Anesthesiology. 2014; 120:4–6 [Article] [PubMed]
Rebel, A, Hester, DL, DiLorenzo, A, McEvoy, MD, Schell, RM . Beyond the “E” in OSCE. Anesth Analg. 2018; 127:1092–6 [Article] [PubMed]
Isaak, RS, Chen, F, Martinelli, SM, Arora, H, Zvara, DA, Hobbs, G, Stiegler, MP . Validity of simulation-based assessment for Accreditation Council for Graduate Medical Education Milestone Achievement. Simul Healthc. 2018; 13:201–10 [Article] [PubMed]
Collins, AB, Chen, L-l . Simulation and safety. ASA Monitor. 2015; 79:50–2
Steadman, RH, Burden, AR, Huang, YM, Gaba, DM, Cooper, JB . Practice improvements based on participation in simulation for the maintenance of certification in anesthesiology program. Anesthesiology. 2015; 122:1154–69 [Article] [PubMed]
Hyman, SA, Shotwell, MS, Michaels, DR, Han, X, Card, EB, Morse, JL, Weinger, MB . A survey evaluating burnout, health status, depression, reported alcohol and substance use, and social support of anesthesiologists. Anesth Analg. 2017; 125:2009–18 [Article] [PubMed]
Martinelli, SM, Chen, F, Hobbs, G, Chidgey, BA, Straube, LE, Zvara, D, Isaak, R . The use of simulation to improve family understanding and support of anesthesia providers. Cureus. 2018; 10:e2262 [PubMed]
Cooper, AZ, Hsieh, G, Kiss, JE, Huang, GC . Flipping out: Does the flipped classroom learning model work for GME? J Grad Med Educ. 2017; 9:392–3 [Article] [PubMed]
Murray, E . Challenges in educational research. Med Educ. 2002; 36:110–2 [Article] [PubMed]
Fig. 1.
Bringing novel educational methods to your learners. This review suggests many novel ways in which educational research and innovation can make a difference to your learners. Each blue circle contains different goals for your learning environment. Ways in which one may utilize novel educational interventions to reach these goals are described in the corresponding boxes.
Bringing novel educational methods to your learners. This review suggests many novel ways in which educational research and innovation can make a difference to your learners. Each blue circle contains different goals for your learning environment. Ways in which one may utilize novel educational interventions to reach these goals are described in the corresponding boxes.
Fig. 1.
Bringing novel educational methods to your learners. This review suggests many novel ways in which educational research and innovation can make a difference to your learners. Each blue circle contains different goals for your learning environment. Ways in which one may utilize novel educational interventions to reach these goals are described in the corresponding boxes.
Fig. 2.
The Master Adaptive Learning model is an advanced model of active learning. The learner works through an iterative cycle of planning, learning, assessing, and adjusting as they master skills or concepts. The skills needed to be a successful learner are listed in each cog below the respective phase headings within the learner. The interactivity of the cogs represent that the learner must engage in all of these areas to develop successfully through a continuous and dynamic process. Reprinted with permission from Cutrer et al. (2017).12 
The Master Adaptive Learning model is an advanced model of active learning. The learner works through an iterative cycle of planning, learning, assessing, and adjusting as they master skills or concepts. The skills needed to be a successful learner are listed in each cog below the respective phase headings within the learner. The interactivity of the cogs represent that the learner must engage in all of these areas to develop successfully through a continuous and dynamic process. Reprinted with permission from Cutrer et al. (2017).12
Fig. 2.
The Master Adaptive Learning model is an advanced model of active learning. The learner works through an iterative cycle of planning, learning, assessing, and adjusting as they master skills or concepts. The skills needed to be a successful learner are listed in each cog below the respective phase headings within the learner. The interactivity of the cogs represent that the learner must engage in all of these areas to develop successfully through a continuous and dynamic process. Reprinted with permission from Cutrer et al. (2017).12 
Fig. 3.
Miller’s Pyramid of Assessment. The face of the pyramid demonstrates varying levels of assessment, which progress in their ability to demonstrate competence moving from the base to the peak. The desired goal is for learners to reach the “DOES” level. The side of the pyramid breaks down the educational domains in which our learners must gain competence. Adapted from Mehay (2012).22 
Miller’s Pyramid of Assessment. The face of the pyramid demonstrates varying levels of assessment, which progress in their ability to demonstrate competence moving from the base to the peak. The desired goal is for learners to reach the “DOES” level. The side of the pyramid breaks down the educational domains in which our learners must gain competence. Adapted from Mehay (2012).22
Fig. 3.
Miller’s Pyramid of Assessment. The face of the pyramid demonstrates varying levels of assessment, which progress in their ability to demonstrate competence moving from the base to the peak. The desired goal is for learners to reach the “DOES” level. The side of the pyramid breaks down the educational domains in which our learners must gain competence. Adapted from Mehay (2012).22 
Table 1.
Educational Methodologies Based on Domains of Utilization
Educational Methodologies Based on Domains of Utilization×
Educational Methodologies Based on Domains of Utilization
Table 1.
Educational Methodologies Based on Domains of Utilization
Educational Methodologies Based on Domains of Utilization×
Table 2.
Sampling of Educational Tools Specific to Anesthesiology Learners
Sampling of Educational Tools Specific to Anesthesiology Learners×
Sampling of Educational Tools Specific to Anesthesiology Learners
Table 2.
Sampling of Educational Tools Specific to Anesthesiology Learners
Sampling of Educational Tools Specific to Anesthesiology Learners×