Education  |   November 2016
Redesign of the System for Evaluation of Teaching Qualities in Anesthesiology Residency Training (SETQ Smart)
Author Notes
  • From the Professional Performance Research Group, Center for Evidence-Based Education, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands (K.M.J.M.H.L.); Department of Anaesthesia and Intensive Care Medicine, Craigavon Area Hospital, Portadown, United Kingdom (A.F.); Department of Anesthesiology, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands (M.W.H.); Centre of Health Sciences Education, Faculty of Health, Aarhus University, Aarhus, Denmark (B.M.); Department of Epidemiology, Fielding School of Public Health, University of California, Los Angeles (UCLA), Los Angeles, California (O.A.A.); and UCLA Center for Health Policy Research, Los Angeles, California (O.A.A.).
  • Submitted for publication September 18, 2015. Accepted for publication July 27, 2016.
    Submitted for publication September 18, 2015. Accepted for publication July 27, 2016.×
  • *Members of SMART Collaborators are listed in appendix 1.
    *Members of SMART Collaborators are listed in appendix 1.×
  • Address correspondence to Dr. Lombarts: Professional Performance Research Group, Academic Medical Center, University of Amsterdam, J1a-119, Meibergdreef 9, 1100 DD Amsterdam, The Netherlands. m.j.lombarts@amc.uva.nl. Information on purchasing reprints may be found at www.anesthesiology.org or on the masthead page at the beginning of this issue. Anesthesiologys articles are made freely accessible to all readers, for personal use only, 6 months from the cover date of the issue.
Article Information
Education / Original Investigations in Education / Education / CPD
Education   |   November 2016
Redesign of the System for Evaluation of Teaching Qualities in Anesthesiology Residency Training (SETQ Smart)
Anesthesiology 11 2016, Vol.125, 1056-1065. doi:10.1097/ALN.0000000000001341
Anesthesiology 11 2016, Vol.125, 1056-1065. doi:10.1097/ALN.0000000000001341
Abstract

Background: Given the increasing international recognition of clinical teaching as a competency and regulation of residency training, evaluation of anesthesiology faculty teaching is needed. The System for Evaluating Teaching Qualities (SETQ) Smart questionnaires were developed for assessing teaching performance of faculty in residency training programs in different countries. This study investigated (1) the structure, (2) the psychometric qualities of the new tools, and (3) the number of residents’ evaluations needed per anesthesiology faculty to use the instruments reliably.

Methods: Two SETQ Smart questionnaires—for faculty self-evaluation and for resident evaluation of faculty—were developed. A multicenter survey was conducted among 399 anesthesiology faculty and 430 residents in six countries. Statistical analyses included exploratory factor analysis, reliability analysis using Cronbach α, item-total scale correlations, interscale correlations, comparison of composite scales to global ratings, and generalizability analysis to assess residents’ evaluations needed per faculty.

Results: In total, 240 residents completed 1,622 evaluations of 247 faculty. The SETQ Smart questionnaires revealed six teaching qualities consisting of 25 items. Cronbach α’s were very high (greater than 0.95) for the overall SETQ Smart questionnaires and high (greater than 0.80) for the separate teaching qualities. Interscale correlations were all within the acceptable range of moderate correlation. Overall, questionnaire and scale scores correlated moderately to highly with the global ratings. For reliable feedback to individual faculty, three to five resident evaluations are needed.

Conclusions: The first internationally piloted questionnaires for evaluating individual anesthesiology faculty teaching performance can be reliably, validly, and feasibly used for formative purposes in residency training.