Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes. Anesthesiology 2018 Apr;128(4):821-831
Date
01/26/2018Pubmed ID
29369062DOI
10.1097/ALN.0000000000002091Scopus ID
2-s2.0-85054040442 (requires institutional sign-in at Scopus site) 16 CitationsAbstract
BACKGROUND: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment.
METHODS: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail.
RESULTS: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room.
CONCLUSIONS: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.
Author List
Blum RH, Muret-Wagstaff SL, Boulet JR, Cooper JB, Petrusa ER, Baker KH, Davidyuk G, Dearden JL, Feinstein DM, Jones SB, Kimball WR, Mitchell JD, Nadelberg RL, Wiser SH, Albrecht MA, Anastasi AK, Bose RR, Chang LY, Culley DJ, Fisher LJ, Grover M, Klainer SB, Kveraga R, Martel JP, McKenna SS, Minehart RD, Mitchell JD, Mountjoy JR, Pawlowski JB, Pilon RN, Shook DC, Silver DA, Warfield CA, Zaleski KL, Harvard Assessment of Anesthesia Resident Performance Research GroupAuthor
Meredith A. Albrecht MD, PhD Associate Professor in the Anesthesiology department at Medical College of WisconsinMESH terms used to index this publication - Major topics in bold
AnesthesiologyClinical Competence
Cross-Sectional Studies
Female
Humans
Internship and Residency
Male
Manikins
Prospective Studies
Reproducibility of Results
