Faculty Collaboration Database

Using Resident-Sensitive Quality Measures Derived From Electronic Health Record Data to Assess Residents' Performance in Pediatric Emergency Medicine. Acad Med 2023 Mar 01;98(3):367-375

Date

11/10/2022

Pubmed ID

36351056

Pubmed Central ID

PMC9944759

DOI

10.1097/ACM.0000000000005084

Scopus ID

2-s2.0-85148678229 (requires institutional sign-in at Scopus site)   7 Citations

Abstract

PURPOSE: Traditional quality metrics do not adequately represent the clinical work done by residents and, thus, cannot be used to link residency training to health care quality. This study aimed to determine whether electronic health record (EHR) data can be used to meaningfully assess residents' clinical performance in pediatric emergency medicine using resident-sensitive quality measures (RSQMs).

METHOD: EHR data for asthma and bronchiolitis RSQMs from Cincinnati Children's Hospital Medical Center, a quaternary children's hospital, between July 1, 2017, and June 30, 2019, were analyzed by ranking residents based on composite scores calculated using raw, unadjusted, and case-mix adjusted latent score models, with lower percentiles indicating a lower quality of care and performance. Reliability and associations between the scores produced by the 3 scoring models were compared. Resident and patient characteristics associated with performance in the highest and lowest tertiles and changes in residents' rank after case-mix adjustments were also identified.

RESULTS: 274 residents and 1,891 individual encounters of bronchiolitis patients aged 0-1 as well as 270 residents and 1,752 individual encounters of asthmatic patients aged 2-21 were included in the analysis. The minimum reliability requirement to create a composite score was met for asthma data (α = 0.77), but not bronchiolitis (α = 0.17). The asthma composite scores showed high correlations ( r = 0.90-0.99) between raw, latent, and adjusted composite scores. After case-mix adjustments, residents' absolute percentile rank shifted on average 10 percentiles. Residents who dropped by 10 or more percentiles were likely to be more junior, saw fewer patients, cared for less acute and younger patients, or had patients with a longer emergency department stay.

CONCLUSIONS: For some clinical areas, it is possible to use EHR data, adjusted for patient complexity, to meaningfully assess residents' clinical performance and identify opportunities for quality improvement.

Author List

Smirnova A, Chahine S, Milani C, Schuh A, Sebok-Syer SS, Swartz JL, Wilhite JA, Kalet A, Durning SJ, Lombarts KMJMH, van der Vleuten CPM, Schumacher DJ

Authors

Adina L. Kalet MD Institute Director, Professor in the Kern Institute for Transforming Medical Education department at Medical College of Wisconsin
Abigail M. Schuh MD Associate Professor in the Pediatrics department at Medical College of Wisconsin

MESH terms used to index this publication - Major topics in bold

Asthma
Child
Clinical Competence
Electronic Health Records
Emergency Medicine
Humans
Internship and Residency
Pediatric Emergency Medicine
Quality Indicators, Health Care
Reproducibility of Results