Jared Clark, Andreia Alexander

Evaluation of Resident Presentations Using a DEI Checklist

Introduction

Evaluation of resident presentations using a dei checklist. Evaluates a DEI checklist's impact on emergency medicine resident presentations. Shows increased health equity attention and direct DEI discussions, a novel evaluation for graduate medical education.

Abstract

Background: In 2021, the Indiana University School of Medicine Department of Emergency Medicine developed a DEI checklist for residents presenting to their colleagues during weekly didactic sessions. The purpose of this present study is to evaluate the implementation of this DEI checklist as determined by its impact on resident presentations. Methods: A Qualtrics survey based on the DEI checklist was produced to collect data from a Kaltura database containing all resident lectures from the years 2018-19 through 2023-24. Ten lectures were analyzed from each year for 50 total lectures. The year 2020-21 was excluded. Data collected was subjected to quantitative statistics using content analysis. Transcripts for each lecture were generated using Otter.ai and analyzed qualitatively to draw conclusions from how residents discussed DEI topics. Results: Each year was assigned an average overall composite score according to the checklist items, with a score of zero indicating complete adherence to the checklist. The results were as follows: 2018-19: 1.8; 2019-20: 2.3; 2021-22: 2.3; 2022-23: 1.2; 2023-24: 0.8. Residents discussed how their topic may affect someone of a diverse background in 70% of lectures in 2018-2019; 70% in 2019-20; 50% in 2021-22; 80% in 2022-23; and 80% in 2023-24. The percentage of presentations with a dedicated DEI/health equity slide gradually increased from 20% of lectures in 2018-19 to 80% in 2023-24. Transcript review revealed residents transitioned from indirectly discussing health equity in 2018 to directly addressing DEI from 2022-24. Conclusion: Our analysis indicates a general trend towards increasing attention to health equity among these residents. Residents showed more intentionality in approaching DEI topics by directly addressing them in later years. Impact and Implications: Current literature provides no widely agreed upon methodology for determining the impact of DEI training in graduate medical education. Our study provides a novel technique for operationalizing this impact.

Review

This study presents a timely and important evaluation of a DEI checklist implemented for resident presentations within an Emergency Medicine department, addressing a recognized gap in the literature regarding methodology for assessing DEI training impact in graduate medical education. The authors utilized a novel, multi-modal approach by analyzing historical resident lecture data from a Kaltura database, generating transcripts via Otter.ai, and applying both quantitative content analysis based on a DEI checklist and qualitative review of transcripts. The longitudinal design, spanning several years before and after the checklist's introduction, is a significant strength, allowing for an examination of trends over time. The findings suggest a positive trajectory, with residents demonstrating increased intentionality, a greater frequency of direct DEI discussions, and a rise in dedicated DEI/health equity slides in their presentations, particularly in the later years of the study. While the study offers valuable insights, several aspects warrant further clarification and consideration. The abstract would benefit from a more explicit description of the DEI checklist itself, including what specific items it contains, as this is central to the evaluation. The methodology surrounding the "Qualtrics survey based on the DEI checklist" requires clarification; it appears researchers used this tool to rate lectures, but this should be stated explicitly to avoid ambiguity. Furthermore, details on the scoring mechanism for the composite score (e.g., maximum possible score) would provide better context for interpreting the results, especially given the initial increase in scores (from 1.8 to 2.3) before the clear downward trend post-implementation. The dip in lectures discussing diverse backgrounds in 2021-22—the first year of checklist implementation—is an interesting counter-trend that could benefit from additional commentary or investigation. Finally, while qualitative analysis was performed, specifying the qualitative methods used (e.g., thematic analysis approach) would enhance the rigor and reproducibility of the study. Despite these areas for potential enhancement, this research makes a meaningful contribution to the nascent field of evaluating DEI interventions in GME. The innovative use of existing lecture data and a composite scoring system offers a practical and adaptable technique for other institutions seeking to operationalize and measure the impact of their DEI training efforts. Future studies could build upon this foundation by expanding the analysis to other departments or institutions to assess generalizability, delving deeper into the specific components of the checklist that proved most effective, and exploring correlations between improved presentation scores and other important outcomes such as resident attitudes, patient care practices, or resident and faculty perceptions of DEI culture within the program.

Full Text

Comments

You need to be logged in to post a comment.