A National Study of Longitudinal Consistency in ACGME Milestone Ratings by Clinical Competency Committees: Exploring an Aspect of Validity in the Assessment of Residents' Competence

Title

A National Study of Longitudinal Consistency in ACGME Milestone Ratings by Clinical Competency Committees: Exploring an Aspect of Validity in the Assessment of Residents' Competence

Creator

Hamstra Stanley J; Yamazaki Kenji; Barton Melissa A; Santen Sally A; Beeson Michael S; Holmboe Eric S

Publisher

Academic Medicine: Journal of the Association of American Medical Colleges

Date

2019
2019-06

Description

PURPOSE: To investigate whether clinical competency committees (CCCs) were consistent in applying milestone ratings for first-year residents over time or whether ratings increased or decreased. METHOD: Beginning in December 2013, the ACGME initiated a phased-in requirement for reporting milestones; emergency medicine (EM), diagnostic radiology (DR), and urology (UR) were among the earliest reporting specialties. The authors analyzed CCC milestone ratings of first-year residents from 2013-2016 from all ACGME-accredited EM, DR, and UR programs for which they had data. The number of first-year residents in these programs ranged from 2,838 to 2,928 over this time period. The program-level average milestone rating for each subcompetency was regressed onto the time of observation using a random coefficient multilevel regression model. RESULTS: National average program-level milestone ratings of first-year residents decreased significantly over the observed time period for 32 of the 56 subcompetencies examined. None of the other subcompetencies showed a significant change. National average in-training examination scores for each of the specialties remained essentially unchanged over the time period, suggesting differences between the cohorts was not likely an explanatory factor. CONCLUSIONS: The findings indicate that CCCs tend to become more stringent or maintain consistency in their ratings of beginning residents over time. One explanation for these results is that CCCs may become increasingly comfortable in assigning lower ratings when appropriate. This finding is consistent with an increase in confidence with the milestones rating process and the quality of feedback it provides.This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

Rights

Article information provided for research and reference use only. All rights are retained by the journal listed under publisher and/or the creator(s).

Citation

Hamstra Stanley J; Yamazaki Kenji; Barton Melissa A; Santen Sally A; Beeson Michael S; Holmboe Eric S, “A National Study of Longitudinal Consistency in ACGME Milestone Ratings by Clinical Competency Committees: Exploring an Aspect of Validity in the Assessment of Residents' Competence,” NEOMED Bibliography Database, accessed March 28, 2024, https://neomed.omeka.net/items/show/6368.