Performance-based assessment, done well, is the best way we have to ensure physicians have the “right stuff”

From the 5/29/2020 newsletter



Performance-based assessment, done well, is the best way we have to ensure physicians have the “right stuff”

Adina Kalet, MD, MPH



The earth just shifted again! And this one worries me more than most of the agile adjustments required in the days of COVID-19 because it may portend a loss of the hard-won support for valid, high stakes, performance-based clinical skills assessments.


This week the National Board of Medical Examiners (NBME) and the Federation of State Medical Boards (FSMB) made the decision to suspend the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) exam for the next year or so. The organizations are concerned about the health and safety of the standardized patients, testing center staff, and medical students at the five free-standing testing centers where students demonstrate that their clinical skills meet the thresholds to be licensed to practice medicine in the US.


Step 2 CS, one of four licensing exams, uses standardized patients to test applicants as they apply their knowledge, gather information from patients, perform focused physical examinations, interpret lab and imaging, reason through clinical challenges, and communicate their findings to patients and colleagues. Decades of research have demonstrated that these principles of clinical sciences and patient-centered skills provide the foundation for the safe and effective practice of medicine. Rigorous performance-based assessment of these skills predicts long term clinical competence and patient outcomes.


The residency selection process is unlikely to be impacted by this decision. Step 2 CS results are usually not available until after residency match lists are submitted and, given the low failure rate (~3%), program directors tend to minimize the importance of the exam. Of course, given we anticipate a virtual interview season, the change may have an outsized impact for a few students. The Kern Institute will convene conversations to anticipate and mitigate any negative impacts the cancellation may have on individuals.


Step 2 CS has spurred a lot of passionate debate over the 16 years since it was first required for licensing. I worry about the long-term survival of this very critical national exam.


In 2013, detractors argued that the “value proposition" is low because the exam has both a high cost and a high pass rate1. Currently, US/Canadian students are charged $1,285 and have a 98% pass rate. The cost is higher ($1,535) and the pass rate lower for non-US/Canadian applicants.


Over the years, medical students, including some from the most elite schools, led a campaign to cancel the exam, and in 2016, the AMA called for the FSMB, NBME, state medical societies, and state medical boards to transition from the Step 2 CS exam to a school-administered clinical skills exam as a licensure requirement2. The debate raged for a while with everyone acknowledging that Step 2 CS is expensive to administer because of its significant fixed costs.


But, what about the students who really benefit from taking the Step 2 CS exam? Frankly, I find the “value proposition” argument worrisome. Given my expertise in remediation, I have met many students who were flagged after failing the exam. They are heterogenous with respect to why they failed but, nonetheless, almost all of them are not ready to move on3. Some need extensive remedial attention. The prognosis is excellent, especially given that they are highly motivated to pass the exam. Occasionally, there is a student who needs a “compassionate off-ramp” out of medical training4. It would be better if there was more feedback provided, since it is rarely enough to provide remediation guidance.


The real value of the Step 2 CS is in fulfilling our social contract that every medical school graduate meets standards for patient safety and satisfaction. The exam gives detailed feedback to schools, helping them improve medical education and strengthen clinical communication and reasoning curricula. It also reminds schools to provide effective remediation programs with “teeth.”


The proposal that we replace the Step 2 CS with school-administered exams sounds logical but is not. A consistent, consequential licensing Objective Structured Clinical Exams (OSCE) would force most medical schools to trade away resources they need to provide an ever-widening range of formative standardized patient (SP) simulations on other topics. This would hobble their educational missions. Even the most sophisticated and well-resourced simulation centers would face tough choices.


And let’s face it, medical schools would not be impartial players. Institutional reputations are impacted by licensing exam pass-rates. Medical school faculty and educational leaders are very opinionated about the role of SP exams, although few are familiar with the evidence and believe me, I have talked with many of them! Of course, many learners do not enjoy the performance component of OSCEs, but the vast majority of students value these experiences and the data they provide.


Performance-based assessments, like the Step 2 CS and other OSCEs, are the most authentic and valid way to assess what our trainees are able to do, and they predict – although imperfectly – what trainees will do in practice. Since clinical communication and reasoning skills are how most physicians conduct the vast majority of their work, it is critical that we assess these skills and holdeach other to high standards. Let’s preserve our ability to do this important task.

Adina Kalet, MD MPH is the Director of the Robert D. and Patricia E. Kern Institute for the Transformation of Medical Education and holder of the Stephen and Shelagh Roell Endowed Chair at the Medical College of Wisconsin.