The Truth About Trust

 From the 10/2/2020 newsletter

Director's Corner

The Truth About Trust

Adina Kalet, MD, MPH

In this Director’s Corner, Dr. Kalet considers the importance and complexity of trust in medical education and encourages us to hone our judgement and have courage …

Anyone who doesn’t take truth seriously in small matters cannot be trusted with large ones either.

-Albert Einstein

The first presidential debate this week has me thinking about the consequences of not being able to trust someone on whom you depend. We rely on our elected officials, like our physicians, to listen, have empathy, engage in respectful - even if sometimes - heated disagreements, make good judgements in very complex situations, have control over intense emotions and, most importantly, consistently tell the truth. To “trust someone” implies that we have confidence in that person, and believe that the individual will be capable, adaptable, and competent now and in the future – even when faced with novel, rapidly evolving circumstances, emotional and physical stressors, and unpredictable challenges.

While always in the background, trust (“entrustment” and “trustworthiness”) has moved to the forefront in the medical education. How we make these trust judgements in medical education – and in life – is worth a hard look.

How do we measure trustworthiness in trainees?

Hodges and Lingard point out that the discourse about what makes a “good” physician – a core responsibility of our work as medical educators – has moved through a series of distinct and overlapping eras over the past seventy years. In the Psychometric Era, we valorized measurable, highly standardized knowledge tests (e.g. MCAT, USMLE Board Exams). The next phase brought great enthusiasm for demonstrable, directly observable, and behaviorally measurable core clinical skills (e.g. oral exams, mini-CEXs, OSCEs). Next, and to the frustration of many program directors, organizations introduced comprehensive, nuanced competency frameworks designed to capture and document each learner’s developmental progress via new standards and milestones.

These changes reflect our evolving grasp of “quality” in medical education. As our understanding improves, we will uncover how to develop rich portfolios of assessment data for each of our trainees. But in the end, data do not make high stakes decisions. We do. And these decisions require making trust judgements and having the courage to act on those judgements.

Trust judgement barriers and opportunities

Unfortunately, clinical faculty are not very good at assigning objective measures of competence. My colleagues and I spent years trying to get experienced clinicians to make reliable (reproducible) measurements of medical student clinical competence. Even with lots of fancy, performance dimension, frame-of- reference, and behavioral observation training, experienced professionals are eccentric and resist standardization. This, I believe, is because there is no single “truth” about clinical competence.

Trust judgments are highly context-dependent and idiosyncratic. We tend to be internally consistent and we know a trustworthy resident when we see one. An experienced professional possesses a highly-honed identity and a strong sense of what a trainee must demonstrate to be trusted to care for “our” patients. Unfortunately, we disagree with our colleagues on when individual trainees can be entrusted to “fly solo” and more independently care for patients. Gingerich has challenged us to embrace this disagreement and see it as a strength rather than a weakness.

Furthermore, experts are also context-dependent! As we collect and collate more-and-more data from larger, diverse pools of experts, we must ensure that trust judgements are appropriately interpreted to protect students from the vagaries of any individual’s bias. This is what van der Vleuten and others call a Program of Assessment for Learning. Ultimately, trained competence“judges” will be charged with making final high stakes assessments regarding decisions such as advancement and graduation. These judges will determine if, based on solid evidence, we can trust a learner to consistently “do the right thing, at the right time, for the right person, and for the right reason” in their next phase of training.

Moving from theory to action

Social and cognitive psychology researchers suggest that competency judges need to both understand the value and limits of the objective data (e.g., exam scores don’t predict clinical skills competence, but they do predict future exam scores) and should explore and develop their judgement “sense.” This sense of who to trust is highly dependent on an individual’s characteristics, experiences and biases. Knowing thyself, in particular understanding one’s biases, is crucialbecause if we are cognizant of them and have integrity, we can make adjustments – “forcing” ourselves to slow down our thinking, toggle to a more analytical rather than intuitive deliberative strategy, when we are in danger of making an error. This takes work, discipline, and practice with feedback.

There is much interesting work to be done to ensure we have trustworthy physicians. Fundamentally, most of us make our trust judgements based not on what students know or can do (we can always teach that stuff), but on who they are as people. Do they always tell the truth even when it leaves them in a “bad light?” Do they admit when they missed a physical exam finding or forgot to check a lab or failed to follow up on something? Do they take the time to listen, attend to details, interact with empathy and kindness, even when stressed emotionally? Do they strive to improve rather than rest on their laurels or test scores? Do they seek to understand the perspectives of others? How do they handle being wrong or making a mistake? Can they sincerely apologize?

We are accountable to society to make defensible promotion and graduation decisions based on each learner’s competence and trustworthiness. These are difficult-to-measure, shifting concepts. We pledge to engage in the ongoing discourses and learn how best to make difficult, discerning judgements.

Adina Kalet, MD MPH is the Director of the Robert D. and Patricia E. Kern Institute for the Transformation of Medical Education and holder of the Stephen and Shelagh Roell Endowed Chair at the Medical College of Wisconsin. 

Fabulous Failures

 From the 9/4/2020 newsletter

Being Human in Medicine

 
Fabulous Failures

 
Himanshu Agrawal, MD – Department of Psychiatry and Behavioral Medicine
 
 

To encourage his students who might be worrying about USMLE board scores and other life challenges, Dr. Himanshu shares the story of one of the darkest and best days in his life …
 
 
It's 2:00 a.m. and I am a junior medical student in India. I haven't eaten in two days and am worsening my heartburn – and my heartbreak – with black coffee and a cigarette. I can feel the sense of doom grip my fundus. A senior medical student whom I barely know staggers into the cafeteria, happy to have a brief respite from his overnight rotation. “Why the long face?” he asks out of genuine concern. The tears erupt, uninvited – “I did horrible in my USMLE Step 1 exam!” I tell him. I hardly know the guy, but I am in mourning, so shame be damned. “There, there! It can’t be that bad. How much did you score?” Envisioning my entire future evaporating in front of me, I manage to say the numbers: “197.”
 
The man takes a step back, and his hand instinctively rises to stroke his chin. It’s as if he has heard someone mutter a terminal diagnosis. “Hmm…That is bad! Well…with a score like that, you won’t be able to get into internal medicine…the only US residency you can get into is psychiatry…” 
 
Suddenly, he meets a reaction he did not expect – a wide grin appears on my tear-smudged face. “Really?! But that’s what I want to do! Psychiatry!!” He looks at me with surprise, then smiles. “Well then what are you crying for? Let’s celebrate! This cup of tea is on you, my friend!”
 
The year was 2000. Psychiatry was not nearly as competitive as it is now, and international medical graduates still got interviews in American programs. So much has changed in the last twenty years, but some things remain the exact same. You see, looking back, this random stranger had no idea what he was talking about – he was certainly no authority on USMLE scores, successes and failures – but, like so many others, he was a speculation-guru, a pundit of pontification. Unknowingly, his prophesizing was exactly the piece of straw I needed to stay afloat!
 
Hopelessness cast as large a shadow on my future back then as it has for several of my medical students. And sometimes, it is as quickly dispelled as mine was that fateful day by that clueless senior student (Sometimes it takes a bit longer).
 
I am writing today for all my students who have recently faced despair or who may one day meet with crippling news. This Distinguished Fellow of the American Psychiatric Association, this latest recipient of Edward J. Lennon Endowed Clinical Teaching Award, this boy from New Delhi who grew up without running water but who now swims in a 29,000 gallon swimming pool (feel free to insert your own yardstick of success) – I was once ready to walk away from it all. I was ready to throw in the towel. 
 
I am so glad I didn’t.
 
Remember two things- firstly, you are not as good as they say you are when you succeed, and you are never as bad as they say you are when you fail. 
 
Secondly, you will never cherish success more sweetly, than when you have had to swallow the bitterness of failure.
 
Do I wish you failure? Of course not. What I am saying is this – there is more to life, so much more, after failure.
 
Failure is not the same as defeat.
 
They say nothing succeeds like success.
 
They have not seen the daily grin on the face of this Fabulous Failure.
 
 
 
(Dedicated to Roda Sir)
 
 
Himanshu Agrawal, MD, DF-APA, is an Assistant Professor in the Department of Psychiatry and Behavioral Health at MCW and co-director of the psychiatry clerkship. He serves as a small group facilitator in the Kern REACH curriculum. 
 

Performance-based assessment, done well, is the best way we have to ensure physicians have the “right stuff”

From the 5/29/2020 newsletter



Performance-based assessment, done well, is the best way we have to ensure physicians have the “right stuff”

Adina Kalet, MD, MPH



The earth just shifted again! And this one worries me more than most of the agile adjustments required in the days of COVID-19 because it may portend a loss of the hard-won support for valid, high stakes, performance-based clinical skills assessments.


This week the National Board of Medical Examiners (NBME) and the Federation of State Medical Boards (FSMB) made the decision to suspend the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) exam for the next year or so. The organizations are concerned about the health and safety of the standardized patients, testing center staff, and medical students at the five free-standing testing centers where students demonstrate that their clinical skills meet the thresholds to be licensed to practice medicine in the US.


Step 2 CS, one of four licensing exams, uses standardized patients to test applicants as they apply their knowledge, gather information from patients, perform focused physical examinations, interpret lab and imaging, reason through clinical challenges, and communicate their findings to patients and colleagues. Decades of research have demonstrated that these principles of clinical sciences and patient-centered skills provide the foundation for the safe and effective practice of medicine. Rigorous performance-based assessment of these skills predicts long term clinical competence and patient outcomes.


The residency selection process is unlikely to be impacted by this decision. Step 2 CS results are usually not available until after residency match lists are submitted and, given the low failure rate (~3%), program directors tend to minimize the importance of the exam. Of course, given we anticipate a virtual interview season, the change may have an outsized impact for a few students. The Kern Institute will convene conversations to anticipate and mitigate any negative impacts the cancellation may have on individuals.


Step 2 CS has spurred a lot of passionate debate over the 16 years since it was first required for licensing. I worry about the long-term survival of this very critical national exam.


In 2013, detractors argued that the “value proposition" is low because the exam has both a high cost and a high pass rate1. Currently, US/Canadian students are charged $1,285 and have a 98% pass rate. The cost is higher ($1,535) and the pass rate lower for non-US/Canadian applicants.


Over the years, medical students, including some from the most elite schools, led a campaign to cancel the exam, and in 2016, the AMA called for the FSMB, NBME, state medical societies, and state medical boards to transition from the Step 2 CS exam to a school-administered clinical skills exam as a licensure requirement2. The debate raged for a while with everyone acknowledging that Step 2 CS is expensive to administer because of its significant fixed costs.


But, what about the students who really benefit from taking the Step 2 CS exam? Frankly, I find the “value proposition” argument worrisome. Given my expertise in remediation, I have met many students who were flagged after failing the exam. They are heterogenous with respect to why they failed but, nonetheless, almost all of them are not ready to move on3. Some need extensive remedial attention. The prognosis is excellent, especially given that they are highly motivated to pass the exam. Occasionally, there is a student who needs a “compassionate off-ramp” out of medical training4. It would be better if there was more feedback provided, since it is rarely enough to provide remediation guidance.


The real value of the Step 2 CS is in fulfilling our social contract that every medical school graduate meets standards for patient safety and satisfaction. The exam gives detailed feedback to schools, helping them improve medical education and strengthen clinical communication and reasoning curricula. It also reminds schools to provide effective remediation programs with “teeth.”


The proposal that we replace the Step 2 CS with school-administered exams sounds logical but is not. A consistent, consequential licensing Objective Structured Clinical Exams (OSCE) would force most medical schools to trade away resources they need to provide an ever-widening range of formative standardized patient (SP) simulations on other topics. This would hobble their educational missions. Even the most sophisticated and well-resourced simulation centers would face tough choices.


And let’s face it, medical schools would not be impartial players. Institutional reputations are impacted by licensing exam pass-rates. Medical school faculty and educational leaders are very opinionated about the role of SP exams, although few are familiar with the evidence and believe me, I have talked with many of them! Of course, many learners do not enjoy the performance component of OSCEs, but the vast majority of students value these experiences and the data they provide.


Performance-based assessments, like the Step 2 CS and other OSCEs, are the most authentic and valid way to assess what our trainees are able to do, and they predict – although imperfectly – what trainees will do in practice. Since clinical communication and reasoning skills are how most physicians conduct the vast majority of their work, it is critical that we assess these skills and holdeach other to high standards. Let’s preserve our ability to do this important task.

Adina Kalet, MD MPH is the Director of the Robert D. and Patricia E. Kern Institute for the Transformation of Medical Education and holder of the Stephen and Shelagh Roell Endowed Chair at the Medical College of Wisconsin.