Measuring Clinical Skills



Medical Students Oppose New Component of Licensing Exam

Texas Medicine Logo(1)

Medical Education Feature -- February 2003  

By  Ken Ortolon
Senior Editor  

Taking a medical history and conducting a physical examination are as basic to practicing medicine as the stethoscope. Yet some believe U.S. medical schools are not adequately assessing whether their graduates have these skills.

Beginning in 2004, the National Board of Medical Examiners (NBME) and the Federation of State Medical Boards (FSMB) plan to roll out a new component to the United States Medical Licensing Examination (USMLE) to measure the basic clinical and communications skills of all medical graduates seeking licensure.

While no one argues that clinical and communication skills are not important to the practice of medicine, the proposed Clinical Skills Examination (CSE) has sparked considerable controversy among medical students, who say it will be costly and inconvenient and likely will weed out only a handful of ill-prepared physicians. And, both the American Medical Association and the Texas Medical Association Council on Medical Education believe implementation of the exam should be delayed or canceled.

"Conceptually, people agree with it, but we're not sure this is the right method to find those people without adequate clinical skills," said Sohail Shah, a fourth-year medical student at Texas A&M University System Health Science Center and a member of the TMA Board of Trustees.

Building a Standard Patient  

Before 1964, medical graduates were required to take a clinical skills test as part of the licensing exam. That test was discontinued, however, because of issues regarding the validity and reliability of its one-on-one bedside examination process. Standardized patients were not used at that time, and the methods available caused concerns about inconsistencies across the multitude of testing sitesand among different examiners.

"It's difficult to come up with a fair and reliable judgment about a practitioner's competence based on watching an interview or examination of one or two patients," said Richard Hawkins, MD, NBME deputy vice president for assessment programs.

Since the original test was dropped, NBME has been developing what it calls "psychometrically sound" methods of assessing clinical skills and has been actively working to develop the new exam for the past 15 years.

The CSE, if implemented on schedule, would be administered as part of Step 2 of the USMLE. The USMLE, jointly sponsored by the NBME and the FSMB, is administered in three parts. Step 1 usually is taken during the second year of medical school and measures knowledge and cognitive skills related to the basic sciences. Step 2 usually is taken during the fourth year and measures knowledge of the clinical sciences. Step 3, administered after graduation and during residency, includes a test of knowledge of general medical issues and computer case simulations designed to measure patient management skills.

The CSE, already pilot tested in Philadelphia and Atlanta and in many medical schools, uses actors who have received extensive training to ensure that they will present to each medical student in exactly the same way, thereby eliminating the inconsistencies that plagued the previous test. It is a one-day exam, during which each medical student is asked to examine 10 to 12 patients for about 15 minutes each. The students are expected to establish a rapport with the patient, elicit pertinent historical information, perform focused physical examinations, answer questions, and provide counseling where appropriate.

Following each patient encounter, the student has 10 minutes to record pertinent history and physical examination findings, list diagnostic impressions, and outline plans for further evaluation, if necessary. The cases cover common situations that physicians are likely to encounter in a general ambulatory clinic.

Each standardized "patient" records what happens during the encounter, using standard checklists and rating scales, and the student's notes are scored by specially trained physician-raters.

"The board feels that the clinical skills examination, incorporating standardized patients, tests skills that are not currently tested using the multiple-choice questions and the computer-based case simulations that are currently part of the USMLE," said Kenneth Cotton, NBME assistant vice president for general services.

Added Dr. Hawkins, "It's important to recognize that the goal of the exam in the licensing process is to assure the public that an applicant for licensure in the United States has met minimal criteria and minimal standards to move to the supervised practice that occurs in residency."

Dr. Hawkins says measuring clinical and communication skills is important because of the key role the patient encounter plays in the diagnostic process.

"There are studies that show that 80 to 90 percent of our diagnostic information comes from the patient encounter, from taking a history, performing an examination, and communicating effectively with patients," he said. "Patient-doctor communication affects patient satisfaction and health outcomes. A high proportion of complaints to state medical boards result from communication skills.

"So the overwhelming evidence indicates that clinical skills are relevant to patient care," Dr. Hawkins said. "For the organization responsible for assuring the public through the licensure process that our successful candidates have mastered a set of skills that are important to practice, it seems critical to include clinical skills assessment in that examination."

Increasing Student Debt  

But medical students say the CSE will place an additional financial burden on them that may not be worth the return.

The test is going to cost each student $950 and will be offered at a limited number of sites. Right now, NBME has a testing site at its home base in Philadelphia and a second in Atlanta, both developed in conjunction with the Education Commission for Foreign Medical Graduates (ECFMG). Four or five additional sites likely will be established across the country. It is believed one of those sites will be located in Texas. These sites would allow more than two-thirds of American medical students to take the examination without overnight travel.

"It poses a huge geographic and financial burden on students across the country," Mr. Shah said. "You're talking about an additional $1,000 in testing fees and, on top of that, travel costs, lodging costs, and a couple of days of missed work. So you're talking about significant financial impact on all medical students who already are coming out with an average debt of $100,000."

Chris Newton, a third-year medical student at Texas A&M and the medical student representative to the TMA Council on Medical Education, says $1,000 may not sound like much when you are talking about $100,000 in debt, "but we have to put the brakes on somewhere. We can't continue to increase the cost of medical education. All we're doing by continually increasing costs is turning away good students who otherwise would want to go into the medical profession."

The students' other major concern is that the test hasn't been proven to be statistically valid. "The NBME has done a lot of studies saying that it is but, unfortunately, these aren't studies that have been published in any peer-reviewed journals," Mr. Newton said.

AMA originally supported the CSE. But at the urging of its Medical Student Section, the AMA House of Delegates in June 2002 adopted a new policy asking the NBME to suspend implementation of the exam until it is demonstrated to be statistically valid, reliable, practical, and evidence-based. That policy also called on state medical boards to exclude the exam from their licensing requirements.

At its interim meeting in December 2002, the AMA house went even farther. It adopted a resolution calling on the Liaison Committee on Medical Education (LCME) and the American Osteopathic Association (AOA) to modify and enforce uniform accreditation standards as soon as possible to require all medical schools to rigorously and consistently assess clinical skills of all students as a requirement of advancement and graduation. The resolution added that "clinical skills assessment is best performed using a rigorous and consistent examination administered by the medical schools, and should not be used in evaluation for licensure of graduates of LCME- and AOA-accredited medical schools."

In September 2002, the TMA Council on Medical Education also approved a draft policy recommending that TMA work with the Texas State Board of Medical Examiners (TSBME) to delay implementation of the CSE until concerns about validity, reliability, and practicality of the test are resolved. The council also urged NBME to work with FSMB to ensure an external source of funding to fund all steps of USMLE. That policy proposal will be presented to the TMA House of Delegates during TexMed 2003 in April.

Fait Accompli?  

But the efforts of the medical students to derail the test may be too little, too late. Houston neurologist William H. Fleming III, MD, a NBME member and past president of the FSMB, says he believes the test will go forward and should.

"The students just don't want another test, that's the bottom line," said Dr. Fleming, who also serves on the USMLE Composite Committee. "What we found, and it's well documented, is that there is a cadre of students who take the USMLE who simply do not have clinical skills. The research shows that there is a need."

Countering the medical students' arguments, Dr. Hawkins says there are more than 200 studies that demonstrate the validity of clinical skills assessment using standardized patients. At the AMA interim meeting, NMBE staff provided a syllabus containing 30 papers that had been published or presented at medical education and assessment conferences on the NBME standardized patient project, as well as a bibliography of 150 published papers on clinical skills assessment using standardized patients. In fact, the ECFMG has required foreign medical graduates entering residency programs in the United States to take a similar exam since 1998. The Medical Council of Canada also has required passage of a clinical skills test for licensure for more than 10 years.

Based on experience with those exams, NBME officials estimate that 8 to 10 percent of candidates will fail the CSE on their initial attempt. Ultimate failure rates, however, are expected to fall somewhere between 1 and 3 percent, or about 300 to 600 candidates per year.

Dr. Hawkins says that is approximately the same number of candidates who currently fail one or more parts of the USMLE, although NBME field tests show that the CSE will screen out a largely different cadre of students. "There's a cohort of students who can pass a knowledge-based examination but can't pass the examination designed to measure their communication skills or their history-taking skills. You're identifying a different set of competencies."

Dr. Hawkins acknowledges that the test would be expensive and that the required travel would be inconvenient. But he says NBME has kept the exam fee as low as possible. "It's more expensive to do these examinations than it is to administer written examinations. Additional costs are incurred by the personnel infrastructure, including standardized patients and standardized patient trainers, as well as site administration and maintenance."

Dr. Fleming says the projected fee is a "bare bones" price. "Neither the federation nor the national board is making any money on this exam."

As for the travel time required, Dr. Hawkins says most medical students will have to travel no more than three or four hours to reach a testing site.

The TSBME has taken no position on the exam other than to say it is statutorily required to use Step 2 of the USMLE in its licensing process. Short of legislation to remove that requirement, candidates for licensure in Texas will have to pass the CSE beginning in 2004.

Texas medical schools also say they have not taken formal positions on the exam, but most believe their students will do well on the exam. Rebecca Kirkland, MD, senior associate dean for medical education at Baylor College of Medicine, says that Baylor already uses an objective structured clinical examination as part of a first-year Patient, Physician, and Society course, as well as for measuring clinical skills of third-year students.

Mr. Shah and Mr. Newton say Texas A&M also uses objective structured clinical examinations.

James Wagner, MD, associate dean for student affairs at The University of Texas Southwestern Medical Center at Dallas, says his institution will use the results of the CSE to measure how well it is preparing graduates for medical practice.

"We'll use it just like we use the written part of the licensing exam," he said. "We use it as an external measure of how well we're doing. We have our own thoughts about what students need to learn, but we don't like to function entirely in a vacuum. We like to hear from outside sources how well our students are learning material that other people think is important."

Marc B. Hahn, DO, dean of the Texas College of Osteopathic Medicine at the University of North Texas Health Science Center, says he thinks the CSE is a step in the right direction.

"The bottom line is we need to begin to teach toward competency and to develop ways of assessing what we deem as appropriate competencies for trainees at all levels, whether it's medical student, intern, or resident," Dr. Hahn said. "Our examinations today test knowledge. Certainly, knowledge is important, but the utilization of that knowledge in a practical manner is what competency is all about."

Ken Ortolon can be reached at (800) 880-1300, ext. 1392, or (512) 370-1392; or by email at Ken Ortolon.  

February 2003 Texas Medicine Contents
Texas Medicine Back Issues