Before they were surgeons and pediatricians and internists, before they were sleep-deprived residents clocking 80-hour work weeks, before they were hard-studying medical students learning every cranny and cavity in the human body, most doctors in the United States and Canada were university students with a common goal: to ace the Medical College Admission Test, universally known as the MCAT. Starting in 2015, medical school hopefuls will share a new goal: acing the new MCAT.
The test is undergoing the fifth major revision in its history. A 22-member committee established by the Association of American Medical Colleges, which administers the exam, has made preliminary recommendations on changes that would bolster its basic science content and introduce questions on broader topics, such as population health, social sciences and ethics.
“It should reflect where medical education is headed,” says Dr. Ronald Franks, vice chair of the MCAT review committee and vice president of health sciences at the University of South Alabama in Mobile, Alabama. “I guess you could say it’s a broadening of the continuum, allowing for more understanding of how molecules work and, at the other end, how groups of patients function and how you can influence the health of cultures and subcultures of patients.”
Introduced in 1928, the MCAT is written by more than 70 000 students in the United States each year, and by about 10 000 in Canada. It was last revised in 1991 and currently consists of four subtests: biological sciences, physical sciences, verbal reasoning and a writing sample. The revised MCAT is expected to be released in 2015 and stay in use until 2030.
Recently, the revision committee made 14 preliminary recommendations on how to improve the test, including eliminating the writing sample, creating a subtest on behavioural and social sciences principles, and testing applicants’ abilities to analyze content in areas such as biostatistics, cross-cultural studies and population health (www.aamc.org/initiatives/mr5/preliminary_recommendations).
The committee solicited and received extensive feedback from various stakeholders, including many faculties of medicine, and one message became clear: The MCAT is a valuable tool, so don’t mess it up. “We had a lot of feedback from medical schools saying don’t do something that makes it less valuable to us,” says Franks.
One of the biggest potential changes will be the elimination of the writing sample, introduced in the 1991 revision to assess communication skills. This section has not proven to be a good predictor of success in medical school, and most admissions departments have not found it very useful, except for assessing the writing skills of people for whom English is a second language.
Franks believes the improved MCAT will become part of a larger toolbox for medical school admissions departments. He would like to see medical schools attempt to standardize ways of assessing particular characteristics, such as reasoning and ethical judgement, that are difficult to test on a written exam.
This could be done, for instance, by having applicants orally reason their way through an ambiguous ethical situation. Franks suggests this scenario: You are working in a food bank and someone asks for a second allotment, for a sick child who isn’t present, but that will mean another patron goes hungry.
“You can see how this is an analogy to medicine, about how to distribute limited resources to large populations,” says Franks. “How thoughtful is the student as they try to work their way through that problem? In medicine, you have to be sensitive to the needs of patients while, at the same time, realizing that you can’t always give them everything they want.”
The MCAT, though, will always be an important part of the admissions process at most institutions, particularly in the early stages when a hoard of applicants must be culled to a manageable number for the interview stage. The University of Toronto in Ontario, for example, only has space to accept 10%–12% of the applicants to its medical school, says Dr. Maureen Shandling, an associate professor in neurology and former dean of admissions at the only Canadian school that was represented on the MCAT review committee.
The original purpose of the MCAT, however, was not to thin the field but to determine who was likely to make it through medical school. Before the test was introduced, many medical students, sometimes up to half of a class, would drop out. Now less than 5% of medical students fail to graduate.
The test was not created to predict who would do well in the profession after graduation, and the intangibles that make someone a good doctor might be impossible to assess in an exam anyway, though Shandling notes that this is indeed the goal of the overall admissions process. “The Holy Grail of the admission process is to get candidates who are successful physicians, not just successful in medical school,” she says.
Studies have shown that, in terms of predicting success in medical school, the MCAT performs well, with some subtests doing a better job than others. A good showing on the biological science section, for example, correlates to good performance in the preclinical stage (years one and two) of medical school. The best predictor of success in years three and four, when focus shifts from basic science to medical practice, is the verbal reasoning subtest.
“Verbal reasoning might be a proxy for a measure of intelligence,” says Tyrone Donnon, associate professor in the medical education research unit, faculty of medicine, at the University of Calgary in Alberta.
As for possible changes to the MCAT, Donnon suggests that the test should not stray too far from its current version, though re-examining parts dropped from past editions couldn’t hurt. “My recommendations would be to stick with what they are doing well now and to look back at previous versions of subtests, particularly ones about general information and understanding society.”
Then again, previous revisions have done little to improve the MCAT’s predictive validity, according to a recent study led by Dr. Clara Callahan, dean of students and admissions for Jefferson Medical College in Philadelphia, Pennsylvania (Acad Med 2010;85:980–86). The study explored the correlation between test performance and performance in medical school and on licensing exams for three versions of the MCAT: 1962–1977, 1978–1991 and 1991–present.
Not only did predictive validity fail to improve much over successful revisions, it decreased with respect to one section of the US medical licensing exam. “Our findings provide broad support for the predictive validity of the three versions of the MCAT administered between 1970 and 2005 in relation to students’ performance in medical school and scores on licensing exams,” the study states. “However, we did not find any measurable improvement in the test’s predictive validity after major revisions to its content and score reporting were introduced in 1978 and, subsequently, in 1991.”
Still, Callahan supports some of the proposed changes to the MCAT, even if history suggests those changes may have modest effects. “There is no question that some of the things that are going to be included in the test that weren’t previously — social and behavioural sciences, research methods, biostatistics, how to read and analyze journal articles better — are things that are important for medical students and physicians,” he says. “The changes are partly to recognize the huge increase in knowledge that goes on every year in medical school, and to give people tools to keep up with that knowledge.”
Of course, there is more to being a good doctor than acquiring knowledge, and like others who work in admissions departments, Callahan would like to see medical schools find new and better ways of assessing applicants’ personal characteristics. “We are not interested in people who just sit in their rooms and study all the time and never do anything else,” she says.
“You want a smart doctor, but you also want an ethical doctor. You want a kind doctor. You want a compassionate doctor. Sometimes doctors get into trouble because they are only smart and not well-rounded. It pains medical schools if they turn out a physician who turns out to not have those characteristics. If you can get the whole package, let’s do it.”