(NewsNation) — From 2018 to 2020, a Department of Defense-run medical school embarked on an experiment to see if it could promote more diversity in the medical field by de-emphasizing the Medical College Admission Test (MCAT).

But the well-meaning reform by the admissions committee at Uniformed Services University of the Health Sciences (USU) may have backfired, actually leading to less diversity.

The committee has long been frustrated at the composition of America’s medical schools, explained Dr. Mike Arnold, a faculty member who served on the committee.

“It’s really a sad thing because I believe 30 years ago, Black men represented 2% of med school admissions, and I think it’s about the same still,” he said. (In 2019, less than 3% of the medical student body consisted of Black men, according to one study.)

Standardized tests, like the SAT, have long been blamed for racially disparate outcomes in admissions. Underrepresented minorities tend to score lower on the MCAT.

An attempt to remove bias

The admissions committee at USU decided to “mask” — that is, conceal — the MCAT scores for applicants at or above the 51st percentile. That means that the admissions committee would know that applicants met a certain standard but could not differentiate between candidates for their scores beyond that.

“The idea was to take away the input of the score that we thought was introducing bias into our admissions process,” Arnold said.

Instead, the committee relied on other admissions criteria to form a more holistic evaluation — everything from college transcripts to letters of recommendation to interviews.

But by 2021, committee members decided to revert to the previous system.

“A majority of the committee members felt when we got rid of it that we were unnecessarily taking data away from ourselves,” Arnold said.

Arnold and some of his colleagues decided to study the impact of the masking with the intention of bringing the system back; the results of that study were published in August.

“We came in with the thought that we were disappointed that the program had gone away, and we wanted to prove the benefit with the thought of re-proposing it the next year,” he said.

‘It was completely the opposite of our hypothesis’

The researchers compared the rate first-generation or underrepresented students were admitted during the masking experiment to the three school years prior (2014-2016).

Minority students’ odds of admission were cut in half when the school masked MCAT scores, they found.

First-generation college students were slightly more likely to get admitted, but their odds still fell far from where they were under the previous system.

“It was completely the opposite of our hypothesis … we would’ve expected it would make it twice as likely for them to get in,” Arnold said.

While the results were eye-opening for Arnold and his colleagues, the authors acknowledge that the outcomes apply to USU during a specific period and may not apply elsewhere.

“We wanted to de-emphasize that test because we know it had bias. But we also know that any process with people has bias, too,” he said.

Arnold pointed to other aspects of admissions processes that carry their own biases, like college essays.

“If I don’t know a physician, it’s a lot different than if my parents are physicians and know other physicians. The people who are going to review my essays are going to know exactly what their colleagues are looking for. So we don’t think about that kind of bias,” he said.

The study made him re-evaluate his view of the MCAT.

“This imperfect test seems to have made our process better,” he said. “Even though it has problems.”