FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records
This paper introduces FoMoH, a comprehensive benchmark of 14 clinically meaningful tasks and over 6 million patients that evaluates state-of-the-art structured EHR foundation models, revealing that while they outperform traditional baselines in discrimination and fairness under limited data, they still face significant challenges in calibration, low-prevalence settings, and cross-institutional transportability.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a computer how to be a doctor. For a long time, the only way to do this was to show the computer thousands of specific examples for every single job: "Here is what a broken arm looks like," "Here is what a fever looks like," "Here is what a heart attack looks like." This is like training a dog to fetch a specific ball; if you want it to fetch a stick, you have to start over. This method is slow, expensive, and the computer often gets confused if it sees a patient it hasn't practiced on before.
Recently, scientists have started using a new trick called "Foundation Models." Think of these as super-smart students who have read every medical textbook in the library before they ever meet a patient. They haven't been told specifically how to diagnose a broken arm yet, but because they understand the general language of medicine, they can learn that specific skill very quickly with just a few examples. The big question everyone is asking is: Do these super-students actually work better than the old way? Can they handle the messy, real-world chaos of a hospital, and are they fair to every kind of patient? This paper sets out to find the answer by putting these models through a rigorous, real-life test.
The Great Hospital Exam: Testing the Super-Students
The authors of this paper decided to stop guessing and start testing. They built a massive "exam" for six of the most advanced medical Foundation Models. Instead of just asking, "Did you get the right answer?" they asked a much deeper set of questions: "Did you get the right answer and know how sure you are?" and "Did you treat rich and poor patients, or different racial groups, fairly?"
They tested these models on over 6 million patients from two major sources: Columbia University Irving Medical Center and a public dataset called MIMIC-IV. The exam consisted of 14 different clinical tasks, ranging from predicting if a patient would die in the hospital to diagnosing chronic conditions like diabetes or schizophrenia.
Here is what the results revealed, broken down into the good, the bad, and the tricky parts.
The Good News: The "Few-Shot" Superpower
The most exciting finding is that these Foundation Models are indeed magic when data is scarce. Imagine you are a doctor in a small clinic where you've only seen 100 cases of a rare disease. A traditional computer model would probably fail because it hasn't seen enough examples to learn. But the Foundation Models, having "read" millions of records during their pre-training, could still make surprisingly good guesses.
In these "low-data" situations, the top-performing models beat the traditional methods at distinguishing between sick and healthy patients. They also seemed to be more fair. Because they learned from such a huge and diverse crowd, they didn't get as confused by different groups of people (like different races or how often people visit the doctor) as the older, specialized models did. It's as if the super-student, having met everyone in the library, is less likely to make assumptions about a stranger than the student who only studied with their own friends.
The Bad News: The "Confidence" Problem
However, there is a catch. While these models were good at saying "This patient is sick," they were sometimes terrible at saying "I am sure this patient is sick."
In medicine, knowing how likely something is to happen is just as important as knowing what will happen. If a model says a patient has a 90% chance of dying, the doctor needs to know if that number is accurate. The paper found that under limited data, these Foundation Models were often poorly calibrated. They might guess correctly, but their confidence numbers were off. It's like a weather forecaster who always predicts rain; they might be right half the time, but if they say "100% chance of rain" when it's actually sunny, they aren't trustworthy. The study suggests that while these models are great at spotting patterns, they haven't quite learned to measure their own certainty accurately yet.
The Tricky Part: The "Rare Disease" Blind Spot
The models also struggled with very rare conditions. If a disease happens in less than 1% of patients, the Foundation Models often missed the boat. The authors suggest that during their massive training, the models might have "ignored" these rare events because they were too busy learning the common stuff. It's like a student who studies for a test by reading a million pages of history but skips the one page about a tiny, obscure event that actually shows up on the exam. When the researchers tried to "fine-tune" the models (teach them the specific answer) with more data, the models didn't always get better; sometimes, they actually got worse, likely because they got confused by the tiny number of examples.
The Final Hurdle: The "New City" Problem
Finally, the paper tested if these models could travel. If a model is trained on patients in New York (Columbia), can it work on patients in California (Stanford)? The answer was a disappointing no.
When they took a model trained at Stanford and tested it at Columbia, it performed worse than a model trained locally. The authors explain that hospitals are like different cities with different dialects. One hospital might record data differently, or have patients with different habits. The "super-student" trained in one city didn't understand the slang of the other. This means that right now, you can't just download a medical AI and expect it to work everywhere; it still needs to be retrained for each specific hospital.
The Bottom Line
This paper doesn't say Foundation Models are the future of medicine yet, but it proves they have a lot of potential. They are fantastic at learning quickly when data is hard to find and are surprisingly fair to different groups of people. However, they still struggle with rare diseases, they aren't always honest about how sure they are, and they can't easily jump between different hospitals.
The authors conclude that before we let these models run the show, we need to fix their "confidence meter," teach them to pay attention to rare conditions, and figure out how to make them understand the unique "dialect" of every hospital. Until then, they are powerful tools, but they need a human doctor to double-check their work.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.