Subgroup performance analysis of adaptation strategies for chest X-ray foundation models
This study reveals that while attention-pooling adaptation strategies for chest X-ray foundation models yield superior overall accuracy, they do not consistently improve subgroup fairness, demonstrating that stronger encoding of protected attributes does not necessarily correlate with larger performance disparities and that fairness must be assessed directly on a per-task basis rather than inferred from representation strength or general performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of medical imaging, artificial intelligence has grown powerful enough to look at a chest X-ray and spot signs of disease with remarkable speed. These systems, often called foundation models, are like vast libraries of visual knowledge trained on millions of images. They learn to recognize patterns in the human body that even experienced doctors might miss. However, a shadow hangs over this progress: these models can inadvertently learn to rely on shortcuts. Instead of looking strictly at the medical evidence, a computer might learn to guess a patient's race or gender based on subtle clues in the image, such as the type of hospital equipment used or the way the X-ray was taken. If the model uses these shortcuts to make its diagnosis, it can become unfair, performing well for some groups of people while failing others. This is a critical concern because medical AI is increasingly being used to make real decisions about patient care, and fairness is just as important as accuracy.
Researchers at Imperial College London set out to understand how the specific methods used to adapt these powerful models for hospital use affect this fairness. They focused on a popular chest X-ray model called Rad-DINO, which acts as a frozen, pre-trained brain that does not change during the study. To make this brain useful for spotting specific diseases like pneumonia or lung lesions, the team had to attach a new, smaller decision-making layer on top. They tested three different ways to build this layer. The first was a simple, direct connection. The second added a small, multi-layered processing unit. The third, more complex method, acted like a sophisticated filter that gathered information from many different depths of the model's internal layers and combined them to make a decision. The team wanted to see if the more complex, expressive method would not only improve the model's ability to find diseases but also reduce the unfair gaps in performance between different groups of patients, such as men and women, or people of different races.
The researchers evaluated these methods using a massive dataset of chest X-rays from a hospital in the United States, ensuring the test group was balanced to represent different demographics fairly. They looked at eight different types of lung conditions. The results showed that the most complex method, which gathered information from multiple layers, did indeed produce the best overall accuracy for finding diseases. It was the most capable at distinguishing between sick and healthy lungs. However, the team found a surprising and counterintuitive truth: being better at the main task did not automatically make the system fairer. In fact, the impact of the complex model on fairness varied depending on the specific disease and subgroup; for some conditions, it reduced performance gaps compared to simpler models, while for others, it introduced larger disparities. There was no consistent trend where one method was universally fairer than the others across all tasks.
Perhaps the most revealing discovery was about how the model "thought" about protected characteristics like race. The researchers measured how strongly the model's internal signals could predict a patient's race, sex, or the angle of the X-ray. They found that the complex model encoded race much more strongly than the simple ones, meaning it had a very clear internal signal for a patient's racial background. A natural assumption might be that a model that "knows" race so well would use that knowledge to bias or create bias. Yet, the data showed no such link. The model that encoded race the most weakly actually produced the largest gaps in performance between racial groups. Conversely, the model that encoded race the strongest did not necessarily create the biggest unfairness. This suggests that simply measuring how much a model knows about a person's identity is not enough to predict if it will treat them fairly.
The study also explored whether changing which layers of the model were combined in the complex method would help. They tried mixing early layers, which capture basic shapes, with later layers, which understand complex concepts. They found that while the choice of layers changed the overall accuracy for some diseases, it did not create a consistent pattern for fairness. Sometimes a specific combination helped one group, while hurting another, with no clear rule to follow. The researchers concluded that there is no single "magic switch" or standard setting that guarantees both high accuracy and fairness. A method that works well for one disease might fail for another, and a method that is fair for one group might not be for another.
Ultimately, the work demonstrates that improving the power of medical AI does not solve the problem of bias on its own. The relationship between how well a model performs and how fairly it treats different people is unpredictable and depends entirely on the specific task at hand. The authors argue that we cannot assume a smarter model will be a fairer one, nor can we assume that a model that hides identity information is safe. Instead, fairness must be checked directly for every single application, looking at the actual results for each group of people, rather than inferring fairness from the model's internal complexity or its overall score. The path to trustworthy medical AI requires this direct, case-by-case scrutiny, ensuring that the benefits of advanced technology are shared equally by all patients.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.