💻 computer science

Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy

This paper proposes AURCC, a label-free selection criterion based on pseudo-label discrepancy that effectively ranks foundational models for medical image classification under distribution shift without requiring target-domain annotations or fine-tuning.

Original authors: Juan Iñaki Larrea, Lucas Mansilla, Enzo Ferrante

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan Iñaki Larrea, Lucas Mansilla, Enzo Ferrante

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medical imaging, computers have become remarkably skilled at reading X-rays, often matching the performance of human specialists. These systems are trained on vast libraries of images, learning to spot signs of disease like pneumonia. However, a significant problem arises when these trained systems are moved from one hospital to another. Just as a driver accustomed to the wide, sunny roads of one country might struggle on the narrow, rainy streets of another, an artificial intelligence model trained on data from one hospital often falters when faced with the different scanners, patient populations, or imaging protocols of a new facility. This shift in conditions is known as a distribution shift, and it can cause a model's accuracy to drop dramatically. The core dilemma for doctors and hospital administrators is this: if they have several powerful, pre-trained AI models available, how do they know which one will work best at their specific hospital before they actually try it? The usual way to find the answer requires testing the models on new images and checking the results against expert human labels, but obtaining those labels is expensive, time-consuming, and often impossible in the very settings that need the technology most.

A team of researchers from Argentina has proposed a new way to solve this selection puzzle without needing any new human labels. They focused on a specific challenge: choosing the best chest X-ray model for a new hospital when the only available data from that hospital is unlabeled images. The researchers built their method on a framework called SUDO, which acts like a quality control check for AI systems. Instead of asking "Is this diagnosis correct?"—which requires a human to say "yes" or "no"—their method asks a different question: "Does the model's confidence in its own predictions match the reality of the data it is seeing?" To do this, the system takes the unlabeled images from the new hospital and groups them based on how confident the AI is in its predictions. For example, it might group all the images where the AI is very sure it sees pneumonia, and another group where it is very sure it does not.

The researchers then run a clever internal test on these groups. They pretend, for a moment, that the images in a specific group are actually the opposite of what the model thinks, and they check if the model's logic holds up. If the model is truly reliable, its internal logic should break down when forced to accept a false assumption about a group of images. If the logic holds up too easily, it suggests the group is "contaminated" with mixed types of images that the model cannot distinguish well. By measuring how much the model's logic wobbles across all these different groups of confidence, the researchers calculate a single score that reflects the model's reliability. They call this score the Area Under the Reliability-Completeness Curve, a number that tells them how well the model is likely to perform without ever seeing a single human-verified label from the new hospital.

To test this idea, the researchers gathered six different advanced AI models designed for medical imaging. They simulated a real-world scenario where these models were trained on data from three different large hospitals and then asked to predict pneumonia in a fourth, distinct hospital where no labels were available. They compared their new scoring method against the traditional approach, which simply ranks models based on how well they performed on a small sample of labeled data from the original training hospitals. The results showed that their new method was highly effective. When the amount of labeled data from the original hospitals was large, the traditional method worked well, as expected. However, in the more difficult and common situation where the labeled data was scarce—representing the reality of many resource-constrained hospitals—the new method consistently outperformed the traditional approach. In these small-data scenarios, the new score correctly identified the best-performing model with a high degree of accuracy, matching the true performance rankings almost perfectly.

The study highlights a crucial insight for the future of medical AI: the best way to choose a model for a new environment is not always to look backward at how it performed on old data, but to look forward at how it behaves on the new, unlabeled data itself. By analyzing the structure of the new data and how the model interacts with it, the researchers found a way to predict success where traditional methods fail. This approach is particularly valuable because it is lightweight and does not require powerful computers; the entire process can be run on standard hardware using only the mathematical representations of the images. While the study focused specifically on pneumonia detection and chest X-rays, the underlying principle offers a promising path for hospitals to safely and efficiently adopt new AI tools, ensuring that the right technology is deployed in the right place, even when the experts needed to verify it are not available.

Technical Summary: Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift

1. Problem Statement

The deployment of foundation models (specifically vision-language models like CLIP and its biomedical variants) in medical imaging faces a critical bottleneck: distribution shift. Clinical images often originate from different institutions, scanners, or patient populations than the training data, leading to unpredictable performance degradation.

Conventionally, selecting the best model for a new, unlabeled target domain requires ground-truth annotations to evaluate performance metrics (e.g., AUC). However, obtaining such annotations in deployment settings is costly and often infeasible due to the need for expert radiologist time. This creates a practical dilemma: given a labeled source dataset and an unlabeled target domain, how can one rank candidate foundation models by their expected target performance without access to target labels?

2. Methodology

The authors propose a label-free selection criterion based on the SUDO framework, adapted here to rank multiple candidate models. The core metric is the Area Under the Reliability-Completeness Curve (AURCC).

The SUDO Procedure:

Partitioning: For a candidate model $\phi_m$ , the unlabeled target data is partitioned into $K$ quantile bins based on the model's predicted positive-class probabilities.
Pseudo-Label Discrepancy: For each bin $b$ $b$ , the method constructs two auxiliary binary training sets:
- Set 1: The bin samples are assigned a pseudo-negative label, combined with ground-truth positive samples from the source training set.
- Set 2: The bin samples are assigned a pseudo-positive label, combined with ground-truth negative samples from the source training set.
Auxiliary Evaluation: A logistic regression classifier is trained on each set and evaluated on a held-out source split. The SUDO score for the bin is the difference in AUC between the negative and positive assumptions: $SUDO_b = s_0(b) - s_1(b)$ $S U D O_{b} = s_{0} (b) - s_{1} (b)$ .
- A large absolute value indicates the bin is dominated by a single class (high reliability).
- A value near zero indicates class contamination (low reliability).
Aggregation: The procedure is repeated over 300 stochastic runs and averaged. The AURCC is computed by sweeping symmetric quantile thresholds from the distribution extremes toward the center, plotting reliability (mean $|SUDO|$ ) against completeness (fraction of samples covered). The area under this curve serves as the final ranking score.

Experimental Regimes:
The method is evaluated under two adaptation strategies using source labels only:

Zero-Shot (ZS): No task-specific training. Probabilities are derived via text-prompt ensembling and image-text similarity.
MLP-Probe (MP): A shallow Multi-Layer Perceptron is trained on frozen source embeddings to create a task-specific decision boundary. The SUDO procedure then operates on the probabilities generated by this probe.

3. Key Contributions

Label-Free Ranking Framework: The paper adapts SUDO from a tool for assessing a single deployed system to a mechanism for model selection among multiple foundation models without target labels.
Systematic Benchmarking: The authors evaluate six chest X-ray foundation models (BioMedCLIP, CXR-CLIP, CheXzero, MedCLIP, MedImageInsight, and CLIP-base) across three inter-hospital distribution shift scenarios (MIMIC-CXR, NIH ChestX-ray14, and CheXpert as sources; PadChest as the target).
Resource-Constrained Analysis: The study explicitly investigates the performance of the ranking criterion as the size of the available labeled source data decreases, simulating resource-constrained settings (RCS).

4. Results

Correlation with Ground Truth: The AURCC ranking strongly correlates with the true target performance (AUC $_{target}$ ). In the MLP-Probe regime, the Spearman correlation ( $\rho$ ) reaches 0.943 ( $p < 0.05$ ) across all scenarios and bin counts ( $K \in \{5, 10, 20\}$ ).
Comparison to Baseline: The natural baseline of ranking models by their accuracy on the held-out source data (AUC $_{source}$ ) performs well when the source dataset is large (e.g., $\rho = 1.000$ for MIMIC). However, as the source size shrinks, the baseline degrades.
Advantage in Small-Source Scenarios:
- For the NIH dataset ( $N \approx 954$ ) and CheXpert ( $N \approx 113$ ), AURCC consistently outperforms the source-AUC baseline.
- At the extreme reduction of $N=50$ held-out source samples, AURCC provides a more accurate ranking than the source baseline in all three scenarios.
- In the Zero-Shot regime, AURCC performs comparably to the source baseline, suggesting that the advantage of AURCC is most pronounced when task-specific adaptation (via the MLP probe) allows the method to exploit structural information in the target distribution that a simple source metric cannot capture.

5. Significance and Claims

The paper claims that AURCC offers a practical solution for the "label-free ranking problem" in medical AI deployment. Its primary significance lies in resource-constrained settings where:

Target labels are unavailable: The method requires no ground truth from the target domain.
Source labels are scarce: Unlike traditional source-based evaluation, which fails when the labeled source set is small, AURCC leverages the structure of the unlabeled target data to maintain ranking quality.
Computational efficiency: The entire pipeline operates on embeddings and runs on CPU, allowing institutions to rank models locally on commodity hardware without expensive fine-tuning or data transfer.

The authors note that while the evaluation is restricted to a binary pneumonia classification task and a single target domain (PadChest), the results suggest that AURCC captures structural information about the target distribution that goes beyond simple aggregate source metrics, making it a robust tool for selecting foundation models under distribution shift.