⚡ electrical engineering

Robustness of transferability estimation metrics for medical imaging

This paper investigates the robustness of transferability estimation metrics in medical imaging, revealing that their performance is highly sensitive to variations in target dataset composition and evaluation metrics, ultimately resulting in low agreement between predicted and actual model rankings.

Original authors: Niclas Claßen, Théo Sourget, Dovile Juodelyte, Rob van der Goot, Veronika Cheplygina

Published 2026-08-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Niclas Claßen, Théo Sourget, Dovile Juodelyte, Rob van der Goot, Veronika Cheplygina

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize different types of fruit. You have a super-smart robot that already knows how to sort apples, oranges, and bananas from a massive library of photos. This is called Transfer Learning: you take a model trained on one big job (the "source") and try to reuse it for a new, smaller job (the "target"), like sorting rare, weirdly shaped tropical fruits. It saves time and energy because you don't have to teach the robot from scratch.

But here's the tricky part: you have a whole shelf of different pre-trained robots. One was trained on fruit, another on cars, and a third on abstract art. Which one should you pick to help you sort your tropical fruits? To answer this, scientists invented Transferability Estimation (TE) metrics. Think of these metrics as "crystal balls" or "compatibility testers." They try to predict, without actually doing the hard work of retraining, which pre-trained robot will work best on your new task. The goal is to save time and money by picking the winner before you start the race.

However, in the world of medical imaging—where we use AI to spot diseases in X-rays or skin scans—things get messy. Medical data is often scarce and unbalanced (there are way more healthy scans than sick ones). Researchers have been using these "crystal balls" to pick the best medical AI, but they've noticed something strange: sometimes the crystal ball says Robot A is the best, and other times it says Robot B. This paper asks a simple, nagging question: Are these crystal balls actually reliable, or are they just reacting to tiny, random changes in the data?

The Great Crystal Ball Test

In this study, the authors decided to put these "compatibility testers" through a rigorous stress test. They treated the problem like a game of musical chairs, but instead of chairs, they were using different slices of medical image data.

The Setup: Miniature Populations
Imagine you have a giant jar of mixed jellybeans representing a medical dataset (like a collection of chest X-rays). Usually, researchers might grab a handful of beans to test their theories. But what if that handful happened to have mostly red beans, while the next handful had mostly blue? The results would look totally different, even though the jar is the same.

To test this, the researchers created "miniature populations." They took their medical datasets and sliced them into different sizes (5%, 10%, 25%, up to 100%). Crucially, they didn't just take one slice; they took five different slices of the same size using different random seeds (like shaking the jar differently each time). They then asked: "If we run our crystal ball tests on these different slices, do we get the same winner?"

The Findings: The Crystal Balls are Shaky
The results were eye-opening. The authors found that the rankings produced by these TE metrics were incredibly sensitive to tiny changes.

The "Seed" Effect: If they changed the random seed just a little bit, the order of the best models would shuffle. A model that was ranked #1 in one slice might drop to #5 in another slice of the same size.
The Size Matters: The smaller the slice of data (the fewer jellybeans in the handful), the more chaotic the rankings became. For very small datasets, the "crystal balls" couldn't agree on anything.
The Metric Trap: The researchers also tested what happens if you judge the "winner" using different rules. In medical imaging, we often care about AUROC (a score that handles uneven classes well) rather than just Accuracy (getting the most answers right). They found that if you switch from judging by Accuracy to judging by AUROC, the entire ranking of the models changes. A model that was #2 under one rule might be #11 under the other.

The Verdict: No Clear Winner
When the researchers compared the predictions of the TE metrics against the actual performance of the models (the "reference ranking"), the agreement was surprisingly low. Even when they used the full dataset (100% of the data), the TE metrics didn't consistently pick the right models.

The paper suggests that the problem isn't just that medical data is hard; it's that the tools we use to predict success are fragile. They change their minds based on:

How much data you give them.
Which random slice of data you pick.
Which scoring rule (Accuracy vs. AUROC) you use to decide who won.

What This Means for the Future
The authors aren't saying these metrics are useless, but they are warning us that we can't trust them blindly. If a researcher picks a single random slice of data and a single scoring rule to declare a "best" model, they might be seeing a fluke rather than a truth. The study concludes that for medical imaging, we need to be much more careful. We can't just look at one test run; we need to understand that these "crystal balls" are sensitive to the tiny details of how we set up our experiments. Until we fix this sensitivity, picking the right pre-trained model for medical AI remains a bit of a gamble.

Technical Summary: Robustness of Transferability Estimation Metrics for Medical Imaging

Problem Definition
Transfer learning (TL) is a standard paradigm in medical image classification due to data scarcity, yet selecting the optimal pre-trained source model for a specific target task remains a significant challenge. While Transferability Estimation (TE) metrics aim to predict the best source model without the computational cost of fine-tuning all candidates, their reliability is questionable. Previous research has reported conflicting performances of TE metrics, often attributed to variations in experimental setups. Furthermore, most TE metrics are designed and evaluated on natural images (e.g., ImageNet) using accuracy as the primary metric. In contrast, medical imaging often requires robustness to class imbalance, typically measured by the Area Under the Receiver Operating Characteristic Curve (AUROC). Current literature lacks a systematic investigation into how TE metrics behave when the target dataset representation varies (e.g., sample size, random seed) and how the choice of evaluation metric for the reference ranking influences TE metric assessment.

Methodology
The authors conducted a systematic study using the MedMNIST v2 dataset collection (comprising 2D biomedical image datasets) and ImageNet as a source pool. The study is divided into two primary experiments:

Impact of Target Dataset Representation (Ex1):
- Miniature Populations: The authors constructed "miniature populations" from eight target datasets by sampling 5%, 10%, 25%, 50%, and 75% of the training data. These subsets were stratified by class label to maintain distribution and generated using five different random seeds to test stability.
- Metrics Evaluated: Seven existing TE metrics were tested: H-score, LEEP, NLEEP, LogME, NCTI, PARC, and SFDA.
- Stability Measures: The study measured intra-metric stability (agreement of rankings across different random seeds for the same subset size), reference stability (agreement between subset rankings and the full dataset ranking), and inter-metric stability (agreement between different TE metrics).
- Correlation Coefficients: To avoid bias from a single metric, the authors utilized Spearman's rank correlation ( $\rho$ ), Kendall's Tau ( $\tau$ ), and weighted Kendall's Tau ( $\tau_w$ ).
Influence of Evaluation Metric for Reference Ranking (Ex2):
- Reference Rankings: Source models were fine-tuned on the full target training sets. The resulting performance was ranked twice: once optimizing for Accuracy and once for AUROC.
- Comparison: The rankings generated by TE metrics were compared against these two distinct reference rankings to determine if TE metrics are robust to the choice of the ground-truth performance metric.

Key Contributions
The paper provides three main contributions:

Sensitivity to Dataset Variations: The study demonstrates that TE metrics are highly sensitive to small variations in the target dataset, including changes in sample size and random seeds. Rankings of source models fluctuate significantly even within the same dataset when using different random seeds.
Impact of Evaluation Metric: The authors show that the choice of evaluation metric (Accuracy vs. AUROC) fundamentally alters the reference ranking of source models. Consequently, the assessment of which TE metric is "best" depends heavily on which reference ranking is used.
Benchmarking: The work provides benchmark results for MedMNIST, highlighting that current TE metrics show low agreement with reference rankings in medical imaging contexts, regardless of the evaluation metric used.

Results

Instability: TE metric rankings are not stable across different random seeds or subset sizes. For example, a source model might be ranked first in one random seed and fifth in another for the same target dataset (e.g., Breast).
Metric-Specific Behavior: While LEEP showed relatively higher consistency on smaller datasets, H-score exhibited the least stability. Some metrics (NLEEP, SFDA) produced NaN values or failed to differentiate models on specific datasets due to mathematical limitations (e.g., ill-conditioned covariance matrices or rank-deficient scatter matrices).
Low Agreement: There is a consistently low agreement between the rankings produced by TE metrics and the reference rankings (both Accuracy and AUROC). This low agreement persists even when using 100% of the training data, suggesting the issue is not merely a data scarcity problem but a domain-specific limitation.
Evaluation Metric Dependency: The reference ranking itself is unstable; models ranked highly by Accuracy are often ranked differently by AUROC. Since TE metrics are typically optimized for accuracy-based rankings, they fail to reliably predict performance when AUROC is the target.

Significance and Claims
The authors claim that the low reliability of TE metrics in medical imaging is not solely due to the choice of evaluation metric but is a broader, domain-specific issue. The study challenges the current practice of relying on single random seed experiments or single evaluation metrics to validate TE methods. The findings suggest that existing TE metrics, largely developed for natural images, do not generalize robustly to medical imaging tasks where data is scarce and class imbalance is prevalent. The paper concludes that future work must account for the instability of target dataset representations and the sensitivity of reference rankings before TE metrics can be considered reliable tools for source model selection in medical applications.

The Great Crystal Ball Test

Technical Summary: Robustness of Transferability Estimation Metrics for Medical Imaging

More like this