🤖 AI

Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift

This paper benchmarks the robustness of 15 foundation models for mammography under domain shifts, revealing that while mammography-specific vision-language models achieve the best out-of-distribution performance, generalization is not guaranteed by domain-specific pretraining alone and requires rigorous dataset-level evaluation.

Original authors: Giang Nguyen, Raghav Mehta, Emma A. M. Stanley, Tian Xia, Thi Hao Nguyen, Hieu Pham, Ben Glocker

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giang Nguyen, Raghav Mehta, Emma A. M. Stanley, Tian Xia, Thi Hao Nguyen, Hieu Pham, Ben Glocker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot eye that's been trained to look at millions of pictures of everything in the world—cats, cars, sunsets, and even regular X-rays. You want to use this robot to spot breast cancer in mammograms (special X-rays of the breast). The big question is: If you take this robot, which learned on "general" pictures, and drop it into a brand-new hospital with different machines and different patients, will it still work? Or will it get confused and start seeing things that aren't there?

This paper is like a giant, rigorous "stress test" for 15 different versions of these robot eyes. The researchers didn't just check if the robots were good at the homework they studied for; they threw them into 12 different countries, 15 different datasets, and 3 tricky medical tasks to see how they handled the unexpected.

The Big Surprise: "Specialized" Isn't Always "Stronger"

You might think that a robot trained only on mammograms would be the ultimate champion. After all, it's like a student who only studied for the final exam, right?

The paper actually argues against this idea. They found that just because a model was trained specifically on mammograms doesn't mean it will be robust when the data changes. In fact, some models that were "adapted" to mammograms performed worse than models that were trained on general natural images or general radiology. It's like a student who memorized the answers to one specific test but fails when the teacher changes the question format, while a student who learned how to think about pictures in general does surprisingly well.

The Real Champions: The "Talkers" vs. The "Lookers"

So, who won the stress test? The paper suggests that the winners are a special group called Vision-Language Models (VLMs).

Think of these models as robots that don't just look at the X-ray; they also read the doctor's notes attached to it.

MaMA and Mammo-FM (the two top contenders) are like detectives who look at the picture and read the case file together. They achieved the strongest average performance across the board.
Mammo-FM was the best at predicting the severity of the condition (BI-RADS), scoring a 0.688 ± 0.016 on average for out-of-distribution tests.
MaMA was the best at spotting breast density and cancer status, scoring 0.865 ± 0.006 for density and 0.718 ± 0.014 for cancer.

However, there's a twist! The paper measured that even the best models aren't perfect. When they moved from the training data to new, unseen data (Out-of-Distribution or OOD), the performance dropped. For example, in the "BI-RADS" task, the average drop was -0.092. This means that even the smartest robot gets a bit less confident when it leaves its comfort zone.

The "Generalist" Who Can't Be Ignored

Here is the most playful part of the story: A model called DINOv3, which was trained on natural images (like photos of landscapes and animals) and never saw a mammogram during its training, turned out to be a fierce competitor.

DINOv3 is like a generalist artist who has never seen a medical scan but has such a good eye for shapes and textures that it can still guess the medical details better than some specialists.

On the cancer task, DINOv3 scored 0.677 ± 0.015, which is very close to the specialized models.
On density, it scored 0.848 ± 0.006.

The paper suggests that having a "natural image" brain is actually a very strong baseline, and sometimes it's hard to beat even with extra medical training.

The "One-Size-Fits-All" Myth is Broken

The researchers also looked at how these models behaved in different places. They found that a model that is great at one hospital might be terrible at another.

Mammo-FM was amazing at handling weird, non-standard X-ray machines (like those that scan film or use contrast dye), but MaMA was consistently better at spotting breast density across all datasets.
Why? The paper suggests it's about how they were taught. MaMA was trained using text reports that explicitly mentioned density (like a recipe listing ingredients), so it learned density really well. Mammo-FM was trained on real clinical reports that focused more on the final diagnosis (cancer or no cancer), so it became better at spotting the "big picture" outcomes rather than the specific texture of the tissue.

The Bottom Line

The main takeaway is that you can't just assume a model is good because it's "specialized" or because it scored high on the training data.

The paper proves that:

Robustness is tricky: A model that works great on the data it was trained on often struggles when it sees new data from different countries or machines.
Language helps: Models that can read medical reports (VLMs) generally do better than models that just look at pictures, but it depends on the specific task.
Context matters: The best model for one job (like checking density) might be the worst for another (like checking cancer severity).

The authors measured these results using a strict method where they froze the robot's brain and only trained a tiny "head" to make the final decision. They suggest that to truly know if a medical AI is ready for the real world, we need to test it on many different external datasets, not just the one it learned from. Until we do that, we can't be sure if the robot is a genius or just a lucky guesser.

Technical Summary: Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift

Problem Statement

Foundation models (FMs) are increasingly adopted as image feature extractors for mammography, typically via a workflow where a pretrained backbone is frozen and a lightweight classification head is trained on a labeled source dataset. However, the robustness of these representations under external domain shifts remains unclear. Prior research indicates that large-scale or modality-specific pretraining does not inherently guarantee robust out-of-distribution (OOD) generalization. This challenge is exacerbated in mammography due to heterogeneous clinical labels (breast density, BI-RADS severity, and cancer status) that vary in granularity, prevalence, and annotation protocols across different datasets. Consequently, high in-distribution (ID) performance often fails to translate to OOD generalization, necessitating a systematic benchmark to evaluate whether mammography representations generalize beyond their source data.

Methodology

Benchmark Construction

The authors constructed a harmonized benchmark comprising 15 public mammography datasets spanning 12 countries/regions (North America, Europe, Asia, Middle East, Africa, and Oceania). The benchmark covers three distinct clinical tasks:

Breast Density: Evaluated at the image level (A, B, C, D).
BI-RADS Assessment: Evaluated at the exam level (1–5).
Cancer Status: Evaluated at the exam level (Binary: Cancer/Non-cancer).

Labels were harmonized to ensure task compatibility. For each task, a specific source dataset served as the training ground for the linear classifier:

BI-RADS: VinDr-Mammo (Source)
Density: EMBED (Source)
Cancer Status: RSNA-site2 (Source)

Model Families

The study evaluated 15 foundation-model backbones grouped into four non-overlapping families:

Natural Image SSL: DINOv2, DINOv3.
General Radiology SSL: RAD-DINO, RayDINO.
Mammography-Adapted SSL: DINOv2-EMBED, DINOv3-EMBED, MAE-EMBED, VersaMammo (further pretrained on EMBED).
Vision-Language Models (VLMs):
- General Medical: BioMedCLIP, UniMedCLIP, MedSigLIP.
- Mammography-Specific: MaMA, GLAM, Mammo-CLIP, Mammo-FM.

Evaluation Protocol

Frozen-Backbone Linear Probe: All models were evaluated as frozen image encoders. A task-specific linear classifier was trained on the source dataset and evaluated on held-out ID test sets and 12 task-compatible OOD datasets.
Preprocessing: Unified preprocessing pipeline with images resized to 1024 × 768.
Aggregation: For exam-level tasks (BI-RADS, Cancer), image-level probabilities were aggregated via per-class max pooling and renormalization.
Metrics: Macro-averaged one-vs-rest AUROC for BI-RADS and Density; Binary AUROC for Cancer. 95% confidence intervals were calculated via bootstrapping.

Key Contributions

Harmonized Benchmark: Creation of a unified benchmark spanning 15 datasets and 12 regions, covering three clinically distinct tasks with harmonized labels.
Unified Evaluation: A systematic comparison of 15 foundation models across three tasks using a consistent frozen-backbone linear-probe protocol.
Multi-Dimensional Analysis: An analysis of OOD behavior that extends beyond aggregate scores to include model-family comparisons, per-dataset evaluation, and feature-space inspection (UMAP).

Results

ID–OOD Performance Alignment

BI-RADS: Strong rank alignment between ID and OOD performance ( $\rho = 0.89$ ), but a mean OOD gap of $-0.092$.
Density: High ID ceiling (0.93–0.95) but similar OOD gap ($-0.092$) and lower rank correlation ( $\rho = 0.76$ ).
Cancer Status: Weakest ID/OOD rank relationship ( $\rho = 0.61$ ) with a slightly positive mean gap ( $+0.017$ ), attributed to noise in label definitions across datasets.

Model-Level Performance

Mammography-Specific VLMs: Achieved the strongest overall mean OOD performance.
- Mammo-FM: Best for BI-RADS ( $0.688 \pm 0.016$ ).
- MaMA: Best for Density ( $0.865 \pm 0.006$ ) and Cancer Status ( $0.718 \pm 0.014$ ).
Vision-Only Baselines: DINOv3 emerged as the most competitive vision-only model, outperforming several mammography-adapted models despite lacking specific mammography pretraining (e.g., $0.635$ on BI-RADS vs. $0.646$ for MaMA).
Mammography Adaptation: Continued pretraining on a single mammography dataset (e.g., EMBED) did not consistently improve OOD generalization. In some cases (e.g., DINOv3-EMBED), it resulted in performance degradation compared to the natural-image baseline.

Pretraining Factors

VLM vs. Vision-Only: VLMs generally achieved higher mean OOD AUROCs than vision-only models across all tasks.
Domain Alignment: Mammography-specific VLMs outperformed general medical VLMs, suggesting that alignment with mammography language is more critical than visual exposure alone.
Task Dependency: The leading model is task-dependent rather than universal. For instance, MaMA's density pretraining (using tabular metadata) correlated with superior density performance, while Mammo-FM's report-based pretraining correlated with better cancer status performance on datasets with exam-level outcomes.

Dataset-Level Heterogeneity

Strong average OOD performance does not imply uniform behavior across all external datasets.
Acquisition Variability: Mammo-FM showed superior generalization to non-standard modalities (e.g., contrast-enhanced and scanned film) compared to MaMA.
Feature Space: UMAP analysis revealed that useful representations preserve clinical signals (e.g., density, BI-RADS) while retaining dataset and acquisition structures (e.g., manufacturer, view). Strong external generalization does not require complete domain invariance.

Significance and Claims

The paper claims that mammography pretraining alone is insufficient for robust OOD generalization. Instead, generalization depends on the interaction between the pretraining objective, model type (vision-only vs. vision-language), and the diversity of pretraining datasets.

Key takeaways include:

VLM Superiority: Mammography-specific VLMs (MaMA, Mammo-FM) provide the strongest OOD performance, but their advantages are task- and dataset-dependent.
Robust Baselines: DINOv3 remains a highly competitive vision-only baseline, challenging the assumption that modality-specific pretraining is always necessary.
Evaluation Standard: Dataset-level OOD evaluation is a central criterion for assessing the clinical utility of FM representations in mammography, as ID performance is a poor proxy for external robustness.
Limitations: The study utilizes a linear-probe protocol to measure representation quality, not the absolute best achievable performance (which might be reached via full fine-tuning). Future work is suggested to explore full fine-tuning, uncertainty estimation, and confidence calibration.