Cross-Scanner Reliability of Brain MRI Foundation Model Embeddings: A Travelling-Heads Study

This study demonstrates that the cross-scanner reliability of brain MRI foundation model embeddings varies significantly based on pretraining strategy, with models incorporating biological metadata achieving scanner-robust performance comparable to traditional morphometric baselines, while purely self-supervised models exhibit substantial scanner-induced variance.

The Gist

The Big Idea: The "Traveling Head" Experiment

Imagine you have a very smart AI assistant that looks at brain scans to understand how healthy a person is, how old they are, or if they have a disease. This AI is like a super-sensor that turns a complex brain image into a single "fingerprint" (a list of numbers) representing that person's brain.

The researchers wanted to know a crucial question: Does this AI fingerprint stay the same if you take the picture with a different camera?

In the real world, hospitals use different MRI machines (made by Siemens, Philips, or GE). These machines are like different camera brands. A photo of a cat taken with a Canon looks slightly different than one taken with a Sony. Usually, we don't care because the cat is still a cat. But in medical AI, if the "fingerprint" changes just because the camera changed, the AI might think the patient's brain has changed, when really, it's just a different machine.

To test this, the researchers used a "Traveling Heads" dataset. They took 20 healthy volunteers and scanned their brains on eight different MRI machines across the UK. This is like taking the same 20 people to eight different photo studios and taking their portraits.

The Contenders: Five AI Models vs. The Old Standard

The team tested five different "Foundation Models" (the new, fancy AI brains) and compared them to FreeSurfer (the old, trusted standard, like a classic film camera).

They asked: When we move from Machine A to Machine B, does the AI's "fingerprint" of the person stay consistent, or does it get confused?

The Results: The Good, The Bad, and The Ugly

The results were dramatic. The models fell into two distinct camps:

🏆 The Champions: "The Biology Guides"

Two models (AnatCL and y-Aware) were the stars of the show.

• The Analogy: Imagine these models were taught by a strict biology teacher. Before they learned to look at the brain, they were told, "Ignore the lighting and the camera brand. Focus only on the shape of the nose and the color of the eyes."
• The Result: When the volunteers moved to a new machine, these models gave almost the exact same fingerprint. They were so reliable that they actually performed better than the old standard (FreeSurfer). They successfully ignored the "camera noise" and focused on the "biological truth."

📉 The Strugglers: "The Self-Taught Students"

Three models (BrainIAC, BrainSegFounder, and 3D-Neuro-SimCLR) failed the test miserably.

• The Analogy: These models were like students who were told to "Just look at the picture and guess." They weren't taught what to ignore. So, they learned to memorize the camera brand.
• The Result: When the same person was scanned on a different machine, these models gave a completely different fingerprint. In fact, the models were so obsessed with the machine that if you showed them a fingerprint, they could guess which machine took the picture with 90% accuracy! They had confused the scanner with the person.

Why Did This Happen? (The Secret Sauce)

The researchers discovered that the architecture (the shape of the AI) or the amount of data it was trained on didn't matter. The only thing that mattered was how it was taught.

• The Biology-Guided Models: These were trained using "contrastive learning" where they were explicitly shown biological facts (like age or brain volume) alongside the images. This forced the AI to learn that these specific features matter, and everything else (like the scanner) is just noise.
• The Self-Supervised Models: These were trained by simply looking at millions of images and trying to guess which ones were similar. Without a teacher pointing out "biology is key," they accidentally learned that "Siemens machines look like this" and "GE machines look like that." They learned the hardware instead of the human.

The Takeaway: Why Should You Care?

This paper is a massive warning sign for the future of medical AI.

1. Don't trust the "Black Box" blindly: Just because an AI is "pre-trained" on millions of images doesn't mean it's ready for the real world. If you use a model that hasn't been taught to ignore scanner differences, your medical diagnosis might depend on which hospital you visit, not your actual health.
2. The "Traveling Head" test is essential: Before we trust these AI models in hospitals, we need to run this specific test: Scan the same person on different machines. If the AI's answer changes, the model is broken for multi-hospital use.
3. Teach the AI what matters: The solution isn't just "more data." It's teaching the AI to care about biology and ignore the equipment.

In short: The best AI models are the ones that know how to look past the camera and see the person. The others are just taking pictures of the camera itself.

Technical Summary

1. Problem Statement

Foundation Models (FMs) for brain MRI are increasingly used as pretrained backbones for clinical tasks (e.g., brain age prediction, disease classification). However, a critical gap exists in understanding whether the internal representations (embeddings) learned by these models are reproducible across different MRI scanners.

• The Risk: If FM embeddings systematically shift based on the acquisition hardware (scanner vendor, model, or site) rather than biological variation, downstream analyses may reflect technical artifacts rather than true biology.
• The Gap: While scanner effects are known in raw images, no study has systematically quantified the cross-scanner reliability of high-dimensional FM embeddings using a rigorous "travelling-heads" design (where the same subjects are scanned on multiple scanners). Previous studies relied on synthetic perturbations or single-site data, which fail to capture real-world hardware variability.

2. Methodology

Dataset: ON-Harmony

The study utilized the ON-Harmony dataset, a multi-site, multi-vendor travelling-heads resource:

• Subjects: 20 healthy adults (13M/7F, age 19–50).
• Scanners: 8 distinct 3T MRI scanners from 3 major vendors (Siemens, Philips, GE).
• Design: Each participant was scanned on 6 different scanners (120 between-scanner sessions). A subset of participants underwent repeat scans on single scanners for within-scanner reliability assessment.
• Modality: T1-weighted (T1w) structural scans.

Models Evaluated

The authors evaluated five architecturally diverse Foundation Models (used in frozen inference mode) and one conventional baseline:

1. AnatCL: 3D ResNet-18. Pretrained with biology-guided contrastive learning (incorporating cortical morphometrics and age).
2. y-Aware: 3D DenseNet-121. Pretrained with biology-guided contrastive learning (incorporating chronological age).
3. BrainIAC: 3D Vision Transformer (ViT-B). Pretrained with pure self-supervised learning (SimCLR) on a large, diverse dataset.
4. BrainSegFounder: 3D Swin Transformer. Pretrained with pure self-supervised learning (reconstruction + rotation + contrastive) on UK Biobank data.
5. 3D-Neuro-SimCLR: 3D ResNet-18. Pretrained with pure self-supervised learning (SimCLR).
6. Baseline: FreeSurfer morphometrics (cortical thickness, surface area, volumes).

Preprocessing

To ensure faithful reproduction, model-specific preprocessing pipelines were used (e.g., CAT12 for AnatCL/y-Aware, TurboPrep for 3D-Neuro-SimCLR, custom pipelines for Transformers).

Reliability Metrics & Analysis

• Intraclass Correlation Coefficient (ICC):
  • Between-scanner (ICC(2,1)): Measures reproducibility across different scanners.
  • Within-scanner (ICC(3,1)): Measures test-retest reproducibility on the same scanner.
  • Thresholds: Poor (<0.50), Moderate (0.50–0.75), Good (0.75–0.90), Excellent (>0.90).
• Variance Decomposition: ANOVA-based method to partition embedding variance into Subject (biological), Scanner (technical), and Residual components.
• Scanner Fingerprinting: A linear SVM trained to predict scanner identity from embeddings (higher accuracy = higher scanner bias).
• Subject Identification: Nearest-neighbor retrieval to see if the same subject can be identified across different scanners.

3. Key Results

Reliability Spectrum

The models fell into three distinct tiers based on between-scanner reliability:

1. Excellent/Good (Biology-Guided Models):
   • AnatCL: ICC = 0.97 (Excellent). Outperformed FreeSurfer (0.93). 100% of dimensions were "good" or better.
   • y-Aware: ICC = 0.81 (Good).
   • FreeSurfer Baseline: ICC = 0.93.
2. Poor (Pure Self-Supervised Models):
   • BrainIAC: ICC = 0.45 (Poor).
   • BrainSegFounder: ICC = 0.31 (Poor).
   • 3D-Neuro-SimCLR: ICC = 0.25 (Poor).
   • Note: These models had 23–58% of their embedding variance attributable to scanner identity.

Variance Decomposition

• AnatCL & FreeSurfer: >84% of variance was attributed to Subject differences; scanner variance was ~12%.
• BrainSegFounder: 57.9% of variance was attributed to Scanner identity (dominated by technical noise).
• 3D-Neuro-SimCLR: Dominated by Residual noise (40.9%), with scanner variance (32.0%) exceeding subject variance (27.0%).

Fingerprinting & Identification

• Scanner Fingerprinting: Pure SSL models (BrainSegFounder, 3D-Neuro-SimCLR) allowed scanners to be identified with >89% accuracy. AnatCL reduced this to 45.5% (still above chance).
• Subject Identification: AnatCL, y-Aware, and FreeSurfer achieved 100% accuracy in identifying the same subject across different scanners. BrainIAC dropped to 49.7%.

Design Factor Analysis

• Pretraining Strategy is Key: The strongest correlate of reliability was the pretraining objective. Models incorporating biological metadata (AnatCL, y-Aware) were robust. Pure self-supervised models were not.
• Architecture & Scale Irrelevant: Reliability did not correlate with model architecture (CNN vs. Transformer), embedding dimensionality, or pretraining dataset size (e.g., BrainIAC used 32k scans but performed poorly; AnatCL used only 4k scans but performed best).

4. Key Contributions

1. First Travelling-Heads Benchmark: This is the first study to systematically quantify the cross-scanner reliability of brain MRI foundation model embeddings using a rigorous travelling-heads design.
2. Identification of the "Scanner Bias" in FMs: The study demonstrates that purely self-supervised foundation models often learn scanner-specific signatures that overwhelm biological signals, leading to poor cross-scanner reliability.
3. Validation of Biology-Guided Pretraining: The authors provide empirical evidence that incorporating biological metadata (age, morphometrics) into the contrastive learning objective is the most effective strategy for achieving scanner-invariant representations.
4. Decoupling Architecture from Reliability: The study proves that simply increasing model size or changing architecture (e.g., to Transformers) does not solve the scanner bias problem; the learning objective is the critical factor.

5. Significance and Implications

• Clinical Deployment Warning: Using "off-the-shelf" foundation models (especially pure self-supervised ones) for multi-site clinical studies or longitudinal monitoring without domain adaptation or harmonization is risky. The embeddings may reflect the scanner rather than the patient's condition.
• Guidance for Future Model Development: To build robust medical FMs, pretraining strategies must explicitly incorporate biological supervision or scanner-aware objectives (e.g., treating scanner as a nuisance factor).
• Role of Harmonization: While statistical harmonization (e.g., ComBat) can reduce scanner variance, the study suggests that selecting a model with inherently robust embeddings (like AnatCL) is a more principled first step than relying solely on post-hoc correction.
• Reproducibility Standard: The authors advocate for routine cross-scanner reliability checks on small travelling-heads cohorts before deploying FMs in multi-site clinical pipelines.

In conclusion, the paper establishes that pretraining strategy is the primary determinant of cross-scanner reliability in brain MRI foundation models, with biology-guided models significantly outperforming pure self-supervised approaches.