💻 computer science

Frozen Brain-MRI Foundation Models Are Site Fingerprints

This paper reveals that frozen brain-MRI foundation models inherently encode acquisition site information as a dominant, linearly decodable fingerprint that often surpasses clinical variables, posing significant attribution risks and creating a trade-off between removing site bias and preserving anatomical segmentation accuracy.

Original authors: Saman Rahbar

Published 2026-08-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Saman Rahbar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery using a super-smart AI assistant. In the world of medical science, this AI is a "Foundation Model"—a massive, pre-trained brain that has already read millions of MRI scans of human heads. Scientists love these models because they are like having a genius medical student who has seen every type of brain imaginable, ready to help diagnose diseases or study anatomy without needing to be retrained from scratch. The big hope is that when you ask this AI to look at a new scan, it sees only the biology: the shape of the brain, the size of the ventricles, and the health of the tissue. It's supposed to be a pure window into the human body, ignoring everything else.

But here is the catch: medical scans aren't taken in a vacuum. They are taken in hospitals, using specific machines (scanners) that might be made by different companies, set to different speeds, or tuned to different settings. Just like how a photo taken with an iPhone looks different from one taken with a Samsung, even if it's the same person, MRI scans carry the "fingerprint" of the machine that took them. For years, scientists have worried that these machine differences might trick the AI, making it think a healthy brain from Hospital A is sick just because the picture looks slightly different from Hospital B. The big question was: Does the AI actually learn to see the brain, or is it secretly memorizing the style of the camera?

This paper investigates exactly that. The researchers took several of these "frozen" AI models—meaning they locked the AI's brain so it couldn't learn anything new—and tested what it was actually seeing. They treated the AI like a suspect and asked: "If I show you a brain scan, can you tell me which hospital it came from?" The answer was shocking. The AI didn't just see the brain; it saw the hospital with crystal-clear vision. In fact, the AI could identify the hospital (the "site") with about 90% accuracy, which was far better than it could identify the patient's age, sex, or even whether they had autism.

The most surprising twist? This wasn't because the AI had been "trained" to be a spy. The researchers tested this by using a version of the AI that had never been trained on any data at all—it was just a random collection of numbers. Even this "dumb," random AI could identify the hospital with nearly the same 90% accuracy. This means the "fingerprint" isn't a secret code the AI learned; it's baked into the raw pictures themselves. The way the machines capture the light and texture of the brain is so distinct that any system looking at the image, smart or dumb, can tell where it came from.

The paper also explored whether we could fix this. They tried to "scrub" the hospital information out of the AI's brain using mathematical tricks. They found that for simple tasks, like just looking at the whole brain and saying "yes" or "no," they could remove the hospital fingerprint without hurting the results. However, for detailed tasks like drawing a map of every tiny part of the brain, removing the fingerprint was disastrous. It turned out that the "hospital style" and the "brain anatomy" were tangled together in the AI's brain. When they cut out the hospital part, they accidentally cut out the tiny, deep structures of the brain too, like the thalamus and the hippocampus, making the AI unable to see them at all.

So, the main takeaway is a warning to scientists: Don't assume these powerful AI tools are purely seeing anatomy. They are also seeing the scanner. And because this "fingerprint" is so deeply rooted in the image itself, you can't just train the AI better to ignore it; you have to be very careful about how you use the AI's answers, especially when you are trying to map the tiny, intricate details of the human brain.

Technical Summary: Frozen Brain-MRI Foundation Models Are Site Fingerprints

Problem Statement
Multi-site pooling is standard in neuroimaging to achieve necessary sample sizes, but it introduces systematic, non-biological variation due to differences in scanner vendors, field strengths, and acquisition protocols. While a substantial literature exists on harmonizing hand-engineered features (e.g., via ComBat) or training domain-invariant models, a newer practice involves using frozen foundation models (FMs) pretrained on large MRI/CT corpora as off-the-shelf encoders. Practitioners assume these embeddings capture anatomy, attaching lightweight heads for downstream tasks without auditing the representation itself. This paper investigates a critical, unasked question: How much of a frozen brain-FM embedding encodes the acquisition site versus clinical or anatomical signal? If site information is deeply embedded, it silently confounds every downstream analysis built upon these shared representations.

Methodology
The author conducted a rigorous audit of frozen 3-D encoders across two independent cohorts (ABIDE-I and ABIDE-II) and five network depths.

Data & Preprocessing: T1-weighted MRI volumes from ABIDE-I (6 sites) and ABIDE-II (15 sites) were used. Preprocessing was deliberately minimal (resampling to 1.7mm isotropic, intensity scaling) to reflect how practitioners feed data to frozen encoders, avoiding skull-stripping or spatial registration.
Encoders: Three frozen encoders from the SwinUNETR family were tested:
1. Brain-pretrained: Self-supervised on ~41k UK Biobank scans.
2. CT-pretrained: Publicly released MONAI self-supervised weights.
3. Random-init: The same architecture with random weights (fixed seed).
- Control Architectures: Plain ViT and 3-D ResNet (randomly initialized) were also tested.
Probing: Global-average-pooled (GAP) embeddings from five hierarchical stages (L0–L4) were probed using:
- Linear Probes: Multinomial logistic regression for categorical targets (site, sex, autism diagnosis) and ridge regression for age.
- Nonlinear Probes: One-hidden-layer MLPs to test for hidden nonlinear structure.
- Metrics: Chance-corrected balanced accuracy ( $d$ ) for classification and $R^2$ for regression.
Controls & Mitigation:
- Raw Voxel Control: Probing site directly from downsampled images (no encoder).
- Population Residualization: Regressing out age, sex, and diagnosis to isolate acquisition effects.
- Removal: Iterative Null-space Projection (INLP) and ComBat were applied to remove the site subspace.
- Downstream Impact: A trained decoder was attached to the frozen encoder for dense segmentation (using SynthSeg silver labels) to measure the cost of site removal on anatomical fidelity (Dice score).

Key Results

Site is the Dominant Signal: Across all encoders and layers, acquisition site is decodable with high accuracy (~0.9 balanced accuracy), significantly exceeding the decodability of clinical variables (sex, age, autism diagnosis). The site-dominance ratio ranges from 2–6× on ABIDE-I and 1.6–3× on ABIDE-II.
Intrinsic, Not Learned: The "site fingerprint" is not a product of pretraining.
- Randomly initialized encoders achieve ~0.9 site decodability, matching or exceeding pretrained models.
- Site is decodable (~0.95) directly from raw downsampled images without any encoder.
- The effect persists across three architecture families (Swin, ViT, ResNet) and multiple random seeds.
- Conclusion: The fingerprint reflects low-level image statistics inherent to the acquisition process, which any architecture preserves.
Acquisition-Driven, Not Population-Driven: Residualizing measured population covariates (age, sex, diagnosis) leaves site decodability essentially unchanged (95–99.8% retained), indicating the signal is driven by acquisition parameters rather than population composition.
Linear Accessibility: Nonlinear (MLP) probes do not outperform linear probes, confirming the site information is linearly accessible and not hidden in complex nonlinear structures.
Depth Profile: Site decodability generally increases with network depth, peaking at the bottleneck (L4), whereas clinical signals (like age) often peak mid-network and drop at the bottleneck.
Mitigation Costs:
- Global Readouts: Removing the site subspace via INLP or ComBat reduces site decodability to near chance (0.94 → 0.07/0.00) with negligible impact on global classification tasks.
- Dense Segmentation: Removing the site subspace is destructive for dense segmentation. While bottleneck removal is Dice-neutral, removing the site subspace across all scales collapses cross-site Dice scores (0.84 → 0.18). A matched-rank random-direction control confirms this is specific to the site subspace.
- Entanglement: Site and anatomy occupy an entangled linear subspace. Removing site directions preferentially destroys small, deep subcortical structures (e.g., thalamus, putamen) while sparing larger peripheral tissues.

Significance and Claims
The paper argues that the assumption that frozen foundation model embeddings are "anatomy-bearing" summaries is flawed; they are, in fact, site fingerprints.

Causal Shift: Unlike previous work suggesting site effects are learned confounds, this study demonstrates the effect is intrinsic to the architecture acting on low-level image statistics. Therefore, pretraining hygiene or larger datasets will not remove it; explicit invariance objectives or downstream removal are required.
Audit Necessity: The author recommends that site decodability be audited and reported as routinely as demographic tables before pooling embeddings.
Context-Dependent Mitigation: The paper claims that while site removal is low-cost for pooled/global readouts, it is not free for dense segmentation tasks due to the entanglement of site and anatomical features. Practitioners must match their mitigation strategy to their downstream head.
Privacy Implication: Frozen embeddings shared across sites leak acquisition provenance (~0.9 accuracy), posing a site-attribution risk for federated learning, though not individual re-identification.

The author releases an open audit toolkit to support the reproduction of these findings and the auditing of future brain-MRI foundation models.

Technical Summary: Frozen Brain-MRI Foundation Models Are Site Fingerprints

More like this