⚡ electrical engineering

Feature-level Site Leakage Reduction for Cross-Hospital Chest X-ray Transfer via Self-Supervised Learning

This paper demonstrates that directly measuring site leakage via post-hoc probes reveals that multi-site self-supervised learning significantly improves cross-hospital chest X-ray transfer performance, whereas adversarial site confusion reduces leakage metrics but fails to reliably enhance classification accuracy, thereby challenging the assumption that feature invariance alone guarantees effective domain adaptation.

Original authors: Ayoub Louaye Bouaziz, Lokmane Chebouba

Published 2026-04-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ayoub Louaye Bouaziz, Lokmane Chebouba

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor who has spent years learning to diagnose pneumonia by looking at chest X-rays. You've become an expert at reading the X-rays from Hospital A. You know exactly how their machines look, how the images are processed, and even the little quirks in their photos.

Now, you get a job at Hospital B. They use different machines, different software, and take pictures in a slightly different way. When you try to use your "Hospital A" skills on "Hospital B" X-rays, you start making mistakes. Why? Because your brain (or in this case, the AI model) learned to rely on clues that aren't actually about pneumonia.

Maybe Hospital A always takes pictures in the morning with bright light, while Hospital B takes them at night with dim light. Your model learned, "Bright light = Pneumonia," instead of "White spots in the lungs = Pneumonia." This is called Site Leakage: the model is "leaking" information about where the photo was taken, which confuses it when it sees a photo from a new place.

This paper is like a detective story trying to solve this problem using two main strategies.

The Problem: The "Accent" in the Data

The authors noticed that most AI research assumes that if you train a model on one hospital, it will magically work on another. They call this "invariance" (meaning "doesn't change"). But the authors say, "Wait a minute, let's actually measure if the model still knows which hospital the photo came from."

They built a "lie detector" test (called a probe). They took the AI's brain (the part that looks at the X-ray) and asked it: "Can you tell me if this photo came from Hospital A or Hospital B?"

If the AI says "Yes, 99% sure," it has high leakage. It's still relying on the hospital's "accent."
If the AI says "I have no idea, it's a 50/50 guess," it has low leakage. It has learned to ignore the accent and focus only on the disease.

The Solution: Two Approaches

The researchers tried two methods to fix this "accent" problem.

1. The "Polyglot" Method (Multi-site Self-Supervised Learning)

The Analogy: Imagine teaching a student to speak a language not by giving them a textbook, but by letting them listen to thousands of conversations between people from different countries, without telling them what the words mean.

How it works: They let the AI look at thousands of X-rays from both Hospital A and Hospital B without telling it which patient has pneumonia. The AI has to figure out, "Hey, these two pictures look similar because they are both lungs, not because they are from the same hospital."
The Result: This was the big winner. By exposing the AI to different "accents" right from the start, it learned to ignore the hospital quirks and focus on the actual lungs. When they tested it on a third hospital (Hospital C), it got much better at diagnosing pneumonia.

2. The "Amnesia" Method (Adversarial Site Confusion)

The Analogy: Imagine trying to teach a student to ignore the accent by hiring a strict teacher who yells, "Wrong! You can't tell me which country this is from!" every time the student tries to guess. This is called "adversarial training."

How it works: They added a special "nagging" component to the AI. Every time the AI's brain tried to guess which hospital an image was from, the system punished it. The goal was to force the AI to forget the hospital identity completely.
The Result: This was a mixed bag.
- Good news: The "lie detector" test showed that the AI did forget the hospital identity better than before. The "leakage" went down.
- Bad news: The AI didn't necessarily get better at diagnosing pneumonia. In fact, it became unstable. Sometimes it worked great, sometimes it failed miserably. It was like trying to amputate a leg to stop a headache; you removed the "leak," but you might have hurt the model's ability to see the disease.

The Big Takeaway

The paper's main conclusion is a reality check for the AI community:

Don't just assume: You can't just say, "We made the model invariant." You have to measure it. The authors proved that even when you try to remove the "accent," a little bit of it often remains.
Exposure is better than erasure: The best way to make an AI work across different hospitals is to show it many different hospitals during its training (The Polyglot Method).
Forcing forgetfulness is risky: Trying to mathematically force the AI to forget the hospital (The Amnesia Method) reduces the "leakage" numbers, but it doesn't guarantee the AI will actually get better at its job. It can make the AI's performance wobbly and unpredictable.

In short: To build a doctor's assistant that works everywhere, don't try to wipe its memory of where it learned. Instead, take it on a tour of every hospital in the country so it learns to see the patient, not the place.

1. Problem Statement

Cross-hospital generalization in medical imaging is a critical deployment risk. Models trained on data from one hospital often fail when deployed to another due to domain shift caused by differences in scanners, acquisition protocols, compression, and patient cohort demographics.

The Core Issue: Models often learn "shortcuts" by associating site-specific signals (e.g., scanner artifacts) with pathology labels rather than learning true disease features.
The Gap: While many papers claim to achieve "site invariance" through domain adaptation, few actually measure the extent to which site information remains in the model's representations. Most assume invariance without quantifying "site leakage."
Goal: To directly measure site leakage in feature representations and evaluate how different training strategies (Self-Supervised Learning vs. Adversarial Confusion) affect both leakage and downstream transfer performance.

2. Methodology

2.1 Experimental Setup

Datasets:
- NIH ChestX-ray14: Used as a labeled source for supervised training (pneumonia detection) and part of the SSL pool.
- CheXpert: Used as an unlabeled source for SSL pretraining and a secondary evaluation target.
- RSNA Pneumonia Detection Challenge: Used exclusively as a held-out target site for transfer evaluation (unseen during supervised training).
Backbone: ResNet-18 initialized with ImageNet weights, converted to single-channel input.
Task: Binary classification for pneumonia.

2.2 Proposed Framework

The authors propose a two-stage evaluation pipeline:

Multi-Site Self-Supervised Learning (SSL) Pretraining:
- The encoder ( $f_\theta$ ) and projection head ( $p_\psi$ ) are pretrained on unlabeled images from NIH and CheXpert using SimCLR-style contrastive learning (InfoNCE loss).
- This aims to learn robust features without relying on pathology labels.
Feature-Level Adversarial Confusion (CanonicalF):
- A variant where a site classifier ( $g_\phi$ ) is attached to the backbone features ( $h$ ) used by the downstream task.
- Gradient Reversal: The site classifier is trained to predict the site, while the encoder is trained to minimize this prediction (via gradient reversal), forcing the backbone features to be "site-invariant."
- This is applied in addition to the SSL loss.
Downstream Evaluation:
- The encoder is frozen.
- A linear pneumonia classifier is trained on labeled data from a single source site (e.g., NIH).
- Performance is evaluated on unseen sites (RSNA, CheXpert).

2.3 Leakage Measurement Protocol (Key Innovation)

Instead of assuming invariance, the authors introduce a post-hoc linear probe to quantify site leakage:

Mechanism: A multinomial logistic regression is trained on the frozen embeddings to predict the acquisition site ( $s$ ).
Metrics:
- Probe on $h$ (Backbone features): Features used by the downstream classifier.
- Probe on $z$ (Projection features): Features used only for SSL.
Baseline: Chance accuracy is 0.50 for binary (NIH vs. CheXpert) and 0.33 for three-site (NIH, CheXpert, RSNA) probes.
Significance: High probe accuracy indicates high "site leakage," meaning the model still relies on site-specific cues.

3. Key Contributions

Leakage Measurement Protocol: A rigorous method to quantify site information in representations using post-hoc probes, moving beyond theoretical claims of invariance to empirical measurement.
Multi-Site SSL Baseline: Demonstrates that multi-site SSL pretraining significantly improves cross-site transfer when used as a frozen backbone, including Leave-One-Site-Out (LOSO) scenarios.
CanonicalF (Adversarial Variant): Introduces a feature-level adversarial head applied specifically to the backbone features used for downstream tasks to test the limits of invariance assumptions.
Comprehensive Evaluation: Reports results across multiple transfer directions (NIH $\to$ others, CheXpert $\to$ others) and LOSO settings, with statistical significance (mean $\pm$ std over 3 seeds).
Empirical Insight: Establishes that multi-site SSL is the primary driver of transfer, while adversarial confusion is a leakage reduction tool that does not guarantee performance gains and may increase variance.

4. Key Results

4.1 Transfer Performance (AUC)

ImageNet Initialization: Baseline RSNA AUC = 0.6736.
SSL-Only (Frozen): RSNA AUC improved to 0.7804 (+0.1068).
CanonicalF (SSL + Adversarial): RSNA AUC dropped to 0.7241 with significantly higher variance (std 0.0956 vs 0.0197 for SSL-only).
- Observation: Adversarial training did not consistently improve AUC and introduced instability.

4.2 Site Leakage (Probe Accuracy)

Before SSL (ImageNet): Probe accuracy on backbone features ( $h$ ) was 0.9962 (near perfect site separation).
After SSL-Only: Accuracy dropped to 0.9890.
After CanonicalF: Accuracy dropped significantly to 0.8504.
- Observation: CanonicalF successfully reduced measurable site leakage on the backbone features (from ~99% to ~85%), but the leakage remained far above chance (50%).

4.3 Stability

Adversarial training (CanonicalF) increased the coefficient of variation for RSNA AUC from ~0.025 (SSL-only) to ~0.13, indicating that forcing invariance makes the model highly sensitive to initialization and hyperparameters.

5. Significance and Conclusion

Reframing Invariance: The paper argues that "site invariance" is often an unproven assumption. Instead, researchers should report leakage metrics alongside transfer performance.
Primary Driver: The study concludes that multi-site SSL is the most effective strategy for improving cross-hospital transfer. It naturally learns features that generalize better than ImageNet initialization.
Role of Adversarial Learning: Feature-level adversarial training can reduce measurable site leakage, but it does not guarantee better downstream performance. In fact, it can degrade performance by removing predictive signals that happen to correlate with the site, or by introducing training instability.
Practical Implication: For medical imaging deployment, relying on SSL pretraining is more robust than relying on adversarial domain adaptation to force invariance. Leakage reduction should be measured explicitly, as "reduced leakage" does not automatically equate to "better clinical performance."

Final Takeaway: The paper shifts the paradigm from "achieving invariance" to "measuring and managing leakage," demonstrating that while SSL drives transfer, adversarial methods are a double-edged sword that requires careful validation.