💬 NLP

Self-supervision drives representational convergence in medical foundation models more than clinical supervision

This study reveals that representational convergence in medical foundation models is primarily driven by self-supervised pretraining objectives rather than clinical supervision or model scale, resulting in modest alignment that supports cross-encoder transfer learning but fails to fully capture clinical judgment or semantic similarity.

Original authors: Soroosh Tayebi Arasteh, Sebastian Ziegelmayer, Mahshad Lotfinia, Lisa Adams, Sven Nebelung, Jakob Nikolas Kather, Daniel Truhn

Published 2026-07-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Soroosh Tayebi Arasteh, Sebastian Ziegelmayer, Mahshad Lotfinia, Lisa Adams, Sven Nebelung, Jakob Nikolas Kather, Daniel Truhn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of very different robots to understand the world. You give them all the same task: look at pictures of medical scans and describe what they see. In the world of artificial intelligence, there is a popular idea called the "Platonic Representation Hypothesis." Think of it like this: if you train enough robots on enough data, they should all eventually start thinking in the exact same way. They should all build an identical internal map of reality, where a "broken bone" looks the same to Robot A as it does to Robot B, no matter who built them or what specific lessons they learned. This idea is exciting because if all medical AI models converged on this single, perfect map, doctors could swap them out like batteries without breaking their tools. But is this actually happening? Do these digital brains really agree on what a disease looks like, or are they just speaking different dialects of the same language?

A team of researchers decided to put this idea to the test with a massive, controlled experiment. They gathered 18 different "image encoders" (the parts of AI that look at pictures) and 7 "text encoders" (the parts that read reports) from various labs and companies. These models ranged from tiny ones with 7 million parameters to giants with 27 billion parameters. The researchers didn't just compare them; they dissected them. They trained new models that were identical in every way except for one thing: the goal they were trying to achieve. Some were trained with "self-supervision" (learning patterns from the images alone), some with "clinical supervision" (learning from doctors' labels), and some with "image-text" pairing (matching pictures to reports).

Here is the twist: the researchers found that the idea of a universal, shared map is mostly a myth, but a very specific kind of reality exists instead. They discovered that self-supervised learning (learning from the images themselves) is the true glue that makes these models agree with each other. In fact, adding a doctor's label to the training actually made the models less similar to their peers, not more. The models trained to just "look and learn" without specific labels ended up aligning much better (about 40% similarity in chest X-rays) than those trained with specific medical labels (only about 21%).

However, this agreement is modest and has strict limits. It only happens within the same type of picture (X-rays agree with X-rays, but not with skin photos). It doesn't get better just because the model gets bigger or smarter; a giant 27-billion-parameter model isn't necessarily more aligned than a smaller one. Most surprisingly, these image models and the text models that read medical reports do not share a common map at all. Their internal languages are completely different, and as you feed them more data, they actually drift further apart.

Despite these limitations, there is a silver lining. Even though the models don't agree on a perfect, shared geometry, they are "good enough" to be interchangeable for practical tasks. The researchers showed that a diagnostic tool trained on one model could be transferred to a completely different model and still work with about 85% of its original accuracy. It's like having two different maps of a city that don't look exactly the same, but if you know how to translate the landmarks, you can still drive from point A to point B without getting lost. The study concludes that we shouldn't wait for AI models to magically converge into one perfect brain through sheer size; instead, we need to design them with the right training goals to ensure they can at least talk to each other when it matters most.

Technical Summary: Self-supervision drives representational convergence in medical foundation models more than clinical supervision

Problem Statement
Medical image encoders from diverse research groups are increasingly treated as interchangeable, predicated on the "Platonic Representation Hypothesis." This hypothesis suggests that as models scale and train on diverse clinical tasks, their internal representations converge toward a shared statistical model of the world. However, whether this convergence is real, what drives it (scale, clinical supervision, or training objectives), and whether it is clinically usable remain untested. Existing similarity measures are often fragile, and claims of convergence may depend heavily on evaluation regimes. Furthermore, it is unknown if such convergence is uniform across patient subgroups or if it extends to cross-modal alignment between images and clinical text.

Methodology
The authors present a controlled dissection of 25 open-weight foundation models (18 image encoders and 7 biomedical text encoders) spanning 7 million to 27 billion parameters, released between 2020 and 2025. The study utilizes a multi-modal evaluation corpus comprising:

Chest Radiography: 650,982 images from six datasets across three countries (US, Spain, Vietnam), including a pediatric cohort.
Other Modalities: Histopathology (22,000 patches), retinal fundus (3,738 images), dermoscopy (5,987 images), and mammography (2,500 images).
Text: Paired image-report corpora (MIMIC-CXR, CheXpert) and image-caption pairs (Quilt-1M).

Key Experimental Designs:

Representational Alignment: The primary metric is Centered Kernel Nearest-Neighbor Alignment (CKNNA) and mutual k-nearest-neighbor (mKNN) alignment at $k=10$ , measuring the overlap of nearest-neighbor graphs between encoder pairs on fixed image pools. A randomly initialized Vision Transformer (ViT) serves as the "floor" for chance-level alignment.
Controlled Training Matrix: To isolate the driver of convergence, the authors trained 12 encoders from a shared self-supervised initialization, varying only the training objective (masked-image self-supervision, clinical-label supervision, or image-text contrastive learning) while holding data, architecture (ViT-S/ViT-B), and scale constant.
Synthetic Generative Model: A synthetic model was constructed where image features mix a clinically supervised signal subspace with nuisance subspaces. Encoders were trained under self-supervised and supervised objectives across varying levels of label informativeness ( $\alpha$ ) to test if supervised objectives inherently drive convergence.
Downstream Usability: The study tested functional interchangeability via:
- Cross-encoder transfer: Training a linear classifier in one encoder's space and applying it to others.
- Feature stitching: Learning an affine map to translate features from one encoder to another's frozen classifier head.
- Drift detection: Using cross-encoder neighbor disagreement to detect distribution shifts.
Clinical Grounding: Two board-certified radiologists provided reference labels for 630 chest radiographs and 300 clinical-similarity triplets to validate whether the shared geometry matches expert diagnostic agreement or similarity judgments.

Key Results

Training Objective Drives Convergence: The primary driver of representational alignment is the self-supervised objective, not clinical supervision. Encoders trained with the same self-supervised objective showed the highest alignment (e.g., 40.4% CKNNA on chest radiography), significantly outperforming label-supervised (21.1%) and image-text (3.3%) encoders. Adding a clinical label objective to an otherwise identical backbone reduced convergence relative to peers.
Modest but Real Convergence: Convergence is real but modest. Within-modality alignment (e.g., 38.3% for histopathology, 10.8% for chest radiography) exceeds the random initialization floor but remains far from perfect agreement.
No Scaling Trend: Convergence does not scale with model size, downstream performance (AUROC), or release year. Larger or more capable models do not sit closer to the shared representation (Spearman $\rho = 0.302, p = 0.223$ for size).
Within-Modality Phenomenon: Convergence is strictly within-modality. Cross-modal alignment between image and text encoders sits at the random floor level and decays toward zero as the evaluation pool size increases, failing to support the hypothesis of a shared image-text geometry in this setting.
Clinical Structure vs. Taxonomy: The shared geometry recovers the empirical clinical co-occurrence of findings (Spearman $\rho = 0.615$ ) but does not align with the administrative ICD-10 hierarchy.
Subgroup Instability: Alignment is uneven across patient subgroups. While differences were not statistically robust after correcting for sample size artifacts, the consensus alignment was least reliable for under-represented groups (e.g., Native American, Patient Refused ethnicity), suggesting representational gaps.
Functional Interchangeability: Despite weak geometric alignment, a linear classifier trained in one encoder's relative coordinate system retained ~85% of its oracle performance when transferred to other encoders and external sites. Feature stitching successfully recovered oracle performance. However, a drift detector based on cross-encoder neighbor disagreement failed to detect distribution shifts.
Expert Judgment: The shared geometry did not reproduce how radiologists judge case-level clinical similarity; encoders performed at chance levels on triplet prediction tasks, even though they maintained diagnostic accuracy against reference labels.

Significance and Claims
The paper claims to provide the first controlled account of what produces representational convergence in medical foundation models, inverting the prevailing assumption that scale and clinical supervision are the primary drivers. The authors conclude that:

Convergence is Objective-Dependent: Interoperability is a property of the self-supervised training objective, not an emergent property of scale or clinical supervision.
Design Implications: To achieve interoperability, the field must design for the specific objective that aligns models rather than waiting for scale to manufacture a universal representation.
Practical Usability: A modest, clinically organized convergence is sufficient to enable "encoder-agnostic" clinical tools (via relative representations and stitching), provided they are built in a common coordinate system.
Limitations of Convergence: The shared geometry is not a model of radiologist similarity judgments, does not bridge the image-text gap, and is not uniform across patient populations. It should be validated against specific clinical tasks and subgroups rather than assumed to be universally robust.

The study emphasizes that alignment is a regime-dependent measurement, sensitive to pool size and evaluation metrics, and that negative results (e.g., lack of cross-modal convergence, lack of scaling trends) are as critical as positive findings for understanding the limits of medical foundation models.

More like this