🤖 machine learning

Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

This paper introduces the Spectral Alignment Score (SAS), an asymmetric metric that reveals hidden modality imbalances in medical Vision-Language Models by showing that clinical reports often contain less structural information than medical images, a critical diagnostic insight that existing symmetric metrics fail to capture.

Original authors: Alessandro Gambetti, Qiwei Han, Cláudia Soares, Hong Shen

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alessandro Gambetti, Qiwei Han, Cláudia Soares, Hong Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: A Broken Translator in the Hospital

Imagine you have a super-smart translator that learned to speak "Image" and "Text" by reading millions of books and looking at millions of photos from the internet. This translator is great at matching a picture of a dog with the word "dog" or a sunset with "beautiful."

But when doctors try to use this translator in a hospital, it starts to fail. A doctor uploads an X-ray of a broken bone, and the translator can't find the matching medical report. It's like the translator is fluent in "Internet English" but has forgotten how to speak "Hospital English."

The researchers in this paper asked: Why is it failing? And more importantly, which side of the conversation is the problem? Is the translator bad at understanding the pictures, or is it bad at understanding the medical reports?

The Old Tools: The "Average" Score

Before this paper, scientists used tools to measure how well the translator was doing. These tools gave a single score, like a grade on a test.

The Flaw: These tools treated the picture and the text as a team. If the team got a "C," the tool just said, "The team is doing poorly." It didn't tell you if the picture was the weak link or the text was. It was like a coach saying, "The team lost," without telling you if the quarterback threw bad passes or the defense missed tackles.

The New Tool: The "Spectral Alignment Score" (SAS)

The authors created a new tool called SAS. Think of SAS as a two-way mirror that lets you see the conversation from both sides separately.

Instead of giving one grade, SAS asks two questions:

If I look at the text, how much of the picture can I guess? (Text $\to$ Image)
If I look at the picture, how much of the text can I guess? (Image $\to$ Text)

By comparing these two answers, SAS can spot an imbalance.

The Big Discovery: The Picture is "Richer" Than the Report

When the researchers tested this on medical data, they found something surprising that the old tools missed:

In the "Internet" world (Natural Data): The pictures and the text were balanced. You could guess the picture from the text, and the text from the picture, equally well.
In the "Hospital" world (Medical Data): There was a huge imbalance.
- The Finding: The medical pictures (X-rays, MRIs) contain a massive amount of detailed structural information. However, the medical reports (the text) are often short, vague, or use specific jargon that doesn't capture all the details in the image.
- The Analogy: Imagine a picture of a complex machine with 100 moving parts. The text description only says, "This machine is broken."
  - If you look at the text, you can't guess the details of the machine (Text $\to$ Image is weak).
  - If you look at the picture, you can guess the text is about a broken machine (Image $\to$ Text is strong).
- The Result: The SAS tool showed that in hospitals, the image holds more information than the text can express. The text is the "bottleneck."

Why This Matters for Doctors

The paper claims that this new tool is the best at predicting whether a system will actually work for finding medical records (retrieval).

Old Way: You train a system, test it, and if it fails, you guess why.
New Way (SAS): You use SAS before you even deploy the system. It tells you: "Hey, your system is failing because the text descriptions aren't detailed enough to match the complex X-rays."

A Real-World Example from the Paper

The researchers tested this on dental X-rays (panoramic radiographs).

Case A: An X-ray showed a simple issue, and the report matched it well. The SAS scores were balanced.
Case B: An X-ray showed a very complex surgery with screws and grafts. The report was a bit generic.
- The SAS tool showed a huge gap: The picture had tons of detail, but the text score was low. The tool correctly identified that the text was failing to capture the complexity of the image.

Summary

The Problem: Medical AI struggles because it was trained on internet data, not hospital data.
The Old Mistake: Previous tools gave a single score that hid why the AI was failing.
The Solution: The new SAS tool splits the score in two, showing exactly which side (image or text) is the weak link.
The Discovery: In medicine, images are often more detailed than the reports describing them. The text is the weak link, not the image.

This tool helps developers know exactly where to fix the AI: they need to make the text descriptions richer to match the detailed pictures, rather than just trying to make the pictures "better."

Technical Summary: Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

Problem Statement
Vision-Language Models (VLMs) frequently underperform when applied to medical image-text data, despite their success in natural domains. While existing representation alignment metrics (e.g., CKA, SVCCA, CORAL, MMD, RMG) can detect that alignment degrades under domain shift, they are inherently symmetric. These metrics collapse cross-modal alignment into a single bidirectional score, rendering them blind to which modality drives the degradation. In medical imaging, where visual structures are often information-dense while paired clinical reports are concise and domain-specific, this directional asymmetry is critical. Current tools cannot distinguish whether the failure stems from the image encoder failing to capture text nuances or the text encoder failing to recover complex visual structures.

Methodology
The authors introduce the Spectral Alignment Score (SAS), a metric designed to decompose cross-modal alignment into directional, eigenmode-specific components.

Mathematical Formulation: Given centered representation matrices for images ( $X$ ) and text ( $Y$ ), SAS projects both modalities onto the principal eigenbasis of an anchor modality (e.g., $X$ ). It computes eigenvalue-weighted per-eigenmode correlations ( $\rho_k$ ) over the top spectral directions defined by a quantile parameter $q$ .
Directional Scores: This yields two directional scores: $S_{X \to Y}$ (how well the text recovers the image's dominant eigenspace) and $S_{Y \to X}$ (how well the image recovers the text's dominant eigenspace).
Imbalance Metric: The difference $\Delta SAS = S_{X \to Y} - S_{Y \to X}$ quantifies modality information imbalance. A positive $\Delta SAS$ indicates the image modality retains richer structural information than the text can recover.
Local Extension: The authors extend SAS to an instance-level formulation, allowing the scoring of individual image-caption pairs using a frozen spectral prior fitted on a reference population.

Experimental Setup
The study benchmarks 15 VLMs (13 general-purpose, 2 medical-specific: PubMedCLIP and PLIP) across four datasets:

Natural: MS-COCO 2014, Flickr30k.
Medical: ROCO, MIMIC-CXR.
The evaluation compares SAS against six existing alignment metrics (CKA, SVCCA, CORAL, MMD, RMG, and cosine similarity differences) and bidirectional retrieval performance (R@1).

Key Results

Detection of Domain Shift: Nine out of ten metrics successfully detect alignment degradation when moving from natural to medical datasets. However, only SAS reveals the direction of this degradation.
Directional Asymmetry in Medicine: In natural datasets, $\Delta SAS$ is statistically indistinguishable from zero (symmetric alignment). In contrast, across all 13 general-purpose VLMs and both medical datasets, $\Delta SAS$ is significantly positive ( $+0.049, p < 0.001$ ). This indicates that medical images retain richer structural information in their dominant eigenspaces than the paired clinical reports can recover. This asymmetry is invisible to all symmetric metrics.
Spectral Sensitivity: The imbalance is most pronounced when focusing on the top eigenmodes (low $q$ ). As $q$ increases to include lower-variance modes, the gap narrows, suggesting that the critical information imbalance resides in the primary spectral directions.
Predictive Power: SAS demonstrates the strongest zero-label correlation with downstream retrieval performance in the medical domain ( $\rho = +0.969$ ), outperforming CKA, SVCCA, and other metrics.
Medical Model Performance: Even specialized medical VLMs (PubMedCLIP, PLIP) struggle with retrieval on MIMIC-CXR (near-zero R@1), despite showing improved alignment scores compared to general models. This suggests that while alignment is necessary, it is not sufficient for robust medical retrieval, likely due to the complexity of clinical jargon.
Case Study: Instance-level SAS applied to dental panoramic radiographs (OPGs) successfully identified specific pairs where the text failed to capture surgical hardware details present in the image (high $S_{X \to Y}$ , low $S_{Y \to X}$ ), validating the metric's diagnostic utility at the sample level.

Significance and Claims
The paper claims that SAS provides a "missing diagnostic dimension" to multimodal evaluation. Its primary contribution is not to replace existing metrics but to operate in synergy with them:

Diagnosis: While cosine similarity ( $\Delta cos$ ) measures whether paired items are separable (retrieval readiness), SAS identifies which encoder is the bottleneck.
Actionable Insight: The authors argue that a positive $\Delta SAS$ suggests the text encoder is the weaker link, implying that interventions such as domain-specific language model pretraining or text-encoder fine-tuning on clinical reports may be more effective than joint fine-tuning.
Clinical Relevance: By exposing directional imbalance, SAS offers a path toward more explainable medical AI, potentially allowing clinicians to inspect which visual concepts lack reliable textual counterparts.

The authors maintain a modest stance, noting limitations such as the requirement for equal feature dimensions and the fact that high alignment scores do not guarantee downstream retrieval success in complex clinical domains. They position SAS as a tool for transparent, interpretable, and actively improvable clinical AI systems.