🧬 biology

Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion

This paper introduces a single-stimulus methodology using the Generalized Procrustes Algorithm to demonstrate that intra-modal representational dispersion among vision models strongly modulates cross-modal alignment with language models, where stimuli with high agreement across vision models elicit significantly stronger cross-modal convergence.

Original authors: Eghbal A. Hosseini, Brian Cheung, Evelina Fedorenko, Alex H. Williams

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eghbal A. Hosseini, Brian Cheung, Evelina Fedorenko, Alex H. Williams

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Idea: Why Do AI Models "Agree" on Some Things but Not Others?

Imagine you have a room full of different art critics. One is a traditional painter, one is a modern photographer, and one is a digital artist. They all look at the same painting.

Sometimes, they all agree perfectly on what the painting is about. They might all say, "This is a sad, rainy day."
Other times, they disagree wildly. One says, "It's a celebration," another says, "It's a nightmare," and the third says, "It's just a blue blob."

This paper asks a simple question: Does it matter which painting we show them?

The researchers found that if you show the critics a painting they all agree on (low disagreement), they are much better at describing it using words that match a language AI. But if you show them a painting they argue about (high disagreement), the connection between their visual understanding and the language AI breaks down.

The Problem: We Usually Look at the "Average"

In the past, scientists looked at how AI models behaved by averaging thousands of images together. It's like asking, "On average, do these critics agree?" The answer is usually "Yes, mostly."

But the authors realized that averaging hides the truth. Just because the average is good doesn't mean every single image is understood the same way. Some images are confusing even to AIs.

The Solution: The "Group Hug" (Generalized Procrustes Analysis)

To figure out which images cause agreement and which cause arguments, the researchers used a mathematical trick called Generalized Procrustes Analysis (GPA).

The Analogy:
Imagine you have three different maps of the same city, drawn by different people.

Map A is rotated 90 degrees.
Map B is flipped upside down.
Map C is stretched.

To compare them, you can't just lay them on top of each other; they won't match. You have to rotate, flip, and stretch them until they all line up perfectly. Once they are aligned, you can see exactly where the maps disagree.

If the maps line up perfectly on a specific street, that street is "low dispersion" (everyone agrees).
If the maps show that street in three completely different places, that street is "high dispersion" (everyone is confused).

The researchers did this with AI vision models (like DINOv2, CLIP, and MAE). They aligned the models' "brains" to create a Joint Consensus. Then, they looked at individual images to see how much each model "drifted" away from the group average.

The Discovery: Agreement Boosts Connection

Once they sorted the images into two piles—"The Agreement Pile" (low dispersion) and "The Argument Pile" (high dispersion)—they tested something new.

They asked: If we feed these images into a Vision AI and a Language AI (like a chatbot), do they understand each other better?

The Result:

The Agreement Pile: When the vision models all agreed on what an image was, the language model understood it almost perfectly. The connection between "seeing" and "speaking" was twice as strong as usual.
The Argument Pile: When the vision models were confused or disagreed, the language model got lost. The connection was weak.

The Metaphor:
Think of the Vision AI and Language AI as two people trying to have a conversation through a translator.

If the Vision AI is looking at a clear, obvious sunset, it says, "Sunset." The translator passes this to the Language AI, which says, "Ah, a sunset!" Perfect alignment.
If the Vision AI is looking at a confusing abstract painting, it might say, "Blue swirl? Red blob? Maybe a cat?" The translator is confused. The Language AI hears nonsense. No alignment.

Why This Matters

This paper changes how we think about AI. It suggests that AI models aren't just "smart" or "dumb" in general. They are situationally smart.

Better Testing: If we want to know if an AI is truly "aligned" with human brains or language, we shouldn't just test it on a random mix of images. We should specifically test it on the images where the models agree.
Understanding the Brain: Humans also agree more on some things than others. By studying which images cause AI models to agree, we might learn what features of the world are "universal" enough for both machines and humans to understand.
The "Platonic" Idea: The paper supports the idea that there is a "true" underlying structure to the world (a Platonic ideal). When AI models learn this structure well, they all converge on the same answer. When they haven't learned it well, they drift apart.

In a Nutshell

The researchers built a tool to measure how much AI models "argue" about a single picture. They found that when AI models agree on what they are seeing, they also speak the same language. This helps us understand how to build better, more aligned AI systems that don't just memorize data, but actually understand the world in a way that matches human perception.

1. Problem Statement

While artificial neural networks (ANNs) across diverse architectures and training objectives often converge to similar representational spaces (supporting the "Platonic Representation Hypothesis"), existing research has primarily established this convergence globally (across groups of stimuli).

The Gap: It remains unclear how individual stimuli elicit convergent or divergent representations. Some stimuli may be interpreted similarly by different models, while others (e.g., ambiguous images) may trigger divergent responses.
The Challenge: Current methods lack a robust, stimulus-specific metric to measure intra-modal convergence (agreement among models of the same modality) and its relationship to cross-modal alignment (agreement between different modalities, e.g., vision and language).

2. Methodology

The authors propose a novel framework to measure and leverage single-stimulus dispersion to modulate cross-modal alignment.

A. Generalized Procrustes Analysis (GPA) for Single-Stimulus Dispersion

To quantify how much different vision models agree on a specific image, the authors adapt Generalized Procrustes Analysis (GPA):

Input: A set of $M$ representation matrices from different vision models (e.g., MAE, DINOv2, CLIP) for $m$ stimuli.
Alignment: GPA finds a set of optimal orthogonal transformation matrices ( $T_i$ $T_{i}$ ) that align all individual model representations into a shared joint consensus representation ( $N_{joint}$ $N_{j o in t}$ ).
- Constraint: Transformations are restricted to orthogonal maps to preserve internal geometry (distances/angles) and avoid distorting covariance structures.
- Preprocessing: Representations are centered, zero-padded to a common dimension, and Frobenius-normalized.
Residual Calculation: For each stimulus $j$ $j$ and model $i$ $i$ , a Procrustes residual ( $r_{ij}$ $r_{ij}$ ) is calculated as the Euclidean distance between the transformed model representation and the joint consensus:
$r_{ij} = \|(N_i T_i)_j - (N_{joint})_j\|_2$
- Low Residual: High agreement (low dispersion) among models.
- High Residual: Low agreement (high dispersion) among models.

B. Dispersion Summarization Strategies

To rank stimuli based on their dispersion across all models, two strategies are employed:

Mean Dispersion: Averaging residuals across all models for each stimulus.
Principal Component of Dispersion: Applying PCA to the residual matrix and using the score along the first principal component (PC1) as a robust measure of the primary axis of disagreement.

C. Cross-Modal Alignment Measurement

The study investigates how these dispersion levels affect alignment between Vision Models (ViT architectures) and Language Models (LLMs like BLOOM, OpenLLaMA, LLaMA).

Metric: Centered Kernel k-Nearest Neighbors Alignment (CKNNA).
- Unlike global metrics (like CKA), CKNNA focuses on local representational structure by computing cross-covariance only between a sample and its $k$ -nearest neighbors (set to $k=10$ ).

3. Experimental Setup

Vision Models: Four distinct ViT architectures trained on different objectives:
- MAE (Masked Autoencoders)
- DINOv2 (Self-supervised)
- CLIP (Contrastive Image-Text)
- CLIP + Fine-tuning on ImageNet-12K
Language Models: Various LLM families (BLOOM, OpenLLaMA, LLaMA) with varying parameter sizes.
Dataset: 1,024 image-caption pairs from the Wikipedia Caption (WIT) dataset.
Procedure:
1. Compute GPA across vision models to generate a joint space.
2. Calculate residuals for all 1,024 stimuli.
3. Select three subsets of stimuli: Low Dispersion (high agreement), High Dispersion (low agreement), and a Random Baseline.
4. Measure CKNNA alignment between vision and language models for each subset.

4. Key Results

The study demonstrates a strong, causal link between intra-modal dispersion and cross-modal convergence:

Dispersion Modulates Alignment: Stimuli with low intra-modal dispersion (where vision models strongly agree) elicit significantly higher cross-modal alignment with language models compared to high-dispersion stimuli.
Magnitude of Effect: The alignment improvement is substantial, reaching up to a two-fold increase in CKNNA scores for specific pairings (e.g., DINOv2 with language models) when using low-dispersion stimuli versus high-dispersion ones.
Robustness:
- The effect holds across different vision model architectures (MAE, DINOv2, CLIP).
- The effect generalizes across different LLM families and sizes.
- The result is consistent regardless of the selection strategy (Mean Dispersion ranking vs. PCA-based selection).
Visual Evidence: Figures 2B–E and 2G–J show clear separation where the "Low Dispersion" curve consistently outperforms the "High Dispersion" and "Random" baselines.

5. Key Contributions

Methodological Innovation: Introduced a stimulus-specific application of Generalized Procrustes Analysis to measure intra-modal representational dispersion at the single-stimulus level.
Discovery of Modulation: Demonstrated that the degree of agreement among vision models on a specific image is a strong predictor of how well that image's representation aligns with language models.
Refined Understanding of Convergence: Provided evidence that the "Platonic Representation" is not uniform across all data; rather, convergence is stimulus-dependent. Some stimuli naturally drive models toward a shared representation, while others induce divergence.
Practical Utility: Offers a method to select "high-quality" or "unambiguous" data points that maximize cross-modal alignment, potentially improving training or evaluation pipelines for multimodal systems.

6. Significance

For AI Research: This work challenges the view of convergence as a global, static property. It suggests that the "structure of the environment" is learned differently depending on the stimulus, and that selecting stimuli based on model agreement can significantly boost multimodal performance.
For Neuroscience: The methodology provides a rigorous tool to probe how individual stimuli are processed by biological vs. artificial systems. By isolating stimuli where artificial models agree (low dispersion), researchers can better test hypotheses about human neural representation without the noise of model disagreement.
Future Directions: The authors suggest that identifying the specific semantic or visual features that characterize "low-dispersion" stimuli could reveal the fundamental building blocks of shared environmental priors, bridging the gap between artificial and biological intelligence.