🤖 machine learning

Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale

This paper challenges the Platonic Representation Hypothesis by demonstrating that reported cross-modal alignment between text and image models is fragile, dependent on small-scale and constrained evaluation settings, and fails to hold when tested on larger datasets and realistic many-to-many scenarios, suggesting that different modalities learn rich but distinct representations of reality rather than a single convergent one.

Original authors: A. Sophia Koepke, Daniil Zverev, Shiry Ginosar, Alexei A. Efros

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: A. Sophia Koepke, Daniil Zverev, Shiry Ginosar, Alexei A. Efros

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how two different people see the world. One person is a Visual Artist who only speaks in pictures, and the other is a Poet who only speaks in words.

For a long time, a popular theory (called the "Platonic Representation Hypothesis") suggested that if you train both of them on enough data, they will eventually start seeing the world in the exact same way. The theory claimed that deep down, the Artist and the Poet are just looking at the same "perfect reality," and their different languages (pixels vs. words) are just surface-level differences.

This paper is like a reality check. The authors say: "Hold on a minute. That theory might be true for small, simple worlds, but when we look at the real, messy, huge world, it falls apart."

Here is the breakdown of their findings using simple analogies:

1. The "Small Room" vs. The "Mega-Mall" Analogy

The previous studies that supported this theory were like testing the Artist and Poet in a tiny, empty room with only 1,000 items on the floor.

In the small room: If you ask the Artist to find a picture of a "dog," and the Poet to find a description of a "dog," they might both pick the same item simply because it's the only dog in the room. They seem to agree perfectly!
In the Mega-Mall: The authors expanded the test to a massive shopping mall with millions of items. Now, when you ask for a "dog," the Artist finds a photo of a Golden Retriever in a park, while the Poet finds a poem about a Chihuahua in a city.
The Result: They are both right! They both found a "dog." But they didn't pick the exact same one. In the huge mall, their agreement drops to near zero. The authors argue that the old studies were fooled because the room was too small to show the differences.

2. The "Library of Babel" Analogy

The paper also points out a problem with how we measure "agreement."

Imagine you have a photo of a lighthouse.
The Poet could describe it a thousand different ways: "A tower by the sea," "A beacon for sailors," "A white stone structure," "A lonely guide in the fog."
The Artist could show you a thousand different photos: a lighthouse at sunset, a lighthouse in a storm, a lighthouse from the side, a lighthouse from above.
The Problem: The old test demanded that if you showed the Poet the "sunset" photo, they had to pick the "sunset" photo back. But the Poet might have picked the "storm" photo because the description "lonely guide in the fog" fits that one better!
The authors say: Real life is messy. One image has many valid descriptions, and one description fits many images. When you account for this "many-to-many" relationship, the models stop looking like they are converging on a single truth.

3. The "Different Caves" Metaphor

The paper uses a famous philosophical idea called Plato's Cave. Plato thought everyone was in a cave watching shadows on a wall, and if we got smart enough, we would all see the same "perfect truth" outside the cave.

The authors suggest a different philosopher, Johann von Uexküll, who argued that every creature lives in its own unique world (called an Umwelt).

A tick lives in a world of heat and smell.
A bat lives in a world of echoes.
They don't see the same "reality"; they see their own version of it.

The Conclusion:
The authors believe that AI models are like these creatures.

The Vision Model lives in a "Visual Cave." It organizes the world by shapes, colors, and angles.
The Language Model lives in a "Word Cave." It organizes the world by grammar, concepts, and stories.

They both learn rich, detailed maps of the world, but they draw the maps differently. They don't converge into one single "Platonic" map. They just build their own unique, high-quality caves.

Why Does This Matter?

It's not bad news: It doesn't mean AI is "dumb." It means AI is specialized. A vision model is great at seeing, and a language model is great at thinking. They don't need to be identical to be useful.
It changes how we build AI: If we want machines that truly understand the world, we can't just rely on text. We can't just say "Language is all you need." We need to respect that seeing and reading are fundamentally different ways of processing reality.

In short: The old theory said, "If we make AI big enough, they will all think exactly alike." This paper says, "Nope. Even the smartest AIs have their own unique perspectives, and that's actually a good thing."

1. Problem Statement

The paper investigates the Platonic Representation Hypothesis (PRH), proposed by Huh et al. [40]. The PRH posits that neural networks trained on different modalities (e.g., text and images) will eventually converge toward a single, shared representation of reality as they scale in size and data consumption. If true, this would imply that modality choice is irrelevant and that language alone could suffice for building machines that understand the world.

The authors challenge the robustness of the experimental evidence supporting PRH. They argue that previous findings rely on fragile evaluation regimes—specifically small datasets and strict one-to-one (bijective) image-caption pairings—that do not reflect the complexity of real-world, large-scale, many-to-many multimodal data.

2. Methodology

The authors re-evaluate cross-modal alignment using the Mutual $k$ -Nearest Neighbor (Mutual $k$ NN) metric, which measures the overlap between the sets of nearest neighbors retrieved by two different models (a vision encoder and a language encoder) for a given query.

Key methodological variations and expansions include:

Scaling the Gallery: Moving from the sparse WIT-1024 (1,024 samples) used in the original study to dense galleries of WIT-1M (1 million) and LAION-15M (15 million) samples.
Relaxing Bijectivity: Testing alignment in many-to-many settings where one image has multiple valid captions and one caption corresponds to multiple distinct images. This was tested using both synthetic data (CycleReward dataset) and natural duplicates in the WIT dataset.
Model Trends: Re-evaluating the correlation between language model capability and cross-modal alignment using 55 LLMs (including recent models like Llama-3, Gemma-3, and DeepSeek-R1) against DINOv2 vision encoders.
Controlled Decomposition: Using the ImageNet validation set with detailed captions to isolate whether low alignment is due to poor representation quality or different internal organization of fine-grained structures.
Sanity Checks: Verifying that the drop in alignment is not an artifact of the metric itself by measuring within-modality alignment (e.g., LLM vs. LLM, Vision vs. Vision) across scales.

3. Key Contributions

The paper provides four major empirical findings that contradict the strong interpretation of the Platonic Representation Hypothesis:

Alignment Degrades with Scale:
- Mutual $k$ NN alignment scores drop precipitously as the retrieval gallery grows from 1K to 1M+ samples.
- On WIT-1024, alignment is non-trivial; on LAION-15M, it approaches zero ( $\approx 0.001$ for $k=1$ ).
- This suggests that the "convergence" observed in small datasets is an artifact of data sparsity, where models agree simply because there are no better alternatives.
Coarse vs. Fine-Grained Structure:
- In controlled ImageNet experiments, both vision and language models individually improve at retrieving correct-class neighbors as the gallery densifies.
- However, they fail to agree on the specific instance (fine-grained structure). For example, a vision model might retrieve a car from a similar angle, while the language model retrieves a caption for the same car model but a different pose.
- Conclusion: Models learn rich, semantically meaningful structures, but they organize these structures differently based on their modality.
The Bijective Assumption is Flawed:
- Real-world data is inherently many-to-many. When the authors relaxed the one-to-one assumption (allowing multiple valid neighbors), mutual $k$ NN alignment dropped consistently.
- The metric penalizes valid semantic matches that are not exact duplicates, indicating that the original evidence for convergence relied on an unrealistic constraint.
The "Stronger Models Align Better" Trend Fails:
- The original study claimed that more capable LLMs align better with vision.
- The authors found this trend holds for older, smaller models on specific benchmarks (HellaSwag, Wikitext) but breaks down for recent, state-of-the-art models on reasoning benchmarks (GSM8K, MMLU, LogiQA2).
- Newer, stronger models do not show increased alignment with DINOv2; in some cases, alignment scores saturate or decrease.

4. Results

Quantitative Drop: Mutual $k$ NN scores for $k=1$ dropped from $\approx 0.135$ (WIT-1024) to $\approx 0.008$ (LAION-15M).
Within-Modality Stability: In contrast to cross-modal drops, within-modality alignment (e.g., DINOv2-base vs. DINOv2-giant) remained stable across scales, proving the metric does not inherently collapse at scale.
ImageNet Decomposition: Even with detailed captions, strict cross-modal agreement remained flat at $\approx 11\%$ even as individual retrieval accuracy improved to $>46\%$ .
Trend Reversal: Regression analysis showed negative $R^2$ values for recent models on reasoning benchmarks, indicating that the linear trend predicted by Huh et al. does not extrapolate to modern LLMs.

5. Significance and Conclusion

The paper fundamentally challenges the notion that different modalities converge to a single "Platonic" representation of reality.

Theoretical Implication: The authors propose an alternative view inspired by von Uexküll's "Umwelt": just as different organisms inhabit different perceptual worlds based on their senses, different AI modalities construct distinct, modality-specific representational structures.
Practical Impact:
- Modality Matters: Language alone may not be sufficient to recover the fine-grained perceptual structure of the world (e.g., spatial relationships, textures) that vision models learn.
- Evaluation Caution: Small-scale, bijective evaluations (like WIT-1024) significantly overstate cross-modal convergence. Future research must evaluate alignment at scale and under many-to-many conditions.
- Future Direction: The authors suggest that while models may share coarse categorical overlaps, they inhabit different "caves." The goal should not be to force convergence to a single representation, but to understand and model the specific "Umwelten" of different modalities and how they intersect.

In summary, the paper argues that models trained on different modalities learn equally rich but structurally distinct representations of the world, rather than converging toward a single shared truth.