A quantitative analysis of semantic information in deep representations of text and images

This paper employs Information Imbalance to demonstrate that semantic information converges across languages, modalities, and architectures in deep models, revealing that predictability is strongest in specific central or final layers and that larger, independently trained models can outperform jointly trained multimodal models in cross-modal alignment.

The Gist

The Big Idea: Finding the "Universal Translator" in AI Brains

Imagine you have two different people: one who speaks only English and one who speaks only Italian. If you show them the same picture of a cat, they both think "cat." But inside their brains, the electrical signals firing might look totally different.

This paper asks a fascinating question: Do the "brains" of different AI models (even ones trained on different languages or types of data) eventually start thinking in the same way?

The authors call this the "Platonic Representation Hypothesis." It's like saying that deep down, all smart systems are trying to map the world onto the same invisible, perfect blueprint. If you look at the right spot in their "brains," the English word "cat" and the Italian word "gatto" should look almost identical.

To test this, the researchers didn't just ask "Are they similar?" They asked, "Who can predict who?"

---

The Tool: The "Information Imbalance" (The Crystal Ball Test)

Usually, scientists measure similarity by looking at two things side-by-side and seeing how much they overlap (like comparing two fingerprints). But the authors used a smarter tool called Information Imbalance.

Think of it like a Crystal Ball:

• If I show you a map of a city (Representation A), can you guess what the traffic report looks like (Representation B)?
• If you can guess it perfectly, the "Imbalance" is low.
• If you have no idea what the traffic report looks like, the "Imbalance" is high.

Crucially, this test is one-way. Maybe knowing the map helps you guess the traffic, but knowing the traffic doesn't help you guess the map. This paper uses this to see which AI model is "smarter" or more "informative" than the other.

---

Key Findings (The Story Unfolds)

1. The "Sweet Spot" in the Middle

AI models are like multi-story buildings with many floors (layers).

• The Ground Floor: This is where the raw data enters. It's messy and specific (e.g., "This is the letter 'A'").
• The Top Floor: This is where the model gives its final answer (e.g., "This is a cat").
• The Middle Floors: The authors found that the middle floors are where the magic happens. This is where the AI strips away the specific language details and finds the pure "meaning."

Analogy: Imagine translating a book. The first few pages are just the alphabet. The last few pages are the conclusion. But the middle chapters are where the actual story lives, regardless of whether the book is in English or Italian. The AI's "meaning" lives in the middle.

2. English is the "King" of Predictability

The study found that English representations are like a high-definition master copy, while other languages are like lower-resolution copies.

• If you take the English version of a sentence and try to guess the Italian version, you do a great job.
• If you take the Italian version and try to guess the English version, you struggle a bit more.
• Why? Because English has more training data on the internet. The AI learned English so well that it became the "universal pivot" for understanding other languages.

3. Bigger Models are Better "Translators"

The researchers compared a giant AI (DeepSeek-V3) with a smaller one (Llama3).

• Analogy: Imagine a giant library (DeepSeek) and a small bookshelf (Llama3).
• The giant library can predict what's on the small bookshelf perfectly.
• But the small bookshelf often fails to predict what's in the giant library because it's missing too many books.
• Takeaway: Size matters. Bigger models capture the "universal meaning" better than smaller ones.

4. Spreading the Wealth (Tokens)

When an AI reads a sentence, it breaks it into chunks called "tokens."

• Old Idea: Maybe the whole meaning of a sentence is hidden in just the last word.
• New Discovery: No! The meaning is spread out across many words.
• Analogy: If you want to understand a joke, you can't just read the punchline (the last token). You need to read the setup, the characters, and the context. The AI needs to look at the average of many words to get the full picture.

5. The Visual vs. Text Surprise

The researchers also looked at how AI handles images vs. text.

• The Expectation: You'd think a model trained specifically to match images with text (like CLIP) would be the best at understanding both.
• The Surprise: Two models trained separately (one just for text, one just for images) actually understood each other better than the model trained to match them together!
• Analogy: Imagine two people who never met but both studied the same encyclopedia. They can have a great conversation. But a third person who was forced to memorize a specific list of "Image-Text Pairs" actually had a harder time connecting the dots.
• Why? It seems that if a model is big enough, it naturally figures out how to connect pictures and words on its own. You don't need to force it to learn the connection explicitly.

---

The Bottom Line

This paper tells us that despite all the differences in how AI models are built (different languages, different sizes, text vs. images), they are all converging on the same universal map of meaning.

• Where? In the middle layers of the network.
• How? By spreading information across many words, not just one.
• Who wins? Bigger models and English tend to be the "leaders" in this universal language.

It suggests that intelligence, whether human or artificial, eventually strips away the noise (like specific words or pixel colors) to find the pure, shared structure of reality underneath.

Technical Summary

1. Problem Statement

Large Transformer models encode information in high-dimensional spaces. The Platonic Representation Hypothesis suggests that as model sizes increase, representations of semantically related inputs (e.g., translations of the same sentence, images of the same class) converge to similar structures, regardless of the specific model architecture or modality.

However, existing methods to measure this convergence have limitations:

• Symmetry: Standard metrics like Central Kernel Alignment (CKA) and Neighborhood Overlap (NO) are symmetric. They measure similarity but cannot quantify directional predictability (i.e., whether representation $A$ contains more information about $B$ than vice versa).
• Computational Cost: Cross-entropy, the ideal measure for information loss, is computationally intractable in high-dimensional spaces (thousands of dimensions).
• Representation Granularity: It is unclear how semantic information is distributed across tokens (e.g., is it concentrated in the last token or spread across the sequence?) and how it varies across network layers.

The paper aims to quantify the relative information content between representations of different languages, modalities, and model scales, specifically addressing information asymmetry and the spatial distribution of semantic information within deep networks.

2. Methodology

Core Metric: Information Imbalance (II)

The authors employ Information Imbalance (II), an asymmetric, rank-based statistical measure that serves as a computationally efficient proxy for cross-entropy in high-dimensional spaces.

• Definition: For two representation spaces $X$ and $Y$, $\Delta(X \to Y)$ is the normalized average rank in $Y$ of the nearest neighbors of points in $X$.
• Interpretation:
  • If $\Delta(X \to Y) \approx 0$, $X$ predicts $Y$ well (neighbors in $X$ are neighbors in $Y$).
  • If $\Delta(X \to Y) \approx 1$, $X$ provides no information about $Y$.
  • Asymmetry: $\Delta(X \to Y) \neq \Delta(Y \to X)$ quantifies directional information flow (e.g., a larger model predicting a smaller one).
• Validation: The authors demonstrate on synthetic Gaussian data that II outperforms CKA and NO in high dimensions, particularly in detecting weak signals and directional relationships where symmetric metrics fail.

Experimental Setup

• Text Data: Parallel sentences from the Opus Books dataset (English paired with Spanish, Italian, German, French, Dutch, Hungarian).
• Image Data: ImageNet-1k (same-class pairs) and Flickr30k (image-caption pairs).
• Models Analyzed:
  • Text: DeepSeek-V3 (671B parameters, MoE), Llama3 (1B, 3B, 8B).
  • Vision: DinoV2-large (Encoder), Image-gpt-large (Autoregressive), CLIP (Jointly trained).
• Representation Aggregation: The study compares three methods for sentence representation:
  1. Last token only.
  2. Concatenation of the last $T$ tokens.
  3. Average of the last $T$ tokens.

3. Key Contributions & Results

A. Semantic Information Distribution in Text (LLMs)

• Layer Depth: Semantic information is not concentrated in a single layer but is most predictive in central layers (roughly 40–60% depth). This "semantic core" is robust across six language pairs.
• Token Aggregation:
  • Averaging vs. Last Token: Averaged token representations yield significantly lower Information Imbalance (better predictability) than single-token or concatenated representations.
  • Implication: Semantic information is spread across many tokens in deep layers, not just the final token. Averaging removes irrelevant positional noise, enhancing the semantic signal.
• Information Asymmetries:
  • Language: English representations are systematically more predictive of other languages than vice versa, particularly in early and late layers. The middle layers show the most symmetry (language independence).
  • Model Scale: DeepSeek-V3 (671B) representations are significantly more predictive of Llama3-8b than the reverse. The gap widens in the latter half of the network.

B. Semantic Information Distribution in Vision

• Architecture Dependence: The location of peak semantic concentration depends on the training objective:
  • Autoregressive Models (Image-gpt): Semantic information peaks in middle layers (similar to LLMs).
  • Encoder Models (DinoV2): Semantic information concentrates in the final layers, designed for downstream tasks like segmentation.
• Cross-Modal Alignment:
  • The layers with the strongest intra-modal semantic concentration (middle for autoregressive, final for encoders) also yield the strongest cross-modal alignment with text captions.
  • Asymmetry: Text representations (DeepSeek-V3) are generally more predictive of image representations than the reverse.

C. The Role of Joint Training vs. Model Scale

• Surprising Finding: Two independently trained models (DeepSeek-V3 and DinoV2) achieve stronger cross-modal alignment (lower II) than the jointly trained CLIP model.
• Implication: Explicit multimodal contrastive training is not strictly necessary for strong alignment. Model scale appears to be the dominant factor; large-scale unimodal training may naturally converge toward a shared semantic manifold.
• Saturation: Increasing the size of vision models (DinoV2 variants) reduces II, but the improvement saturates once a certain capacity is reached.

4. Significance and Conclusion

• Refining the Platonic Hypothesis: The paper supports the idea of a shared semantic manifold but clarifies that convergence is layer-specific and asymmetric. Convergence is strongest in intermediate processing stages, while early/late layers retain modality or language-specific features.
• Methodological Advancement: The use of Information Imbalance provides a nuanced view of representation quality that symmetric metrics (CKA, NO) miss, specifically revealing directional information flow and the impact of model scale.
• Practical Implications:
  • Representation Extraction: For tasks requiring semantic alignment, using averaged token representations from central layers is superior to using the last token.
  • Model Design: The findings suggest that massive scale alone can drive cross-modal convergence, potentially reducing the reliance on expensive joint multimodal training for certain alignment tasks.
  • Language Bias: The systematic asymmetry favoring English highlights a potential bias in current LLMs, where English acts as a "pivot" with richer representational capacity.

In summary, the paper provides a rigorous, quantitative framework for understanding how semantic information is encoded, distributed, and aligned across different modalities and model architectures, challenging the notion of uniform convergence and highlighting the critical roles of layer depth, token aggregation, and model scale.