Technical Summary: The Wittgensteinian Representation Hypothesis

1. Problem Statement

A fundamental question in multimodal representation learning is why independently trained neural networks from different modalities (e.g., vision, point clouds, language) converge toward shared representational structures. While the Platonic Representation Hypothesis (PRH) posits that such convergence occurs toward a shared "statistical structure of reality," it remains agnostic about the direction of this convergence and the specific nature of the endpoint.

Existing empirical evidence for convergence relies exclusively on symmetric similarity measures (e.g., CKA, mutual kNN, RSA). By construction, these metrics satisfy $s(X, Y) = s(Y, X)$, allowing them to detect that convergence exists but rendering them structurally blind to where or in which direction the convergence is heading. Consequently, the question of whether convergence is mutual or directional, and if directional, which modality serves as the attractor, has remained unexamined.

2. Methodology

The authors introduce Directional Convergence Analysis to address the limitations of symmetric metrics.

2.1 Asymmetric Alignment Measure: CYCLE-KNN

The core methodological innovation is the application of CYCLE-KNN, an inherently asymmetric alignment measure, to investigate convergence direction.

• Mechanism: CYCLE-KNN computes alignment via a two-step nearest-neighbor cycle. For representations $X$ and $Y$, it finds the $k$ nearest neighbors of a point $i$ in space $Y$, then finds the $k$ nearest neighbors of those points back in space $X$. The score is the fraction of points $i$ that return to themselves (or their original neighborhood) after this round trip.
• Asymmetry: The metric is asymmetric for $k \ge 2$. If $CYCLE\text{-}KNN(X \to Y) > CYCLE\text{-}KNN(Y \to X)$, it implies that the neighborhood structure of $Y$ is more coherent (compact) than $X$, making the return hop more likely to succeed.
• Directional Gap ($\Delta$): The authors define $\Delta(X, Y) = CYCLE\text{-}KNN(X \to Y) - CYCLE\text{-}KNN(Y \to X)$. A consistently positive $\Delta$ indicates that $Y$ acts as a more stable reference frame (attractor) for $X$.

2.2 Experimental Setup

• Models: The study evaluates 58 independently trained unimodal models spanning three modalities arranged along an abstraction hierarchy:
  • 3D Point Clouds: 7 models (e.g., PointMAE, PointBERT, PointGPT).
  • 2D Vision: 22 models (e.g., ViT, MAE, DINOv2, DINOv3, CLIP).
  • Language: 29 models (e.g., BLOOMZ, LLaMA, Qwen2.5/3, InternLM).
  • Note: Except for cross-modal controls (CLIP, ReCon++), no model was trained on data from other modalities.
• Datasets:
  • WiT: 1,024 image-text pairs for Vision-Language alignment.
  • ShapeNet55: Matched 3D objects, rendered images, and text descriptions for Point Cloud-Vision-Language alignment.
• Analysis: The authors compute $\Delta$ across all cross-modality pairs, analyzing layer-wise density, intra-modality consensus, and scale invariance (parameter counts ranging from 5.7M to 72B).

3. Key Findings

3.1 Directional Asymmetry

The study uncovers a consistent directional asymmetry invisible to symmetric measures:

• Non-language modalities converge toward language: Vision and point cloud representations align significantly more toward language than language aligns toward them.
  • $\Delta(\text{Vision} \to \text{Language}) = +0.010$ ($p < 0.05$).
  • $\Delta(\text{Point Cloud} \to \text{Language}) = +0.030$ ($p < 0.01$).
  • $\Delta(\text{Point Cloud} \to \text{Vision}) = +0.046$ ($p < 0.01$).
• Robustness: This asymmetry holds across 638 model pairs, 10 model families, and four orders of magnitude in parameter size. It is invariant to the choice of $k$ (tested from 1 to 50).

3.2 Intra-Modality Consensus

Language models exhibit the highest intra-modality representational consensus (Mean CKA = 0.904), followed by vision (0.857), and point clouds (0.440). This high agreement among independently trained language models suggests they occupy a deep, stable basin in the representational space, consistent with the "attractor" hypothesis.

3.3 Geometric Mechanism: Feature Density

The directionality is explained by feature density asymmetry:

• Compactness: Language representations occupy the most compact regions of the representational space (lowest pairwise mean distance $D$), particularly in intermediate-to-late layers.
• Trajectory: Vision models show monotonically increasing density (sparse) with depth, while language models follow an inverted-U trajectory, reaching maximum compactness at layers where cross-modal alignment peaks.
• Mechanism: Compact neighborhoods in language facilitate the "return hop" in CYCLE-KNN, driving the observed directional bias.

3.4 Information Bottleneck Interpretation

The authors interpret these findings through the Information Bottleneck (IB) framework. Under IB, optimization drives representations toward maximal compression of task-irrelevant information. Natural language, as a highly compressed, discrete, and compositional structure of perceptual experience, acts as the optimal compression target. As models compress information, their representations naturally converge toward the structural geometry of language.

4. The Wittgensteinian Representation Hypothesis (WRH)

Based on these findings, the paper proposes the Wittgensteinian Representation Hypothesis:

"The semantic structure of language constitutes the asymptotic attractor of multimodal representation convergence."

This hypothesis posits that as independently trained models become sufficiently powerful, their learned representations—regardless of input modality—converge toward the representational geometry defined by language's semantic structure. This reframes Wittgenstein's dictum ("The limits of my language mean the limits of my world") as a statement about representation learning: language imposes structural constraints that define the boundary toward which all perceptual representations converge under sufficient optimization.

5. Significance and Claims

The paper claims the following contributions and significance:

1. Methodological Correction: It identifies a systematic blind spot in representation learning: the exclusive reliance on symmetric measures has left the direction of convergence unexamined. The authors demonstrate that CYCLE-KNN's asymmetry, previously cataloged but unanalyzed, encodes critical directional structure.
2. Resolution of PRH: While PRH established that convergence happens, WRH answers where it goes. The endpoint is not a generic "statistical structure of reality" but a specific, discrete, compositional structure: language.
3. Universality: The phenomenon is scale-invariant and robust across diverse architectures (transformers, autoencoders) and training paradigms (supervised, self-supervised, contrastive), suggesting it is a fundamental property of the modality representations rather than an artifact of specific training objectives or model sizes.
4. Theoretical Grounding: The paper provides a principled interpretation via the Information Bottleneck framework, linking geometric compactness to the compression of perceptual experience into symbolic language.

The authors explicitly state that WRH is a falsifiable, directional prediction confirmed by their experiments across 58 models. They caution that this describes a structural tendency in current neural network training, not a normative claim that language is the "correct" or "complete" representation of reality, noting potential risks regarding cultural biases inherent in language data.