Technical Summary: Characterizing Universal Object Representations Across Vision Models

Problem Statement

Deep neural networks (DNNs) trained with diverse architectures, objective functions, and datasets have been observed to converge on similar visual representations, a phenomenon sometimes referred to as the "Platonic Representation Hypothesis." However, the specific visual properties these shared representations capture and the factors driving this convergence remain poorly understood. Key open questions include:

1. Do universal representations reflect high-level conceptual properties (e.g., object categories) or low-level perceptual properties (e.g., edges, colors)?
2. Which inductive biases (architecture, objective, training data, model size, or performance) drive the emergence of these universal dimensions?
3. How do these shared dimensions relate to biological vision (neural activity and human behavior)?

Previous work has established that models overlap in representational structure but has not systematically decomposed these structures into interpretable, non-negative dimensions to distinguish between universal and model-specific components across a large, diverse set of models.

Methodology

The authors analyzed 162 diverse vision models (spanning CNNs, Transformers, MLP-Mixers, and Hybrids) trained on various datasets (ImageNet, LAION, Taskonomy, etc.) and objectives (supervised, self-supervised, vision-language). The analysis pipeline involved the following steps:

1. Data and Representation Extraction:

   • Stimuli: 22,248 object images from the THINGS database, covering 1,854 basic-level object categories.
   • Features: Penultimate-layer representations were extracted from all 162 models. The penultimate layer was chosen as it integrates high-level conceptual and fine-grained perceptual information closest to behavioral output.

2. Symmetric Non-Negative Matrix Factorization (Symmetric NMF):

   • To make representations comparable across models with different feature spaces, the authors first computed pairwise object similarity matrices ($S_m$) for each model using a Radial Basis Function (RBF) kernel.
   • Each similarity matrix was decomposed into a small set of non-negative dimensions ($r=50$) using Symmetric NMF: $S_m \approx W_m W_m^\top$.
   • This yielded a non-negative embedding $W_m$ for each model, where each column represents a dimension defined by loadings over the same set of images. This approach ensures dimensions are interpretable as "parts-based" features.

3. Quantifying Universality:

   • A universality score was computed for each dimension in each model. This involved finding a one-to-one correspondence (via the Hungarian algorithm) between dimensions of a target model and all other models based on squared cosine similarity.
   • The score for a dimension $k$ in model $m$ is the average similarity to its best-matching dimensions across all other $M-1$ models.
   • Scores were calibrated against a permutation null distribution to correct for chance alignment and baseline similarity due to non-negativity.

4. Characterization and Validation:

   • Interpretability: 1,059 representative dimensions were labeled by crowd-sourced participants as semantic, visual, both, or neither.
   • Structural Analysis: Category consistency ($\eta^2$) was calculated to determine if dimensions grouped images by object category (between-category variance) or by visual features within categories (within-category variance).
   • Biological Alignment: The predictive power of universal vs. model-specific dimensions was tested against:
     • Multi-unit activity in macaque inferior temporal (IT) cortex.
     • Human behavioral similarity judgments (odd-one-out triplet tasks).
   • Controlled Comparisons: The authors isolated the effects of architecture, objective function, training data, model size, and performance to determine their influence on universality.

Key Results

1. Existence of Universal Dimensions

The study identified a set of interpretable universal dimensions that recur consistently across the 162 diverse models. Universality scores ranged from near-chance (0.0003) to 0.54, with the most universal dimensions reaching 64% of the within-model stability ceiling. These universal dimensions are not artifacts of specific model families or image sets, as rankings remained stable under bootstrap resampling and image set changes (THINGS vs. ObjectNet).

2. Nature of Universal vs. Model-Specific Dimensions

• Universal Dimensions: These are significantly more interpretable to humans. At high universality levels, 80% of dimensions were labeled as semantic or mixed semantic-visual, and they showed strong category consistency (grouping images by object category).
• Model-Specific Dimensions: These are largely uninterpretable (78% labeled as "neither" at low universality) and are driven by perceptual image properties (texture, color, viewpoint) rather than conceptual structure. They exhibit high within-category variance.
• Reconstruction Importance: Crucially, universality does not correlate with a dimension's contribution to the model's overall similarity structure (reconstruction importance). Both universal and model-specific dimensions contribute equally to the model's representational geometry.

3. Drivers of Universality

The authors found that no single inductive bias accounts for the emergence of universal dimensions:

• Architecture: No significant difference in universality across CNNs, Transformers, MLP-Mixers, or Hybrids.
• Objective Function: While semantic tasks produced more universal dimensions than geometric tasks within the Taskonomy family, no single objective (e.g., contrastive vs. non-contrastive, vision-language vs. visual-only) consistently drove universality across the broader model zoo.
• Training Data: Neither dataset scale (ImageNet-1K vs. 21K) nor specific web-scale datasets (LAION, DFN, MetaCLIP) significantly altered universality scores.
• Model Size/Performance: Universality showed no correlation with parameter count or ImageNet top-1 accuracy.

4. Alignment with Biological Vision

Models with a higher proportion of universal dimensions better predict:

• Neural Activity: Multi-unit activity in macaque IT cortex ($r = 0.74$).
• Human Behavior: Human similarity judgments in odd-one-out tasks ($r = 0.76$).
  Furthermore, the "universal half" of a model's dimensions (top 25 by score) significantly outperforms the "model-specific half" in predicting both neural and behavioral data.

Significance and Claims

The paper claims that universality is an emergent property of successful visual representation learning rather than a consequence of specific architectural or training choices. The findings suggest that:

• Diverse models converge on a shared set of conceptual, interpretable object representations that align with biological vision.
• The convergence is driven by the constraints of the visual world itself, rather than model-specific inductive biases.
• Interpretability, conceptual content, and biological alignment co-occur in the structure that models converge on.

The authors position their work as a step toward understanding the "why" of visual representation learning, demonstrating that despite the "fractured" nature of model-specific features, a robust, universal core exists that mirrors the organization of the biological visual system. They explicitly state that this convergence is not complete (universal dimensions do not have perfect scores, and model-specific dimensions remain significant), but the shared structure is sufficient to explain alignment with brains and behavior.

The paper concludes that this framework offers a generalizable approach for understanding shared vs. specific representations in any set of models processing common inputs, moving beyond simple representational alignment metrics to characterize the content of convergence.