Revealing the core dimensions underlying representations in brains, behavior and AI

Technical Summary: Revealing the Core Dimensions Underlying Representations in Brains, Behavior, and AI

Problem Statement
The study of representations across neuroscience, psychology, and artificial intelligence relies heavily on characterizing how stimuli are organized relative to one another, typically via similarity matrices. While methods like Representational Similarity Analysis (RSA) allow for the comparison of these matrices across systems, they do not recover the underlying dimensions that generate the observed similarities. Conversely, dimensionality reduction techniques such as Multidimensional Scaling (MDS), Principal Component Analysis (PCA), and t-SNE can summarize data but often yield axes that are difficult to interpret as meaningful representational properties. Furthermore, existing methods struggle with two critical practical constraints: (1) the dimensionality of the representational space is usually unknown, and (2) empirical similarity data is often sparse or incomplete (e.g., behavioral judgments where not all stimulus pairs are compared). Current approaches fail to jointly solve the problems of recovering interpretable dimensions, determining the necessary number of dimensions, and learning from incomplete data without distorting the similarity structure through imputation.

Methodology: Similarity-Based Representation Factorization (SRF)
The authors introduce Similarity-Based Representation Factorization (SRF), a general computational framework designed to recover low-dimensional, non-negative, and interpretable embeddings directly from similarity matrices.

• Core Formulation: SRF is built upon Symmetric Non-negative Matrix Factorization (SNMF). It models a symmetric, non-negative similarity matrix $S \in \mathbb{R}^{n \times n}_{\ge 0}$ as the product of a low-rank embedding matrix $W$ and its transpose: $S \approx WW^\top$, where $W \in \mathbb{R}^{n \times r}_{\ge 0}$ and $r \ll n$. The non-negativity constraint encourages additive, parts-based factors, which are often more interpretable than the subtractive components of PCA.
• Handling Incomplete Data: Unlike standard matrix factorization, SRF operates directly on observed entries without imputing missing values. It defines a binary mask $M$ where $M_{ij}=1$ if the similarity $S_{ij}$ is observed. The objective function minimizes the Frobenius norm of the reconstruction error only over observed entries: $\min_{W \ge 0} \frac{1}{2} \| M \odot (S - WW^\top) \|_F^2$. This allows the model to learn from sparse graphs (e.g., word associations) and predict unobserved similarities.
• Optimization: The authors employ the Alternating Direction Method of Multipliers (ADMM) to solve the optimization problem. They provide theoretical proofs regarding the convergence of the algorithm and the stability of the solution under rank misspecification.
• Dimensionality Selection: To address the unknown dimensionality $r$, SRF utilizes a specialized cross-validation protocol. Because entries in a similarity matrix are statistically dependent (coupled by the low-rank structure), standard entry-wise cross-validation can lead to information leakage and inflated rank estimates. SRF uses a "restricted hold-out" scheme that ensures the training set remains below the matrix-completion threshold, preventing the validation set from being implicitly determined by the training set. The optimal rank is selected at the minimum of the validation error curve.

Key Results
The paper validates SRF through simulations and applications across diverse empirical datasets:

1. Simulation Performance:

   • Recovery Accuracy: SRF accurately recovers ground-truth latent dimensions from synthetic similarity matrices, even when dimensions are diffuse or overlapping.
   • Robustness to Noise: The method maintains high recovery accuracy under moderate levels of measurement noise.
   • Missing Data: SRF significantly outperforms standard imputation baselines (k-nearest neighbor and median imputation) in predicting held-out similarities and recovering underlying dimensions, particularly in under-determined regimes where observations are fewer than parameters.
   • Rank Detection: The cross-validation approach for rank selection outperforms traditional methods like the Scree test and Horn's Parallel Analysis in identifying the true dimensionality.

2. Statistical Power:

   • In hypothesis testing, SRF demonstrates greater statistical power than RSA. By decomposing the similarity matrix into distinct dimensions, SRF allows researchers to test specific hypotheses against the dimension that best matches them, rather than against the full, noisy similarity matrix. This advantage is most pronounced for dimensions contributing low variance to the overall structure, which are often obscured in RSA.

3. Empirical Applications:

   • Datasets: SRF was applied to behavioral judgments (THINGS, Peterson datasets, Mur-92), neural recordings (macaque IT cortex, human fMRI from NSD), deep neural network activations (CLIP), and semantic associations (Small World of Words, SWOW).
   • Interpretability: The method recovered stable, interpretable dimensions across all domains. Examples include animacy/inanimacy distinctions in object recognition, taxonomic groupings in animals, perceptual features (color) and semantic features (animacy) in CLIP, and abstract themes (e.g., physical strength, affection) in word associations.
   • Generalization: Dimensions derived from the sparse SWOW word-association graph successfully predicted independent behavioral ratings (e.g., valence, arousal, concreteness) from the Glasgow Norms, demonstrating that the recovered dimensions capture generalizable semantic information.
   • Efficiency: In the THINGS odd-one-out task, SRF achieved behavioral prediction accuracy comparable to the state-of-the-art SPoSE neural network model but with roughly half the number of dimensions (35 vs. 66).

Significance and Claims
The paper positions SRF as a general-purpose method that bridges the gap between descriptive similarity analysis and mechanistic understanding. Its primary significance lies in its ability to move beyond scalar comparisons of similarity matrices to identify the specific factors that organize representations.

• Unified Framework: SRF unifies the strengths of similarity-based analysis (comparing systems via a common format) with the ability to infer the latent dimensions generating that structure.
• Handling Sparsity: By learning directly from incomplete data without imputation, SRF enables the analysis of datasets that were previously difficult to process, such as large-scale word association networks or sparse behavioral judgments.
• Enhanced Inference: The method offers a more sensitive tool for confirmatory hypothesis testing compared to RSA, particularly for detecting low-variance dimensions.
• Interpretability: The non-negative, parts-based nature of the recovered embeddings facilitates direct interpretation, supporting theories about brain mechanisms, stimulus properties, and model failures.

The authors conclude that SRF provides a principled, general approach for uncovering, understanding, and leveraging the dimensions underlying representations in brains, behavior, and AI, applicable wherever pairwise relationships are sparse or partially observed.