Technical Summary: Multi-way Representation Alignment

Problem Statement

Deep neural networks trained independently often converge to similar latent representation spaces, a phenomenon described by the Platonic Representation Hypothesis. While existing methods successfully align pairs of models, they face significant limitations when scaling to $M \ge 3$ models. Current pairwise strategies scale quadratically ($O(M^2)$) in the number of maps, require fitting $M-1$ new maps to incorporate a single new model, and fail to enforce global consistency. Specifically, composing pairwise maps (e.g., mapping $X \to Y$ then $Y \to Z$) often yields different results than learning a direct map $X \to Z$, making pairwise alignment unreliable as a shared reference for large collections of models.

Methodology

The paper proposes a framework to align multiple representation spaces $\{X_m\}_{m=1}^M$ through a shared "universe" reference space $U$. This approach reduces complexity to $O(M)$ by learning a single map $\Omega_m$ from each model to the universe, ensuring that translation between any two models is unique and path-independent ($\Omega_{m \leftarrow n} = \Omega_m \Omega_n^\top$).

The authors explore three specific alignment strategies:

1. Generalized Procrustes Analysis (GPA):
   Adapted for neural representations, GPA constructs an orthogonal universe. It seeks a consensus centroid $U$ and a set of orthogonal rotations $\{\Omega_m\}$ that minimize global dispersion. Because the maps are constrained to the orthogonal group $O(d)$, GPA preserves the internal geometry (distances and angles) of each model. This is critical for tasks like model stitching and latent probing but may be suboptimal for retrieval where maximizing cross-model agreement is prioritized over strict isometry.

2. Generalized Canonical Correlation Analysis (GCCA):
   GCCA relaxes the orthogonality constraint, allowing linear projections that deform spaces to maximize shared correlation. While GCCA minimizes pairwise mismatch energy and excels at retrieval tasks, it distorts the internal geometry of the models, rendering it less suitable for tasks requiring geometric fidelity.

3. Geometry-Corrected Procrustes Alignment (GCPA):
   To bridge the gap between geometric fidelity and retrieval performance, the authors propose GCPA. This two-stage method:

   • Stage 1 (Initialization): Solves the standard GPA problem to establish a robust, orthogonal universe and base representations.
   • Stage 2 (Correction): Applies a lightweight, shared residual map $T_\theta$ (a small MLP) to the universe coordinates. This correction minimizes the distance to a per-sample consensus direction (the average of normalized vectors across all models) while strictly penalizing deviation from the original GPA geometry via a "trust" penalty. This allows the method to maximize agreement (improving retrieval) without discarding the geometric scaffold.

Key Contributions

1. Extension to Multi-way Alignment: The authors extend orthogonal representation alignment to three or more models by adapting GPA to the neural representation setting, establishing a shared coordinate system that guarantees cycle consistency.
2. Identification of Trade-offs: The paper empirically demonstrates that while GPA provides a consistent global reference superior to pairwise maps for geometric tasks, it lags behind agreement-maximizing methods like GCCA on retrieval tasks.
3. GCPA Proposal: The introduction of Geometry-Corrected Procrustes Alignment, which combines a robust orthogonal universe with a post-hoc consensus-driven correction. This achieves state-of-the-art performance in any-to-any retrieval while retaining the practical benefits of a shared Procrustes frame.

Experimental Results

The authors evaluate their methods on benchmarks spanning inter-architecture probing, multilingual translation, cross-camera person re-identification, and multimodal retrieval.

• Geometric Interoperability: In "weak-link" experiments where two models share little surface similarity, a multi-way universe stabilizes alignment better than direct pairwise maps. Furthermore, the shared universe allows for efficient extension; adding a new model requires fitting only one map, matching or outperforming the need to refit the entire universe or fit multiple pairwise maps.
• Clustering: On the MASSIVE dataset, standard alignment (GPA, GCCA) did not improve clustering quality over unaligned spaces. However, GCPA significantly improved the Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI), indicating that the geometric correction successfully pulls semantically similar items together.
• Retrieval Performance:
  • Multilingual Translation (TED-MULTI): GCPA outperformed pairwise, GPA, and GCCA baselines across varying numbers of languages ($M=3, 5, 10$), improving both average and worst-case retrieval accuracy.
  • Robustness to Noise: Under 75% corrupted training correspondences, GCPA degraded more gracefully than GCCA and maintained higher absolute accuracy than GPA.
  • Person Re-identification (MARKET-1501): GCPA achieved the highest Mean Average Precision (mAP) across all camera pairs, outperforming both strict isometries (GPA) and agreement-maximizing methods (GCCA).
  • Multimodal Retrieval (FLICKR8K): GCPA substantially improved hard alignment cases, such as audio-to-image retrieval (e.g., HuBERT to DINOv2), nearly doubling performance compared to GPA. Analysis showed GCPA reduced the mean cross-modal distance and increased the agreement of high-coherence triplets.

Significance and Claims

The paper claims that multi-way alignment through a shared universe is necessary for scaling representation interoperability. The primary significance lies in resolving the tension between preserving internal model geometry (essential for stitching) and maximizing cross-model agreement (essential for retrieval).

The authors assert that GCPA effectively synthesizes the best of both worlds: it uses the GPA universe as a geometric scaffold to ensure stability and consistency, while the consensus-driven correction refines the alignment for semantic agreement. This allows for a practical, composable reference space that achieves state-of-the-art retrieval performance without sacrificing the structural integrity required for other downstream tasks. The work suggests that such shared-universe alignment can support interoperability across independently trained systems, though it notes that this may also propagate biases if not carefully managed.