🤖 machine learning

Improving Relative Representations with Learned Anchors and Whitened Inner Products

This paper proposes a robust framework for cross-model communication that improves upon traditional Relative Representations by learning semantic anchor prototypes and employing a geometry-aware similarity metric, thereby enabling stable, nearly lossless information transfer between heterogeneous neural architectures.

Original authors: Oscar Thorsted Svendsen, Nikolaj Holst Jakobsen, Fabian Mager, Hiba Nassar

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Oscar Thorsted Svendsen, Nikolaj Holst Jakobsen, Fabian Mager, Hiba Nassar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two different translators who speak the same language but use completely different dictionaries and sentence structures. One translator (Model A) describes a "king" as a tall, golden figure, while the other (Model B) describes a "king" as a short, blue figure. If you try to give Model B a note written by Model A, it won't understand it because the coordinates don't match.

This is the problem Relative Representations tries to solve. Instead of trying to force Model A and Model B to agree on the exact coordinates of a "king," they agree on a set of landmarks (anchors). They say, "Okay, regardless of how you draw the map, how close is the 'king' to the 'queen' and the 'castle'?" If both models agree on the distances to these landmarks, they can talk to each other, even if their internal maps look totally different.

However, the old way of doing this had two big flaws, which this paper fixes with a new system called PARAM and WIP.

The Problem with the Old Way

Random Landmarks: The old method picked landmarks by throwing darts at a map. Sometimes you'd pick a bunch of darts that landed in the same spot (redundancy), or you'd miss huge areas of the map entirely. This meant the "translation" was often blurry or lost important details.
The Wrong Ruler: The old method used a ruler that only measured angles (direction) but ignored distance (magnitude). Imagine trying to measure a room where the floor is stretched unevenly. If you only look at the angle of a corner, you might think a small, stretched room is the same as a large, normal one. This caused the "translation" to break when the models were very different from each other (like a small language model talking to a giant one).

The New Solution: PARAM and WIP

1. PARAM: Learning the Best Landmarks

Instead of throwing darts randomly, the authors teach the computer to learn the perfect landmarks.

The Analogy: Imagine you are trying to describe a city to a friend. Instead of picking random street corners, you ask the city itself: "Where are the most important, stable hubs that represent the whole area?"
How it works: The system creates landmarks by averaging groups of data points. This smooths out the noise (like averaging a few shaky photos to get one clear picture). These new landmarks are "robust semantic prototypes"—they are stable, cover the whole map evenly, and don't get confused by the quirks of a specific model.

2. WIP: The Smart Ruler

The authors replaced the old "angle-only" ruler with a Whitened Inner Product (WIP).

The Analogy: Think of the old ruler as a rubber band that stretches and shrinks depending on who is holding it. If Model A stretches the space and Model B squishes it, the rubber band gives the wrong answer.
How it works: The new WIP ruler is "whitened." Before measuring, it mathematically flattens out the rubber band. It corrects for stretching, squishing, and shifting. Crucially, it also pays attention to how strong a signal is (magnitude), not just which way it points. This allows the models to keep important details (like "how confident" the model is) that the old method threw away.

The Results: "Lossless" Translation

The authors tested this by trying to connect different types of AI models:

Vision: They connected a CNN (a type of image model) with a Transformer (a different, more modern image model).
Language: They connected different language models (like BERT variants) that speak different languages (English, German, French, etc.).
Small vs. Big: They even connected tiny language models to much larger ones.

The Outcome:
In the past, trying to stitch these models together resulted in a disaster (the new model would guess randomly). With PARAM and WIP, the models communicated almost perfectly.

They achieved zero-shot stitching: You could train a tool on Model A's data and immediately use it on Model B's data without any retraining.
The performance was nearly identical to using the original model directly. It was as if the "translation" was lossless—no information was lost in the process.

Summary

This paper introduces a new way to make different AI models talk to each other.

Stop guessing landmarks: Learn the best, most stable reference points (PARAM).
Use a better ruler: Use a measurement tool that accounts for stretching and shrinking in the data (WIP).

The result is a universal "plug-and-play" system where you can swap out the brain (encoder) of an AI for a different one, and the rest of the system still works perfectly without needing to be retrained.

Technical Summary: Improving Relative Representations with Learned Anchors and Whitened Inner Products

1. Problem Statement

Independently trained neural models often converge to latent representations that are incompatible due to differences in rotation, anisotropic scaling, and translation. This creates a fundamental barrier to building highly modular AI systems where encoders and decoders can be swapped without retraining.

Relative Representations (RR) address this by mapping absolute embeddings to a shared space defined by similarities to a common set of anchor points, rather than relying on absolute coordinates. However, standard RR implementations suffer from two primary limitations:

Random Anchor Sampling: Traditional methods use randomly sampled datapoints as anchors. This often leads to poor coverage of the data manifold, redundancy, and low-rank relative features, causing a loss of task-relevant information.
Cosine Similarity: The standard similarity metric relies solely on angular relationships, discarding magnitude information (which can encode feature confidence) and failing to account for the anisotropic geometries common in modern architectures like Transformers. It is also unstable under affine shifts and near the origin.

These limitations result in poor cross-space alignment and information preservation, preventing effective zero-shot stitching between heterogeneous models.

2. Methodology

The authors propose a robust framework comprising two core components: PARAM (Parameterized Anchor Representations via Adaptive Matrices) and WIP (Whitened Inner Product).

2.1 PARAM: Learning Anchors as Convex Mixtures

Instead of treating anchors as a fixed, randomly sampled subset of data, PARAM learns anchors as convex mixtures of a support set of parallel datapoints.

Mechanism: A shared mixture matrix $P$ is optimized to generate anchors $A_i$ for each encoder space $i$ via $A_i = P X_{sub}^i$ , where $X_{sub}^i$ is a subset of embeddings.
Constraints: $P$ is constrained to be row-stochastic, ensuring anchors remain within the convex hull of the data.
Benefits: This approach provides a denoising effect through averaging, creating stable semantic prototypes. It allows for continuous anchor placement within the latent space, avoiding the limitations of discrete datapoint selection.
Training Objectives: The matrix $P$ $P$ is optimized using a weighted sum of losses:
- Single-space objectives: Coverage ( $L_{cov}$ , soft k-means), Orthogonality ( $L_{orth}$ , penalizing correlated anchors), and Length control ( $L_{len}$ , constraining norms).
- Multi-space objective: Symmetric InfoNCE ( $L_{symNCE}$ ) to align relative coordinates across different encoders using paired samples.

2.2 WIP: Whitened Inner Product

To address the geometric distortions (mean shifts and anisotropic scaling) between independently trained spaces, the authors replace cosine similarity with the Whitened Inner Product.

Mechanism: For each embedding space, the empirical mean $\mu_i$ is removed, and a symmetric whitening transformation $L_i = (\Sigma_i)^{-1/2}$ is applied based on the covariance matrix $\Sigma_i$ .
Formula: The similarity is defined as $s_{WIP}(x, a_r^i) = \langle L_i(x - \mu_i), L_i(a_r^i - \mu_i) \rangle$ .
Benefits: This metric is robust to affine distortions (satisfying Cross-Space Alignment) while preserving both angular and magnitude information (satisfying Information Preservation), which is critical for discriminative signal.

3. Key Contributions

Learned Anchors (PARAM): A method to learn anchors as convex mixtures of data, ensuring robust semantic coverage and stability across embedding spaces, replacing random sampling.
Geometry-Aware Similarity (WIP): A covariance-aware similarity metric that is invariant to affine shifts and preserves magnitude information, addressing the anisotropy of modern architectures.
Unified Framework: A joint approach that significantly improves zero-shot stitching performance across highly heterogeneous architectures (e.g., CNNs vs. Transformers, and varying scales of Language Models).

4. Experimental Results

The framework was evaluated on zero-shot stitching tasks across vision and language modalities, comparing against a baseline of random anchors with cosine similarity.

Image Classification (CIFAR-100): When stitching CNN-based (RexNet) and Transformer-based (ViT) encoders, the proposed PARAM-WIP method achieved near-lossless transfer. For instance, training a classifier on ViT features and testing on RexNet features yielded an F1 score of 80.94% (PARAM Multi) compared to 56.67% for the Random-Cosine baseline. In many cases, zero-shot performance matched the absolute performance of the original latent space.
Text Classification (MARC): On cross-lingual sentiment analysis (e.g., English, German, French, Spanish, Chinese), the framework consistently outperformed the baseline. For example, stitching French to English encoders resulted in 93.75% F1 (PARAM Multi) versus 92.06% (Random-Cosine).
Small Language Models (SLMs): In a zero-shot retrieval task between models of varying scales (GPT-2, TinyLlama, Gemma), the baseline showed extreme instability (R@1 $\approx$ 0.5% - 10%). In contrast, PARAM-WIP achieved R@1 scores between 87% and 95% and R@5 scores near 100%, demonstrating stable geometric alignment even between models with vastly different hidden dimensionalities.
Ablation Studies: Results confirmed that both components are necessary. While WIP improves retrieval quality, its accuracy gains are maximized when combined with learned anchors. Conversely, random anchors with WIP often underperformed random anchors with cosine similarity due to noise sensitivity near the origin.

5. Significance and Claims

The paper claims that the reliance on random sampling and angular projection (cosine similarity) is a fundamental limitation in current relative representation research, particularly for heterogeneous model families.

Robustness: The proposed framework enables nearly lossless information transfer and stable zero-shot communication between encoders that were previously incompatible.
Modularity: By formalizing the relationship between latent geometry and relative coordinate systems, the work provides a practical pathway toward modular AI architectures where encoders and decoders can be interchanged without costly retraining.
Geometric Understanding: The results suggest that preserving magnitude information and accounting for affine distortions are critical for aligning modern, anisotropic embedding spaces, a requirement that standard cosine similarity fails to meet.

The authors note limitations, including the dependence of representational capacity on the size of the support subset and the requirement for stable covariance estimation for whitening. They suggest future work could explore adaptive subset selection, structured sparsity, and non-linear similarity functions.