📊 statistics

Scalable and Interpretable Representation Alignment with Ordinal Similarity

This paper introduces a scalable and interpretable ordinal-similarity framework, instantiated by Triplet and Quadruplet Similarity Indices, to overcome the interpretability, robustness, and scalability limitations of existing representation alignment metrics.

Original authors: Diogo Soares, Pankhil Gawade, Andrea Dittadi, Ewa Szczurek

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Diogo Soares, Pankhil Gawade, Andrea Dittadi, Ewa Szczurek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge how similar two different maps of the same city are. One map is drawn by a human, and the other by a robot.

The Problem with Current Tools
Right now, scientists use tools to compare these "maps" (which are actually complex data representations inside AI models). But these tools have three big flaws:

They are confusing: The scores they give are like a raw number without a ruler. Is a score of 0.7 good? Bad? It depends on the size of the map or the type of ink used, making it hard to know what "good" actually looks like.
They are fragile: If you throw one weird, giant rock (an "outlier") onto the map, the tool might scream that the two maps are completely different, even if 99% of them match perfectly.
They are slow: To get an accurate score, these tools often need to measure every single street and alleyway. On a massive city (a huge dataset), this takes forever, so people have to guess using shortcuts that aren't always reliable.

The New Solution: The "Ordinal" Approach
The authors of this paper propose a new way to compare maps called Ordinal Similarity. Instead of measuring the exact distance between two points (e.g., "Point A is exactly 5 miles from Point B"), they only care about the order or ranking of distances.

Think of it like this:

Old Way: "Is A 5 miles from B, and is C 10 miles from B?"
New Way: "Is A closer to B than C is?"

If the answer to the second question is the same in both maps, that's a point of agreement. They don't care about the exact miles, just the relative order.

The Two New Tools: TSI and QSI
The paper introduces two specific tools to do this:

TSI (Triplet Similarity Index): Imagine you pick one person (the "anchor") and two friends. You ask: "Is Friend 1 closer to the Anchor than Friend 2?"
- If both maps agree on who is closer, that's a "win."
- TSI counts how many times the maps agree on these "who is closer?" questions.
- Analogy: It's like checking if two people agree on the order of their favorite ice cream flavors, regardless of how much they love each one.
QSI (Quadruplet Similarity Index): This is a step further. Imagine two pairs of people. You ask: "Is the distance between Pair A smaller than the distance between Pair B?"
- This checks if the maps agree on the scale of distances between different groups, not just relative to one person.
- Analogy: It checks if both maps agree that "the park is closer to the school than the library is to the stadium."

Why This is a Game-Changer

It's Easy to Understand: The score is a simple probability. If you get a score of 0.5, it means the maps are completely random (like flipping a coin). If you get 0.8, it means 80% of the time, the maps agree on who is closer. You don't need a PhD to know that 0.8 is better than 0.5.
It's Tough: If you throw a giant rock (an outlier) onto the map, it only messes up a tiny fraction of the "who is closer?" questions. The rest of the map stays perfect. The score barely moves, so the tool doesn't panic.
It's Fast: Because the tool only cares about order, it can use smart math tricks to guess the answer by checking just a tiny sample of the city. It can give you a highly accurate answer in seconds, even for massive datasets, without needing to measure every single street.

What They Proved
The authors mathematically proved that:

These tools are robust: They won't break if the data is messy.
They are scalable: They work fast on huge amounts of data.
They are equivalent to checking if the "neighborhoods" on the map are the same. If the maps agree on who is closest to whom, they are essentially the same map.

Real-World Tests
The team tested this on:

Training AI: Watching how an AI's internal "map" changes as it learns. Their tool showed the map getting better and more consistent over time.
Multimodal Models (like CLIP): Checking if an AI understands that a picture of a cat and the word "cat" are similar. Their tool showed that bigger, smarter models had better alignment, whereas older tools got confused when the models changed size.

In a Nutshell
This paper gives scientists a new, reliable, and easy-to-read ruler for measuring how similar AI models are. Instead of getting lost in complex numbers and fragile measurements, they can now simply ask: "Do these two models agree on who is closer to whom?" and get a clear, trustworthy answer.

Technical Summary: Scalable and Interpretable Representation Alignment with Ordinal Similarity

Problem Statement

Evaluating the similarity between representation spaces is a foundational challenge in representation learning, essential for interpreting model behavior, aligning artificial systems with human cognition, and designing transfer learning or generative models. However, existing similarity metrics suffer from three critical limitations:

Lack of Interpretability: Most metrics yield raw scores without intuitive meaning or consistent baselines, making it difficult to distinguish meaningful alignment from spurious similarity.
Sensitivity to Outliers: Current metrics often lack robustness; a small number of extreme outliers can drastically alter similarity scores, causing perfectly aligned representations to appear misaligned.
Computational Intractability: Exact computation for large datasets is often infeasible due to high memory and time requirements, forcing reliance on heuristic approximations that lack theoretical bounds for correctness.

Methodology

The authors propose a shift from metrics relying on precise nominal distance values to those emphasizing robust relational properties, inspired by Non-metric Multidimensional Scaling (NMDS). The core hypothesis is that preserving ordinal relationships (rank orders of distances) captures the essential structure needed for alignment.

The paper introduces two primary metrics based on ordinal consistency:

Triplet Similarity Index (TSI): Measures the consistency of anchor-based relative similarity. It evaluates the fraction of triplets $(i, j, k)$ where the ordinal relationship "point $i$ is closer to $j$ than to $k$ " is preserved across two representation spaces.
Quadruplet Similarity Index (QSI): Measures the consistency of relative similarity between disjoint pairs. It evaluates the fraction of quadruplets $(i, j, k, l)$ where the relationship "distance $(i, j)$ is less than distance $(k, l)$ " is preserved.

Theoretical Foundations

Probabilistic Interpretation: TSI and QSI are defined as the probability that a randomly chosen ordinal relationship is preserved. This provides a direct semantic meaning (e.g., a TSI of 0.8 implies 80% of triplets agree on ordering).
Baseline for Dissimilarity: The authors prove that for statistically independent representations, the expected score converges to approximately $0.5$ (assuming negligible ties), establishing a universal, interpretable baseline for random alignment regardless of dataset size or dimensionality.
Neighborhood Equivalence: A formal equivalence is established between perfect TSI alignment and the preservation of Mutual Nearest Neighbors (MNN) across all scales ( $k=1$ to $N-2$ ).
Robustness: Theoretical bounds demonstrate that TSI and QSI are inherently robust to outliers and corruptions. The deviation of the similarity score is bounded by the proportion of transformed points, ensuring that sparse outliers cannot disproportionately shift the metric.
Metric Axioms: The complementary distances ( $1 - \text{TSI}$ and $1 - \text{QSI}$ ) are proven to satisfy the axioms of a proper metric (equivalence, symmetry, and triangle inequality).

Computational Efficiency

While naive implementations would require $O(N^3)$ and $O(N^4)$ time, the authors leverage connections to the Kendall rank correlation coefficient to develop efficient algorithms:

Exact Computation: TSI can be computed in $O(N^2 \log N)$ time and $O(N)$ space; QSI in $O(N^2 \log N)$ time and $O(N^2)$ space.
Scalable Approximation: As bounded U-statistics, TSI and QSI admit principled Monte Carlo approximations. By sampling a small number of triplets or quadruplets, the metrics can be estimated with additive error $\epsilon$ and confidence $1-\delta$ in time independent of the dataset size $N$ .

Key Contributions

Ordinal Similarity Formulation: The formalization of TSI and QSI as metrics based on the agreement of relative similarity and distance comparisons.
Theoretical Interpretability: Demonstration of a direct probabilistic interpretation and the establishment of a stable, universal baseline ( $\approx 0.5$ ) for dissimilar representations.
Connection to Neighborhood Consistency: Proof that perfect TSI alignment is equivalent to joint Mutual Nearest Neighbors alignment across all scales.
Guaranteed Robustness: Theoretical bounds guaranteeing robustness to outliers and corruptions, addressing a key fragility in existing metrics like CKA.
Computational Scalability: Provision of efficient exact algorithms and a principled approximation scheme with theoretical error bounds, enabling application to massive datasets.
Empirical Validation: Validation across diverse scenarios, including tracking representation convergence during training and evaluating multimodal alignment in CLIP models.

Experimental Results

The authors validate the proposed metrics through several experiments:

Consistent Baselines: In tests with statistically independent representations, TSI and QSI maintained stable scores near 0.5 across varying sample sizes and dimensions. In contrast, metrics like CKA, SVCCA, and PWCCA showed high sensitivity to these parameters, sometimes yielding near-perfect alignment scores for independent data.
Outlier Robustness: When 2% of data points were perturbed with increasing outlier strength, TSI and QSI remained remarkably stable, adhering to their theoretical lower bounds. Conversely, neighborhood-based metrics (MutualNN, CKNNA) and kernel-based metrics (CKA) dropped to near-zero values due to "hubness" effects induced by outliers.
Approximation Quality: The principled sampling approximation for TSI and QSI demonstrated rapid, monotonic convergence with negligible deviation, significantly outperforming standard batched approximations used for other metrics.
Real-World Applications:
- Training Dynamics: TSI and QSI accurately tracked the convergence of Vision Transformer (ViT) representations during training on CIFAR-10, distinguishing between models trained with the same seed (converging to 1.0) and distinct seeds (converging to a sub-perfect alignment).
- Multimodal Alignment: In evaluating CLIP models of varying scales, TSI and QSI correctly captured the monotonic increase in alignment. Notably, CKA exhibited counter-intuitive behavior (a drop in score) when transitioning to a model with a different embedding dimension, highlighting its sensitivity to dimensionality changes.

Significance

The paper claims that TSI and QSI provide a rigorous, scalable, and theoretically grounded framework for representation alignment. By prioritizing distance ordering over raw values, these metrics offer:

Interpretability: Scores have clear probabilistic meanings and stable baselines, allowing practitioners to judge alignment quality without case-specific calibration.
Robustness: They are theoretically guaranteed to be insensitive to sparse outliers and corruptions, a property lacking in many current state-of-the-art metrics.
Scalability: The ability to compute exact values efficiently or approximate them with guaranteed error bounds makes them applicable to modern large-scale datasets where other methods fail or require unreliable heuristics.

The authors conclude that ordinal similarity enables practitioners to better understand and design representations, offering a reliable alternative to existing metrics that often suffer from interpretability issues, fragility, and computational constraints.