💬 NLP

A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures

This paper systematically evaluates four multilingual interpretability methods across 21 models, finding that anisotropy distorts most metrics while ILO uniquely and robustly correlates with cross-lingual transfer performance, leading the authors to recommend ILO as the primary sharing metric alongside anisotropy diagnostics.

Original authors: Oskar Holmström, Marcel Bollmann, Marco Kuhlmann

Published 2026-09-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Oskar Holmström, Marcel Bollmann, Marco Kuhlmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems that understand language are built on a foundation of mathematics that maps words and sentences into vast, multi-dimensional spaces. In these spaces, the meaning of a word is represented by a specific point, and the relationship between words is defined by the distance and direction between those points. When these models are trained on many different languages at once, they develop a remarkable ability: they begin to use the same internal points to represent similar concepts across those languages, effectively creating a shared mental dictionary. This phenomenon, known as cross-lingual sharing, is the engine that allows a model trained mostly on English to answer questions in Swahili or translate poetry from Japanese. However, for years, scientists have struggled to measure exactly how well this sharing happens. Different research teams have built different rulers to measure the same thing, and often, those rulers gave conflicting answers. One team might say a model shares its knowledge perfectly, while another, using a different method, might say it barely shares anything at all. This confusion made it difficult to know which models were truly multilingual and which were just pretending.

A team of researchers set out to resolve this confusion by comparing four representative measurement tools selected to contrast different methodological families against a wide variety of language models. They gathered twenty-one different models, ranging from small, efficient versions to massive, complex ones, representing five distinct families of artificial intelligence. These models varied in size from 125 million to 14 billion parameters, a scale that captures the diversity of the field. The researchers fed these models a standard set of sentences in ten different languages, including Arabic, Chinese, English, and Turkish, and then applied each of the four measurement tools to see how much the models mixed their languages together. They also tested the models' real-world performance by teaching them a new task in English and seeing how well they could apply that knowledge to other languages without further training. This functional test served as a validation criterion, a way to see if the measurements actually predicted the models' ability to transfer knowledge, though the authors note that transfer is driven by factors beyond just representational alignment.

The results revealed that the disagreement between the measuring tools was not a reflection of the models themselves, but rather a flaw in how the tools were built. The researchers discovered that the internal representations of these models are often "anisotropic," a term describing a shape that is stretched into a narrow, needle-like cone rather than spreading out evenly in all directions. Imagine the difference between a sphere and a thin pencil; in the pencil-shaped space, almost any two points look close to each other simply because the space is so narrow. Three of the four measuring tools were fooled by this shape. They interpreted the fact that all points were crowded together as evidence of strong sharing, even when the models were not actually mixing their languages. One tool, in particular, relied on a method that stripped away the most important directions of information to find patterns, only to discover that the language identity had been pushed into the discarded directions. It was like trying to find a specific flavor in a soup by removing the salt, only to conclude the soup was bland because the salt was gone.

In contrast, one specific measurement, called Interlingual Local Overlap, proved to be the only reliable ruler. This tool looked at the immediate neighbors of each word in the model's internal space, asking whether words from different languages were standing next to each other. Unlike the other tools, this method survived rigorous testing. It consistently predicted which models would perform well on cross-lingual tasks, showing a strong correlation that held true regardless of the model's size or family. The researchers found that models with high scores on this specific metric were the ones that could successfully transfer knowledge from English to other languages. The other tools either failed to distinguish between models that shared knowledge and those that did not, or they were so influenced by the narrow shape of the data that they produced misleadingly high scores for models that were actually quite poor at sharing.

The study concludes that the field has been misled by tools that are sensitive to the geometric distortions of the data rather than the actual mixing of languages. The researchers recommend that future studies use Interlingual Local Overlap as the primary sharing metric, but to be reported alongside anisotropy diagnostics. Furthermore, they suggest that whenever anyone reports a score for language sharing, they must also report a diagnostic check for the shape of the data, ensuring that a high score reflects genuine mixing and not just a narrow, distorted space. This work does not claim to have solved the mystery of how language models learn, but it provides a clear, trustworthy way to measure what they have learned. By identifying the tools that work and the ones that fail, the researchers have cleared the fog, allowing the scientific community to finally compare models on a level playing field and understand the true nature of multilingual intelligence. These results are corrective rather than dismissive of global sharing metrics: a single global score remains the right instrument for quantifying and comparing sharing across models, while explaining how sharing is implemented calls for mechanistic methods.

Technical Summary: A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures

Problem Statement

Multilingual language models (MLMs) develop shared cross-lingual representations without explicit alignment supervision. While various interpretability methods claim to quantify this sharing, they have largely been developed in isolation. Consequently, when different metrics yield conflicting conclusions about the degree of sharing in a model, it remains unclear whether the disagreement reflects genuine architectural differences or is an artifact of the measurement technique itself.

A critical confounding factor is representational anisotropy: the tendency of contextual embeddings to occupy a narrow cone in the embedding space rather than spreading isotropically. This geometry inflates apparent similarity between vectors and confines meaningful variation to low-dimensional subspaces. The paper investigates whether existing sharing metrics are robust to this distortion or if they inadvertently measure anisotropy rather than genuine cross-lingual alignment.

Methodology

The authors conducted a systematic comparison of four representative sharing metrics across 21 base models from five distinct families (BLOOM, OPT, XGLM, Qwen3, SmolLM3), ranging from 125M to 14B parameters. The study utilized the FLORES-200 devtest set (10 languages) for representation analysis and five downstream tasks (Belebele, XNLI, XCSR, SIB-200, XQuAD) for functional validation.

The Four Metrics Evaluated

CKA (Centered Kernel Alignment): Measures global geometric similarity between cross-lingual representation matrices.
ANC (Average Neuron Correlation): Measures per-neuron alignment via Pearson correlation of activations on parallel sentences.
$\lambda$ (GMM Dominance per Token): Fits a Gaussian Mixture Model (GMM) in a PCA subspace (retaining 98% variance) and calculates the fraction of tokens more likely under a foreign-language component than their own.
ILO (Interlingual Local Overlap): A neighborhood-based metric using $k$ -nearest neighbors ( $k=5$ ) in the full hidden state space to measure bridge and reachability scores between languages.

Experimental Framework

Functional Validation: The authors correlated each metric with zero-shot cross-lingual transfer performance (after fine-tuning on English).
Statistical Controls: They employed Spearman rank correlations, partial correlations (controlling for model size), and permutation tests (shuffling within families) to distinguish genuine relationships from confounds.
Anisotropy Diagnostics: They measured effective dimensionality ( $n_{pca}$ ) and mean random cosine similarity to quantify the degree of anisotropy.
Probing: Language probes were trained on the full hidden state, the PCA principal subspace, and the residual subspace to locate where language identity is encoded.

Key Findings

1. Three Distinct Sharing Patterns

The analysis revealed three distinct patterns of cross-lingual sharing, rather than a binary split:

Concentrated Sharing (BLOOM $\ge$ 1B, Qwen3, SmolLM3): High $\lambda$ and ILO, with variance concentrated in just 2 dimensions ( $n_{pca}=2$ ) at peak layers.
Distributed Sharing (XGLM): High ILO but near-zero $\lambda$ , with variance distributed across thousands of dimensions.
Minimal Sharing (OPT): Low values across all metrics, consistent with English-centric training.

2. Anisotropy-Driven Failures

The disagreement between metrics is directly traceable to anisotropy:

CKA is inflated by the shared cone direction, showing a narrow, uninformative range across families.
$\lambda$ (GMM Dominance) fails catastrophically in distributed models (like XGLM). Because it operates in a PCA subspace retaining 98% of variance, it discards the dimensions where language identity actually resides in these models. Probing experiments showed that in high- $\lambda$ models, language identity is relocated to the residual subspace (discarded by $\lambda$ ), while in distributed models, the principal subspace contains no cluster structure for the GMM to fit. Thus, $\lambda$ measures the behavior of a mixture model in a subspace from which language information has been removed.
ANC is more robust as it subtracts neuron means, making it invariant to the common-shift component of anisotropy.
ILO remains robust because it operates on the full hidden state and is invariant to per-vector scaling. A centered variant (ILOc) confirmed that the metric is not driven by the shared anisotropic mean shift.

3. Prediction of Cross-Lingual Transfer

Only ILO demonstrated a robust association with cross-lingual transfer performance:

ILO: Spearman $\rho = 0.90$ across all models. This correlation survived controls for model size, family membership, and per-task variation.
ANC: Showed a strong correlation ( $\rho = 0.84$ ) but this relied heavily on between-family contrasts; it did not survive permutation tests that shuffled data within families.
$\lambda$ : Moderate correlation ( $\rho = 0.52$ ) that failed to hold consistently across tasks and families.
CKA: No significant association ( $\rho = 0.14$ ).

4. Training Dynamics

Sharing does not emerge gradually. In models like BLOOM, $\lambda$ and ILO undergo abrupt phase transitions at specific training steps (e.g., between 10K and 100K steps for BLOOM-1.1B). Instruction tuning can induce sharing in small models (e.g., BLOOMZ-560m) but has negligible effects on larger models already exhibiting strong sharing.

Significance and Recommendations

The paper argues that geometric sharing scores cannot be interpreted independently of the geometry in which they sit. The primary contribution is the identification of anisotropy as the driver of metric disagreement, specifically showing that PCA-based methods like $\lambda$ can report "sharing" where language identity is merely relocated to discarded dimensions, or miss sharing where it is distributed.

Recommendations:

Adopt ILO: The authors recommend ILO as the primary metric for quantifying cross-lingual sharing, as it is the only one whose association with transfer survives rigorous controls.
Report Anisotropy Diagnostics: Any geometric sharing metric should be reported alongside anisotropy diagnostics (e.g., effective dimensionality $n_{pca}$ and mean random cosine similarity) to allow readers to distinguish genuine sharing from anisotropic collapse.
Caution with PCA: Metrics relying on variance-ranked projections (like $\lambda$ ) are unreliable in highly anisotropic models where language information may reside in low-variance directions.

The authors conclude that while global sharing metrics remain useful instruments for comparing models, their reliability is contingent on the underlying representational geometry, necessitating a shift toward metrics that are robust to anisotropy.