Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings

Technical Summary: Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings

Problem Statement

The standard method for comparing text embeddings is cosine similarity. However, scattered literature suggests that alternative metrics (e.g., Euclidean distance, Tanimoto coefficient, rank-based measures) can outperform cosine in specific settings. These studies have failed to establish a clear geometric condition determining when a non-cosine metric is superior or why the difference occurs. The central question addressed is: For a given encoder and task, can any simple, parameter-free metric beat cosine, and what property of the encoder controls this outcome?

Methodology

The authors conducted a comprehensive empirical study involving:

• Encoders: Nineteen distinct encoders ranging from compact sentence transformers (e.g., MiniLM, MPNet) to large language models (up to 7B parameters, including Qwen, Mistral, and GPT-2). The set includes both models fine-tuned with contrastive objectives (designed for retrieval) and base language models used without fine-tuning.
• Metrics: Nineteen parameter-free similarity metrics drawn from existing literature, categorized into:
  • Angular/Inner-product (Cosine, Dot product, Tanimoto, Dice, etc.)
  • Minkowski/Distance (L1, L2, L∞, Fractional Lp, Canberra, Bray-Curtis)
  • Information-geometry (Jensen-Shannon, Hellinger, Bhattacharyya)
  • Rank-based (Spearman rank correlation)
• Datasets: Seven datasets across three task types: Semantic Similarity (STS-B, SICK-R, STS16), Paraphrase Detection (Quora, PAWS), and Inference (SNLI, MultiNLI).
• Evaluation: Performance was measured using Spearman rank correlation for ranking tasks and ROC-AUC/PR-AUC for binary tasks. Calibration was also assessed.
• Geometric Analysis: The study measured the "shape" of the embedding space using metrics such as IsoScore, participation ratio, and specifically rogue-dimension dominance (the fraction of total variance held by the single most dominant dimension).
• Causal Testing: To verify if anisotropy is the cause rather than a correlate, the authors performed a "removal test" by projecting out the top principal directions (the "all-but-the-top" trick) and re-evaluating performance.

Key Findings

The study reveals a clean split in performance based on the geometric properties of the embedding space, specifically anisotropy (the concentration of variance into a few dominant directions).

1. Well-Spread (Isotropic) Encoders:

   • On encoders where variance is spread evenly (typically fine-tuned sentence transformers like BGE, E5, and MPNet), cosine similarity is the best parameter-free choice.
   • Alternative metrics provide no usable margin (average gain of ~0.001 in Spearman correlation).
   • Many proposed "alternatives" (e.g., angular similarity, correlation similarity) are mathematically equivalent to cosine when vectors are normalized, merely rescaling the same ranking.

2. Crowded (Anisotropic) Encoders:

   • On encoders where variance concentrates into a few dominant directions (typically base language models like GPT-2, RoBERTa, and Qwen used without contrastive fine-tuning), cosine similarity performs poorly.
   • Rank-based and L1-type metrics (Manhattan, Canberra, Bray-Curtis) beat cosine by a clear margin (average gain of ~0.055 in Spearman correlation).
   • This represents a relative improvement of approximately 20% on average, and up to 59% for extreme cases like GPT-2, because cosine starts from a low baseline on these models.

3. The Deciding Factor:

   • The outcome is determined by the geometry of the embedding space, not the training objective itself.
   • Rogue-dimension dominance (the variance fraction of the single largest dimension) predicts the performance gain of alternative metrics with a rank correlation of 0.86 and a linear correlation of 0.95.
   • While training objectives (contrastive vs. plain) often correlate with geometry, they are not the direct cause. For instance, E5-Mistral (trained with a cosine objective) remains anisotropic and benefits from alternative metrics, whereas Multilingual BERT (a plain model) is well-spread and does not.

4. Mechanism:

   • Cosine similarity is an L2-normalized inner product. When a single direction holds a massive share of the variance, it dominates the inner product, causing unrelated vectors to appear similar (angles shrink toward zero).
   • Rank-based metrics ignore the magnitude of coordinates, focusing only on order, while L1-type metrics sum absolute differences, preventing a single large coordinate from dominating the distance calculation.

Validation and Causal Evidence

To confirm that anisotropy is the cause, the authors removed the top principal directions from the embeddings:

• On Crowded Encoders: Removing the top 10 directions caused cosine performance to recover significantly (from mean Spearman 0.32 to 0.42), and the advantage of alternative metrics nearly vanished.
• On Well-Spread Encoders: The same removal slightly degraded cosine performance (as those directions held signal) and did not change the metric advantage (which remained near zero).
• Control: Removing random directions instead of the top ones did not yield the same recovery, confirming the effect is specific to the dominant directions.

Significance and Claims

The paper's primary contribution is a diagnostic rule rather than a new metric or model.

• Practical Guidance: For the vast majority of deployed systems using fine-tuned embedders (which are typically well-spread), cosine similarity remains the correct parameter-free choice. Switching metrics in these scenarios yields negligible gains.
• Risk Identification: For users employing raw or instruction-tuned language model embeddings without fine-tuning (common in retrieval or clustering pipelines where fine-tuning is skipped), cosine may be unreliable.
• Actionable Check: Practitioners can calculate a single number (rogue-dimension dominance) to determine if their encoder is anisotropic. If the value is high, they should either use a rank-based/L1 metric or remove the dominant dimensions to restore cosine's effectiveness.
• Unification: The study resolves conflicting prior literature by showing that both claims (cosine is best; alternatives are better) are correct depending on the geometric state of the specific encoder being used.

The authors emphasize modesty regarding absolute gains (approx. 0.05 Spearman), noting that while significant relative to the low baseline of anisotropic models, it does not elevate raw LLM embeddings to the performance level of well-spread, fine-tuned embedders. The work separates the role of the embedding space geometry from the role of the comparison metric.