On the Robustness of Multilingual Text Embedding Rankings Across Learning Tasks, Languages, and Benchmark Datasets

Technical Summary: Robustness of Multilingual Text Embedding Rankings

Problem Statement
Large-scale multilingual text embedding models are foundational to modern NLP systems, including retrieval-augmented generation (RAG) and semantic search. Despite the existence of comprehensive benchmarking platforms like MTEB (Massive Text Embedding Benchmark) that evaluate models across 250+ languages and numerous tasks, the robustness of their reported rankings remains insufficiently understood. Current leaderboards typically rely on simple averaging of performance scores across heterogeneous datasets. This approach implicitly assumes metric comparability, ignores dataset correlations, and can bias rankings toward overrepresented tasks or languages. Consequently, conclusions about model superiority may depend heavily on implicit choices regarding dataset composition and performance aggregation methods, obscuring whether observed rankings reflect genuine model capabilities or artifacts of the evaluation design.

Methodology
The authors present a meta-study analyzing the robustness of multilingual embedding rankings within the MTEB Multilingual v2 framework. The study employs a two-pronged approach to quantify evaluation uncertainty:

1. Robustness Indicators:

   • Dataset-Composition Robustness (DS): Measures the sensitivity of model rankings to changes in dataset composition. This is achieved through correlation-aware subsampling, where datasets with highly correlated performance profiles are clustered, and representative subsets are randomly selected to generate uncorrelated evaluation scenarios.
   • Ranking-Scheme Robustness (RS): Measures the sensitivity of rankings to the choice of score aggregation methodology. The study utilizes a diverse set of Multi-Criteria Decision-Making (MCDM) schemes (including WSM, TOPSIS, VIKOR, and PROMETHEE II) combined with different objective weighting strategies (Equal, MEREC, CRITIC) to generate multiple rankings for the same data.

2. Experimental Design:

   • Scope: The analysis covers nine learning tasks (classification, clustering, retrieval, reranking, pair classification, STS, multiclass classification, instruction reranking, and bitext mining) across five languages (English, French, German, Hindi, and Spanish) in depth, with artifacts released for approximately 230 additional languages.
   • Constraints: The study focuses on open, 100% zero-shot models to avoid data leakage from task-specific training.
   • Analysis: For each language-task pair, the authors compute rank variance across different MCDM schemes (for RS) and across different dataset subsamples (for DS). They also calculate a cross-task consistency (CT) score to identify models that remain top performers across diverse downstream applications within a single language.

Key Contributions

1. Novel Robustness Indicators: The paper introduces the first systematic indicators for quantifying evaluation uncertainty in multilingual benchmarks, distinguishing between rankings that are stable under varying evaluation designs and those that are fragile.
2. Language-Granular Analysis: Unlike previous studies that aggregate results globally, this work provides a granular analysis of robustness at the language level, revealing significant disparities in benchmark reliability across different languages.
3. Comprehensive Empirical Study: The authors release a detailed analysis of ~230 languages and provide specific robustness profiles for five major languages across nine tasks, offering a more nuanced view than standard leaderboards.

Results

• Task-Specific Robustness:
  • Bitext Mining and Retrieval demonstrate the highest robustness, with bitext mining achieving full robustness (stable across both DS and RS) in all five analyzed languages.
  • Instruction Reranking shows limited cross-lingual coverage, with robust evaluation possible only for English.
  • Classification and Pair Classification generally show RS-only robustness due to limited dataset diversity in many languages.
• Model Performance:
  • Large-scale LLM-based models (e.g., llama-embed-nemotron-8b, Qwen3-Embedding-8B) frequently emerge as robust top performers, particularly in classification and clustering tasks.
  • Specialized models dominate specific tasks: bge-m3 is the sole fully robust model for bitext mining across all languages, while bilingual-embedding-large shows strong robustness in retrieval.
  • Task-Agnostic Consistency: When analyzing consistency across all nine tasks within a language, only a small subset of models remains consistently strong. llama-embed-nemotron-8b is identified as the only model achieving task-agnostic robustness across all five languages.
• Drivers of Robustness: The study identifies two primary factors predicting higher robustness:
  1. Training Objective Alignment: Models trained with objectives tightly aligned with the evaluation task (e.g., contrastive objectives for retrieval) show less sensitivity to dataset composition.
  2. Training Data Breadth: Models trained on broad, multilingual, and multi-task synthetic corpora generalize more stably than those fine-tuned on narrow, task-specific data.

Significance and Claims
The paper claims that current benchmark leaderboards often obscure meaningful differences in language-specific robustness and cross-task generalization. By applying these robustness indicators, the authors demonstrate that many "top-performing" models are only robust under specific evaluation designs or for specific tasks. The study asserts that llama-embed-nemotron-8b stands out as a uniquely robust default choice for multilingual deployments due to its stability across tasks, languages, and evaluation schemes.

The authors position this work as a "benchmarking meta-study" rather than a model selection guide. They emphasize that their findings highlight the need for more diverse and robust benchmark designs, particularly for non-English languages where dataset availability often limits the ability to assess dataset-composition robustness. The results suggest that practitioners should not rely solely on average-based leaderboards but should consider the stability of rankings under different evaluation conditions when selecting models for critical applications.