Technical Summary: Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models

Problem Statement

Multilingual text embedding models are critical for cross-lingual knowledge transfer in NLP, yet their evaluation remains highly uneven across high-, mid-, and low-resource languages. Existing benchmarks, such as MTEB and MMTEB, aggregate performance using simple averaging of heterogeneous metrics. This approach implicitly assumes dataset comparability while ignoring correlations between datasets and imbalances in task and language coverage.

The core problem addressed is that apparent stability in benchmark rankings may be an artifact of dataset scarcity rather than genuine model robustness. In low-resource settings, a model may consistently rank first simply because only a single dataset exists, or because multiple available datasets are highly correlated (redundant), providing no independent evidence of robustness. Current evaluation frameworks fail to distinguish between genuine model stability and "apparent stability" arising from sparse or redundant benchmark resources. This is particularly acute for Slavic languages, which, despite being spoken by over 300 million people, suffer from underrepresentation in NLP resources and exhibit linguistic complexities (e.g., rich morphology, mixed scripts) that challenge cross-lingual generalization.

Methodology

The authors propose a two-dimensional framework for analyzing multilingual embedding benchmarks under conditions of dataset scarcity and imbalance. The framework operates across two evaluation scopes and analyzes three complementary aspects:

1. Evaluation Scopes

• Task-Specific Analysis: Evaluates ranking stability and top-$k$ transfer consistency within a fixed language and task.
• Cross-Task Analysis: Evaluates whether models generalize consistently across different task families within a single language.

2. Three Analytical Aspects

• Ranking Stability: Assesses whether benchmark rankings remain stable under:
  • Aggregation Stability: Different ranking/aggregation methods (e.g., WSM, TOPSIS, VIKOR, PROMETHEE II with various weighting strategies).
  • Composition Stability: Different dataset compositions generated by decorrelating highly similar datasets (using Pearson correlation thresholds).
• Top-$k$ Transfer Consistency: A quantitative extension of prior work that measures how reliably individual models remain among the top performers. Unlike binary membership, this metric assigns rank-weighted credit, giving higher weight to models consistently appearing near the top of the ranking.
• Evidence Strength: A novel component introduced to quantify the reliability of the underlying evaluation evidence.

3. Evidence Strength Score (ESS)

The paper introduces a qualitative and quantitative measure to characterize the reliability of conclusions for each language-task pair:

• Qualitative Evidence Levels: Pairs are categorized into five levels ($E_0$ to $E_{RS+DS}$) based on dataset availability ($n_{t,l}$) and the number of decorrelated dataset clusters ($q_{t,l}$). These levels range from "No evidence" ($E_0$) to "Full stability assessability" ($E_{RS+DS}$), distinguishing between single-dataset evidence, redundant multi-dataset evidence, and genuinely diverse evidence.
• Quantitative Score (ESS): A score from 0 to 1 combining four factors:
  1. Normalized dataset availability.
  2. Effective dataset diversity (accounting for redundancy).
  3. Ability to evaluate aggregation stability.
  4. Ability to evaluate composition stability.
     $$ESS(t, l) = \frac{1}{4}(A(t, l) + Div(t, l) + RS(t, l) + DS(t, l))$$

The framework applies these metrics to the Slavic-language subset of the MTEB benchmark, utilizing 15 ranking schemes derived from multi-criteria decision-making methods and weighting strategies.

Key Results

Benchmark Sparsity and Redundancy

The analysis reveals severe benchmark sparsity across Slavic languages:

• Task Coverage: Russian achieves 100% task coverage, while languages like Ukrainian, Bosnian, and Belarusian cover only 25–37.5% of tasks.
• Evidence Levels: Many language-task pairs rely on a single dataset ($E_1$) or highly correlated datasets ($E_{SC}$). Tasks like Semantic Textual Similarity (STS), Reranking, and Pair Classification are severely underrepresented, often falling into the "no evidence" ($E_0$) or "single-dataset" ($E_1$) categories.
• Bitext Mining Exception: Bitext mining is the only task with sufficient coverage and diversity to support full stability analysis (both aggregation and composition stability) for a large subset of languages.

Ranking Stability vs. Evidence Strength

• High Apparent Stability: Where stability can be assessed (e.g., in Bitext Mining), rankings are highly stable across aggregation methods (mean Kendall's $W \approx 0.986$) and dataset compositions.
• The Scarcity Trap: High stability scores in low-ESS settings (e.g., single-dataset scenarios) do not indicate genuine robustness; they merely reflect the lack of variability in the evaluation setup. The authors emphasize that stability conclusions must be interpreted jointly with ESS values.

Model Performance

• Task-Specific Specialists: Different models dominate specific tasks. For example, Qwen3-Embedding variants dominate classification and pair classification; LaBSE-ru-turbo and bilingual-embedding-large lead in retrieval; and KaLM-Embedding-Gemma3 leads in multilabel classification.
• Cross-Task Generalists: A small group of models demonstrates stable cross-task transfer consistency across Slavic languages:
  • llama-embed-nemotron-8b: Emerges as the strongest overall cross-task model, leading in 55.6% of analyzed languages.
  • multilingual-e5-large-instruct: Consistently performs well, leading in 33.3% of languages.
  • Qwen3-Embedding variants: Perform strongly, particularly in Polish and Russian.
• Clustering Dominance: llama-embed-nemotron-8b shows near-perfect consistency ($TC=1.00$) in clustering across all languages, a task with relatively better coverage.

Cross-Task Transfer Consistency

Unlike task-specific results, which vary significantly by language and dataset, cross-task rankings are highly stable. The analysis suggests that task specialization is the primary source of variability in evaluation, rather than language variation. Generalist models (like llama-embed-nemotron-8b) maintain high rankings across diverse task families, whereas task specialists (like bge-m3 in bitext mining or Octen-Embedding in STS) rarely appear as top cross-task performers.

Significance and Claims

The paper claims that benchmark scarcity is a major obstacle to trustworthy multilingual evaluation. Its primary contributions are:

1. Methodological Framework: It provides a structured approach to distinguish between genuine model robustness and apparent stability caused by sparse or redundant data.
2. Evidence Strength Score (ESS): It introduces a metric to explicitly qualify the confidence one can place in benchmark conclusions, arguing that rankings without high ESS should be interpreted with caution.
3. Empirical Insight: It demonstrates that for Slavic languages, current benchmarks are insufficient to draw robust conclusions for most language-task pairs. The "winners" in many low-resource scenarios are often artifacts of limited evaluation coverage.
4. Identification of Robust Models: Despite data limitations, the study identifies a small cluster of large multilingual models (llama-embed-nemotron-8b, multilingual-e5-large-instruct, Qwen3-Embedding) that consistently generalize across tasks and languages, suggesting they are the most reliable choices for general-purpose multilingual embedding tasks.

The authors conclude that future evaluations must move beyond simple aggregate scores to include evidence strength quantification, and that the NLP community needs richer, more balanced, and less redundant benchmark resources to support reliable evaluation in low-resource multilingual settings.