The Multilingual Curse at the Retrieval Layer: Evidence from Amharic

Technical Summary: The Multilingual Curse at the Retrieval Layer (Evidence from Amharic)

Problem Statement
The paper challenges the prevailing assumption that strong zero-shot performance on aggregate multilingual benchmarks guarantees reliable cross-lingual retrieval for all languages. The authors argue that this assumption fails for underrepresented, morphologically rich languages. Specifically, they posit that when retrieval representations transfer poorly to such languages, downstream systems (e.g., Retrieval-Augmented Generation and cross-lingual QA) inherit a "retrieval-imposed quality ceiling." This degradation is often obscured by aggregate multilingual scores. The authors use Amharic—a Semitic language with over 58 million speakers, a non-Latin script (Ge'ez), and complex root-and-pattern morphology—as a diagnostic case to demonstrate that zero-shot multilingual retrieval is insufficient for equitable information access in the LLM era.

Methodology
To stress-test multilingual retrieval transfer, the authors constructed Dataset V2, an expanded Amharic passage retrieval benchmark containing 68,000 query–passage pairs derived from AMNEWS, XL-SUM, Amharic Wikipedia, and AmQA. Following weakly supervised practices, headlines and questions serve as queries, with source-aligned text segments as positive passages.

The study evaluates four retrieval paradigms under a shared protocol:

1. Dense bi-encoders
2. Late-interaction retrievers (ColBERT-style)
3. Learned sparse retrievers (SPLADE-style)
4. Cross-encoder re-rankers

The evaluation compares three categories of models:

• Zero-shot multilingual retrievers: Five state-of-the-art multilingual embedding models (e.g., multilingual-e5, snowflake-arctic-embed, gte-multilingual, embeddinggemma, harrier-oss) evaluated without language-specific fine-tuning.
• Amharic-fine-tuned multilingual retrievers: Two recent multilingual models (embeddinggemma-300m and harrier-oss-v1-270m) fine-tuned on the Amharic Dataset V2 training split.
• Monolingual Amharic retrievers: Models initialized from Amharic-only backbones (roberta-medium-amharic and roberta-base-amharic) and trained specifically for Amharic retrieval across all four paradigms. BM25 serves as a sparse lexical baseline.

Key Contributions

1. Diagnostic Case Study: The paper uses Amharic to empirically test whether strong zero-shot multilingual retrieval transfers to underrepresented, morphologically rich languages.
2. Expanded Benchmark: The release of an expanded Amharic retrieval benchmark (Dataset V2) with 68K pairs and a multi-source composition, moving beyond single-source news datasets.
3. Comprehensive Evaluation: A unified comparison of zero-shot multilingual, fine-tuned multilingual, and monolingual retrievers across dense, late-interaction, sparse, and cross-encoder paradigms.
4. Quantification of the Gap: The authors quantify a persistent performance gap that remains even after in-language fine-tuning of multilingual models.
5. Open Resources: Public release of the dataset, codebase, evaluation scripts, and trained model checkpoints to support reproducible research in low-resource language IR.

Results

• Zero-Shot Deficit: The strongest zero-shot multilingual retriever (snowflake-arctic-embed-l-v2.0, 568M parameters) significantly underperforms the strongest monolingual Amharic first-stage retriever (ColBERT-Base-Amharic, 110M parameters). The gap is 23% relative MRR@10 (0.653 vs. 0.803) and 19.1% relative NDCG@10. Notably, the zero-shot multilingual model has over five times the parameters of the monolingual model yet fails to match its performance.
• Lexical Competitiveness: BM25 (MRR@10: 0.612) outperforms four of the five zero-shot multilingual dense retrievers, indicating that dense multilingual representations do not uniformly improve over sparse lexical matching for Amharic.
• Impact of Fine-Tuning: Fine-tuning multilingual models on Amharic supervision yields substantial gains (32–60% relative MRR@10 improvements over their zero-shot counterparts). However, even the best fine-tuned multilingual model (harrier-oss-v1-270m + FT, 0.760 MRR@10) remains below the strongest monolingual Amharic retriever (0.803 MRR@10), leaving a residual gap of roughly 5.4% relative MRR@10.
• Architecture and Scaling: The gap persists across all monolingual paradigms (dense, sparse, late-interaction). Furthermore, parameter scale is not a sufficient substitute for language-specific modeling; the 270M–300M parameter fine-tuned multilingual models do not match the performance of the 110M parameter monolingual ColBERT model.
• Re-ranking: Two-stage re-ranking using a monolingual Amharic cross-encoder on top of a dense bi-encoder achieves the highest score in the evaluation (MRR@10: 0.830), demonstrating that joint query-passage scoring resolves distinctions missed by independent encoders.

Significance and Claims
The paper claims that zero-shot multilingual retrieval is not a sufficient proxy for equitable information access in the LLM era. The findings suggest that:

• Aggregate benchmarks are misleading: High aggregate scores can mask severe language-specific deficits, particularly for languages with divergent scripts and rich morphology.
• In-language adaptation is necessary but insufficient: While fine-tuning multilingual models on language-specific data significantly narrows the performance gap, it does not eliminate it. The starting representation, tokenizer, and pretraining distribution of multilingual models still limit their effectiveness compared to models built natively for the target language.
• Evaluation must be language-specific: For underrepresented languages, retrieval systems must be evaluated and adapted in-language rather than inferred from aggregate multilingual benchmarks. Relying on zero-shot retrievers for languages like Amharic may impose a measurable quality ceiling on downstream RAG and QA systems.
• Future Research Direction: The authors call for a shift in evaluation focus from whether a model "nominally supports" a language to whether it retrieves effectively for that language under realistic supervision and deployment constraints. They do not claim that every language requires a separate retrieval stack, but they argue that zero-shot performance should not be treated as evidence of sufficient language coverage.