BM25-Augmented Many-Shot Translation for Low-Resource North-Eastern Indian Languages

Technical Summary: BM25-Augmented Many-Shot Translation for Low-Resource North-Eastern Indian Languages

Problem Statement
This paper addresses the challenge of machine translation (MT) for eleven low-resource North-Eastern Indian languages participating in the WMT26 Low-Resource Indic Language Translation shared task. These languages span four distinct language families: Indo-Aryan (Assamese), Austroasiatic (Khasi), eight Tibeto-Burman languages (Mizo, Manipuri in Bengali and Meitei Mayek scripts, Bodo, Kokborok, Karbi, Nyishi, Tagin), and an English-based creole (Nagamese). Several of these, particularly Nyishi, Karbi, and Tagin, have almost no prior MT work. The task is divided into two categories based on data availability: Category 1 (moderate data) and Category 2 (very limited data). The authors aim to translate between English and these languages in both directions without relying on model fine-tuning, which has dominated previous editions of the task.

Methodology
The proposed system, submitted by the "gators" team from the University of Florida, adapts a retrieval-augmented many-shot translation pipeline previously used for Indigenous American languages (AmericasNLP 2026). The core architecture is inference-time retrieval-augmented generation (RAG) using a large language model (LLM) and a sparse retrieval index.

1. Data Preparation:

   • Retrieval Banks: The system constructs language-specific training banks ($R$) by merging official WMT26 data with publicly available corpora (e.g., Samanantar, Endangered Recipes, IndicGenBench FLORES) and prior WMT releases (WMT23).
   • Augmentation: Significant data augmentation was applied, particularly for Category 2 languages. For instance, the Bodo corpus was expanded 9.3× to over 114,000 pairs, while others received smaller augmentations.
   • Filtering: Data underwent SHA-256 deduplication, Unicode normalization (NFC/NFKC), and length-ratio filtering. Cross-lingual similarity filtering (LaBSE) was optionally applied to noisier external sources.
   • Domain Alignment: The authors explicitly excluded synthetic data from the Multi30k dataset (image captions) after finding that domain mismatch negatively impacted performance, despite the volume increase.

2. Retrieval Mechanism (BM25):

   • The system uses BM25 (Okapi variant) to retrieve the top-$r$ most similar parallel examples from the training bank for a given source sentence $q$.
   • Index Direction: A critical correction from the authors' previous work involved building the BM25 index on the source language side. For English-to-Indic ($En \to X$), the index is built on English; for Indic-to-English ($X \to En$), the index is built on the Indic language.
   • Deduplication: To prevent prompt flooding with near-identical examples from overlapping corpora, the system skips exact matches and deduplicates based on a 60-character normalized prefix.

3. In-Context Translation:

   • Model: Translations are generated by Gemini 2.5 Flash via the Google GenAI SDK. No model fine-tuning is performed.
   • Prompt Construction: The prompt consists of the input sentence $q$, $r$ retrieved training exemplars, and $d$ development exemplars ($D_d$) selected via word-overlap ranking with leave-one-out exclusion.
   • System Prompts: Each language has a custom system prompt encoding specific linguistic features (e.g., warning Khasi about VOS order, clarifying Bodo's Tibeto-Burman nature despite Devanagari script).
   • Post-processing: Includes stripping model prefixes, Unicode normalization, and validating output characters against expected Unicode blocks (crucial for Meitei Mayek to prevent fallback to Bengali).

4. Hyperparameter Optimization:

   • A grid search was conducted over retrieval count $r \in \{0, 20, 40, 80\}$ and development exemplar count $d \in \{0, 10, 20, 49\}$ for all 22 language-direction pairs.
   • Configurations were evaluated on held-out development sets (50 sentences per cell) using the ChrF metric.

Key Contributions

• Transfer of RAG to New Domains: The paper demonstrates that a retrieval-augmented, fine-tuning-free approach, previously successful in the AmericasNLP task, transfers effectively to the North-Eastern Indian language family, despite different linguistic structures and script systems.
• Critical Bug Fix: The authors identified and resolved a bug in their previous pipeline where the BM25 index was incorrectly built on the target language (Spanish) regardless of translation direction. Correcting this to build indices on the source language significantly improved $X \to En$ performance.
• Domain Sensitivity Analysis: Through ablation studies, the authors show that for BM25-based RAG, domain alignment is more critical than sheer data volume. Synthetic data from image captioning tasks (Multi30k) was detrimental when applied to news/government text domains.
• Comprehensive Grid Search: The study provides a detailed analysis of how retrieval ($r$) and development exemplars ($d$) interact across 22 distinct language pairs, revealing asymmetries in optimal configurations between $En \to X$ and $X \to En$ directions.

Results
The system was evaluated on the WMT26 test set using BLEU and ChrF++ metrics.

• Performance: The submission achieved the best BLEU score in 9 out of 19 translation tasks and the best ChrF++ score in 11 out of 19 tasks.
• Directional Asymmetry:
  • $En \to X$: Retrieval ($r$) was generally beneficial, with $r=80$ being optimal for ten of the eleven languages. This suggests RAG effectively aids in generating low-resource languages.
  • $X \to En$: Retrieval was often less critical or even detrimental. In seven out of eleven $X \to En$ tasks, the optimal $r$ was 0, suggesting that when generating a high-resource language (English) that the LLM already knows well, retrieval does not add value unless the retrieved examples are nearly identical to the query.
• Data Scarcity: For languages with extremely small corpora (e.g., Karbi with ~900 pairs), the performance gains from increasing $r$ plateaued, indicating that retrieval strength is limited by corpus size.
• Script Effects: The paper notes a significant performance gap between Manipuri in Bengali script and Meitei Mayek script, likely due to the LLM's pre-training exposure being higher for the former.

Significance and Claims
The authors claim that their approach isolates the capabilities of retrieval-augmented MT, proving that fine-tuning is not strictly necessary to compete with other submissions in low-resource settings. The system's success relies on:

1. High-quality, domain-aligned retrieval banks.
2. Correct index construction (source-language indexing).
3. Strategic use of in-context learning with curated development exemplars ($d$) which often proved more impactful than training retrieval ($r$) for small corpora.

The paper concludes that while the system depends on API access (Gemini) and faces limitations regarding safety filters and rate limits, it represents a robust, training-free alternative for low-resource translation. The authors modestly note that performance decay from development to test sets was observed, consistent with previous findings, but the system still outperformed the competition in the majority of metrics.