💬 NLP

An Empirical Study of Many-Shot In-Context Learning for Machine Translation of Low-Resource Languages

This paper empirically demonstrates that many-shot in-context learning significantly improves machine translation for low-resource languages, with BM25-based example retrieval offering substantial data efficiency gains by matching the performance of much larger non-retrieved example sets.

Original authors: Yinhan Lu, Gaganpreet Jhajj, Chen Zhang, Anietie Andy, David Ifeoluwa Adelani

Published 2026-04-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yinhan Lu, Gaganpreet Jhajj, Chen Zhang, Anietie Andy, David Ifeoluwa Adelani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but inexperienced chef (a Large Language Model) how to cook a specific, rare dish from a remote village that the chef has never seen before. This village speaks a language the chef doesn't know, and there are no cookbooks written in that language.

This paper is about how to teach this chef using examples instead of a full textbook. The researchers wanted to see if giving the chef more and more examples (like showing them 1,000 different recipes) would help them cook better, and how to do this without burning a fortune on electricity.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Empty Kitchen"

Most AI models are trained on the "internet," which is full of English, French, and Chinese, but very poor in languages like Anaang, Efik, or Tamazight. It's like the chef has a massive library of French cookbooks but zero books on how to make Nigerian or Quechua dishes. When you ask the chef to translate these languages, they usually guess poorly.

2. The Solution: "Many-Shot" Learning

Instead of just showing the chef one or two examples (which is called "few-shot"), the researchers tried showing them hundreds or even thousands of examples right before asking for the translation.

The Analogy: Imagine you are trying to learn a new dialect. If someone whispers one sentence to you, you might get it wrong. But if you sit in a room with 1,000 people speaking that dialect, you start to pick up the rhythm, the slang, and the grammar naturally.
The Result: The paper found that more examples = better cooking. As they increased the number of examples from 1 to 1,000, the AI's translation quality got significantly better, sometimes doubling in accuracy.

3. The Catch: The "Expensive Buffet"

There is a problem with showing 1,000 examples: it's expensive and slow.

The Analogy: Imagine the chef has to read 1,000 recipe cards before cooking a single meal. It takes a long time, and the paper costs a lot of money. For small communities, this "buffet" is too pricey to afford.

4. The Magic Trick: The "Smart Librarian" (BM25)

The researchers asked: Do we really need to show the chef 1,000 random recipes? Or can we just show them the 50 best ones?

They used a simple search tool called BM25 (think of it as a Smart Librarian). Instead of handing the chef a random pile of 1,000 cards, the librarian looks at the specific dish you want to cook and finds the 50 most similar recipes from the pile.

The Finding: This was a game-changer.
- 50 Smartly Chosen Examples worked just as well as 250 Random Examples.
- 250 Smartly Chosen Examples worked just as well as 1,000 Random Examples.
Why it matters: You get the same high-quality translation but use 4x to 10x less computing power. It's like getting a gourmet meal without having to read the entire library.

5. The "Foreign Cookbook" Test (Domain Mismatch)

The researchers also tested if the type of example mattered. They used examples from Wikipedia (everyday topics) versus examples from the Bible (religious texts).

The Analogy: If you want to learn how to order food at a restaurant, reading a cookbook is good. Reading a religious text might help with some words, but it won't teach you how to order a burger.
The Result: Generally, Wikipedia examples were better because they matched the "everyday" nature of the test. However, for some languages, the Bible examples were still surprisingly helpful. It's like saying, "If you don't have a restaurant menu, a religious text is better than nothing!"

6. The "Order of the Cards" (Curriculum Learning)

They wondered if the order of the examples mattered. Should you show the chef the easiest recipes first, then the hard ones? Or the hard ones first?

The Result: It didn't really matter. Whether the examples were sorted by length or shuffled randomly, the chef performed the same. The content of the examples mattered much more than the order.

The Big Takeaway

This paper proves that for languages that AI usually ignores, giving the AI a massive amount of context helps it learn.

However, you don't need to feed it everything. If you use a simple search tool to pick the most relevant examples, you can get the same great results with a fraction of the cost. This makes it possible for low-resource communities to use advanced AI without needing a supercomputer budget.

In short: To teach AI a rare language, don't just throw a thousand random books at it. Hire a smart librarian to pick the best 50, and the AI will cook up a perfect translation.

1. Problem Statement

Large Language Models (LLMs) have demonstrated strong capabilities in natural language processing, particularly through In-Context Learning (ICL), where models adapt to tasks using few-shot examples. However, performance for Low-Resource Languages (LRLs) remains suboptimal due to limited representation during pre-training.

While recent "many-shot" prompting (using hundreds or thousands of examples) has shown promise for scaling performance, it faces two critical challenges for LRL communities:

Inference Cost: Processing thousands of examples per query is computationally expensive and often prohibitive for low-resource communities.
Example Selection: It is unclear how to efficiently select examples from large pools to maximize performance without incurring the full cost of random sampling at scale.
Domain and Ordering Effects: The impact of out-of-domain data (e.g., religious texts vs. Wikipedia) and the ordering of examples (e.g., short-to-long) on many-shot scaling is not well understood for extremely low-resource languages.

2. Methodology

The authors conducted a systematic empirical study focusing on 10 truly low-resource languages recently added to the FLORES+ benchmark. These include Nigerian languages (Anaang, Efik, Ibibio, Oro), Sudanese Arabic, Emakhuwa, Ladin, Mauritian Creole, Tamazight, and Quechua.

Experimental Setup:

Models: Evaluated on four LLMs: Gemini 2.5 Flash, GPT-4.1, Llama 3.3 70B, and Gemma 3 27B.
Task: Machine Translation (MT) in two directions: English $\to$ LRL (eng $\to$ X) and LRL $\to$ English (X $\to$ eng).
Shot Scaling: Tested $k \in \{0, 1, 5, 10, 25, 50, 100, 250, 500, 1000\}$ examples per prompt.
Metrics: Primary evaluation using chrF++ (character n-gram F-score), with spBLEU as a secondary metric.

Three Core Experiments:

Scaling Analysis: Measured performance gains as the number of randomly sampled examples increased from 0 to 1,000.
Example Selection: Compared random sampling against BM25 retrieval (based on English source text similarity) to determine if fewer, high-quality examples could match the performance of many random examples.
Domain & Ordering:
- Domain Mismatch: Compared in-domain examples (Wikipedia) against out-of-domain examples (Bible translations) for languages where Bible data was available.
- Ordering: Tested if sorting examples by length (Short-to-Long vs. Long-to-Short) influenced translation quality, inspired by curriculum learning.

3. Key Contributions

Systematic Many-Shot Study for LRLs: This is one of the first comprehensive studies applying many-shot ICL (up to 1,000 shots) specifically to extremely low-resource languages, moving beyond the typical few-shot (e.g., <20) constraints.
Data Efficiency via Retrieval: Demonstrated that BM25-based retrieval drastically improves data efficiency. The study quantifies that 50 BM25-retrieved examples perform roughly equivalent to 250 randomly selected examples, and 250 BM25 examples match 1,000 random examples.
Domain Insights: Provided empirical evidence that while in-domain data is generally superior, out-of-domain data (like religious texts) can still yield significant gains for specific LRLs when in-domain data is scarce.
Ordering Insignificance: Found that the order of examples (length-based sorting) has negligible impact on many-shot performance, suggesting that semantic relevance (via retrieval) is more critical than structural ordering.

4. Key Results

A. Scaling Effects

Performance Gains: Performance generally improves log-linearly as the number of examples increases.
Directionality: Gains are significantly larger for eng $\to$ X (translation into LRLs) than for X $\to$ eng. For eng $\to$ X, some languages saw performance more than double (e.g., Tamazight saw a +35.6 chrF++ gain from 0-shot to 1,000-shot).
Model Differences: Proprietary models (Gemini 2.5 Flash, GPT-4.1) consistently outperformed open-weight models (Llama 3.3, Gemma 3) in many-shot settings. Open-weight models often failed or degraded at $k=500$ or $k=1000$ due to context window limitations and weaker instruction following with massive context.

B. Example Selection (BM25 vs. Random)

Superiority of Retrieval: BM25 retrieval on the English source side consistently outperformed random selection, especially at lower shot counts ( $k=5$ to $k=50$ ).
Cost Reduction: The "equivalence" findings (50 BM25 $\approx$ 250 Random; 250 BM25 $\approx$ 1,000 Random) suggest that communities can achieve high performance with a fraction of the inference cost by using simple retrieval methods.

C. Domain and Ordering

Domain: In-domain (Wikipedia) examples generally scaled better than out-of-domain (Bible) examples. However, for languages like Efik and Ibibio, Bible data provided competitive or superior results, indicating that out-of-domain data is a viable fallback when in-domain data is unavailable.
Ordering: Sorting examples by length (Short-to-Long or Long-to-Short) showed no significant impact on translation quality compared to random or default ordering.

5. Significance and Implications

Practical Scalability: The findings provide a roadmap for low-resource language communities to leverage LLMs without prohibitive costs. By using simple retrieval (BM25) to select a smaller subset of examples, communities can achieve performance comparable to massive random-shot prompts.
Resource Allocation: The study highlights that for LRLs, example quality (relevance) is more important than sheer quantity once a certain threshold is reached.
Limitations & Future Work: The authors note that the study relied on English-centric translation pairs and automatic metrics (chrF++), which may be less reliable for LRLs. They also highlight the high cost of API credits ($30,000+) required for the study, emphasizing the barrier to reproducibility for low-resource researchers.

In conclusion, the paper validates that many-shot ICL is highly effective for low-resource machine translation, provided that retrieval-based example selection is used to optimize the trade-off between performance and inference cost.