Beyond Many-Shot Translation: Scaling In-Context Demonstrations For Low-Resource Machine Translation

Technical Summary: Beyond Many-Shot Translation: Scaling In-Context Demonstrations For Low-Resource Machine Translation

Problem Statement
Machine Translation (MT) for low-resource languages is fundamentally constrained by the scarcity of high-quality parallel data. While Large Language Models (LLMs) have improved MT performance generally, adapting them to lesser-represented languages remains difficult. Traditional approaches like supervised fine-tuning require large amounts of supervision that are often unavailable. Although In-Context Learning (ICL) offers a parameter-free alternative by conditioning models on demonstration examples at inference time, the scaling behavior of ICL in low-resource settings is not fully understood. Specifically, it is unclear whether performance continues to improve as the number of in-context examples grows toward the limits of modern long-context models (up to 1M tokens), or if diminishing returns and performance degradation occur. Furthermore, the impact of different data sources (monolingual vs. parallel) on this scaling behavior remains an open question.

Methodology
The study investigates the scaling of many-shot ICL for low-resource MT by systematically varying the number of in-context examples and the type of training corpus.

• Models: The authors evaluated two decoder-only Transformer models with ultra-long context support (up to 1M tokens): Qwen2.5-7B-Instruct-1M and Nemotron-8B-UltraLong-1M-Instruct.
• Target Languages: The experiments focused on Javanese and Sundanese, two low-resource Austronesian languages.
• Corpus Types: Three distinct types of in-context demonstrations were tested:
  1. Unsupervised Monolingual Data: Crawled text from the target languages (Javanese/Sundanese) without translation pairs.
  2. Instruction-Style Data: Task-oriented examples (e.g., from the Alpaca dataset) translated into the target languages, formatted as instruction-input-output triples.
  3. Parallel Data: Synthetic parallel corpora created by translating target language sentences into English and Indonesian. Indonesian was included as a reference language due to its typological proximity to the target languages.
• Experimental Setup: The context budget was scaled from 27 tokens up to approximately 1 million tokens (2^20). The study utilized the FLORES+ dataset for evaluation. Performance was measured using COMET-22 (neural-based metric) and ChrF++ (character n-gram overlap). The authors compared "first-k" sampling (taking the first $k$ examples) against random sampling to test for order sensitivity.
• Synthetic Data Construction: To address the lack of parallel data, the authors constructed synthetic parallel corpora by extracting sentences from the Aya collection dataset and translating them into English and Indonesian using OpenAI's GPT-5 model.

Key Contributions

1. Scaling Characterization up to 1M Tokens: The paper provides the first empirical characterization of many-shot ICL for low-resource MT at the scale of 1 million tokens. It establishes that effective context sizes are often significantly smaller than the maximum supported window.
2. Data-Type Sensitivity Analysis: The study demonstrates that the type of in-context corpus critically influences both the magnitude of performance gains and the scaling trajectory. Notably, it finds that instruction-style monolingual data can be competitive with parallel data in certain configurations.
3. Resource Release: The authors constructed and released synthetic parallel corpora for English-Indonesian-Javanese and English-Indonesian-Sundanese triplets to facilitate future research.

Results

• Non-Linear Scaling and Saturation: Performance does not scale linearly with context size. Gains from additional context saturate quickly, typically around $2^{14}$ to $2^{16}$ tokens (16K–64K tokens). Beyond this point, adding more examples yields diminishing returns.
• Performance Collapse: As the context window approaches its maximum limit (near 1M tokens), performance often degrades, sometimes falling below zero-shot baselines. This "collapse" is attributed to factors such as dispersed attention mechanisms and the "lost-in-the-middle" phenomenon. The Qwen model exhibited more pronounced degradation and output length instability near the limit compared to Nemotron.
• Corpus Type Effects:
  • Parallel Data: Generally provided the highest performance gains, particularly for translation from English to the target languages.
  • Instruction Data: Surprisingly, instruction-style monolingual data achieved performance comparable to parallel data in several scenarios, suggesting that task-following signals and formatting are highly effective.
  • Unsupervised Monolingual Data: Consistently yielded the lowest scores.
  • Reference Language: The choice of reference language (English vs. Indonesian) showed direction-specific effects. Indonesian-parallel examples were more beneficial for target-to-English translation, while English-parallel examples benefited English-to-target translation.
• Sampling Robustness: Experiments comparing "first-k" and random sampling showed no significant performance differences, suggesting that the observed scaling trends are robust to the specific selection of examples, though positional effects remain a potential factor.

Significance and Claims
The paper claims that simply maximizing the context window size does not yield proportional quality gains in low-resource MT. Instead, there is an "effective context limit" well below the model's maximum capacity, beyond which performance may degrade. The study highlights that the type of data and model architecture are critical determinants of scaling behavior.

The authors emphasize that for low-resource languages where high-quality parallel data is scarce, instruction-style monolingual data offers a viable and competitive alternative to parallel supervision. This finding suggests that the structure and task definition of in-context examples may be as important as the alignment provided by parallel pairs. Finally, the results caution against blindly scaling context windows to the maximum limit without considering the specific corpus type and model architecture, as doing so can be counter-productive. The work underscores the need for careful sampling strategies rather than brute-force context expansion.