💬 NLP

No Optimal Language Set Exists for Multilingual Instruction Tuning: Insights from a Linguistically-Informed Study

This paper systematically evaluates linguistically-informed language selection strategies for multilingual instruction tuning and finds that no universal optimal language set exists, as performance is highly dependent on specific tasks and models, with adding more languages often triggering the curse of multilinguality.

Original authors: Gürkan Soykan, Gözde Gül Şahin

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gürkan Soykan, Gözde Gül Şahin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of artificial intelligence as a massive, hungry library. In recent years, scientists have been teaching these digital brains to follow instructions, not just in one language like English, but in dozens or even hundreds. This is called "multilingual instruction tuning." Think of it like training a new employee who needs to speak to customers from every country on Earth. The goal is to make the employee smart enough to understand a request in French, answer in Swahili, and explain a joke in Japanese, all without getting confused.

However, there's a catch. The library has limited shelf space (computing power) and a limited amount of time to train the employee. If you try to cram every single language into the training schedule, the employee might get overwhelmed and start forgetting how to speak any of them well. This is known as the "curse of multilinguality." Because of this, many researchers have wondered: Is there a "magic recipe" for picking the perfect group of languages? Maybe if we pick languages that are geographically spread out, or ones that sound very different from each other, we could create a super-smart employee who works perfectly for everyone, everywhere. It's a hopeful idea: find the perfect mix, and you get the best possible AI.

This paper, titled "No Optimal Language Set Exists for Multilingual Instruction Tuning," goes into the lab to test that exact hope. The researchers, Gürkan Soykan and Gözde Gül Şahin, set up a series of experiments to see if they could find that magic recipe. They took three different types of AI models (think of them as three different trainees with different learning styles) and tried to teach them using various "linguistically-informed" strategies. These strategies were like different ways of picking a team: one team was chosen based on where the languages are spoken on a map (Geography), another based on how the languages are built grammatically (Typology), and others based on how similar their meanings are or how they were learned by computers. They compared these smart picks against a simple "random" pick, like drawing names out of a hat.

The results were a bit of a plot twist. The authors found that there is no single "best" team of languages that wins every time. It's not like finding a universal key that opens every door. Instead, the best choice depended entirely on which specific AI model you were training and what specific task you asked it to do. For one model, picking languages based on geography worked great; for another, picking them based on grammar was better. Sometimes, the "smart" picks actually did worse than just picking languages randomly!

The study also discovered a hard limit. When they kept adding more languages to the training mix, the AI's performance didn't keep getting better forever. Once they hit a certain point—around 13 or 14 languages—the performance actually started to drop. It's like trying to eat a buffet: the first few plates are delicious, but if you keep piling more food on your plate, you eventually get too full to enjoy any of it. This confirmed the "curse of multilinguality": with a fixed amount of training time, adding too many languages hurts the AI's ability to learn them all well.

Furthermore, the researchers found that what works for one AI model might be terrible for another. A strategy that made one model smarter could make a different model dumber. They also noticed that sometimes, teaching the AI in just one language (like Vietnamese) was just as good as, or even better than, teaching it in a mix of many languages, depending on the model.

In the end, the paper suggests that the search for a single, perfect "optimal language set" is a dead end. There is no one-size-fits-all solution. The best approach seems to be a bit more cautious: picking a diverse group of about 10 to 15 languages is usually a safe bet, but you have to be ready to test different combinations for every new model and every new task. The authors warn that simply averaging scores across different tests can be misleading, hiding the fact that a strategy might be great for one job but terrible for another. So, while we can't find a magic recipe, we do know that being thoughtful about how we pick our languages—and knowing when to stop adding more—is the key to building better, more helpful multilingual AI.

Technical Summary: No Optimal Language Set Exists for Multilingual Instruction Tuning

Problem Statement

Multilingual instruction tuning (MIT) aims to improve the cross-lingual performance of large language models (LLMs) but faces three primary challenges: the curse of multilinguality (where per-language performance degrades as model capacity is split among more languages), data scarcity, and high computational costs. A prevailing hypothesis suggests that carefully selecting a linguistically diverse subset of languages should yield universally superior models that generalize better across tasks and languages than random selection or monolingual tuning. However, prior work has not systematically investigated whether linguistically-informed selection strategies yield a universally optimal MIT strategy under fixed computational budgets.

Methodology

The authors conduct a systematic evaluation of various language selection strategies against a random baseline across three model families and five multilingual benchmarks.

Dataset: The study utilizes Bactrian-X, a parallel multilingual instruction dataset containing 3.4 million instruction-response pairs across 52 languages.
Experimental Setup:
- Budget Control: Following Chen et al. (2023a), the total training budget is fixed at 67,000 instances per experiment. For 14-language subsets, this equates to approximately 4,786 instances per language.
- Selection Strategies: The authors select 14-language subsets using $k$ $k$ -means clustering ( $k=14$ $k = 14$ ) on different feature spaces:
  - GEO: Geographical vectors.
  - TYPO: Typological vectors.
  - SEM: Semantic typology embeddings.
  - LEARN: Learned language vectors.
  - FAM: One language per language family.
  - RND: Random selection (baseline).
  - ALL: All 52 languages (used to test the limits of the budget).
- Models: Three model families are instruction-tuned using LoRA:
  - mGPT-1.3B
  - mT5-XL-3.7B
  - BLOOM (1.7B, 3B, and 7B variants)
- Evaluation: Models are evaluated zero-shot on five benchmarks: XNLI, PAWS-X, XCOPA, XStoryCloze, and XWinograd.
- Statistical Rigor: Each configuration is run with three random seeds. Significance is assessed via paired t-tests, with effect sizes measured using Cohen's $d_z$ .

Key Results

1. No Universal Optimal Strategy

The central finding is that no single language selection strategy consistently outperforms others across all models and tasks.

Performance is highly task- and model-dependent. For instance, the GEO strategy leads for BLOOM-7B on XStoryCloze, while TYPO achieves the best average for BLOOM-7B overall. FAM performs best on average for mT5-xl, and GEO or TYPO lead for mGPT on NLU tasks.
Improvements over the random (RND) baseline are often not statistically significant. Significant gains were observed only in specific model-task combinations (e.g., GEO vs. RND for mGPT on XNLI), but these did not generalize.
In some configurations, linguistically-informed subsets performed significantly worse than the random baseline, indicating that informed selection can backfire.

2. The Curse of Multilinguality Re-emerges

Increasing the number of languages beyond a modest threshold triggers performance degradation.

For BLOOM-7B, performance peaks at 13–14 languages and consistently declines as the language count increases to 52 (the "ALL" setting).
The "ALL" setting never achieved the best average performance for either model tested, confirming that under a fixed computational budget, maximizing language count is detrimental.

3. Model-Dependent Generalization

Cross-lingual generalization to unseen languages (languages present in benchmarks but not in the training subset) varies significantly by model architecture.

While linguistically-informed subsets generally improved over the base model for mGPT, mT5-xl, and BLOOM-3B, results diverged for larger BLOOM models (7B), where only two of five subsets improved over the base.
This suggests a non-trivial interaction between pretraining corpus composition and subset selection; a strategy effective for one model's pretraining distribution may be harmful for another.

4. Model Size and Monolingual Competition

Scale Effects: Strategy rankings shift as model scale increases. For example, GEO excels at 3B and 7B scales, while FAM starts strong at 1.7B but diminishes. No strategy remains uniformly best as scale increases.
Monolingual vs. Multilingual: In a case study on Vietnamese, monolingual tuning (67k examples) ranked first for mT5 and second for mGPT, performing competitively with the best multilingual subsets. This challenges the assumption that multilingual subsets are always preferable.

5. The Role of Overlap

The study analyzes the overlap between pretraining corpora, training subsets, and evaluation languages.

BLOOM has a small overlap between its pretraining corpus and XNLI evaluation languages (8/15), and most informed subsets scored significantly below the random baseline on XNLI for BLOOM-7B.
mGPT covers 14/15 XNLI languages in pretraining, and informed subsets scored significantly higher there.
However, overlap alone does not dictate outcomes; GEO improved XStoryCloze significantly on both models despite sharing few evaluation languages, suggesting that benchmark language sets themselves barely intersect, making universal optimality structurally impossible.

Significance and Implications

The paper argues that the field must move away from seeking a "one-size-fits-all" optimal language set for multilingual instruction tuning.

Methodological Warning: Reporting single aggregate scores across tasks and models is deeply misleading. A strategy that appears superior in aggregate may underperform significantly on specific tasks (e.g., FAM had the best average for mT5-xl but the worst score on XNLI). The authors urge the community to report per-task and per-model breakdowns with confidence intervals and avoid over-generalizing from limited benchmarks.
Practical Takeaway: While no universal optimum exists, linguistically-informed selection (particularly GEO and TYPO) tends to be a safer default than random selection, especially at larger model scales. For practitioners, the key insight is to target 10–15 linguistically diverse languages rather than maximizing language count, as adding more languages beyond this threshold triggers the curse of multilinguality under fixed budgets.
Negative Findings as Guidance: The absence of a universal winner is attributed to three compounding factors: pretraining corpus mismatch, task-language alignment constraints, and seed sensitivity. These findings caution against simple universal prescriptions for MIT data curation and highlight the need for task- and model-specific evaluation with proper statistical reporting.

The authors conclude that while linguistically-informed selection generally outperforms random selection on average, the complexity of interactions between model architecture, pretraining data, and task requirements precludes the existence of a single optimal language set for all scenarios.