💬 NLP

Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models

This paper demonstrates that target language typological features significantly influence the translation quality of state-of-the-art multilingual large language models like NLLB-200 and Tower+, even after controlling for data availability and script, while also suggesting that languages with specific typological properties could benefit from alternative decoding strategies.

Original authors: Vitalii Hirak, Jaap Jumelet, Arianna Bisazza

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vitalii Hirak, Jaap Jumelet, Arianna Bisazza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, multilingual translator robot. You might think that if you feed it enough data, it will speak every language perfectly. But in reality, some languages are still much harder for the robot to translate than others.

This paper asks a simple question: Is the difficulty of translating a language just about how much data the robot has seen, or is there something "in the DNA" of the language itself that makes it harder?

The authors decided to test this using two of the most advanced translator robots currently available (NLLB-200 and Tower+). They didn't just look at the amount of data; they looked at the typological features of the languages. Think of typological features as the "personality traits" of a language—things like how complex its grammar is, whether it puts the verb at the beginning or end of a sentence, and how many different forms a single word can take.

Here is a breakdown of their findings using some everyday analogies:

1. The "Data vs. DNA" Debate

For a long time, researchers thought the only reason a language was hard to translate was because the robot hadn't read enough books in that language (the "Data" argument).

The Paper's Finding: Even when they controlled for how much data the robot had, the "DNA" of the language still mattered.
The Analogy: Imagine two students taking a test. One student has a thick textbook (lots of data), and the other has a thin one. You'd expect the first student to win. But this paper found that even if both students had the same thick textbook, the student whose native language had a very different structure from the test language (e.g., different word order) still struggled more. The structure of the language itself creates an intrinsic difficulty.

2. The "Labyrinth" of Word Order

The researchers looked closely at how languages arrange their words. Some languages are like a straight highway (Subject-Verb-Object, like English), while others are like a winding maze where the verb might be at the very end, or the object might jump around.

The Finding: Languages with "winding" word orders (flexible word order) and complex grammar (lots of word endings) were consistently harder to translate.
The Analogy: Translating a straight highway is easy; you just drive forward. Translating a maze is harder because you have to constantly check your map to see where you are going. The robot gets confused more easily in these linguistic mazes.

3. The "Flashlight" Problem (Beam Search)

This is the most technical part, but it has a great real-world analogy. When the robot translates, it doesn't just pick the first word that comes to mind. It looks ahead at several possible paths to see which one leads to the best sentence. This is called "beam search."

The Standard Practice: Usually, the robot uses a "narrow flashlight" (a small beam size), looking at only 3 or 5 possible paths at a time. This is fast and usually works well for simple languages.
The Paper's Discovery: For complex languages (the "mazes"), the narrow flashlight isn't enough. The robot needs a wider flashlight (looking at more paths, like 7 or more) to find the best sentence.
The Analogy: If you are walking through a simple hallway, you only need to look at the path directly in front of you. But if you are in a giant, confusing cave with many tunnels, you need to shine your light in many directions at once to make sure you don't miss the right exit. The paper found that for certain languages, the standard "narrow flashlight" strategy is actually suboptimal, and widening the search significantly improves the quality of the translation.

4. The Two Robots

The authors tested two different types of robots:

NLLB-200: A specialized translator built specifically for this job.
Tower+: A general-purpose "Large Language Model" (like the ones powering chatbots) that was fine-tuned for translation.
The Result: Both robots showed the same pattern. The "DNA" of the target language (morphology and word order) predicted how well they would perform. However, the specialized robot (NLLB) was generally better at handling low-resource languages, while the general robot (Tower+) excelled when the language was one it was explicitly trained on.

The Bottom Line

The paper concludes that we can't just throw more data at a problem and expect it to be solved. The structure of the language itself dictates how hard it is to translate.

Furthermore, the "one-size-fits-all" approach to how the robot thinks (using the same narrow search strategy for every language) is flawed. Just as you wouldn't use a tiny flashlight to navigate a massive cave, we shouldn't use the same narrow search strategy for complex languages. The paper suggests that for languages with flexible word orders and complex grammar, we should give the robot a "wider flashlight" (a larger beam size) to help it find the best translation.

In short: Some languages are just harder to translate because of how they are built, and to translate them well, the robot needs to look at more possibilities than it usually does.

Problem Statement

Despite significant advancements in multilingual modeling, substantial quality disparities persist across languages. While uneven training resources are a known factor, prior research suggests that intrinsic typological properties may also determine the difficulty of modeling specific languages. However, existing evidence largely relies on small monolingual models or bilingual models trained from scratch on limited, typologically narrow datasets (e.g., Europarl or the Bible). Consequently, it remains an open question whether typological properties can explain the performance variations of state-of-the-art (SOTA) Large Language Models (LLMs) and multilingual Machine Translation (MT) systems, which are typically trained on massive, uncontrolled web-scale corpora. Furthermore, it is unclear if standard decoding strategies, such as left-to-right beam search with a fixed beam size, are optimal across languages with diverse typological characteristics.

Methodology

The authors investigate these questions by analyzing two large, pre-trained multilingual translation models:

NLLB-200 (3.3B): An encoder-decoder Transformer model representing traditional SOTA MT.
Tower+ (9B): A decoder-only LLM (based on Gemma2) post-trained for MT, representing the current trend of using general-purpose LLMs for translation.

Experimental Setup:

Dataset: The study utilizes the FLORES+ benchmark, comprising 997 English sentences (and sentences from six additional source languages: Arabic, Italian, Dutch, Turkish, Ukrainian, and Vietnamese) translated into 124 typologically diverse target languages.
Language Properties: To estimate the impact of typology, the authors compile a set of fine-grained, continuous features for 212 languages. These include:
- Resourcedness: Approximated using language size data from the GlotCC CommonCrawl corpus.
- Distances: Typological distances (genetic, geographic, syntactic, etc.) from the URIEL database and a binary feature for shared writing scripts.
- Morphological Complexity: Eight continuous measures (e.g., Word/Lemma Entropy, Mean Size of Paradigm, Inflectional Synthesis, Type/Token Ratios) derived from Universal Dependencies and FLORES+ data.
- Word Order: Gradient measures including word order entropy, Subject-Object order proportion, and the percentage of head-final phrases.
Decoding Strategy: The authors vary the beam size ( $k \in \{1, 3, 5, 7\}$ ) during inference to assess how widening the search space affects translation quality and model confidence.
Evaluation: Translation quality is measured using chrF++, a character-level n-gram metric suitable for morphologically rich languages. The study also analyzes generation probabilities to evaluate decoding optimality independent of surface-level metrics.

Key Contributions

Typological Predictors of Quality: The authors identify specific typological features that predict the translation quality of both NLLB-200 and Tower+ across 7 source and 124 target languages.
Decoding Strategy Interplay: The study analyzes the relationship between optimal beam size and the typological properties of the target language, challenging the assumption that a single decoding strategy is optimal for all languages.
Resource Release: The authors release a dataset of fine-grained, continuous typological properties for the 212 target languages in the FLORES+ benchmark to facilitate future research on language-specific decoding strategies.

Results

1. Impact of Typology on Translation Quality (NLLB-200):

Correlation Analysis: Target language typology significantly correlates with translation quality (chrF++ scores), even after controlling for data resourcedness and script similarity.
Key Predictors:
- Morphological Complexity: Languages with higher word and lemma entropy (packing more information into word structure) tend to have lower translation quality.
- Word Order: Languages with less predictable word orders (higher negative SO proportion) and a stronger preference for head-final phrases are more challenging to translate.
- Distances: Greater genetic, geographic, and syntactic distances between source and target languages inversely correlate with quality.
Regression Analysis: A linear regression model confirms that target language head-finality and morphological complexity (measured by Moving Average Type/Token Ratio, MATTR) are significant negative predictors of chrF++ scores, alongside source-target genetic distance and data resourcedness.

2. Impact of Beam Size and Decoding:

Variability in Gain: There is significant variability in the improvement of chrF++ scores when increasing beam size from $k=1$ to $k=7$ . Roughly half of the languages benefit from a wider beam.
Probability vs. Quality: While predicting chrF++ gain based on typology yielded low model fit, predicting probability gain (the increase in model confidence) showed strong correlations.
Typological Influence on Decoding: Languages with complex morphology and flexible word orders (head-finality) show significantly higher probability gains with wider beam searches. This suggests that standard narrow beam search may be a suboptimal approximation of the model's highest-probability sequence for these specific language types.

3. Decoder-Only Models (Tower+):

Performance Differences: Tower+ outperforms NLLB-200 on languages it was explicitly trained on but underperforms on others, highlighting the importance of model coverage.
Predictors: For Tower+, official language coverage and target language resourcedness are the dominant factors for quality. However, typological features (morphological complexity and head-finality) still play a significant, albeit smaller, role.
Decoding Gains: The relationship between typology and probability gain in Tower+ is less conclusive than in NLLB-200, though genetic distance and head-finality remain significant predictors.

Significance and Claims

The paper claims that target language typology drives translation quality in state-of-the-art multilingual models, even when controlling for trivial factors like data size and script. This finding extends previous work on modeling difficulty to the era of large pre-trained models.

The authors argue that the standard practice of using a uniform decoding strategy (e.g., fixed beam size) across all languages is likely suboptimal. Their results suggest that languages with specific typological properties (complex morphology, flexible word order) benefit significantly from wider beam searches, not necessarily in terms of surface-level metric gains for all cases, but in terms of accessing higher-probability sequences generated by the model.

Finally, the authors emphasize the need for language-specific decoding optimization. They suggest that future work should explore dynamically setting beam widths based on target language properties and evaluating alternative decoding strategies that better account for typological diversity. The paper concludes modestly, noting limitations such as the reliance on surface-level metrics (chrF++), the use of approximations for training data proportions, and the focus on only two models and one decoding paradigm.