Technical Summary: Unraveling the Token Dynamics of Large Language Models for Machine Translation

Problem Statement

While Large Language Models (LLMs) have demonstrated strong performance in machine translation (MT), existing research primarily focuses on benchmarking translation quality or improving it through prompting and retrieval. There is a significant gap in understanding why LLMs fail, particularly in low-resource and non-English-centric settings. Standard benchmarks often assess tasks with definitive correct answers, failing to capture the nuances of open-ended multilingual generation. This paper addresses the lack of insight into the root causes of translation failures in LLMs, specifically investigating the relationship between token-level dynamics, language resource availability, and translation performance.

Methodology

The authors conducted a large-scale empirical analysis involving 15 models (including instruction-tuned and reasoning variants from Qwen, DeepSeek, Llama, Gemma, and specialized translation models like NLLB and Tower-Plus) across 22 language pairs (LPs). These pairs were categorized into English-centric and non-English-centric groups, spanning high-, medium-, and low-resource languages.

Key Experimental Components:

1. Data: The study utilized 66,000 instances sampled from various parallel corpora (TED, SwissAdmin, WMT20, etc.), ensuring a mix of translation directions and resource levels.
2. Prompting: To ensure evaluation validity, the authors designed specific prompt templates (Prompt 2) to elicit translation-only outputs, filtering out the "noise" (explanatory text) common in LLM generations.
3. Metrics: Translation quality was evaluated using COMET-22 and SacreBLEU.
4. Token Activation Rate (TAR): A novel metric introduced to quantify language representation. TAR measures the proportion of a model's vocabulary activated when processing text in a specific language. It serves as a proxy for the language's representation in the model's pre-training data.
5. Typological Distance: The study utilized the URIEL database to measure distances between languages (genetic, geographic, syntactic, phonological, inventory, and featural) to assess their impact on performance.
6. Reasoning Token Analysis: The authors investigated whether reasoning LLMs generate more tokens as a compensatory mechanism for low TAR and whether this correlates with improved translation quality.

Key Contributions

1. Systematic Failure Analysis: The paper demonstrates that non-English-centric language pairs consistently yield lower COMET and BLEU scores compared to English-centric pairs, with significant performance degradation in low-resource settings.
2. Introduction of Token Activation Rate (TAR): The authors propose TAR as a quantifiable metric for language representation in model vocabularies. They validate TAR against known training data distributions (using Bloomz and EuroLLM), showing a strong correlation between TAR and actual language data proportions.
3. Correlation with Performance: The study establishes that lower TAR and greater typological distance are strongly associated with reduced translation quality (lower COMET and BLEU scores).
4. Reasoning Token Dynamics: The research reveals that reasoning LLMs tend to generate more tokens when translating into languages with low TAR, suggesting a compensatory mechanism. However, the impact of these additional tokens on translation quality is highly model-dependent; for some models (e.g., Qwen), more reasoning tokens correlate with quality improvements, while for others (e.g., DeepSeek), the correlation is weak or negative.

Results

• Performance Disparities: Non-English-centric pairs (e.g., Arabic-Hebrew, English-Tamil) showed substantially lower average COMET scores than English-centric pairs. Even among high-resource languages, pairs with greater typological distance (e.g., Chinese-French) performed worse than closer pairs (e.g., French-Italian).
• TAR and Translation Quality: A strong positive correlation exists between TAR and translation scores. Languages with low TAR (such as Khmer, Tamil, and Hebrew) consistently corresponded to the lowest COMET scores across nearly all models.
• Validation of TAR: In models with disclosed training data (Bloomz-7b1 and EuroLLM-22B), TAR showed robust Spearman and Kendall rank correlations with actual language data distributions, confirming its utility as a proxy for language coverage.
• Reasoning Mechanisms: Reasoning models (e.g., Qwen3-4B-Thinking) generated significantly more reasoning tokens for target languages with low TAR. In Qwen models, this increase in token count correlated positively with improvements in COMET and BLEU scores (explaining ~62.8% of variability in one case). Conversely, DeepSeek models showed negative or weak correlations, indicating that simply generating more tokens does not universally guarantee better translation quality.
• Model Specificity: Translation-specific models (Tower-Plus-72B, Google Translate) and large multilingual LLMs (aya-expanse-32b, DeepSeek-V3.2-Exp) generally outperformed smaller or less multilingual models.

Significance and Claims

The paper claims that token-level dynamics are critical to understanding the multilingual performance of LLMs. The introduction of TAR provides a concrete, measurable way to diagnose why models fail on specific languages, linking vocabulary activation directly to translation success.

The authors modestly frame their findings as identifying correlations rather than establishing causal mechanisms. They emphasize that while increased reasoning token generation acts as a compensatory mechanism for low TAR, its effectiveness is contingent on model-specific factors such as training data, alignment strategies, and output noise. The study underscores the need to jointly consider token dynamics and model architecture when evaluating reasoning LLMs for machine translation, moving beyond simple benchmarking to a deeper understanding of failure modes in low-resource and non-English-centric contexts.