Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

1. Problem Statement

The paper identifies a critical flaw in how frontier Large Language Models (LLMs) are evaluated for multilingual capabilities. Current state-of-the-art benchmarks (e.g., MT-AIME24, INCLUDE, Global-MMLU) are structured as mathematical reasoning tasks or multiple-choice knowledge questions translated into various languages.

The authors argue that these benchmarks fail to measure multilingual proficiency (the ability to generate fluent, semantically faithful text in a target language). Instead, they primarily measure:

• Mathematical Reasoning: Performance correlates strongly with English-only math benchmarks (e.g., AIME25), not language generation skills.
• Factual Recall: Performance correlates with English knowledge benchmarks (e.g., MMLU-Pro), indicating models are retrieving facts rather than understanding language.
• Misleading "Thinking" Variants: Models with "reasoning" or "thinking" chains often outperform standard "instruct" models on these benchmarks but perform worse or equally on real-world user preference platforms (like LMArena), suggesting the benchmarks reward internal reasoning in English rather than external multilingual generation.

2. Methodology

A. Critique of Existing Benchmarks

The authors conducted a correlation analysis between benchmark scores and LMArena Elo ratings (a proxy for real-world human preference).

• MT-AIME24 vs. LMArena: Spearman correlation $\rho = -0.09$ (near zero/negative).
• INCLUDE vs. LMArena: Spearman correlation $\rho = -0.26$ (negative).
• Error Analysis: Manual categorization of errors revealed that failures in MT-AIME24 were logical/arithmetic (not semantic), and failures in INCLUDE were factual (not linguistic). Furthermore, trace analysis showed models overwhelmingly default to English for internal reasoning regardless of the input language, confirming that the "multilingual" aspect is superficial.

B. Proposed Solution: Round-Trip Translation

The authors propose Round-Trip Translation as a reference-free evaluation method.

• Process: Source text (English) $\to$ Target Language $\to$ Back to English.
• Metric: The semantic gap between the original and the back-translated text is measured. If meaning is lost, the model failed to preserve semantics during the intermediate translation.
• Advantages:
  • No Human References: Eliminates the need for expensive human-translated gold standards.
  • No Stronger Judge Needed: The evaluation can be performed by the model itself or a standard LLM judge, avoiding the "oracle problem" where a stronger model is needed to judge a frontier model.
  • Semantic Sensitivity: Directly tests the preservation of meaning, tone, and nuance across languages.

C. The LiT Benchmark (Lost in Translation)

To operationalize this, the authors introduced LiT, a new benchmark with the following specifications:

• Scale: 1,600 samples (200 unique texts $\times$ 8 language sequences).
• Language Sequences: 8 chains spanning high, medium, and low-resource languages (e.g., Japanese $\to$ Korean $\to$ Chinese $\to$ Russian $\to$ English; Swahili $\to$ Arabic $\to$ Hausa $\to$ Amharic $\to$ English).
• Content Categories:
  1. Abstracts (20%): Humanities and STEM (testing terminological precision and argumentative nuance).
  2. Pragmatics (60%): Core semantics, discourse coherence, implicit content, pragmatic inference, and social interaction.
  3. Informal (20%): Slang, idioms, and register shifts.
• Evaluation Framework: Uses MQM (Multidimensional Quality Metrics) via an LLM-as-a-Judge (Grok 4.1 Fast). Errors are penalized by severity: Minor (-1), Major (-5), Critical (-25). The primary metric is the percentage of translations scoring MQM $\ge$ 80 (considered "fit-for-purpose").

3. Key Contributions

1. Demonstration of Benchmark Misalignment: Proved that current multilingual benchmarks correlate poorly with human preferences and primarily measure reasoning/knowledge retrieval rather than generation capability.
2. Introduction of LiT: A scalable, reference-free benchmark that covers a diverse range of linguistic phenomena and resource levels.
3. Robustness Validation: Showed that round-trip translation is robust against classical criticisms (e.g., error cancellation) when applied to modern instruction-following LLMs.
4. Empirical Discovery of Capability Gaps: Revealed a "catastrophic cliff" in performance between high-resource and low-resource languages that existing benchmarks fail to capture.

4. Key Results

A. Correlation with Human Preference

• LiT vs. LMArena: Round-trip translation scores on LiT correlate almost perfectly with LMArena Elo ratings ($\rho = 0.94$).
• This contrasts sharply with the near-zero correlation of MT-AIME24 and INCLUDE, validating LiT as a superior proxy for real-world utility.

B. Performance Across Resource Levels

• High-Resource Languages: Frontier models (e.g., Gemini-3-Flash, Qwen3.5) perform well on high-resource sequences (East Asia, Central Europe), with MQM $\ge$ 80 rates often exceeding 80-90%.
• Low-Resource Languages: Performance collapses drastically.
  • Gemini-3-Flash maintained a pass rate of 59.2% on low-resource chains (Africa, South America).
  • Most other models dropped to <10%, with several (e.g., Qwen3-235B, Qwen3-30B) achieving near 0% pass rates, indicating unusable output.
  • This reveals a "capability cliff" where models are effectively broken for low-resource languages, a fact masked by current benchmarks.

C. Impact of "Thinking" (Reasoning) Variants

• Informal Text: Reasoning variants often underperform compared to instruct variants on informal text. The authors suggest reasoning models over-complicate simple colloquial inputs, leading to register mismatches.
• Low-Resource Languages: Reasoning variants showed slight improvements over instruct variants in low-resource settings (e.g., DeepSeek-V3.2-Exp Thinking scored 7.2 vs. 1.0 for Instruct), suggesting reasoning can mitigate some lexical gaps but cannot compensate for missing language knowledge.

D. Robustness

• A separate robustness benchmark targeting idioms, polysemy, and syntactic ambiguity showed high rank-order correlation ($\rho > 0.77$) with the main LiT results, confirming that round-trip translation effectively captures genuine cross-lingual weaknesses rather than being derailed by classical failure modes.

5. Significance and Implications

• Paradigm Shift: The paper argues for a fundamental shift in multilingual evaluation from knowledge/reasoning proxies to generative fidelity.
• Real-World Impact: Current benchmarks give a false sense of progress, particularly for billions of users speaking low-resource languages. LiT exposes that many frontier models are effectively unusable for these populations.
• Scalability: The round-trip method allows for the continuous, automated evaluation of multilingual models without the prohibitive cost of human annotation, enabling the community to track genuine progress in language preservation.
• Future Directions: The authors call for benchmarks that disentangle single-language performance and move toward native, culturally grounded generation tasks that do not rely on English pivots.

In conclusion, LiT provides a rigorous, scalable, and human-aligned method to evaluate multilingual LLMs, revealing that while models are improving at reasoning, their ability to faithfully generate meaning in diverse, low-resource languages remains critically underdeveloped.