💬 NLP

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

This paper introduces "Recovered in Translation," an automated framework that leverages test-time compute scaling strategies like Universal Self-Improvement and a novel T-RANK method to generate high-quality, semantically preserved translations of benchmarks into eight Eastern and Southern European languages, thereby overcoming the limitations of existing resources and enabling more reliable multilingual LLM evaluation.

Original authors: Hanna Yukhymenko, Anton Alexandrov, Martin Vechev

Published 2026-02-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hanna Yukhymenko, Anton Alexandrov, Martin Vechev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a giant, global library of tests to see how smart different AI computers are. Right now, most of these tests are written in English. To test AI in other languages, someone has to translate these tests.

The Problem: The "Broken Telephone" Game
Currently, translating these tests is like playing a game of "Broken Telephone" with a broken phone.

The Old Way: Existing translations often treat the question and the answer like two separate pieces of paper. They translate the question, then translate the answer, and just tape them back together.
The Result: The connection breaks. In languages like Ukrainian or Greek, where words change their shape depending on their role in a sentence (grammar cases), this separation causes the answer to accidentally give away the correct choice. It's like translating a riddle where the answer is hidden in the grammar of the question itself. The AI doesn't solve the puzzle; it just cheats because the translation gave it a hint.

The Solution: "Recovered in Translation"
The authors of this paper built a new, automated factory called R-Translation to fix this mess. Instead of a human translator or a simple robot, they use advanced AI to act as a team of expert editors.

Here is how their factory works, using a creative analogy:

1. The "T-RANK" Method: The Olympic Judges

Imagine you have five different translators trying to translate a tricky sentence.

Old Method (Best-of-N): You ask a judge to give each translator a score from 1 to 10. The problem? The judge might get tired or biased, giving the first translator a high score just because they spoke first.
The New Method (T-RANK): Instead of scoring, the AI acts like an Olympic judge panel. It puts the five translations in a ring and makes them fight it out.
- Round 1: Translator A fights Translator B.
- Round 2: Translator B fights Translator C.
- The Twist: The AI rotates who goes first so no one gets a "home-field advantage."
- The Final Step: Once the winner is picked, the AI looks at all the translations again, finds the best parts of each one, and stitches them together into a perfect "Frankenstein" translation that has the best grammar and meaning from everyone.

2. The "USI" Method: The Self-Improving Chef

Think of this as a chef who makes five different versions of a soup.

Instead of just picking the best one, the chef tastes all five, realizes, "Hmm, Soup #1 has the right salt, but Soup #3 has the perfect spice," and then mixes them into one Super-Soup that is better than any single attempt. This is called Universal Self-Improvement.

3. Why This Matters

The researchers tested this factory on 8 different languages (like Ukrainian, Turkish, and Greek) using famous AI tests like MMLU and Winogrande.

The Result: Their new translations were much more accurate.
The Impact: When they ran AI models on these new, high-quality tests, the scores changed. Some models looked smarter, others looked dumber. This proves that the old tests were lying to us because the translations were bad.

The Big Picture

Think of the old translations as a blurry photo of a landscape. You can kind of see the trees, but you can't tell if it's a forest or a park.
The authors' new framework is like sharpening the lens. Suddenly, you can see the details.

Why should you care?
If we want AI to be fair and helpful for everyone in the world, not just English speakers, we need to make sure the tests we use to grade them are perfect. If the test is broken, the grade is wrong. This paper provides the tools to fix the tests, ensuring that when an AI speaks Ukrainian or Greek, we are actually testing its intelligence, not its ability to spot a translation error.

In short: They built a smart, automated system that acts like a team of expert editors to translate AI tests perfectly, ensuring that AI models get a fair grade in languages other than English.

1. Problem Statement

The reliability of evaluating Multilingual Large Language Models (LLMs) is currently compromised by the poor quality of existing translated benchmarks. Key issues identified include:

Semantic Drift and Context Loss: Existing resources often translate questions and answers separately, leading to grammatical mismatches and loss of contextual nuance, particularly in languages with complex grammatical features (e.g., case systems, gender agreement).
Answer Leakage: In sentence completion tasks (e.g., Winogrande), translating options separately can inadvertently reveal the correct answer through grammatical gender markers, allowing models to succeed via language proficiency rather than reasoning.
Inadequate Tools: Many current benchmarks rely on classical Machine Translation (MT) tools (e.g., Google Translate, DeepL) or older LLMs (e.g., GPT-4) that lack the multilingual capabilities of frontier models.
Lack of Scalability: There is a gap in scalable, fully automated frameworks that can adapt translation strategies to specific language resource levels (mid-resource vs. low-resource) while preserving benchmark integrity.

2. Methodology: The "Recovered in Translation" (RIT) Framework

The authors propose a fully automated, configurable pipeline designed to translate datasets and benchmarks into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). The framework supports two modes: Dataset (straightforward text) and Benchmark (complex Question-Answer structures requiring context preservation).

The framework employs four distinct translation methods, leveraging test-time compute scaling strategies:

Self-Check (SC): A lightweight 0-shot translation followed by a self-correction step where the model evaluates and fixes its own output. Suitable for high-resource languages.
Best-of-N (BoN): Generates $N$ translation candidates at high temperature. An LLM-as-a-judge scores them (1–10), and the highest-scoring candidate is selected. Limitation: Prone to positional bias and numerical scoring inaccuracies.
Universal Self-Improvement (USI): Generates $N$ candidates. The model is instructed to synthesize the "best features" from all candidates into a single refined output, rather than just selecting one. This mimics a "Fusion-of-N" approach but tailored for translation.
Translation Ranking (T-RANK) [Novel]:
- Multi-Prompt Sampling: Generates diverse candidates using different prompts.
- Multi-Round Competitive Ranking: To mitigate positional bias (where LLMs favor early candidates), the framework rotates candidates across different input positions over $N$ rounds.
- Refinement: After ranking, the judge model reviews all candidates again to correct and refine the top-ranked translation. This method forces the model to identify subtle errors and inconsistencies across the entire set of candidates.

Implementation Details:

Models: Used GPT-4o-mini and Gemini-2.0-Flash for translation and evaluation.
Target Languages: Ukrainian, Romanian, Slovak, Lithuanian, Bulgarian, Turkish, Greek, Estonian.
Benchmarks Translated: MMLU, Hellaswag, ARC-Challenge, and Winogrande.

3. Key Contributions

Automated Framework: A novel, fully configurable pipeline that balances cost, time, and translation quality, supporting both open-weight and closed-source models.
T-RANK Method: A new ranking strategy that uses multi-round positional rotation to eliminate bias and improve error detection compared to standard Best-of-N or USI methods.
Comprehensive Analysis: A deep dive into the root causes of translation failures in existing benchmarks (e.g., MuBench, Global-MMLU), specifically highlighting how separate translation of Q&A pairs breaks grammatical coherence.
Resource Release: The authors released the framework code and high-quality translations of major benchmarks into 8 under-served languages.

4. Results and Evaluation

The authors evaluated their methods using reference-based metrics (COMET), reference-free Quality Estimation (QE), and LLM-as-a-judge comparisons.

Translation Quality (COMET Scores):
- On WMT24++ and FLORES benchmarks, USI and T-RANK consistently outperformed baselines and Best-of-N.
- T-RANK showed particular strength in handling complex grammatical structures in benchmarks, while USI was slightly more efficient for shorter dataset texts.
Benchmark Integrity (LLM-as-a-Judge):
- When comparing their translations against Global-MMLU (a major existing resource), the RIT translations (using T-RANK/USI) won significantly more comparisons across Ukrainian, Romanian, and Lithuanian.
Downstream Model Performance:
- Evaluating mid-sized models (Gemma 3, Qwen 3, Llama 3.1) on the new translations yielded higher accuracy scores compared to existing benchmarks.
- Winogrande showed the most significant improvement (+3.42% average gain), confirming that preserving the question-answer context prevents answer leakage and yields more reliable reasoning metrics.
- Language-wise gains: Greek (+3.89%), Ukrainian (+2.7%), and Turkish (+2.65%) showed the highest improvements.

5. Significance and Conclusion

Reliability: The study proves that translation quality directly impacts the validity of LLM evaluations. Poor translations lead to misleading performance metrics, often favoring models that exploit grammatical leaks rather than reasoning.
Scalability: The framework demonstrates that high-quality, multilingual benchmark creation can be automated without massive human curation, provided the right test-time scaling strategies (like T-RANK) are used.
Future Direction: The work highlights that frontier LLMs, when guided by sophisticated prompting and ranking strategies, outperform traditional MT tools. It calls for a shift away from static, human-translated benchmarks toward dynamic, automated pipelines that respect linguistic nuances.

Limitations Noted:

Reliance on closed-source models for the core pipeline (though the method is applicable to open-weight models).
Uniform application of methods without per-input difficulty estimation (though the framework allows user configuration).
Focus remains on Eastern/Southern European languages; generalizability to other low-resource language families requires further study.