💬 NLP

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

This paper investigates the reliability of multilingual LLM evaluations by demonstrating that translation errors in machine-translated benchmarks significantly contribute to accuracy drops and that automatic error detection methods show non-trivial agreement with human annotations.

Original authors: Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber, Jens Lehmann

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber, Jens Lehmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to grade a class of students who speak different languages. You have a famous test written in English (the "Gold Standard"). To see how well your students speak Spanish, French, or German, you take that English test, run it through a translation machine, and give the translated versions to your students.

The big question this paper asks is: What happens if the translation machine makes mistakes?

The authors, a team of researchers, discovered that these translation errors aren't just tiny typos; they are like potholes in the road that can cause even the smartest students to crash, even if they knew the answer perfectly in English.

Here is a breakdown of their findings using simple analogies:

1. The "Robot Grader" vs. The Human Expert

The researchers wanted to know: Can we trust other AI models to spot these translation potholes, or do we need a human expert?

The Experiment: They took a set of translated test questions and asked four different "Robot Graders" (AI models like GPT-5.2, Mistral, etc.) to highlight exactly where the translation went wrong. They compared the robots' work to a "Gold Standard" created by professional human linguists.
The Result: The robots were okay, but not perfect. One robot, GPT-5.2, was the best at finding the potholes, agreeing with the human experts about half the time. The other robots were much less accurate.
The Catch: The robots often got confused about where the error started and ended. It's like two people trying to draw a circle around a stain on a shirt; they might agree there is a stain, but one draws a tiny circle around the center, while the other draws a huge circle including the whole sleeve.

2. The "Source of the Problem" (Who is to blame?)

Sometimes, the English test question itself is weird or confusing. Sometimes, the translation makes it worse. The researchers wanted to know: Is the student failing because the English question was bad, or because the translation messed it up?

The Analogy: Imagine a math problem that says "If a car travels 50 miles in 2 hours..."
- Source Issue: The English question is broken and says "If a car travels 50 miles in 2 apples..." (The original is nonsense).
- Translation Issue: The English is fine, but the translator changes "miles" to "kilometers" and "hours" to "minutes," making the math impossible to solve.
The Finding: The researchers found that translation errors are the real troublemakers. Even when the English question was perfect and solvable, the translated version caused the AI students to get the answer wrong about 6 to 11 percentage points more often just because of the translation mistakes.

3. The "Score Drop" vs. The "Ranking"

This is the most surprising part. The researchers asked: If we magically fixed all the translation errors, would the leaderboard of AI models change?

The Analogy: Imagine a race where every runner starts 10 meters behind the starting line because of a translation error.
- The Result: If you move everyone forward 10 meters (fix the translation), everyone gets faster. The runner who was in 1st place is still in 1st place. The runner in 10th place is still in 10th place. The order of the winners doesn't change.
- The Problem: However, the times are all wrong. You think the winner ran a 10-second race, but they actually ran a 9-second race. You are underestimating everyone's true ability.
The Conclusion: Translation errors act like a uniform weight attached to every AI model. They drag everyone's score down equally. So, while the "Leaderboard" (who is #1) stays the same, the actual scores are biased and too low. We are telling the world, "This AI is only 80% smart," when it might actually be 85% smart, just because the test was translated poorly.

Summary of the Takeaway

The paper concludes that using machine-translated benchmarks to test AI is risky.

Robots aren't perfect at spotting translation errors yet.
Translation errors cause real, measurable drops in performance (about 6–11 points lower accuracy).
The rankings of AI models are stable (the winner is still the winner), but the scores are unfairly low.

It's like judging a cooking competition where the ingredients were translated from a different language. The best chef might still win, but the judges might think the food tastes worse than it actually does because the recipe was slightly garbled in translation.

Technical Summary: Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

Problem Statement

Machine-translated benchmarks are the standard for assessing the multilingual capabilities of Large Language Models (LLMs). However, the quality of these translations is frequently overlooked, introducing "translation artifacts" that undermine the reliability and comparability of evaluation results. While prior work has established that translation errors can bias cross-lingual benchmarks (e.g., XNLI, MMLU), existing studies often rely on manual inspection of small samples or coarse heuristics (e.g., sentence length ratios) that lack the precision to identify specific error types and locations. Furthermore, it remains unclear how well automated span-level error detection tools (such as LLMs acting as judges or quality estimation models) align with expert human annotations on naturally occurring benchmark translations, and to what extent target-side translation errors specifically drive accuracy drops compared to source-side anomalies.

Methodology

The authors address these gaps through a three-pronged approach involving human annotation, automated span localization, and statistical performance modeling.

1. Human and LLM Span Annotation (EU20 Subset)

The study utilizes a subset of the EU20 benchmark suite, comprising DeepL translations of MMLU, ARC, HellaSwag, GSM8K, and TruthfulQA into 20 EU languages.

Human Reference: Professional linguists manually annotated 225 segments across nine target languages (Germanic, Romance, Uralic, Slavic, Baltic) using an MQM-inspired, span-based protocol. Annotators identified error spans, assigned severity (Major/Minor), and distinguished between Accuracy (faithfulness) and Fluency/Style errors.
Automated Annotation: Four instruction-tuned LLMs (GPT-5.2, GPT-4o-mini, LLaMA-4, Mistral-Large) and a span-aware Quality Estimation baseline (xCOMET-XXL) were prompted to generate MQM-style error spans on the same data.
Metrics: Agreement was measured using greedy 1:1 matching with position-based overlap coefficients (OC) and string-based 3-gram Dice similarity (SIM). Threshold-free character-overlap metrics (Char-F1) and Krippendorff's unitizing alpha were also employed to assess robustness.

2. Meta-Evaluation (Span-ACESRef)

To complement the naturally occurring EU20 data, the authors constructed Span-ACESRef, a cleaned projection of the SPAN-ACES dataset. This resource isolates single, targeted error phenomena by mapping edits between "good" and "incorrect" translations onto a human reference. This dataset (1,407 items) allows for a controlled evaluation of span localization precision, distinguishing between "classic" (exact match) and "tolerant" (boundary-flexible) matching metrics.

3. Performance Impact Analysis

The authors linked span-level error annotations to downstream LLM performance using logistic regression.

Data: A paired subset of 651 item-language pairs (4,993 observations) from MMLU, ARC, and GSM8K.
Variables: Binary indicators for Target-side errors (T) and Source-side issues (S).
Controls: The models controlled for whether the evaluation model solved the English original ( $y_{EN}$ ), as well as fixed effects for language, dataset, and the evaluation model itself.
Analysis: Average Marginal Effects (AMEs) were computed to estimate the probability-point drop in accuracy associated with translation errors. A counterfactual analysis ( $T=0$ ) was performed to determine if removing translation errors would alter relative system rankings.

Key Contributions

Human-LLM Span Agreement: The paper releases a professional span-level MQM reference for an EU20 subset and evaluates automated annotators against it. GPT-5.2 demonstrates the highest agreement with human annotations (mean OC Span-F1 of 0.55), outperforming other LLMs and the xCOMET-XXL baseline.
Span-ACESRef Validation: The authors introduce Span-ACESRef as a controlled benchmark for span localization. Results indicate that while classic matching scores are low, tolerant matching scores are substantially higher, suggesting that many disagreements are boundary near-misses rather than complete detection failures.
Quantified Performance Impact: Using logistic regression with English controls, the study isolates the impact of translation errors. Target-side errors are consistently associated with 6–8 percentage point (pp) drops in accuracy in the full model and 6–11 pp drops when restricted to items solvable in English.

Results

Annotation Quality: GPT-5.2 is the strongest automated annotator, though absolute agreement with humans remains non-trivial (F1 ~0.55). Other models (Mistral-Large, GPT-4o-mini, LLaMA-4) show significantly lower agreement. xCOMET-XXL performs better than smaller LLMs but worse than GPT-5.2.
Source vs. Target Errors: Source-side anomalies (S) show a weaker and less stable negative association with accuracy compared to target-side errors (T). The effect of translation errors persists even after controlling for source issues and English correctness.
Impact Magnitude: A translation containing at least one annotated error is associated with a significant accuracy loss. For example, using GPT-5.2 annotations, the estimated overall loss is approximately 4.3 pp in the full model and 5.6 pp in the English-solvable subset.
Ranking Stability: Counterfactual analysis reveals that while translation errors significantly lower absolute scores, they behave as a relatively uniform penalty across systems. Consequently, relative system rankings (leaderboard orderings) remain highly stable (Spearman's $\rho \approx 1.0$ ) even when translation errors are hypothetically removed.

Significance and Claims

The paper claims that translated leaderboards can remain internally stable in their relative ordering while simultaneously understating absolute multilingual performance. The presence of translation errors introduces a systematic downward bias in measured accuracy that is not merely an artifact of source-side issues.

The authors emphasize that while current evaluation practices may preserve the ranking of models, the absolute performance estimates are biased. This suggests that the community cannot rely solely on machine-translated benchmarks for precise performance quantification without accounting for translation quality. The work advocates for the integration of span-level error analysis into benchmark release protocols to better understand and mitigate these biases.

The authors remain modest regarding their claims, noting that their findings are correlational and specific to the selected benchmarks, MT systems, and evaluated LLMs. They acknowledge limitations such as the scarcity of high-quality span-annotated references for low-resource languages and the inherent subjectivity in defining span boundaries.