The Translation Tax Is Not a Scalar: A Counterfactual Audit of English-Source Cue Inheritance in Chinese Multilingual Benchmarks

Technical Summary: The Translation Tax Is Not a Scalar

Problem Statement
The evaluation of Large Language Models (LLMs) in multilingual settings frequently relies on translated benchmarks (e.g., MMMLU, Belebele, XCOPA). A prevailing assumption, termed the "Translation Tax," posits that translated benchmarks artificially inflate model scores because models exploit residual English-source cues preserved through translation, rather than demonstrating genuine language understanding. However, the literature lacks rigorous quantification of this effect. Existing concerns regarding translation artifacts (e.g., Artetxe et al., 2020) and representativeness gaps (e.g., Sitaram, 2025; Wu et al., 2025) suggest that translation-induced score shifts are complex, yet no single study has systematically decomposed these shifts into cue-driven inflation versus semantic degradation across multiple estimation strategies.

Methodology
This paper conducts a counterfactual audit of English-to-Chinese benchmarks using a triangulation approach with four distinct estimators to isolate the "Translation Tax" (TT). The study utilizes a stratified subset of 228 items from MMMLU and 100 items from Belebele, evaluated across nine models (Group A: English-centric frontier; Group B: Chinese-optimized; Group C: Open-source multilingual).

1. Estimator E1 (Back-Translation): Measures the score gap between original English items and items round-tripped through Chinese back to English. This serves as a proxy for structural cue residue but is acknowledged as a noisy upper bound due to back-translation noise and naturalness loss.
2. Estimator E2 (Bilingual Native Annotation): A calibration attempt where annotators score items on cue identifiability, cultural residue, and syntactic residue. In this study, this was executed as a 30-item single-annotator rubric calibration using an LLM, serving as a check on rubric coverage rather than a full human estimator.
3. Estimator E3 (INCLUDE Non-Translation Control): Compares performance on translated MMMLU Chinese against INCLUDE Chinese, a dataset sourced natively from regional exams. This provides an external validity comparison, though it lacks item-level matching.
4. Estimator E4 (Matched LLM-Naturalization Stress Test): A counterfactual where item content (answer, options, semantics) is held fixed while the Chinese surface form is rewritten by an LLM to be more "natural." This is the only estimator that varies surface naturalness while holding content fixed. A critical methodological step involved correcting a prompt-construction bug (duplicate choice labels) in the initial run, which significantly altered the results.

Key Results
The study finds that the Translation Tax is not a uniform scalar correction but a set of estimator- and item-dependent validity risks.

• E1 (Back-Translation): Point estimates are small (ranging from 0.000 to +0.047). While five of six model-benchmark cells showed positive estimates, the confidence intervals for most included zero. The only cell narrowly excluding zero (MMMLU/gpt-5.4-mini) was identified as parser-fragile. Item-clustered bootstrapping yielded a small positive average effect (+0.022, 95% CI [0.001, 0.044]), suggesting a minor translation tax exists but is highly sensitive to parser validity.
• E2 (Annotation): The calibration run found no correlation between cue scores and item-level performance gains. The gpt-4o model showed a negative correlation, and the data was dominated by tied scores, rendering this estimator inconclusive for cue exploitation at this scale.
• E3 (INCLUDE Control): English-centric models (Group A) showed near-zero gaps between translated MMMLU and native INCLUDE. Chinese-optimized models (Group B) showed mixed results: two models (deepseek-chat, qwen-max) performed significantly worse on translated MMMLU than on native INCLUDE, while a third (glm-4.5) showed a positive gap. This suggests model-family interactions may exist but are confounded by domain differences (academic vs. regional exam).
• E4 (Naturalization): After correcting the prompt bug, the study found no statistically significant model-family interaction. The earlier draft's claim of a strong interaction was an artifact of the bug combined with the parser fragility of glm-4.5. However, a dose-response effect persisted: items with high "translation residue" (where models previously failed on Chinese but succeeded on English) showed a performance gain when naturalized (+0.103), whereas low-residue items showed no gain (-0.015).

Significance and Claims
The paper argues that the "Translation Tax" cannot be treated as a single scalar value to be subtracted from benchmark scores. Instead, validity risks are heterogeneous:

1. Estimator Dependence: Different proxies yield different magnitudes and directions of effect.
2. Item Dependence: The tax is concentrated on specific items with high translation residue, not distributed uniformly across the benchmark.
3. Model Dependence: While aggregate model-family effects were largely absent or confounded by domain fit, parser fragility and prompt sensitivity play a critical role in observed scores.

The authors conclude that translated benchmarks introduce validity risks that are best understood as a set of conditions rather than a fixed penalty. They release a reporting checklist for future translated benchmark papers, emphasizing the need to disclose estimator scope, parser fragility, and subgroup contrasts. The study does not claim to have measured the "latent" Translation Tax directly but provides a framework for identifying when and where translation cues may artificially inflate scores.