GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation

1. Problem Statement

Current agent benchmarks (e.g., GAIA) are predominantly English-centric. When these benchmarks are adapted for non-English languages, the standard workflow involves Machine Translation (MT) followed by minimal post-editing. The authors argue that this "minimal workflow" is insufficient for agentic tasks (tasks requiring multi-step reasoning and tool use) because it introduces three critical failure modes that compromise benchmark validity:

• Functional Misalignment: MT often fails to preserve task constraints. For example, translating an IOC country code (e.g., "CUB") as a common noun ("lion cub") or leaking the answer into the prompt. This causes agents to fail not due to reasoning errors, but due to broken task logic.
• Cultural Misalignment: Literal translations often retain source-region assumptions (e.g., US geography, imperial units, or specific recycling laws) that are unnatural or irrelevant to the target population. This forces agents to either solve a "translated" version of a US-centric task or translate the prompt back to English to solve it, bypassing true multilingual reasoning.
• Difficulty Calibration: Translation can inadvertently alter task difficulty. For instance, if relevant information is scarce in the target language, the task becomes harder; conversely, if a translation simplifies a complex riddle, it becomes easier.

2. Methodology

The authors propose a refined adaptation workflow that moves beyond simple translation to ensure functional alignment, cultural alignment, and difficulty calibration. This workflow was applied to create the GAIA-v2-LILT dataset.

The GAIA-v2-LILT Dataset

• Scope: An extension of the GAIA benchmark covering five non-English languages: Arabic, German, Hindi, Korean, and Portuguese (Brazil).
• Size: 165 query-answer pairs per language (validation set).
• Source: Based on the machine-translated version of GAIA from the MAPS package.

The Hybrid Review Workflow

To address the limitations of standard MT, the authors implemented a three-stage review process designed to mitigate LLM self-preference bias and human fluency bias:

1. Deterministic Filtering (Automated):
   • Rule-based scripts check for objective defects before human/LLM review.
   • Checks include: Language identification, answer leakage detection (ensuring the gold answer isn't in the prompt), and placeholder recall (ensuring numbers/URLs are preserved).
2. LLM Judges (Decomposed Evaluation):
   • Instead of holistic scoring, separate LLMs evaluate specific axes: fluency, adequacy, query-answer compatibility, and cultural appropriateness.
   • This narrow, single-axis approach reduces bias and tracks human judgment more closely than global analysis.
3. Human Audit (Bilingual Linguists):
   • Bilingual linguists perform a final review, trained to prioritize functional integrity over mere fluency.
   • They verify that the task is solvable, culturally appropriate, and maintains the original difficulty level.
   • Meta-review: A second linguist and an ML researcher review the linguist's corrections to ensure consistency.

3. Key Contributions

• Identification of Agent-Specific Pitfalls: The paper categorizes specific translation failures unique to agent benchmarks, distinguishing between "under-translation" (localizing terms that should remain English) and "over-translation" (localizing terms that break constraints).
• A Rigorous Adaptation Workflow: A novel pipeline combining deterministic filters, decomposed LLM judging, and structured human auditing to ensure task solvability and cultural relevance.
• GAIA-v2-LILT Dataset: A high-quality, audited multilingual benchmark dataset that serves as a baseline for evaluating non-English agent performance.
• Empirical Evidence: Demonstration that standard MT-based benchmarks significantly underestimate agent capabilities due to measurement errors introduced by poor alignment.

4. Experimental Results

The authors evaluated three frontier models (GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.6) using the Open Deep Research agent framework on both the raw MT drafts and the human-audited GAIA-v2-LILT.

• Performance Gains: Correcting the alignment issues resulted in massive performance improvements. Agent success rates increased by 10.9% to 32.7% across the five languages compared to the raw MT versions.
• Closing the Gap: The audit reduced the performance gap between non-English languages and the English baseline. In the best cases, the gap was reduced to within 3.1% of English performance. However, significant gaps remained in some cases (e.g., Arabic showed a 30.3% gap), indicating that while alignment helps, other factors (like data scarcity) still exist.
• Analysis of Edits:
  • Functional Alignment had the highest "flip rate" (67.9%), meaning correcting functional errors (like answer leakage or format violations) most frequently changed a model's outcome from wrong to right.
  • Fluency had the lowest flip rate (40.6%), suggesting that stylistic perfection is less critical to agent success than functional correctness.
  • Cultural Alignment and Difficulty Calibration also significantly impacted correctness (approx. 50% flip rate).

5. Significance and Conclusion

The paper concludes that a substantial portion of the "multilingual performance gap" in AI agents is actually benchmark-induced measurement error rather than a true lack of model capability.

• Implication for Research: Future multilingual agent evaluations cannot rely on simple machine translation. They require task-level alignment to ensure that the benchmark tests reasoning, not the model's ability to decode broken instructions or navigate cultural mismatches.
• Resource Allocation: The authors suggest that localization efforts should prioritize functional and cultural fidelity over stylistic fluency, as these factors have the highest impact on evaluation validity.
• Limitations: The current workflow does not include automated agent testing (e.g., checking if search APIs work in the target region) or the localization of file attachments (images/PDFs), which are left for future work.

In summary, GAIA-v2-LILT demonstrates that rigorous, human-in-the-loop alignment is essential for creating valid, fair, and accurate benchmarks for autonomous agents in non-English languages.