💬 NLP

Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation

This paper demonstrates that the performance rankings of developer-agent frameworks are highly sensitive to the evaluation language and judge backbone, revealing that no single model dominates across all languages and that localizing judge instructions is critical for accurate multilingual assessment.

Original authors: Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of expert architects (the AI coding agents) to build a house. You want to know if they did a good job. Usually, you'd send in a single, strict inspector (the AI Judge) who only speaks English to check their work.

This paper asks a simple but revolutionary question: What if the inspector speaks a different language than the blueprints?

The researchers discovered that the language the inspector speaks doesn't just change how they talk; it completely changes who they think is the best architect.

Here is the breakdown of their findings using everyday analogies:

1. The "Accent" Problem

Imagine you have two famous chefs, Chef GPT and Chef Gemini.

When a judge who speaks English tastes their food, Chef GPT gets a 5-star rating, and Chef Gemini gets a 3-star.
But, when a judge who speaks Arabic or Hindi tastes the exact same food, the results flip! Chef Gemini suddenly gets 5 stars, and Chef GPT drops to a 3.

The Lesson: The "best" AI isn't a universal truth. It depends entirely on the language the person grading them is speaking. If you only test in English, you might be promoting the wrong AI for the rest of the world.

2. The "Translator vs. The Brain" Experiment

The researchers ran a fascinating experiment to see why this happens. They took the same task and the same AI judge, but they changed the instructions:

Scenario A (Full Localization): The judge gets the task description in Hindi, and the instructions on how to grade it are also in Hindi.
Scenario B (Partial Localization): The task description is in Hindi, but the instructions on how to grade it are still in English.

The Result: In Scenario B, the judge's performance crashed. It was like asking a brilliant chef to cook a complex Indian dish but giving them the recipe in English while they only speak Hindi. They got confused and failed.

Takeaway: You can't just translate the homework; you have to translate the teacher's instructions too. If you don't, the AI gets lost in translation.

3. The "Weak vs. Strong" Student

The study looked at six different AI "judges" (backbones).

The Top Students (GPT-4o and Gemini): These are smart enough to understand nuance. Because they are smart, they react differently to different languages. One might be great at English logic but struggle with Arabic nuances, while the other is the opposite. Their rankings flip-flop depending on the language.
The Struggling Students (DeepSeek and Qwen): These models were so confused by the coding tasks that they failed almost everything, no matter what language was used. Because they were failing so hard, the language didn't matter much to them—they were just "bad" in every language.
The Lesson: You only see these language quirks when the AI is actually smart enough to have an opinion.

4. The "Agreement" Problem

The researchers asked all six judges to grade the same 55 coding projects.

The Result: They barely agreed with each other. It was like asking six different art critics to grade a painting, and they all gave it completely different scores.
The Metaphor: If you ask a panel of judges to rate a movie, and one loves it while another hates it, you can't trust the "average" score. The paper shows that in AI coding, the "score" is often just a reflection of which specific AI model happened to be the judge that day.

Why Does This Matter?

Currently, most AI benchmarks are like a monolingual sports league. They only test in English.

The Risk: If you build an AI app for a company in India or the Middle East, and you only tested it with an English-speaking AI judge, you might pick the wrong tool. You might think "Model A" is the best because it won the English tournament, but "Model B" might actually be the champion in Hindi or Arabic.

The Bottom Line

The paper is a wake-up call: Language is not just a setting; it's a variable.

If you want to know if an AI is good at coding, you can't just ask it in English. You have to ask it in the language it will actually be used in, and you have to make sure the "teacher" grading it speaks that language fluently too. Otherwise, you're just playing a game of chance with your technology.

1. Problem Statement

Current benchmarks for evaluating AI coding agents (Agentic Code Benchmarks) predominantly rely on English-only evaluation prompts and assume that the "Judge" (the LLM evaluating the agent's output) performs consistently regardless of the language used. This paper identifies a critical, previously unmeasured confound: the interaction between the evaluation language and the Judge's backbone model.

The authors argue that treating evaluation language as a fixed default (English) introduces a measurable bias. Specifically, changing the language of the judge's prompt can invert the relative rankings of different backbone models. This means a model deemed "best" in English might be suboptimal in other languages, leading to misleading conclusions about model capabilities in multilingual deployment scenarios.

2. Methodology

The study employs a systematic, large-scale experimental design to isolate the effects of language and backbone models.

Benchmark & Tasks: The authors utilized the DevAI benchmark, consisting of 55 realistic AI development tasks (spanning data processing, modeling, training, etc.). These tasks are decomposed into 365 explicit, hierarchical requirements, allowing for granular, requirement-level evaluation rather than just binary "task solved" metrics.
Frameworks: Three distinct developer-agent frameworks were tested:
- MetaGPT (Multi-agent decomposition)
- GPT-Pilot (Tool-oriented workflows)
- OpenHands (Interactive agents)
Judge Backbones: Six diverse LLM backbones served as the "Agent-as-a-Judge":
- GPT-4o, GPT-5.4, Claude Sonnet 4.6, Gemini 3 Flash Preview, DeepSeek V3.2, and Qwen3.5-9B.
Languages: Five typologically diverse languages were used for prompt localization:
- English (Baseline)
- Arabic (Right-to-left, morphologically rich)
- Turkish (Agglutinative)
- Chinese (Non-segmented script)
- Hindi (Devanagari script)
Experimental Scale: The study executed 4,950 judge runs (55 tasks × 5 languages × 3 frameworks × 6 backbones).
Localization Strategy: The "Agent-as-a-Judge" prompt stack (including system instructions, requirement descriptions, and justification prompts) was fully localized into the target languages while preserving the underlying task structure and logic. Translations were verified by native speakers with graduate-level training.
Metrics:
- Requirement Satisfaction Rate: Percentage of individual requirements satisfied.
- Task Solve Rate (TSR): Percentage of tasks where 100% of requirements were met.
- Inter-Backbone Agreement: Measured using Fleiss' $\kappa$ to assess consistency between different judges.
Ablation Study: A controlled experiment on GPT-4o compared Full Localization (all prompts in target language) vs. Partial Localization (benchmark text in target language, but judge instructions remained in English) to determine if instruction language alone drives performance differences.

3. Key Contributions

Discovery of Language-Backbone Interaction: The paper provides the first systematic evidence that evaluation language and judge backbone interact non-linearly, causing backbone rankings to invert.
Multilingual Benchmark Study: A comprehensive evaluation across 55 tasks, 5 languages, 3 frameworks, and 6 backbones, totaling nearly 5,000 runs.
Localization Methodology: A practical framework for localizing the AAAJ prompt stack while maintaining structural integrity, demonstrating that instruction localization is critical for evaluation stability.
Open Data Release: Release of full requirement-level judgments, runtime statistics, and summary metrics to ensure reproducibility.

4. Key Results

A. Backbone Rankings Invert by Language

The most significant finding is that no single backbone dominates across all languages.

English: GPT-4o is the strongest (44.72% satisfaction).
Arabic: Gemini leads significantly (51.72% vs. GPT-4o's 26.25%, $p < 0.001$ ).
Hindi: Gemini leads (53.22% vs. GPT-4o's 44.02%).
Chinese: GPT-4o leads, but the gap with Gemini is not statistically significant.
Turkish: GPT-4o and Gemini perform similarly.

B. Capability Gates Multilingual Effects

The study found that "weaker" backbones (DeepSeek, Qwen) showed near-zero cross-language variation because their baseline performance was too low to detect language-specific nuances. Significant language-dependent effects were only observable in high-capability models (GPT-4o, Gemini).

C. Requirement-Type Sensitivity

Operational tasks (e.g., Data Loading, Training) remained relatively stable across languages.
Semantically dense tasks (e.g., Model Construction, Evaluation Metrics) showed high sensitivity to language. For instance, under GPT-4o, satisfaction for "Evaluation Metrics" dropped from 58.67% in English to 14.67% in Arabic.

D. Inter-Backbone Agreement is Low

Agreement between different judge backbones on individual requirement judgments was modest (Fleiss' $\kappa \leq 0.231$ ). This indicates that the choice of judge backbone substantially alters the verdict, and there is no "ground truth" consensus among current LLMs.

E. Ablation: Instruction Localization Matters

The ablation study revealed that translating only the benchmark content is insufficient.

Hindi: Switching from Full Localization to Partial Localization (English instructions) caused satisfaction to crash from 42.8% to 23.2%.
Arabic: Performance remained relatively stable between Full and Partial localization, suggesting that for languages with strong representation in training data (like Arabic), English instructions may still suffice, whereas lower-resource languages (like Hindi) require full localization for stable reasoning.

5. Significance and Implications

Re-evaluating Benchmarks: The paper challenges the standard practice of using English-only evaluation for multilingual agent benchmarks. It suggests that current rankings may be artifacts of the evaluation language rather than true model capabilities.
Evaluation Protocol: The authors recommend that evaluation language and judge backbone must be reported as first-class metadata. Non-English evaluations should be validated with at least two different judge backbones.
Process-Aware Evaluation: The study reinforces the value of "Agent-as-a-Judge" and requirement-level analysis over simple Task Solve Rates (TSR), as TSRs were consistently low (often <5%) even when partial progress was substantial.
Token Length Confound: The authors ruled out token length as the primary driver of these differences, as correlations between input token counts and satisfaction scores were inconsistent and non-significant.

In conclusion, this paper establishes that language is not a neutral parameter in agentic evaluation. It acts as a dynamic variable that interacts with model architecture to produce divergent outcomes, necessitating a shift toward multilingual, multi-backbone evaluation protocols for robust AI agent benchmarking.