💬 NLP

Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge

This paper introduces Judge-LS, a meta-evaluation protocol demonstrating that while LLM-as-a-Judge systems exhibit significant preference instability and accuracy drops when evaluating Chinese or language-switched content compared to English, they do not systematically favor English over translation-equivalent Chinese responses.

Original authors: Shaojie Yin

Published 2026-06-15✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shaojie Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, automated referee (an AI) whose job is to watch two people answer a question and decide who did a better job. This is how many modern AI systems are tested today: one AI acts as the judge for other AIs.

This paper asks a simple but tricky question: Does this referee care about the language the answers are spoken in, or does it only care about the quality of the answers?

Think of it like a cooking competition. If two chefs make the exact same delicious soup, but one chef presents it in a fancy French restaurant and the other in a casual diner, does the judge give the French presentation a higher score just because it sounds more "fancy"? Or does the judge taste the soup and realize they are identical?

The Experiment: The "Language Translator" Test

The researchers took a standard set of 419 questions and answers (a benchmark called LLMBar) and ran them through four different AI judges. They did this in three different "costumes":

English: The original version.
Chinese: The exact same questions and answers, but translated perfectly into Chinese.
Code-Switching: A mix of English and Chinese (like saying "Please send the email to the manager," mixing languages naturally).

They also did a special "tie-breaker" test. They took a perfect English answer and its perfect Chinese translation and asked the judge to compare them. Since the content is identical, the judge should say, "It's a tie!"

What They Found

The results were a bit surprising and showed that the referees aren't as neutral as we'd hope.

The "English Accent" Bias: Every single judge performed best when the answers were in English. When the answers were in Chinese or a mix of languages, the judges made more mistakes. It's as if the referee gets a little confused or less sharp when the contestants speak a different dialect, even if the logic is the same.
The "Flip-Flop" Problem: In about 11% to 14% of the cases, the judge changed its mind just because the language changed.
- Analogy: Imagine Judge A says, "Chef 1 wins!" when the menu is in English. But if you hand them the same menu translated into Chinese, Judge A suddenly says, "Actually, Chef 2 wins!" even though the food hasn't changed at all. This is called a "preference flip."
It's Not Just "English is Better": You might think the judges just love English and hate everything else. But the "tie-breaker" test showed something more complex. When the judges did pick a winner between an English answer and its Chinese translation, they actually picked Chinese more often than English!
- The Takeaway: The problem isn't that the judges blindly prefer English. The problem is that they are unstable. They are easily swayed by how the information is presented, whether it's the language, the order the answers appear in, or a mix of both.

Why This Matters

If you are building an AI system to help people in China, or to handle mixed-language conversations, you can't just trust an English-trained judge to be fair.

The "Brittle" Judge: A good judge should be like a rock-solid scale. If you put the same weight on it, it should give the same reading, whether you describe the weight in kilograms or pounds. These AI judges are more like a wobbly scale; the reading changes depending on how you hold it.
The Cost of Confusion: Because the judges flip their decisions so often (about 1 in 10 times), they could accidentally rank a worse AI as the winner just because the test was written in a different language.

The Solution Proposed

The authors suggest a new, lightweight "health check" called Judge-LS. Before you trust an AI judge to rank models for a multilingual world, you should run this simple test:

Translate the test into the target language.
Run the judge again.
Check if the judge changes its mind.

If the judge changes its mind too often, it's not ready for the job. It's like hiring a referee who gets confused by a different accent; you need a referee who judges the game, not the language.

In short: The paper proves that AI judges are currently sensitive to language changes. They aren't just "English lovers"; they are "unstable observers" that need to be tested for consistency before we trust them to decide which AI is the best.

Technical Summary: Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge

1. Problem Statement

The widespread adoption of Large Language Models (LLMs) as automatic judges for open-ended instruction-following evaluation introduces a critical reliability question: Does a judge evaluate the semantic quality of an answer, or does its preference react to the language in which the comparison is presented?

While LLM-as-a-Judge offers scalability and semantic awareness superior to reference-based metrics, it inherits the biases of generative models. Prior work has documented position bias, verbosity bias, and prompt sensitivity. This paper isolates a specific reliability failure mode: language-switching invariance. Specifically, it investigates whether a judge preserves its preference when a pairwise comparison is presented in English, Chinese, or a Chinese-English language-switched variant, assuming the underlying quality relation (the "gold label") remains unchanged.

2. Methodology: The Judge-LS Protocol

The authors propose Judge-LS, a lightweight, training-free meta-evaluation protocol designed to test language-switching invariance using only API calls.

2.1 Base Data and Transformation

Dataset: The study utilizes the full LLMBar benchmark, consisting of 419 pairwise items (one natural subset and four adversarial subsets) with objective gold labels indicating which response better follows an instruction.
Language Variants: For every item, three versions are generated:
1. EN: The original English version.
2. ZH: A natural Simplified Chinese translation of the instruction and both candidate outputs.
3. LS: A natural Chinese-English language-switched variant, preserving English task terms, named entities, and technical terms where appropriate.
Transformation Model: gpt-4.1-mini is used for translation. The prompt explicitly instructs the model to preserve all factual, mathematical, formatting, and instruction-following errors to ensure the gold label validity is not compromised by "silent repairs."
Audit: An automatic audit verifies field completeness and flags high-risk variants (e.g., severe length shrinkage, numeric-token mismatches). 19 items (4.5%) were flagged, and a sensitivity analysis excludes them.

2.2 Experimental Design

Judges: Four API-accessible models were evaluated:
- GPT-4.1 Mini
- Claude Haiku 4.5
- Gemini 2.5 Flash
- DeepSeek V4 Flash
Judgment Construction: Each item is evaluated under all three language conditions. For each condition, the judge receives both the original and swapped answer orders (A vs. B). Additionally, translation-equivalent tie probes are created by comparing the English gold answer against its Chinese translation to test for language preference in "Tie" scenarios.
Total Volume: The experiment generated 13,408 successful unique pairwise judgments.

2.3 Metrics

The study reports several diagnostic metrics beyond simple accuracy:

Strict and Tie-Half Accuracy: Measures agreement with gold labels, with the latter giving partial credit to "Tie" predictions.
Language-Invariance Flip Rate: The percentage of judgments where the preference changes solely due to the language transformation (EN vs. ZH/LS).
Gold-Correctness Flip Rate: The percentage of cases where a correct judgment becomes incorrect (or vice versa) after translation.
Position Inconsistency: The rate at which the winner changes when the answer order is swapped.
Tie-Probe Language Preference: The distribution of wins between English and Chinese in translation-equivalent tie probes.
Statistical Rigor: The paper employs Wilson 95% confidence intervals and exact two-sided McNemar/binomial tests for paired comparisons.

3. Key Results

3.1 Accuracy and Stability

English Superiority: All four judges achieved their highest accuracy in English. For example, DeepSeek's tie-half accuracy dropped from 90.5% (EN) to 87.8% (ZH) and 88.9% (LS).
Significant Flip Rates: Language transformations induced preference flips in 10.7% to 14.4% of cases relative to English.
- GPT-4.1 Mini and Claude Haiku were most sensitive to Chinese translations (14.4% and 14.2% flip rates, respectively).
- DeepSeek and Gemini showed lower but still substantial flip rates (~11.1%).
Correctness Impact: The gold-correctness flip rates closely mirrored preference flip rates, indicating that these are not harmless swaps between a correct answer and a tie, but actual changes in the judge's alignment with the objective truth. Paired significance tests rejected symmetry for all model-language comparisons ( $p \le 0.011$ ).

3.2 Language Preference vs. Instability

No Systematic English Bias in Ties: Contrary to the hypothesis that judges inherently prefer English, translation-equivalent tie probes were judged as "Tie" in the vast majority of cases (e.g., DeepSeek: 97.5%, GPT-4.1 Mini: 95.6%).
Chinese Preference in Non-Ties: When judges did not call a tie, they more often favored the Chinese version over the English version. This suggests the issue is not a simple "English-first" prior, but rather instability in how judges process different linguistic presentations.

3.3 Interaction with Position Bias

Language-Switching Amplifies Position Sensitivity: Position inconsistency (changing the winner when order is swapped) increased significantly in language-switched (LS) conditions.
- Average position inconsistency rose from 12.1% (EN) to 18.1% (LS) across models.
- Gemini Flash showed the largest increase, from 10.3% (EN) to 22.2% (LS).
Adversarial Difficulty: Adversarial subsets (e.g., Adv-GPTOut) exhibited lower accuracy and higher flip rates, suggesting that language transformations add a layer of difficulty to already challenging reasoning tasks.

4. Contributions

Judge-LS Protocol: Introduction of a simple, low-resource protocol for testing language-switching invariance in LLM-as-a-Judge evaluation without model training.
Comprehensive Benchmarking: Transformation of the full 419-item LLMBar benchmark into English, Chinese, and language-switched variants with preserved gold labels.
Large-Scale Evaluation: Evaluation of four diverse API judges across 13,408 judgments, including answer-order swaps and translation-equivalent tie probes.
Diagnostic Framework: Reporting of a multi-dimensional set of metrics including flip rates, position inconsistency, uncertainty estimates, and sensitivity analyses that exclude mechanically risky variants.

5. Significance and Claims

The paper claims that the central risk in multilingual LLM evaluation is not necessarily that judges "prefer English," but that they are sensitive to language presentation in ways that alter correctness and interact with position bias.

Reliability Implication: A 10–14% flip rate is large enough to alter model rankings when systems have close scores. If a leaderboard uses an English judge prompt to evaluate Chinese or language-switched answers, the reported performance may conflate answer quality with judge-language sensitivity.
Design Principle: Evaluation pipelines should be treated as systems under test. A robust judge must preserve core preferences under label-preserving surface transformations, similar to how robust classifiers must handle irrelevant input perturbations.
Practical Workflow: The authors recommend a five-stage workflow for multilingual evaluation: (1) baseline English evaluation with order swaps, (2) transformation to target languages, (3) audit and sensitivity analysis, (4) calculation of invariance flips with confidence intervals, and (5) tie probes to distinguish bias from instability.

The study concludes that current LLM judges are not yet invariant to language presentation, and relying on a single language condition for multilingual model selection is a fragile practice.