💬 NLP

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

The paper introduces BabelJudge, an open-source benchmark and reliability audit framework that evaluates LLM-as-a-Judge models across multiple languages and agent trajectories by generating gold-standard labels through controlled perturbations to expose hidden failure modes like position and verbosity biases that raw accuracy metrics fail to capture.

Original authors: Shreyas KC

Published 2026-06-23✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shreyas KC

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have hired a very smart, very fast judge (an AI) to decide which of two answers is better. You want this judge to be fair, accurate, and consistent. But what if the judge has secret habits that make them unfair? What if they always pick the answer written first, or the one that is longer, even if it's nonsense?

This is exactly what the paper BabelJudge investigates. It's a "report card" system designed to audit these AI judges before we trust them with important decisions.

Here is the breakdown of the paper's findings using simple analogies:

1. The Problem: The "Biased Judge"

The authors found that AI judges aren't neutral. They have three main "bad habits" (biases) that hide behind a high score of "accuracy":

The "First Seat" Bias (Position Bias): The judge always prefers the answer written in the top slot (Slot A), regardless of which answer is actually better. It's like a movie critic who always gives a higher rating to the first movie they watch, just because they haven't seen the second one yet.
The "Longer is Better" Bias (Verbosity Bias): The judge loves long, wordy answers. Even if a short answer is perfect and a long answer is just fluff, the judge picks the long one. It's like a teacher giving an 'A' to a student who wrote 10 pages of nonsense just because they wrote more than the student who wrote one perfect paragraph.
The "Language Barrier" Bias: The judge is great at English but gets confused and unreliable in other languages (like Swahili).

2. The Solution: The "Sabotage Test"

How do you test a judge without asking humans to grade every single answer (which is expensive and slow)?

The authors invented a clever trick called "Gold-Labelling by Degradation."

The Analogy: Imagine you have a perfect, golden apple. You then take a knife and deliberately cut a chunk off it, or add some dirt to it. You now have two apples: the Perfect One and the Damaged One.
The Test: You show these two apples to the judge and ask, "Which is better?"
The Logic: You know the answer is the Perfect One because you made the other one worse. If the judge picks the Damaged One, you know the judge is failing.
The Twist: They do this in two ways:
1. They sometimes make the Damaged One longer (to test if the judge likes length over quality).
2. They swap the order (sometimes the Perfect One is first, sometimes second) to test if the judge has a "First Seat" bias.

3. The Results: The "Reliability Score"

The paper tested a popular AI judge (Qwen2.5) in four languages: English, Hindi, Arabic, and Swahili.

The Trap of Raw Accuracy: If you just ask, "How often did the judge pick the right answer?" the scores look okay (around 77% on average). It looks like the judge is doing a good job.
The Reality Check: When the authors applied their "Reliability Score" (which punishes the judge for being biased), the scores dropped significantly.
- English & Hindi: The judge passed, but barely.
- Swahili: The judge failed.
  - In Swahili, the judge's "Reliability Score" was only 0.55 (below the passing line of 0.65).
  - Why? Because in Swahili, the judge was essentially flipping a coin. When they swapped the order of the answers, the judge's decision changed completely. It wasn't judging quality; it was just guessing based on which side the answer was on.

4. The "Agent" Extension: The "Robot Worker"

The paper also tested how these judges handle AI Agents (robots that use tools, like checking the weather or booking a flight).

They found new ways the judge can fail:

Hallucination Blindness: The judge didn't notice that the robot used a tool that doesn't exist.
Argument Blindness: The judge didn't notice the robot gave the wrong information to a tool (e.g., asking for the weather in "Paris, France" but typing "Paris, Texas" by mistake).
The "Step Pad" Bias: Just like with text, if the robot's plan was longer (even if it had extra, useless steps), the judge preferred it.

5. The Takeaway

The paper concludes that you cannot trust an AI judge just because it gets high accuracy scores.

The Warning: If you deploy a judge for a language like Swahili without checking these biases, your results will be noisy and unreliable. The judge might be picking answers based on which one was written first, not which one is true.
The Recommendation: Before you let an AI judge grade anything important, run it through BabelJudge.
- If the score is low, don't use it alone.
- Instead, use a "team" of judges (a majority vote) or switch to a human for that specific language.

In short: BabelJudge is a "lie detector" for AI judges. It reveals that while they might look smart on paper, they often have hidden biases that make them unreliable, especially in languages other than English.

Technical Summary: BabelJudge

Problem Statement

The adoption of "LLM-as-a-judge" has become the dominant approach for scalable evaluation in NLP pipelines, including benchmarks, RLHF preference collection, and production quality gates. However, the judges themselves are not neutral; they carry systematic biases that raw accuracy metrics fail to capture. Prior work has documented specific failure modes:

Position Bias: Judges systematically prefer responses placed in a specific slot (e.g., slot A).
Verbosity Bias: Judges reward longer responses regardless of information value.
Self-Enhancement Bias: Judges prefer outputs from models within the same family.
Cross-Lingual Degradation: Judge reliability collapses in lower-resource languages, yet most existing benchmarks evaluate exclusively in English.

Currently, these biases are documented in scattered studies with incompatible experimental setups, leaving practitioners without a unified, reusable instrument to audit an arbitrary judge model across multiple languages and failure modes.

Methodology: Gold-Labelling by Degradation

BabelJudge introduces a framework to measure judge reliability without requiring expensive human preference labels. The core insight is gold-labelling by degradation:

Synthetic Construction: Starting from a high-quality reference response ( $r$ ) drawn from a multilingual corpus, a controlled degradation function is applied to generate a perturbed response ( $p$ ).
Known Ground Truth: By construction, the reference response is the "gold" label (the better response).
Dual-Order Presentation: Each pairwise item is presented to the judge in both slot orderings (reference-first and perturbed-first). This allows for the direct measurement of position bias and order consistency.

Perturbation Types

The framework utilizes five controlled, language-agnostic perturbation types to probe specific failure modes:

Truncate: Drops trailing sentences to test information loss tolerance.
Shuffle: Reorders sentences to test coherence sensitivity.
Number Corrupt: Perturbs numeric tokens to test factual accuracy detection.
Drop Entities: Removes entity-like tokens to test entity completeness.
Repeat Pad: Appends redundant sentences to the perturbed response. Crucially, this makes the perturbed response longer but strictly worse, serving as a specific probe for verbosity bias.

Agentic Extension

The framework extends to agentic settings (tool-use trajectories) via nine trajectory-level perturbations, including:

Argument Corruption: Mutating argument values.
Tool Name Swaps: Replacing tools with incorrect alternatives.
Hallucinated Tools: Inserting non-existent tool calls.
Missing/Extra Arguments: Removing required args or adding nonsense keys.
Step Manipulation: Dropping intermediate steps, early termination, or padding with redundant steps (to probe trajectory-length bias).

Reliability Metrics

BabelJudge moves beyond simple accuracy to a composite Reliability Score ( $R$ ) that penalizes bias multiplicatively. The score is calculated as:
$R = (\text{Competence Base}) \times (\text{Position Penalty}) \times (\text{Verbosity Penalty})$

Where:

Competence Base: A weighted average of raw accuracy and order consistency.
Position Penalty: Derived from the win-rate difference between slot orderings ( $\Delta_{pos}$ ).
Verbosity Penalty: Derived from the susceptibility to preferring longer, lower-quality responses.

This multiplicative gating ensures that a judge with high raw accuracy but significant bias is penalized more severely than additive averaging would suggest. The paper proposes a deployment threshold of $R \geq 0.65$ .

Experimental Results

The framework was evaluated using Qwen2.5-7B-Instruct-4bit across four languages: English, Hindi, Arabic, and Swahili.

Key Findings

Raw Accuracy Overstates Reliability: While macro accuracy remained relatively high across languages (ranging from 0.660 to 0.835), the composite reliability score dropped significantly, particularly in lower-resource languages.
Cross-Lingual Collapse:
- Hindi: High reliability (0.714) with high accuracy (0.835).
- Swahili: Reliability collapsed to 0.550 despite an accuracy of 0.660.
- Driver of Failure: The primary cause for Swahili's low reliability was order consistency (0.480), indicating the judge's verdict was effectively random when slot order was swapped.
Verbosity Susceptibility: This metric was highly language-dependent. English showed low susceptibility (0.075), while Hindi showed significantly higher susceptibility (0.250), suggesting the judge is more easily fooled by length in Hindi than in English.
Position Bias: Position bias ( $\Delta_{pos}$ ) remained within acceptable ranges (<0.10) for all languages in this sample, suggesting partial debiasing in the model.

Significance and Contributions

The paper claims the following contributions:

Methodology: A "gold-labelling-by-degradation" approach that eliminates annotation costs by manufacturing pairwise items with known labels.
Unified Instrument: A single framework that operationalizes position, verbosity, and order consistency biases into comparable metrics.
Composite Metric: A reliability score that makes the gap between raw accuracy and actual judge quality visible, revealing that accuracy alone is an incomplete signal.
Agentic Extension: The first extension of judge auditing to agent trajectories, introducing metrics for tool accuracy, hallucination detection, and trajectory-length bias.
Open-Source Tool: An open-source Python package (BabelJudge) with adapters for 11 judge backends, designed to serve as a "living leaderboard" for judge reliability.

The authors conclude that BabelJudge should be used as a standard checkpoint before deploying LLM judges in multilingual or agentic production settings, particularly to identify languages or scenarios where the judge's reliability falls below the 0.65 threshold.