💬 NLP

Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

This paper introduces Consensus-Based Calibration (CBC), a label-free post-hoc calibrator derived from two-way ANOVA that eliminates language-induced rank reversal in multilingual LLM judges by recovering and removing language-backbone interaction terms, thereby significantly improving ranking consistency and alignment with human preferences without requiring ground-truth labels.

Original authors: Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers often need to know which computer program is the best at solving a problem. To find out, they use other computer programs, known as "judges," to read the answers and assign them a score. This method has become a standard way to test new models because it is fast and scalable. However, a hidden complication has emerged: these judges do not always agree with themselves. When a judge evaluates the same set of answers in different languages, such as English, Arabic, or Spanish, it often changes its mind about which model is the best. A model that receives a top score in one language might drop to the bottom in another. This inconsistency creates a confusing landscape where the "winner" depends entirely on the language used to ask the question, making it difficult to trust the results or choose the right tool for the job.

A team of researchers has now identified the source of this confusion and developed a way to fix it without needing human experts to re-grade thousands of answers. They discovered that the problem is not random noise, but a specific, predictable pattern where the judge's language and the model being tested interact in a way that skews the scores. By treating the scores as a collection of data points rather than absolute truths, they created a mathematical adjustment that strips away this language bias. The result is a clearer, more consistent ranking that holds true regardless of whether the prompt is written in English, Swahili, or Japanese.

The researchers began by testing this phenomenon on a large scale. They took six different powerful AI models and asked them to solve fifty-five different software engineering tasks. They then used six other AI models to judge the quality of those solutions. Crucially, they ran this entire experiment in eight different languages. The results were striking. In seven out of fifteen possible pairings of models, the ranking flipped completely depending on the language. For instance, one model might be the clear favorite when the judge speaks English, but fall behind a competitor when the judge speaks Spanish. In some cases, the difference was so large that the preferred model in one language was the least preferred in another. This proved that the judges were not simply being inconsistent; they were reacting differently to the same content based on the language of the prompt.

To solve this, the researchers treated the scores like a grid of measurements. They realized that the final score a model received was made up of three parts: how hard the task was, how skilled the model actually was, and a third, invisible factor that mixed the language with the model. This third factor was the culprit. It acted like a filter that distorted the view of the model's ability depending on the language used. The team developed a method called Consensus-Based Calibration to remove this filter. The process is surprisingly simple: they looked at the average scores across all languages and all models, calculated the specific distortion for each language-model pair, and subtracted it out. This operation, which they call double-centering, effectively cancels out the language bias while leaving the true skill of the models intact.

The effectiveness of this method was tested rigorously. Before the adjustment, the rankings of the models changed wildly from language to language, with a consistency score of only 0.650. After applying their calibration, the rankings became almost perfectly stable across all eight languages, with the consistency score jumping to 0.902. In a separate test using a different set of 10,500 items across seven languages, the improvement was even more dramatic, with consistency rising from 0.430 to 0.900. Perhaps most importantly, when they compared their corrected rankings against a set of human-preferred answers, the agreement rose from 68.7% to 76.6%. This suggests that by removing the language bias, the computer judges were actually getting closer to what human experts would have decided.

The researchers are careful to note that this method does not create a perfect, universal truth. It works best when the same tasks are evaluated across the same set of languages. It also cannot fix a situation where every single judge is harsher or kinder to one specific language across the board; it only fixes the way the language interacts with the specific model being tested. Furthermore, the method relies on having a complete set of data where every model is judged in every language. If data is missing, the calculation becomes more difficult. However, for the vast majority of multilingual evaluations where data is available, this approach offers a powerful tool. It allows researchers to stop guessing which model is truly better and start seeing the results clearly, free from the distortion of the language used to ask the question.

This work changes how we think about evaluating artificial intelligence in a global context. It moves the field away from accepting inconsistent, language-dependent scores as the final word. Instead, it provides a reliable way to align different languages onto a single, shared standard of quality. By doing so, it ensures that when we say one AI model is better than another, that statement holds true whether the conversation happens in Tokyo, Cairo, or New York. The solution does not require expensive human labor or new data collection; it simply requires a smarter way of looking at the data we already have.

Technical Summary: Rank Reversal in Multilingual LLM Judges

Problem Statement
Multilingual Large Language Model (LLM) judges exhibit unstable behavior depending on the prompt language. On an eight-language "Agent-as-a-Judge" benchmark, the ranking of evaluator backbones (the underlying LLMs used for scoring) alternates significantly across languages (English, Arabic, Chinese, Hindi, Japanese, Spanish, Turkish, and Swahili). Specifically, 7 out of 15 backbone pairs show statistically significant pairwise rank reversals, meaning a backbone that outperforms another in one language may underperform it in another. This phenomenon is treated as a measurement problem where the observed score is confounded by a language-backbone interaction term. Existing solutions often rely on expensive human annotations or ensemble methods that do not directly target the additive language-backbone interaction inherent in multi-evaluator, multilingual benchmarks.

Methodology: Consensus-Based Calibration (CBC)
The paper proposes a label-free, post-hoc calibrator called Consensus-Based Calibration (CBC). The approach models the judge score $S(t, \ell, b)$ for a task $t$ , language $\ell$ , and backbone $b$ using a two-way additive layout with interaction:
$S(t, \ell, b) = \mu(t) + \alpha(b) + \beta(\ell, b) + \gamma(t, \ell) + \epsilon$
Where:

$\mu(t)$ is task difficulty.
$\alpha(b)$ is backbone skill.
$\beta(\ell, b)$ is the language-backbone interaction (the target for removal).
$\gamma(t, \ell)$ is a residual task-language effect.
$\epsilon$ is noise.

The Estimator:
CBC estimates the interaction term $\beta(\ell, b)$ by double-centering the cell-mean score matrix. Given the mean score $\bar{S}(\ell, b)$ for each language-backbone pair:
$\hat{\beta}(\ell, b) = \bar{S}(\ell, b) - \bar{S}(\cdot, b) - \bar{S}(\ell, \cdot) + \bar{S}(\cdot, \cdot)$
The calibrated score is then obtained by subtracting this estimate: $\hat{S}(t, \ell, b) = S(t, \ell, b) - \hat{\beta}(\ell, b)$ .

Theoretical Guarantees:

Identifiability: Under sum-to-zero constraints, $\beta(\ell, b)$ is uniquely identifiable from population cell means via double-centering.
Finite-Sample Concentration: Assuming independent homoskedastic Gaussian noise, the estimator converges at a rate of $O(1/\sqrt{n})$ with an explicit variance constant of $(1 - 1/m)(1 - 1/k)$ , where $m$ is the number of backbones and $k$ is the number of languages.
Robustness to Misspecification: Crucially, the estimator remains unbiased for $\beta(\ell, b)$ even when task-language interactions $\gamma(t, \ell)$ are present, because these terms cancel out during the double-centering operation. The method does not, however, correct for genuine three-way interactions $\gamma(t, \ell, b)$ or shared language-level shifts where all backbones are uniformly biased in a specific language.

Key Contributions

Empirical Characterization: The authors demonstrate that rank reversal is a pervasive issue, with the top-ranked backbone alternating across all 8 languages in their expanded benchmark.
Label-Free Estimator: CBC requires no human labels, utilizing only existing multi-backbone evaluation data.
Explicit Theoretical Bounds: The paper provides a finite-sample concentration bound and proves unbiasedness under task-language misspecification, distinguishing this work from standard ANOVA applications which assume correct model specification.
Validation: The method is validated on both an internal benchmark and a separately collected external panel (M-RewardBench).

Results

Internal Benchmark (8 Languages, 6 Backbones, 55 Tasks):
- Rank Consistency: CBC increased the held-out cross-task Kendall rank correlation ( $\tau$ ) from 0.650 (raw) to 0.902.
- Decision Accuracy: In per-language deployment decisions, CBC agreed with the held-out additive-model oracle in 100% of cases, compared to 68.5% for raw scores.
- Rank Alignment: Post-calibration, all 8 languages aligned on a single backbone ordering (Gemini > GPT-4o > GPT-5.4 > Sonnet > DeepSeek > Qwen).
External Validation (M-RewardBench, 7 Languages, 5 Evaluators):
- Rank Consistency: $\tau$ improved from 0.430 to 0.900.
- Human Anchor Agreement: Agreement with public human gold preferences increased from 68.7% to 76.6% (a gain of 7.9 percentage points, 95% CI [6.0, 9.9]).
- Comparison: CBC significantly outperformed baselines including quantile normalization, ComBat-EB, and adapted pairwise Bradley-Terry-Luce models.

Significance and Claims
The paper claims that CBC offers a practical, label-free solution to the instability of multilingual LLM judges. By removing the language-backbone interaction term, it enables the derivation of a language-invariant evaluator ranking, which is the operational goal for many deployment scenarios.

The authors are modest in their claims, emphasizing that:

The results are consistency diagnostics and model-reference improvements, not absolute measures of human-grounded correctness.
The method assumes a complete, balanced panel of scores; it does not solve zero-shot transfer to unseen languages (LOLO results showed only modest gains).
It does not correct for shared language-level shifts (e.g., if all judges are uniformly harsher in one language) or three-way interactions (backbone-specific task adaptation in specific languages).
The "human-anchor" improvement on M-RewardBench is presented as the strongest external evidence of downstream usefulness, but it is aggregate evidence, not a per-language diagnosis of bias versus genuine specialization.

The work positions CBC not as a universal fix for all evaluation biases, but as a specific, statistically grounded tool for recovering interaction terms in multilingual evaluation matrices where multiple backbones score the same items across multiple languages.

Technical Summary: Rank Reversal in Multilingual LLM Judges

More like this