📊 statistics

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

This paper proposes a judge-aware ranking framework that extends the Bradley-Terry-Luce model to jointly estimate latent model quality and judge reliability from pairwise comparisons without ground truth, thereby improving ranking accuracy, data efficiency, and uncertainty calibration in LLM evaluation.

Original authors: Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to rank the best chefs in the world. You don't have a "gold standard" recipe book to check their work against (no ground truth). Instead, you ask a panel of food critics to taste dishes and vote on which one is better.

The Problem: Not All Critics Are Created Equal
In the world of Large Language Models (LLMs), we often use other AI models to act as these critics (judges). The paper points out a major flaw in how we usually do this: we treat every critic as if they are equally good.

Imagine one critic is a Michelin-starred chef who can taste the difference between salt and sugar. Another critic is someone who just guesses randomly. If you give them both equal weight in your final ranking, the random guesser's noise will muddy the results. Worse, if you ask more random guessers to vote, you might become very confident in a completely wrong ranking. It's like asking a thousand people who can't read to vote on a spelling bee winner; more votes just make the wrong answer look more certain.

The Solution: A "Judge-Aware" System
The authors propose a new framework that acts like a smart tournament organizer. Instead of just counting votes, this system figures out how good each critic is while it's calculating the rankings.

Here is how it works, using a few analogies:

The "Sensitivity" Dial: Imagine every critic has a dial on their head called "Sensitivity."
- A high-sensitivity critic (a reliable judge) has the dial turned up. If they see a tiny difference between two dishes, they confidently pick a winner.
- A low-sensitivity critic (an unreliable judge) has the dial turned down. They can't tell the dishes apart, so their votes are basically random noise.
- The system automatically turns down the volume on the low-sensitivity critics and turns up the volume on the high-sensitivity ones.
Learning on the Fly: The system doesn't need to know who the good critics are beforehand. It looks at the pattern of votes. If Critic A always agrees with the majority of other smart critics, the system learns, "Ah, Critic A is reliable," and gives their votes more weight. If Critic B disagrees with everyone randomly, the system learns, "Critic B is noisy," and ignores their votes.
The "Confidence" Meter: Because the system knows which critics are shaky, it can tell you how sure it is about the final ranking.
- Old way: "Model X is #1." (But it might be wrong, and we don't know how wrong).
- New way: "Model X is #1, and we are 95% sure of this because the top critics agreed." Or, "Model X is #1, but the margin is tiny and the critics are confused, so we aren't very sure."

What They Found
The researchers tested this idea on real data where they had thousands of AI models comparing each other.

Better Alignment: Their method produced rankings that matched what human experts preferred much better than the old "equal weight" method.
Data Efficiency: They got accurate results with fewer comparisons. It's like needing fewer votes to find the best candidate if you know which voters are experts.
Fixed the "Confidently Wrong" Problem: In the old method, adding more data sometimes made the system more confident in a bad ranking. The new method prevents this by filtering out the noise.

In a Nutshell
This paper introduces a smarter way to rank AI models when you don't have a "correct answer key." Instead of treating every AI judge as an equal expert, it builds a system that figures out which judges are actually good at spotting differences and listens to them more, while tuning out the ones who are just guessing. This leads to fairer, more accurate leaderboards and tells us exactly how much we should trust the results.

Technical Summary: A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

1. Problem Statement

The evaluation of Large Language Models (LLMs) on open-ended tasks increasingly relies on the "LLM-as-a-judge" paradigm, where automated judges assess pairwise comparisons of model outputs in the absence of ground-truth labels. While this approach scales evaluation, a critical, under-modeled limitation exists: judge reliability varies substantially across different LLMs.

Existing ranking pipelines typically treat all judges as equally reliable, aggregating pairwise comparisons uniformly. This assumption is statistically misspecified. As demonstrated in this work, ignoring judge heterogeneity leads to:

Biased Rankings: Aggregating noisy or low-quality judgments with high-quality ones distorts the latent quality scores of the models being evaluated.
Invalid Uncertainty Quantification: Under misspecified models, increasing the amount of data does not necessarily improve accuracy; instead, it can lead to "confidently wrong" rankings where confidence intervals shrink around biased estimates.

Current solutions either retain the homogeneous judge assumption (e.g., standard Bradley-Terry-Luce models) or rely on heuristic weighting schemes (e.g., REM, GTR/FTR) that lack a unified statistical interpretation and do not support valid inference or confidence intervals.

2. Methodology

The authors propose a Judge-Aware Ranking Framework that extends the classic Bradley-Terry-Luce (BTL) model to explicitly account for judge heterogeneity.

2.1. The Model

The framework introduces a judge-specific discrimination parameter ( $\gamma_k$ ) for each judge $k$ . For a pairwise comparison between model $i$ and model $j$ by judge $k$ , the probability of preferring model $i$ is modeled as:
$P(Y=1 | i, j, k) = \sigma(\gamma_k (s_i - s_j))$
Where:

$s_i, s_j$ are latent quality scores for the models.
$\gamma_k > 0$ is the discrimination parameter for judge $k$ .
$\sigma(\cdot)$ is the logistic function.

Interpretation of $\gamma_k$ :

High $\gamma_k$ : The judge is highly sensitive to quality differences; even small gaps in latent scores lead to decisive preferences.
Low $\gamma_k \approx 0$ : The judge's preferences approach random chance, contributing little information.
The standard BTL model is recovered as the special case where $\gamma_k \equiv 1$ for all judges.

2.2. Estimation and Identifiability

The framework jointly estimates the latent model scores ( $s$ ) and judge discrimination parameters ( $\gamma$ ) via Maximum Likelihood Estimation (MLE) without requiring ground-truth labels.

Identifiability: The model is identifiable up to natural normalizations. Specifically, scores are identifiable up to a global shift, and discrimination parameters are identifiable up to a common scale. The authors impose constraints $\sum s_i = 0$ and $\sum \log \gamma_k = 0$ to ensure uniqueness.
Optimization: The authors employ a projected Adam optimizer to maximize the log-likelihood over the normalized parameter space, reparameterizing $\alpha_k = \log \gamma_k$ to enforce positivity.

2.3. Theoretical Guarantees

The paper establishes rigorous asymptotic properties for the MLE:

Consistency: The estimator converges in probability to the true parameters as the number of comparisons $T \to \infty$ .
Asymptotic Normality: The estimator follows a normal distribution asymptotically, enabling the construction of Wald confidence intervals for score differences and rank comparisons.
Uncertainty Quantification: Unlike heuristic methods, this framework provides principled confidence intervals that account for both model uncertainty and judge reliability.

3. Key Contributions

Identification of Bias: The paper identifies judge heterogeneity as a fundamental source of bias and invalid uncertainty in LLM-as-a-judge rankings, demonstrating that "more data" can exacerbate errors under misspecified aggregation.
Judge-Aware Generalization: It proposes a statistical generalization of the BTL model that jointly estimates model quality and judge reliability, automatically down-weighting unreliable judges through the likelihood function.
Statistical Foundations: The work establishes identifiability, consistency, and asymptotic normality for the resulting estimator, enabling valid confidence intervals for LLM rankings.
Empirical Validation: Through simulations and real-world benchmarks, the authors demonstrate that the method improves alignment with human preferences and achieves higher data efficiency compared to unweighted baselines.

4. Experimental Results

The framework was evaluated on multiple public benchmarks (MT-Bench, Chatbot Arena, UltraFeedback) and a newly collected in-house dataset involving 45 candidate models and up to 20 judge LLMs.

4.1. Simulation Studies

Convergence Rate: The MLE achieves the theoretical $O(1/T)$ convergence rate.
Confidence Intervals: The judge-aware model maintains empirical coverage close to the nominal 95% level across sample sizes. In contrast, the unweighted BTL model exhibits systematic under-coverage that worsens as data increases, confirming the "confidently wrong" phenomenon.
Efficiency: The weighted model produces consistently shorter confidence intervals than the unweighted model, indicating higher statistical efficiency by extracting more information from high-quality comparisons.

4.2. Real-World Benchmarks

Alignment with Human Preferences: The learned scores show strong positive correlations with established human-preference benchmarks (e.g., MT-Bench scores and UltraFeedback dimensions like helpfulness and truthfulness).
Data Efficiency: In subsampling studies, the weighted model reaches high correlation with full-data references using fewer comparisons than the unweighted model, particularly in low-budget regimes.
Handling Heterogeneity: In experiments with mixed-quality judge pools (containing both high and low-quality judges), the weighted model significantly improved ranking fidelity (Pearson correlation increased from 0.899 to 0.940) compared to the unweighted approach.
Case Study (Chatbot Arena): The weighted ranking corrected anomalies present in the unweighted ranking, such as placing GPT-4 correctly at rank 1 (vs. rank 3 in unweighted) and correctly ordering model variants by size, aligning more closely with human evaluation rankings (Spearman $\rho = 0.9955$ vs. $0.9699$).

4.3. Judge Reliability Estimation

The model successfully recovered known judge qualities purely from pairwise data. For instance, it assigned high discrimination parameters ( $\gamma$ ) to judges manually identified as high-quality (e.g., kimi-k2-thinking-turbo) and low parameters to noisy judges (e.g., deepseek-ai/DeepSeek-R1-Distill-Llama-70B with $\gamma \approx 0.046$ ).

5. Significance and Claims

The paper claims that this framework provides a statistically principled infrastructure for scalable, reference-free LLM evaluation. Its primary significance lies in:

Correcting Systematic Bias: By explicitly modeling judge reliability, the framework prevents the dilution of informative signals by noisy judges, leading to more accurate leaderboards.
Enabling Valid Inference: It is the first method to provide asymptotic guarantees and confidence intervals for LLM rankings derived from heterogeneous automated judges, allowing researchers to distinguish statistically significant differences from noise.
Data Efficiency: The method is particularly valuable in low-budget scenarios where data is scarce, as it maximizes the utility of high-quality judgments.

The authors note that while the current model assumes a global scalar for judge discrimination, future work could extend this to covariate-dependent discrimination (e.g., task-specific reliability) and address correlated errors among similar judges. However, the current framework offers a robust, generalizable solution for the immediate challenges of LLM evaluation.