📊 statistics

Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence

This paper introduces Heterogeneous Judge-Aware (HJA) ranking, a structured framework that decomposes multi-judge pairwise comparisons into identifiable consensus rankings, judge-specific sensitivities, and residual disagreements to improve recovery, robustness, and uncertainty calibration in large language model evaluation.

Original authors: Shibo Yu, Yingzhou Wang, Yan Chen, Guodong Li, Jin-Hong Du

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shibo Yu, Yingzhou Wang, Yan Chen, Guodong Li, Jin-Hong Du

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to rank the best restaurants in your city. Instead of asking just one food critic, you ask a panel of 20 different people.

In the past, if you asked 20 people, you might just take their answers, average them out, and create one single "Top 10" list. You'd assume that if two people disagree, it's just random noise or a mistake.

The Problem:
The authors of this paper argue that this "averaging" approach is flawed. Not all judges are the same.

Judge A might be a "food snob" who hates spicy food but loves fancy plating.
Judge B might be a "spice lover" who thinks fancy plating is a waste of money.
Judge C might be very strict and only likes the top 3 restaurants, while Judge D is easy-going and likes almost everything.

If you just average their scores, you lose the nuance. You don't know why they disagree, and you might end up with a ranking that satisfies no one.

The Solution: HJA (Heterogeneous Judge-Aware) Ranking
The paper proposes a new way to handle this called HJA. Think of it like a smart team manager who doesn't just count votes, but understands the personality of each voter.

The HJA model breaks the final ranking down into three distinct parts, like separating ingredients in a recipe:

The Consensus (The "Common Ground"):
This is the part where everyone agrees. Maybe everyone agrees that "fresh ingredients" are good. The model finds this shared "truth" or baseline ranking that most judges are trying to describe.
The Sensitivity (The "Volume Knob"):
This measures how strongly a specific judge follows that common ground.
- Analogy: Imagine the consensus is a radio station playing a song. Judge A has the volume turned up to 10 (they agree strongly). Judge B has the volume at 2 (they are a bit fuzzy or unsure). Judge C has the volume at -5 (they actually hate the song and prefer the opposite!).
- HJA measures this "volume" for each judge separately, rather than assuming everyone hears the radio at the same loudness.
The Disagreement (The "Secret Sauce"):
This is the most important part. It captures the structured reasons why judges don't agree.
- Analogy: Even if the radio is loud, Judge A might still prefer the "Spicy" track, while Judge B prefers the "Sweet" track. This isn't random noise; it's a specific, predictable preference.
- HJA isolates these specific preferences. It realizes that Judge A isn't just "wrong"; they just have a different "flavor profile" that the model can map out.

Why is this better?

It's Honest about Uncertainty: The model doesn't just give you a list; it tells you how confident it is. If two restaurants are very close in quality, the model says, "We aren't sure which is #1, and here is the range of possibilities."
It Handles "Near-Ties": In the real world, the difference between the #1 and #2 restaurant is often tiny. Old models get confused here. HJA is designed to handle these "toss-up" situations better.
It Finds the "Bad Actors": The model can spot if a judge is being weird or biased. For example, if a judge consistently rates their own company's products higher than everyone else (a "self-preference" bias), HJA can detect this pattern in the "Disagreement" section and adjust for it, rather than letting it ruin the whole list.

How they tested it:
The authors tested this on both fake data (where they knew the "true" answer) and real data from the internet (like people ranking AI chatbots).

The Result: HJA was better at finding the true rankings, more robust when noisy or biased judges were added to the mix, and provided better "confidence intervals" (telling you how sure you can be about the ranking) compared to the old "average everything" methods.

In a Nutshell:
Instead of treating a panel of judges as a single, blurry voice, HJA listens to each judge individually. It figures out what they all agree on, how strongly they feel about it, and exactly why they disagree. This leads to a clearer, more reliable, and more honest ranking system.

Technical Summary: Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence

Problem Statement

Large language model (LLM) evaluation increasingly relies on pairwise comparisons from panels of multiple judges (human annotators, model-based judges, or varying protocols). Standard ranking pipelines typically pool these judgments into a single latent score vector, treating systematic disagreement among judges as exchangeable noise. However, recent evidence indicates that judge disagreement is often structured, varying significantly across judge models, languages, tasks, and protocols. This heterogeneity is particularly consequential in subjective evaluations and "near-tie" comparisons, where small perturbations can drastically alter final leaderboards.

The core problem is that moving from a single judge to a panel does not merely increase sample size for a classical ranking problem; it fundamentally changes the object of inference. The goal shifts from recovering a single pooled ordering to understanding a structured multi-judge ranking problem that accounts for consensus, judge-specific sensitivity, and systematic residual disagreement.

Methodology: Heterogeneous Judge-Aware (HJA) Ranking

The authors propose Heterogeneous Judge-Aware (HJA) ranking, a framework that decomposes judge-specific latent scores into three distinct, identifiable components:

1. Model Formulation

The framework models the latent score matrix $S \in \mathbb{R}^{K \times N}$ (where $K$ is the number of judges and $N$ is the number of items) using a structured decomposition:
$S = \gamma\mu^\top + UV^\top$

Consensus Ranking ( $\mu$ ): A row-average consensus direction representing the population-level ranking target.
Consensus Sensitivity ( $\gamma$ ): A vector of judge-specific parameters ( $\gamma_k$ ) measuring how strongly judge $k$ tracks the shared consensus signal. Unlike global quality metrics, $\gamma_k$ specifically measures alignment with the consensus direction.
Structured Disagreement ( $UV^\top$ ): A low-rank residual component capturing systematic deviations from the consensus that cannot be explained by scalar rescaling alone. This term is constrained to be orthogonal to the consensus direction.

The pairwise preference probabilities follow the Bradley-Terry-Luce (BTL) model:
$\text{logit}(p_{kij}) = \gamma_k(\mu_i - \mu_j) + U_{k,:}(V_{i,:} - V_{j,:})^\top$

2. Identifiability and Constraints

To ensure the decomposition is statistically well-defined, the authors impose specific normalization and rank conditions (Condition 1):

Centering: $\mu$ and columns of $V$ are centered; rows of $U$ sum to zero.
Scaling: $\gamma$ sums to $K$ ; $U$ and $V$ are scaled to unit norms.
Orthogonality: The consensus direction $\mu$ is orthogonal to the heterogeneous subspace $V$ .
Subspace Anchoring: Singular values of the residual component are distinct and ordered.

Under these conditions and assuming the cross-judge aggregate score is non-zero, the authors prove (Proposition 1) that the consensus direction, sensitivity parameters, and residual disagreement subspace are identifiable from the pairwise comparison data, unique up to column-wise sign changes.

3. Estimation Algorithm

The authors develop a constrained maximum likelihood estimator (MLE). Due to the non-convex nature of the bilinear factorization and the nonlinear anchoring constraints, they propose an anchored proximal alternating algorithm (Algorithm 1):

Block Updates: Alternating minimization over judge parameters ( $\gamma, U$ ) and item parameters ( $\mu, V$ ) with proximal terms to ensure stability.
Re-anchoring: A critical step (Algorithm C.2) that restores the nonlinear identification constraints after each block update, preserving the identifying geometry of the decomposition.
Convergence: The algorithm is proven to enjoy local linear convergence (Theorem 2) under standard regularity assumptions.

4. Uncertainty Quantification

The framework supports statistical inference in a fixed-panel repeated-comparison regime, where the number of judges $K$ may remain fixed while information grows through repeated comparisons.

Asymptotic Normality: The constrained MLE is shown to be asymptotically normal (Theorem 3).
Confidence Intervals: Using the Fisher information matrix derived from the graph structure of comparisons, the method provides confidence intervals for consensus score contrasts, judge-specific contrasts, sensitivity parameters, and pairwise probabilities. This allows for uncertainty quantification even when the judge panel is small or fixed.

Key Results

Synthetic Experiments

Recovery: HJA successfully recovers consensus rankings, judge-specific behaviors, and structured heterogeneity as comparison information increases.
Robustness: Misspecified baselines (pooled BTL and sensitivity-only models like JA-Ranking) degrade significantly under strong heterogeneity, concentrating around biased score matrices.
Uncertainty Calibration: HJA provides well-calibrated uncertainty statements, whereas baselines fail to capture the true variance in structured disagreement.

Real-Data Benchmarks

Evaluated on four datasets (Chatbot Arena, MT-Bench, UltraFeedback, and an in-house dataset), HJA demonstrates:

Improved Robustness: Higher stability in rankings when noisy judges are injected.
Better Prediction: Superior held-out pairwise prediction accuracy.
Near-Tie Performance: Significant gains in accuracy for empirically ambiguous near-tie pairs, where standard methods are fragile.
Diagnostic Utility: The fitted model identifies high-leverage judges (those with high sensitivity but also high residual disagreement) and reveals structured judge-model affinity patterns (e.g., family-based self-preference effects) without treating them as universal findings.

Significance and Claims

The paper claims to provide a statistically grounded framework for ranking under heterogeneous comparative judgments. Its primary contributions are:

Decomposition of Targets: It separates consensus ranking, judge sensitivity, and structured disagreement into separate inferential targets, rather than collapsing them into a single score or treating disagreement as noise.
Identifiability: It establishes conditions under which these components are uniquely recoverable from pairwise data, moving beyond arbitrary latent space factorizations.
Fixed-Panel Inference: It enables rigorous uncertainty quantification (confidence intervals) for ranking targets even when the judge panel does not grow, a common constraint in LLM evaluation.
Diagnostic Power: The framework serves as a diagnostic tool to detect unreliable judges, quantify sensitivity to consensus, and localize systematic affinity patterns (such as self-preference bias) within the residual disagreement term.

The authors modestly note that the framework is not intended to certify rankings under adversarial judge panels where severe coordinated bias may affect the consensus estimate, nor does it guarantee global optimality due to the non-convex nature of the optimization. Instead, it is positioned as a diagnostic layer for heterogeneous evaluation to improve the robustness and interpretability of ranking conclusions.