📊 statistics

Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges

This paper demonstrates that in multi-judge LLM evaluation with labeled calibration data, retaining all judges and calibrating their outputs yields significantly lower error rates than curating a subset based solely on accuracy, because even weak or biased judges provide non-redundant signals that improve probabilistic calibration.

Original authors: Yanran Li

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanran Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Don't Fire the Weak Judges; Teach Them to Be Honest

Imagine you are trying to decide which of two AI chatbots is better at answering a question. You don't have a human expert to ask, so you ask a panel of 100 different AI models to vote.

The Old Way (Curating):
Traditionally, people look at the results and say, "Okay, these 5 AI judges are really smart and get the right answer most of the time. The other 95 are kind of dumb or confused. Let's fire the 95 and only listen to the top 5." This is called curation. It makes sense if you just want to know who won.

The New Way (Calibrating):
This paper argues that if you want to know how confident you should be in the answer, firing the "dumb" judges is a mistake. Instead, you should keep all 100 judges, but you need to calibrate them.

Think of calibration like teaching a group of weather forecasters.

The Expert: Always says "100% chance of rain" when it rains.
The Novice: Always says "100% chance of rain" even when it's sunny.
The Contrarian: Always says "0% chance of rain" when it rains.

If you only keep the Expert, you get a good answer, but you might miss the nuance. If you keep the Novice and the Contrarian, you can actually learn a lot! You can teach the Novice, "Hey, you're usually too optimistic," and tell the Contrarian, "Hey, you're usually wrong, so flip your answer."

The paper's main discovery is: A panel of 100 judges, once "taught" (calibrated) how to adjust their confidence, gives you a much more accurate picture of reality than a panel of just the top 5 "smart" judges.

The Experiment: The Four Arenas

The researchers tested this idea on four different "arenas" (datasets) where they knew the correct answers in advance (like a practice exam):

JudgeBench: 350 questions with 32 judges.
RewardBench: Nearly 3,000 comparisons with 100 judges.
RewardBench 2: Harder questions with 174 judges.
LLMBar: Instruction-following tests with 53 judges.

In every single arena, they compared two teams:

Team A (The Curators): Picked the top 3, 5, or 10 judges based on who got the most answers right, and threw the rest away.
Team B (The Inclusives): Kept every single judge, even the ones who were bad, and used a mathematical "teacher" (calibration) to adjust their votes.

The Result:
Team B (keeping everyone) almost always won.

On the hardest test (RewardBench 2), the "Keep All" team was twice as accurate at estimating confidence as the "Top 5" team.
Even when the researchers tried to be super smart about picking the best subset of judges, they couldn't beat the "Keep All" team.

Why Does This Work? (The Magic of "Noise")

You might ask: "If a judge is bad, why keep them?"

The paper explains that even a "bad" judge carries useful information, provided you know how to read them.

The "Anti-Expert": Imagine a judge who is wrong 70% of the time. That's actually very useful! If they say "Option A is better," you can confidently say "Option B is better." You just flip their vote.
The "Confused" Judge: Imagine a judge who is right 50% of the time (pure guessing). Their votes don't help, but they also don't hurt if you know they are guessing.
The "Signal" of Disagreement: When the smart judges and the dumb judges disagree, it tells you that the question is hard. If everyone agrees, the question is easy. By keeping the whole panel, you get a better sense of which questions are tricky.

The paper uses a mathematical concept called the "Calibrated Jury Theorem." It proves that as long as you have a way to learn from the judges (calibration), adding more judges—even weak ones—never makes your probability estimates worse. It's like adding more sensors to a car; even a slightly broken sensor can tell you something about the road if you know how to interpret its glitch.

The Recipe for Success

The paper suggests a simple recipe for anyone evaluating AI models:

Don't Fire Anyone Yet: Gather all your judges (LLMs or reward models).
Use a "Teacher" (Calibration): Use a small set of questions where you know the right answers (the calibration set).
Teach the Judges: Run the math to learn how each judge tends to be biased.
- "Judge A is too confident."
- "Judge B always picks the first option."
- "Judge C is actually an anti-expert; flip their vote."
Aggregate: Combine all the adjusted votes.

When should you not do this?
The paper notes two rare exceptions where you might want to fire a judge:

The Broken Judge: If a judge's answers are so messy that the computer can't even read them (e.g., it outputs gibberish 50% of the time), throw them out.
The Redundant Room: If you already have 174 judges and they are all clones of each other, adding one more clone won't help. But usually, having a diverse group is better.

The Bottom Line

In the past, we thought the best way to get a good answer was to find the smartest people and ignore the rest. This paper says that's wrong.

If you want to know how sure you can be about an answer, you should keep the whole crowd, teach them how to be honest, and listen to the whole chorus. The "weak" voices often hold the key to understanding the uncertainty.

The slogan of the paper: Calibrate, Don't Curate.

Technical Summary: Calibrate, Don't Curate

Problem Statement

In the evaluation of Large Language Models (LLMs) and reward models, multi-judge pairwise comparison is the dominant paradigm. The prevailing heuristic for managing heterogeneous judge panels (ranging from near-random to near-expert) is curation: selecting a subset of the most accurate judges (top- $k$ ) and discarding weaker ones to maximize point accuracy.

However, modern evaluation systems increasingly require calibrated probabilistic outputs (reliable confidence estimates) rather than just accurate winners. This paper investigates whether the curation heuristic holds when the objective shifts from point accuracy to calibrated probabilistic estimation using a labeled calibration set. The central question is: Should we curate a small panel of high-accuracy judges, or calibrate the full panel including weaker judges?

Methodology

The authors propose a four-step statistical pipeline for multi-judge aggregation, comparing two arms: Accuracy-Based Selection (top- $k$ pruning) versus Full-Panel Inclusion. Crucially, the aggregation and calibration procedures are held fixed across both arms to isolate the effect of judge panel composition.

The Pipeline

Heterogeneity-Aware Aggregation ( $M_\omega$ ):
- Judges' verdicts are aggregated into a raw probability using an integrated Bayesian one-coin model.
- Judge weights are derived from posterior mean accuracy ( $\alpha_k/(\alpha_k+\beta_k)$ ) estimated on the calibration split.
- This implements a log-odds weighting principle where judges with uncertain accuracy ( $\alpha_k \approx \beta_k$ ) receive near-zero weight, while strong and weak judges contribute based on their learned reliability.
- Note: The paper also considers Dawid–Skene EM as an alternative aggregator.
Residual Bias Correction ( $M_{\omega, \delta}$ ):
- A low-capacity post-hoc map (Platt-style logistic regression) corrects global scale and shift biases in the aggregated score.
- This step allows the model class to expand from $M_\omega$ to $M_{\omega, \delta}$ without increasing asymptotic calibrated loss.
Distribution Calibration ( $M_{\omega, \delta, G}$ ):
- A Beta calibration map is applied to the corrected probabilities to ensure distribution-level calibration.
- The map $g(p) = \sigma(a \log p + b \log(1-p) + c)$ generalizes Platt scaling, allowing asymmetric corrections near confident positives and negatives.
- Elastic net regularization is used to prevent overfitting, particularly when calibration data is limited.
Conformal Coverage (Optional):
- Split conformal prediction wraps the calibrated probabilities to provide finite-sample marginal coverage guarantees.

Experimental Setup

Datasets: Four labeled pairwise-evaluation benchmarks: JudgeBench, RewardBench, RewardBench 2, and LLMBar.
Judges: Panels ranging from 38 to 174 judges, including native reward models and augmented API/GPU-hosted LLMs.
Metrics: Negative Log-Likelihood (NLL) and Brier Score (strictly proper scoring rules), Expected Calibration Error (ECE), and point accuracy.
Protocol: 50/50 calibration/evaluation splits repeated over 100 permutations. Statistical significance is assessed via item-level paired permutation tests.

Theoretical Rationale

The paper provides two theoretical justifications for why inclusion outperforms selection in calibrated settings:

Monotonicity of Oracle-Calibrated Risk (Proposition 1):
Under strictly proper scoring rules (Brier, NLL), the oracle-calibrated risk is monotone with respect to the judge set. If $S \subseteq T$ , then $R(T) \leq R(S)$ . Adding judges cannot increase the optimal calibrated loss because it provides additional conditional information ( $I(Y; X_{T \setminus S} | X_S) \geq 0$ ).
Calibrated Jury Theorem (Theorem 1):
Extending the Condorcet Jury Theorem, the authors show that even "anti-expert" judges (accuracy $p_k < 0.5$ ) contribute positively to the error bound if their bias is learnable. The optimal weight $\alpha_k = \log(p_k / (1-p_k))$ flips the vote of an anti-expert rather than discarding it. As long as judges are not pure noise ( $p_k \neq 0.5$ ), their signals are non-redundant and improve the exponential decay of the error bound.

Key Results

1. Calibration Reversal

The primary finding is a reversal of the selection-inclusion tradeoff:

Uncalibrated: Pruning weak judges (top- $k$ ) often improves raw accuracy and sometimes raw NLL by removing extreme low-quality signals.
Calibrated: After applying Beta calibration, using the full judge panel consistently outperforms accuracy-based selection.
- On RewardBench 2, retaining all judges achieved an NLL of 0.006, compared to 0.013 for top-5 selection (halving the calibration error).
- Similar gains were observed across JudgeBench (+19% NLL reduction), RewardBench (+31%), and LLMBar (+22%).

2. Robustness to Stronger Curators

The advantage of full-panel inclusion persists even against stronger pruning baselines:

Greedy Forward Selection: Selecting judges to minimize inner-split calibrated NLL failed to beat the full panel.
$\ell_1$ -Sparse Logistic Regression: Failed to sparsify the panel, effectively recovering the full set.
Oracle Top-5: Even with perfect knowledge of test-set accuracy, selecting only 5 judges could not match the full panel's calibrated performance.
Subset Search: Exhaustive search on deduplicated families and beam search on full panels confirmed that inclusion is either the winner or statistically tied (with the full panel being the simpler default).

3. Role of Disagreement

The paper explains that weak judges contribute value through disagreement. Their votes mark uncertain items and produce less extreme probabilities, which the calibrator can learn to trust. Accuracy-based pruning removes this information, whereas full-panel calibration retains it and learns the appropriate trust levels.

4. Budget and Scope Diagnostics

Label Budget: The inclusion advantage is most pronounced with small calibration budgets (e.g., on RewardBench, full-panel inclusion reaches NLL 0.01 with only 75 labels, while top-5 requires >300).
Boundary Cases: Inclusion fails only when judges are pathological (e.g., unparseable verdicts, reliability below chance with no learnable bias) or when the panel is already saturated (redundant judges at the sharpness floor). The authors propose operating diagnostics (parseability, non-saturation, aggregator compatibility) to filter such cases.

Significance and Claims

The paper claims to present the first systematic evidence of a calibration reversal in multi-judge LLM evaluation. Its significance lies in shifting the operating principle for labeled evaluation regimes:

Do not discard weak judges by accuracy alone.
Keep them if they are parseable, non-redundant, and calibratable.
The "weak" signal from a diverse panel, when properly aggregated and calibrated, provides more information about uncertainty and correct answers than a curated subset of "strong" judges.

The authors emphasize that this result holds specifically for labeled benchmarks with held-out calibration data. In open-ended evaluation without ground truth, the ability to learn judge biases and calibrate probabilities is not available, and the curation heuristic may remain valid. The work suggests that in labeled regimes, the budget tradeoff favors retaining a diverse judge panel and investing in calibration data over curating a small, high-accuracy panel.