🤖 machine learning

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

This paper introduces Conformal Elo, a low-cost evaluation framework that enhances LLM ranking accuracy by propagating calibrated win probabilities to estimate local uncertainty and applying split conformal prediction to bound global disagreement with human judgments, thereby delivering reliable Elo estimates without large-scale human annotation.

Original authors: Bora Kargi, David Salinas

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bora Kargi, David Salinas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to rank the best chefs in the world. Usually, you'd ask a panel of human food critics to taste every dish and vote on who wins. But asking thousands of humans to do this is expensive and slow. So, instead, you hire a "Robot Judge" (a Large Language Model) to taste the dishes and decide the winners.

The problem? Robot Judges are quirky. They might prefer the chef who speaks first, the one who writes longer reviews, or even chefs who sound like the Robot itself. If you just ask the Robot, "Did Chef A beat Chef B?" and it says "Yes," you lose a lot of information. You don't know how much better Chef A was. Was it a landslide victory, or was it a tiny, shaky edge?

This paper introduces a new method called Soft-Elo to fix this. It's like upgrading from a simple "Win/Loss" scoreboard to a high-tech "Confidence Meter."

Here is how it works, broken down into two simple steps:

1. The Local Fix: From "Hard" Labels to "Soft" Probabilities

The Old Way (Hard-Elo):
Imagine the Robot Judge sees two dishes. It gives them scores: Dish A gets an 8, Dish B gets a 2. The old system just looks at this and says, "A wins!" It treats this exactly the same as a battle where Dish A got a 9 and Dish B got an 8. In both cases, the system records a simple "1" for a win.

The Flaw: This throws away the nuance. The Robot knows the first battle was a blowout, but the system treats it the same as a close call. This makes the final rankings look "stretched out" and inaccurate compared to what humans would actually think.

The New Way (Soft-Elo):
Instead of forcing the Robot to pick a winner, Soft-Elo asks: "How confident are you?"

If the Robot gives an 8 vs. 2, it calculates a 94% chance that A is better.
If the Robot gives a 9 vs. 8, it calculates a 52% chance that A is better.
The Magic: The system feeds these percentages (probabilities) into the ranking math instead of just "Win/Loss." This tells the math, "Hey, this win was a huge deal," or "This one was basically a tie."
The Result: The final rankings become much more accurate. The paper shows this reduces the error in the rankings by about 70%, bringing the Robot's scores much closer to what human judges would say.

2. The Global Fix: Adding a "Safety Net"

Even with the better "Soft" method, the Robot Judge isn't perfect. There is still a small gap between what the Robot thinks and what humans think. Sometimes the Robot is consistently too harsh on new chefs or too nice to old ones.

The Old Way:
You'd just give the Robot's ranking and hope for the best. If you tried to guess how far off it might be, your guess would be a huge, useless range (like saying, "The chef is between 10th and 100th place").

The New Way (Conformal Prediction):
The authors add a "Safety Net" using a statistical trick called Split Conformal Prediction.

Think of it like a weather forecast. Instead of just saying "It will rain," a good forecast says, "There is a 90% chance it will rain between 2 PM and 4 PM."
The authors look at how the Robot did on a bunch of known chefs (the calibration group). They see the pattern of errors.
When a new chef is tested, the system doesn't just give a single number. It gives a range (e.g., "This chef is likely between 1400 and 1450 Elo").
The Result: Because the "Soft" method made the initial ranking so much better, this safety net is now narrow and useful (about 60% narrower than before). It gives developers a honest, tight estimate of where the model stands without needing to pay humans to check it.

The Big Picture

The paper argues that we shouldn't just ask an AI Judge, "Who won?" We should ask, "How sure are you?" and then use that confidence level to build a better ranking.

By doing this, they created a tool that:

Saves Money: It doesn't need thousands of human votes to get a good ranking.
Is More Accurate: The rankings are much closer to human reality.
Is Honest: It tells you exactly how much you can trust the result with a "confidence interval" (a safety range).

In short, they turned a rigid, error-prone robot judge into a flexible, self-aware one that knows when it's guessing and when it's sure, giving us a much clearer picture of who the best AI models really are.

Technical Summary: From Uncertain Judgments to Calibrated Rankings

Problem Statement

Evaluating Large Language Models (LLMs) typically relies on costly human annotation campaigns. While "LLM-as-a-judge" offers a scalable, low-cost alternative, it introduces systematic errors that miscalibrate resulting rankings. These errors include position bias, length bias, and self-preference. Furthermore, standard evaluation pipelines often collapse nuanced judge scores into hard binary labels (win/loss/tie), discarding information regarding the strength of preference.

The paper identifies two specific failure modes in current "Hard-Elo" pipelines:

Scale Distortion: While judges often recover the correct ranking of models, the resulting Elo distances are mis-scaled. Judges tend to be overconfident, producing Elo gaps that do not align with human-derived gaps.
Wide Uncertainty Intervals: When attempting to quantify uncertainty using conformal prediction on these mis-scaled residuals, the resulting confidence intervals are excessively wide (spanning 132–261 Elo points), rendering them operationally useless for distinguishing model performance.

The core issue is that Hard-Elo treats a decisive victory (large score difference) and a close contest (small score difference) identically, ignoring the local uncertainty encoded in the judge's score gap.

Methodology: Soft-Elo

The authors propose Soft-Elo, a two-layer framework that improves Elo estimation by preserving uncertainty information at the local level and calibrating it at the global level.

1. Local Level: Calibrated Win Probabilities

Instead of converting judge scores into hard labels ( $y \in \{0, 0.5, 1\}$ ) for the Bradley-Terry (BT) model, the authors propagate the raw score differences ( $s$ ) into calibrated win probabilities.

Score-to-Probability Mapping: The raw score difference $s(x)$ is mapped to a soft target $\tilde{y}(x)$ using a logistic function with a learned temperature parameter $\beta$ :
$\tilde{y}(x) = P[B \prec A | x] = \sigma(\beta s(x))$
Calibration: The parameter $\beta$ is estimated via Maximum Likelihood Estimation (MLE) on a calibration set of battles (excluding the held-out model) to ensure the predicted probabilities match the observed human agreement rates.
Integration: This soft target replaces the hard label in the standard BT log-likelihood objective. The BT model structure and Elo conversion remain unchanged; only the training targets are "softened" to reflect the judge's confidence.

2. Global Level: Split Conformal Prediction

Even with local calibration, a residual gap often remains between LLM-derived and human-derived Elo ratings. To quantify this irreducible discrepancy and provide honest uncertainty bounds:

Residual Modeling: The authors compute the signed residual $\epsilon_i = \text{Elo}_{\text{LLM}, i} - \text{Elo}_{\text{Human}, i}$ for a set of calibration models.
Normalized Nonconformity: To account for varying sample sizes and noise, residuals are normalized by the bootstrap standard error ( $\hat{SE}_i$ ) of the LLM Elo estimate: $S_i = |\epsilon_i| / \hat{SE}_i$ .
Interval Construction: Using split conformal prediction, a quantile $\hat{q}$ is determined from the calibration set. For a new model, the final human Elo estimate is provided as a conformal interval:
$C = [\text{Elo}_{\text{LLM}, n+1} - \hat{q}\hat{SE}_{n+1}, \text{Elo}_{\text{LLM}, n+1} + \hat{q}\hat{SE}_{n+1}]$
This provides distribution-free marginal coverage guarantees for new models drawn from the same population.

Key Contributions

Local Uncertainty Handling: Demonstrated that regressing calibrated win probabilities from raw judge scores (rather than collapsing them to binary labels) drastically improves Elo estimation accuracy and reduces structured residuals.
Global Uncertainty Quantification: Applied split conformal prediction to LLM-human Elo residuals, generating prediction intervals with distribution-free marginal coverage guarantees.
Low-Cost Evaluation Pipeline: Established a protocol that estimates Elo ratings and uncertainty bounds without requiring human annotations for the model under test, relying instead on battles against diverse opponents.
Benchmarking: Validated the approach on LMArena (55 held-out models), showing that Soft-Elo brings LLM-derived ratings within 17.9 Elo Mean Absolute Error (MAE) of human-derived ratings.

Results

Accuracy Improvement: Across eight different judge models, Soft-Elo reduced the held-out Elo MAE by 39% to 73%. The average MAE dropped from 45.9 Elo (Hard-Elo) to 17.9 Elo (Soft-Elo).
Ranking Stability: The method improved scale fidelity without altering the model rankings; changes in Spearman rank correlation were negligible ( $\Delta\rho \in [-0.011, +0.014]$ ).
Interval Narrowing: By reducing the structured residuals fed into the conformal step, Soft-Elo reduced the median width of the 90% conformal intervals by 39% to 70%. For example, on the DeepSeek-V3.2 judge, the median interval width decreased from 261 Elo (Hard-Elo) to 78 Elo (Soft-Elo), while maintaining empirical coverage near the 90% target.
Sample Efficiency: Soft-Elo outperformed Hard-Elo across all annotation budgets, with the performance gap being most significant in low-data regimes.

Significance and Claims

The paper claims that Soft-Elo provides a low-cost evaluation tool that offers developers calibrated Elo estimates and "honest uncertainty bounds" without the need for large-scale human annotation campaigns for the target model.

The significance lies in shifting the focus from merely reporting a point estimate (which can be biased) to providing a calibrated interval that accounts for both local judge uncertainty and global systematic disagreement. The authors emphasize that this approach targets the specific quantity leaderboard users inspect: distances on the human Elo scale. By surfacing calibrated uncertainty at the model level, the method aims to prevent the misinterpretation of model rankings caused by judge biases and fragility.

Limitations Acknowledged:

The method only models uncertainty encoded in the judge's score difference; it does not account for epistemic uncertainty, hallucinated scores, or prompt ambiguity.
The conformal guarantee relies on the exchangeability of calibration and test models. Significant shifts in prompt distribution, model families, or temporal changes could violate this condition, requiring re-calibration.