💬 NLP

When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs

This paper introduces the Accuracy-Controlled Evaluation (ACE) framework to demonstrate that traditional global calibration metrics are confounded by accuracy differences, often leading to frequent ranking reversals, and argues that fair cross-model comparisons of Large Language Models require accuracy-aware evaluation methods.

Original authors: Zhichao Yang, Caiqi Zhang, Ruihan Yang, Chengzu Li, Nigel Collier, Deqing Yang

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhichao Yang, Caiqi Zhang, Ruihan Yang, Chengzu Li, Nigel Collier, Deqing Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Smart Student" Trap

Imagine you are trying to judge how well two students, Alex and Jamie, understand a subject. You want to know not just what they got right, but how confident they were when they answered.

In the world of AI (Large Language Models), this "confidence" is called calibration. A well-calibrated model is like a student who says, "I'm 90% sure this is right," and is actually right 90% of the time. A poorly calibrated model might say, "I'm 90% sure," but only be right 50% of the time.

For a long time, researchers have used a single score (like a report card grade) to compare these models. The rule was simple: The lower the score, the better the calibration.

The Paper's Discovery:
The authors found a major flaw in this rule. They realized that being smarter (more accurate) automatically makes your "confidence score" look better, even if your confidence is actually just as bad as the other student's.

The Analogy:
Imagine Alex gets 90% of the answers right, and Jamie gets 50% right.

Alex says, "I'm 100% sure" on every answer. Since Alex is right 90% of the time, their "confidence score" looks amazing.
Jamie also says, "I'm 100% sure" on every answer. But since Jamie is only right 50% of the time, their score looks terrible.

If you just look at the scores, you'd think Alex is a genius at judging their own knowledge. But in reality, both students are equally overconfident. They both said "100%" when they shouldn't have. The only reason Alex looks better is that Alex knew more facts.

The paper calls this a "confounder." The raw score confuses "being smart" with "being honest about your confidence."

The Solution: The "Fair Play" Framework (ACE)

To fix this, the authors created a new way to compare models called ACE (Accuracy-Controlled Evaluation). Instead of letting the "smartest" model win automatically, ACE forces the models to play on a level playing field.

They use three different "lenses" or views to look at the data:

The "Same Outcome" Lens (Instance-Aligned):
- How it works: We only compare the models on the specific questions where both got the answer right, or both got it wrong.
- The Analogy: Imagine a debate where we only look at the arguments where both debaters agreed on the facts. If Alex still sounds more confident than Jamie on the questions they both got wrong, then Alex is genuinely more overconfident, not just smarter.
The "Same Mix" Lens (Distribution-Aligned):
- How it works: We take the smart model's answers and mathematically "dial down" their success rate to match the less smart model. We pretend the smart model got the same number of questions right as the other one.
- The Analogy: It's like taking a basketball player who makes 90% of their shots and pretending they only made 50%. Now, if they still claim to be "90% sure" they will make the next shot, we know they are lying about their confidence, regardless of their actual skill.
The "Same Options" Lens (Candidate-Aligned):
- How it works: Instead of letting each model generate its own answer, we give them both the same list of possible answers and ask them to pick the best one.
- The Analogy: Instead of asking two chefs to cook their own dishes, we give them the exact same ingredients and ask them to rate how good their dish will be. This stops them from just saying, "My dish is great because I picked easy ingredients."

The Shocking Result: Rankings Flip!

When the authors applied this new "Fair Play" framework, something surprising happened: The rankings often reversed.

Before (Raw Scores): The bigger, smarter models (like the 72B parameter model) usually looked like they had perfect confidence.
After (ACE): Once you controlled for their higher accuracy, many of them actually looked worse at judging their own confidence than the smaller models.

The Takeaway:
The paper argues that we have been fooled. We thought bigger models were better at "knowing what they don't know." But often, they were just better at knowing the answers. When you strip away the advantage of being smarter, their confidence isn't actually that impressive.

Summary in One Sentence

The paper proves that comparing AI confidence using standard scores is unfair because smarter models get a "free pass" on their scores, and when you fix this unfairness, the models that looked best often turn out to be the worst at judging their own confidence.

Technical Summary: When Calibration Rankings Reverse

Problem Statement

The reliable deployment of Large Language Models (LLMs) in high-stakes domains requires not only high accuracy but also well-calibrated confidence estimates, where a model's predicted confidence aligns with its empirical correctness. Currently, the community relies heavily on global calibration metrics, such as Expected Calibration Error (ECE) and Brier Score (BS), to compare the calibration quality of different models.

The paper identifies a critical flaw in this practice: raw global calibration metrics are confounded by differences in model task accuracy. When comparing two models with different accuracy levels, a model with higher accuracy can achieve a better global calibration score simply because it makes fewer mistakes, even if its underlying confidence behavior (uncertainty estimation) is identical or worse than a less accurate model. This leads to misleading conclusions where "stronger" models appear better calibrated solely due to their superior task performance. The authors term the resulting phenomenon ranking reversal, where a model favored by raw metrics ceases to be favored once accuracy is controlled.

Methodology: ACE Framework

To address this confound, the authors propose ACE (Accuracy-Controlled Evaluation), a framework designed to disentangle calibration quality from task accuracy. ACE evaluates models under matched or partially matched correctness conditions before drawing calibration conclusions. The framework consists of three complementary views:

Instance-Aligned (IA) Calibration:
- Mechanism: Compares models only on examples where they share the same outcome (both correct or both wrong).
- Goal: Provides strict instance-level control. It isolates calibration differences to confidence assignment rather than task success.
- Decomposition: It separates shared competence ( $D_{both-right}$ ) from shared blind spots ( $D_{both-wrong}$ ), revealing if a stronger model is overconfident on questions neither model understands.
Distribution-Aligned (DA) Calibration:
- Mechanism: Matches the aggregate outcome composition of the compared models to a common effective accuracy level. Instead of discarding data (as in simple resampling), it uses reweighting (inspired by propensity score weighting in causal inference) to adjust the contribution of correct and incorrect predictions.
- Goal: Enables broader distribution-level comparison by ensuring the weighted proportion of correct/incorrect predictions is identical across models, effectively simulating a scenario where both models operate at the same accuracy level.
Candidate-Aligned (CA) Calibration:
- Mechanism: Designed for evaluative settings where models judge a shared pool of candidate answers (e.g., $U_i = \{s^{(A)}_i, s^{(B)}_i\}$ ). Both models assign confidence to every candidate in the shared pool.
- Goal: Removes asymmetry in the generation space and isolates calibration in "evaluation mode." It also exposes self-preference bias, where a model might assign higher confidence to its own answer regardless of its correctness relative to a competitor.

Key Contributions

Theoretical and Empirical Demonstration of Confounding: The authors prove analytically (via limit cases) and empirically (via strong negative correlations between accuracy gaps and raw ECE gaps) that global metrics like ECE and BS are linearly dependent on accuracy. Higher accuracy mechanically improves these scores even if confidence behavior is unchanged.
Identification of Ranking Reversal: The paper documents that ranking reversals are frequent. Models that appear better calibrated under raw metrics often become less calibrated when evaluated under ACE. This phenomenon is observed across different model families, dataset types, and confidence elicitation methods.
The ACE Framework: The proposal of a three-view evaluation framework (IA, DA, CA) that explicitly controls for accuracy to enable fair cross-model comparison.
Empirical Re-evaluation of Model Advantages: The study re-evaluates common comparison axes (Small vs. Large models; Thinking vs. Non-thinking models) using ACE, finding that many reported calibration advantages weaken substantially or disappear entirely once accuracy is controlled.

Key Results

The authors evaluated 14 models across five benchmarks (FreshQA, TriviaQA, GPQA Diamond, LiveBench, MMLU-Pro) using three confidence elicitation methods (Verbalized, P(True), Self-Consistency).

Prevalence of Ranking Reversal: Ranking reversals occur frequently (ranging from 13% to 65% depending on the setting and metric). In many cases, the model favored by raw ECE/BS is no longer favored after accuracy control.
Scale-Based Comparisons: The apparent calibration advantage of larger models (e.g., 72B vs. 7B) weakens markedly under ACE. For instance, under verbalized confidence, reversal rates reached 60% in Distribution-Aligned settings. This suggests that the raw advantage is largely attributable to higher task accuracy rather than superior confidence quality.
Thinking vs. Non-Thinking Comparisons: Similar patterns emerge when comparing reasoning-enabled modes against non-reasoning modes. Reversal rates were as high as 65% under Candidate-Aligned settings with Self-Consistency. This indicates that raw calibration gains attributed to "thinking" are often mediated by improved answer accuracy rather than a pure metacognitive improvement in confidence estimation.
View Consistency: DA and IA views tend to agree on reversal status (79.2% agreement), suggesting they capture related corrections to raw rankings. CA often acts as a separate axis, particularly sensitive to candidate-pool alignment and self-preference biases.
Dataset and Method Dependence: The stability of raw rankings varies by dataset structure (e.g., MMLU-Pro showed high stability, while FreshQA was dominated by CA reversals) and confidence elicitation method (e.g., Self-Consistency was highly sensitive to CA alignment).

Significance and Claims

The paper claims that raw global calibration metrics are not robust for cross-model comparison when models differ in accuracy. The primary contribution is not the assertion that larger or thinking models are poorly calibrated, but rather that standard evaluation practices can overstate the metacognitive advantage of stronger models by conflating confidence quality with outcome composition.

The authors argue that for fair cross-model comparison, accuracy control is not a minor refinement but a necessary condition that can fundamentally change the conclusion about which model is better calibrated. They position ACE as a useful companion to conventional calibration reporting to ensure that assessments of LLM confidence quality reflect genuine uncertainty estimation capabilities rather than mere task success. The paper concludes that fair calibration comparison requires accuracy-aware evaluation to avoid misleading diagnostic conclusions.