💻 computer science

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

This paper challenges the reliability of using self-consistency and cross-model agreement as confidence signals for Large Language Models, demonstrating through a large-scale study that while agreement is a positive but weak predictor of correctness, it often leads to over-confidence in frontier models where high agreement coincides with frequent errors.

Original authors: Kaihua Ding

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaihua Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive talent show to find the best AI brain. You have a panel of judges (the AI models) and you ask them a million tricky questions. The big rule everyone follows is: "If all the judges agree on an answer, it must be right." It feels like common sense, right? If ten people all say the sky is green, maybe we're all seeing a weird filter? But if a hundred people say it, it's probably true!

This paper is like a detective story that goes behind the scenes to check if that rule actually works. The author, Kaihua Ding, ran a huge experiment with 53 different "runners" (people running the AI tests) and asked them to generate 265,000 answers to some of the hardest math and science questions available (called GPQA Diamond and AIME).

Here is the twist the paper uncovers: Agreement is not the same thing as being correct.

The "Echo Chamber" Trap

Think of the AI models like a group of students who all studied from the same slightly flawed textbook. If they all memorized the same wrong answer to a question, they will all raise their hands and agree on that wrong answer. They aren't agreeing because they are smart; they are agreeing because they share the same bias or a "memorized heuristic."

The paper found that when AI models agree with themselves (or each other), it's often just them echoing a shared mistake, not a sign of truth.

The "Over-Confident" Star Student

The most surprising discovery involves the "frontier" models—the super-smart, most advanced AI versions. You might think the smartest model would be the most reliable. The paper shows the opposite is true in this specific setup.

The Finding: The most advanced model (gpt-4.1) agreed with itself on 89% of the time (a very high confidence score). But when it agreed, it was only right about 48% of the time.
The Analogy: Imagine a student who is so sure of their answer that they shout it out with 100% confidence, but they are actually wrong half the time. The paper calls this "over-confident." In fact, this super-smart model was worse at signaling when it was right than a slightly less advanced "mid-tier" model.
The Proof: The study measured this using a statistical link (called a correlation) between "agreement" and "correctness." For the smartest model, this link was very weak (around 0.20), meaning its high confidence was almost useless as a trust signal.

Does "Chain-of-Thought" Help?

Some people think if you ask the AI to "show its work" (a method called Chain-of-Thought), it will be more honest.

The Result: Yes, asking the AI to show its work did make it slightly more accurate (it got more questions right).
The Catch: However, it didn't make the "agreement signal" much better. The AI still couldn't reliably tell you when it was right just by how much it agreed with itself. It's like a student who writes out a long, detailed explanation but still gets the final answer wrong, and you can't tell the difference just by looking at how neatly they wrote it.

The "Shared Bias" Mystery

The researchers also checked if different AI families (like GPT and Claude) made the same mistakes.

The Discovery: They found that when different AI models got a question wrong, they often picked the exact same wrong answer.
The Analogy: It's like two different students from different schools both getting the same math problem wrong because they both learned a specific, wrong trick in class. This suggests that when AIs agree on a wrong answer, it's not a lucky coincidence; it's a shared bias baked into their training.

So, Should We Stop Using Agreement?

Not exactly. The paper doesn't say "never trust agreement." It says, "Use it carefully, and know your limits."

Good for Budgeting: Agreement is great for deciding how much computer power to spend. If the models are all unsure (low agreement), you might want to spend more time or money to get a better answer.
Bad for Trusting: You should never use agreement as a standalone "trust me" button. If a super-smart model says, "I am 90% sure," and it agrees with itself, you still can't be sure it's right. In fact, the paper suggests that routing hard questions to the "smartest" model based on confidence is a bad idea because that model is the most over-confident.

The Bottom Line

The paper concludes that self-consistency (agreement) is a "conditional proxy." It's a helpful hint in some situations (like with mid-level models), but it is not a guarantee of truth.

What is proven: In these specific tests, high agreement often meant high confidence but low accuracy, especially for the most advanced models.
What is suggested: The "over-confidence" might be a side effect of how these models are trained, and shared errors happen because different models learn from similar data.
What is unknown: The paper notes that these results are specific to the models and questions tested. We don't know if this happens with every type of AI or every kind of question, but the warning is clear: Don't assume that just because everyone agrees, they are right. Sometimes, they are just all looking at the same wrong map.

Technical Summary: When LLMs Agree, Are They Right?

Problem Statement

Large Language Models (LLMs) are increasingly deployed as "LLM-as-judge" systems, often in ensembles or "mixture-of-experts" panels, to evaluate AI outputs in enterprise pipelines. These systems operate on a pervasive intuition: that agreement (either among multiple judges or via self-consistency across stochastic samples from a single model) signals correctness. Consequently, agreement is used as a confidence proxy for cost-based routing, adaptive sample-budget allocation, and selective prediction (abstention).

However, the paper argues that this assumption is unreliable. Agreement is not synonymous with accuracy. A model may agree with itself or other models due to shared biases, memorized heuristics, option-position priors, or systematic hallucinations rather than truth. The central question is: Under what conditions does the agreement signal function as a usable proxy for correctness, and when does it fail?

Methodology

The authors conduct a large-scale, cross-runner audit to separate recurring confident errors from stochastic noise.

Data Provenance: The study analyzes 265,000 samples generated by 53 runners (graduate course participants) who submitted model outputs. While runners shared a course context, their runs were treated as independent submissions. The dataset covers 394 unique cases across GPQA Diamond (graduate-level 4-option multiple choice) and AIME (competition mathematics).
Experimental Design:
- Models: Experiments focus on the gpt-4.1 family (nano, mini, and frontier gpt-4.1) and an exploratory check on three tiers of Anthropic's Claude (haiku, sonnet, opus).
- Axes of Comparison:
  - Axis A (Tier): Nano vs. Mini.
  - Axis B (Prompting): Zero-shot vs. Chain-of-Thought (CoT) on the Mini model.
  - Axis C (Scale): Mini vs. Frontier gpt-4.1.
- Parameters: Temperature = 1.0, $K=50$ samples per case.
Metrics:
- Self-Consistency ( $C$ ): The proportion of samples agreeing with the majority answer ( $n_{maj}/K$ ).
- Majority-Correctness ( $M$ ): A binary indicator if the majority answer matches the ground truth.
- Sample Accuracy ( $A$ ): The proportion of individual samples matching ground truth.
- Correlation: Spearman's $\rho$ between $C$ and $M$ (and $A$ ).
Statistical Inference: Due to non-i.i.d. data (runners contribute multiple rows; cases are shared), the authors use a hierarchical runner-clustered bootstrap (resampling runners, then cases within runners) and a case-clustered bootstrap for robustness.

Key Contributions

Large-Scale Audit: A 265,000-sample audit of self-consistency as a confidence signal on hard reasoning benchmarks, utilizing hierarchical bootstrapping to control for runner and case dependencies.
Frontier Degradation Evidence: Demonstration that the agreement signal degrades on the most consistent frontier models. These models exhibit over-confidence (high $C$ ) without corresponding gains in accuracy, making confidence-based routing toward them counterproductive.
Recurrence Analysis: A separation of stochastic errors from recurring confident-wrongness. The study shows that many confident errors are not sampling noise but recur across different runners and prompts.
Positional Bias Control: An option-shuffle experiment revealing that a significant portion of "confidence" in multiple-choice settings is driven by option-position priors (e.g., avoiding option D) rather than semantic understanding.
Cross-Family Check: An exploratory analysis showing that confident errors are partly shared across providers (GPT and Claude), suggesting that cross-model agreement may reflect shared pretraining biases rather than independent verification.
Dataset Release: Public release of a de-identified, schema-validated dataset and analysis pipeline.

Results

1. Agreement is a Positive but Weak Predictor

Across all 12 experimental cells (combinations of model, prompt, and benchmark), the correlation between self-consistency ( $C$ ) and majority-correctness ( $M$ ) is positive but weak ( $\rho \approx 0.20–0.59$ ).

No cell exceeds $\rho = 0.60$ .
A "high-agreement auto-accept" rule still admits a substantial number of wrong answers.
The signal is regime-dependent: it is stronger for mid-tier models and weaker for frontier models.

2. The Frontier Model Paradox

The frontier model (gpt-4.1) exhibits the highest self-consistency ( $C=0.89$ on GPQA) but the lowest correlation between agreement and correctness ( $\rho(C, M) = 0.20$ ).

Over-confidence: 77% of GPQA cases for the frontier model have $C \ge 0.8$ , yet 48% of these high-agreement answers are wrong.
Calibration: The frontier model has the worst Expected Calibration Error (ECE = 0.41) among all tested configurations.
Implication: Routing hard cases to the frontier model based on high agreement is counterproductive, as it inherits a high error rate.

3. Chain-of-Thought (CoT) Effects

Accuracy: CoT robustly improves accuracy ( $\Delta A \approx +0.067$ on GPQA).
Signal: CoT provides only a marginal improvement to the agreement–correctness signal ( $\rho(C, M)$ ). While it improves the correlation with sample accuracy ( $A$ ), its effect on the deployment label ( $M$ ) is borderline or null depending on the resampling method.

4. Recurring and Shared Errors

Recurrence: A non-trivial subset of confident errors (e.g., 28% of unique GPQA cases for gpt-4.1) are confidently wrong ( $C \ge 0.8$ ) for every runner, indicating these are systematic biases rather than sampling noise.
Positional Bias: In GPQA, shuffling options significantly reduces accuracy and alters majority-answer distributions, confirming that "confidence" often reflects position priors (e.g., under-selecting "D") rather than truth.
Cross-Provider Sharing: On shared GPQA items, GPT and Claude models frequently agree on the same wrong answers at high confidence levels. This rate exceeds a label-permutation null, suggesting shared biases across providers.

5. System Implications

Adaptive Sampling: Agreement is useful for allocating compute. An adaptive rule using agreement to stop sampling early saves ~60% of samples while maintaining error rates.
Routing Policies: A confidence-routed cascade (cheap $\to$ expensive) performs better than random routing but is dominated by simply always using the mid-tier model. Routing to the over-confident frontier model degrades performance.

Significance and Claims

The paper concludes that self-consistency is not accuracy, but rather a conditional, regime-dependent empirical proxy for it.

Usable Regimes: Agreement is a useful, coarse signal for allocating compute (adaptive sampling) and is most reliable for unsaturated mid-tier models.
Unusable Regimes: Agreement is a poor standalone confidence score, particularly for frontier models where high agreement is decoupled from correctness due to over-confidence and shared biases.
Safety Implication: Systems that treat self-consistency as a definitive confidence threshold risk routing confidently-wrong answers into trusted paths. The paper cautions against using agreement as a reason to escalate hard cases to the most consistent frontier models.

The authors emphasize modesty in their claims: the study is an audit of specific benchmarks (GPQA, AIME) and model families, and the findings highlight the need to treat agreement as a "coarse, cost-saving signal" rather than a trustworthy threshold for correctness.