💬 NLP

Decomposing Wrong-Consensus Agreement in LLM Self-Consistency: A GPT-4.1 Case Study

This paper quantitatively analyzes the erratic performance of LLM self-consistency voting by introducing a "pluralistic agreement index" that decomposes consensus into mechanical preference-driven and residual components, revealing that while shared biases dominate on multiple-choice tasks, unexplained heterogeneity persists on open-domain problems and high agreement does not guarantee correctness.

Original authors: Lizhuo Zhang, Mengmeng Tang, Chenfeng Long, Xiaoyong Tang, Xiang Luo

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lizhuo Zhang, Mengmeng Tang, Chenfeng Long, Xiaoyong Tang, Xiang Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, large language models are often treated as oracle-like machines that can solve complex problems by thinking through them. To make these systems more reliable, researchers developed a technique called self-consistency. The idea is simple: instead of asking the model for a single answer, you ask it the same question many times and let the answers vote. If the model produces the same answer most of the time, the assumption is that this consensus is likely correct. This method has become a standard tool for improving accuracy, operating on the intuitive belief that agreement among many attempts signals truth. However, this belief faces a stubborn reality: sometimes, a model can agree with itself very strongly and still be completely wrong. This happens most often with difficult questions, where the model might confidently settle on a plausible but incorrect answer, creating a false sense of security. Understanding why this happens is crucial, because if we cannot distinguish between a confident correct answer and a confident wrong one, we cannot trust these systems in high-stakes situations.

A recent study by researchers at Hunan Agricultural University and the Yuelushan Laboratory investigates this specific failure mode using a powerful new model, GPT-4.1. Rather than just observing that errors happen, the team designed a way to measure exactly how much of that wrong agreement comes from the nature of the question itself versus a deeper, shared flaw in the model's thinking. They treated the problem like a forensic audit of the model's voting process using public per-run data from a prior release. When the model gets a question wrong, they asked: is the group of wrong answers clustering together simply because there is one very attractive wrong option that everyone picks, or is there something else causing them to agree on the wrong path? To answer this, they created a simulation where they could strip away the model's specific preferences for certain answers and see what agreement remained.

The researchers found that the answer depends heavily on the type of question being asked. When the model faced multiple-choice questions with a fixed set of options, the story was relatively straightforward. In these cases, the strong agreement on a wrong answer was almost entirely explained by the fact that the model had a mechanical preference for a specific, attractive distractor. The model was essentially drawn to the same wrong door every time, and the voting system simply reflected that single attraction. In these scenarios, the "shared bias" that causes the error was captured by the model's individual preference for that specific wrong option. The study showed that for these multiple-choice tasks, the model's internal preference explained between 81% and 93% of the wrong consensus. This suggests that on constrained tasks, the error is not a mysterious, correlated failure of the whole group, but rather a predictable attraction to a specific trap.

However, the picture changed dramatically when the researchers looked at open-ended questions, such as those found in math competitions where there is no list of choices to pick from. Here, the model's internal preference for a specific answer explained only about 59% to 78% of the agreement. A significant portion of the wrong consensus remained unexplained even after accounting for the model's preferences. This leftover agreement suggests that when the model is generating its own answers from scratch, it is not just picking a popular wrong option; the different attempts are somehow coordinating to land on the same error in a way that goes beyond simple preference. The researchers noted that this unexplained agreement was consistent with the idea that the model carries a shared, systematic bias from its training that causes it to make the same mistake repeatedly, even when it tries to think differently.

The study also confirmed that high agreement is not a guarantee of correctness. The researchers looked at the most confident groups of answers—those where the model agreed with itself almost perfectly—and found that even in these cases, the accuracy was far from perfect. On the hardest questions, the most agreed-upon answers were correct only about 42% to 83% of the time. This means that even when the model is extremely consistent, it is still wrong a significant portion of the time. Furthermore, the study showed that on difficult questions, using the voting method could actually make performance worse than just taking a single guess. This "backfire" effect was most pronounced on the hardest problems, where the model's tendency to agree on a wrong answer was strong enough to override the occasional correct guess.

Ultimately, this research reframes how we should view the confidence of an AI. The study demonstrates that agreement is not a certification of truth, but rather a graded signal that must be interpreted with caution. On multiple-choice tests, a wrong consensus is often just the model latching onto a single attractive wrong answer. But on open-ended tasks, a wrong consensus can hide a deeper, unexplained correlation where the model's different attempts reinforce each other's errors. The researchers conclude that while self-consistency is a useful tool, it has a ceiling: it can lift accuracy significantly over a single guess, but it cannot push it to perfection, especially when the questions are hard. The findings suggest that we must stop treating high agreement as a green light for correctness and start treating it as a nuanced piece of evidence that requires further scrutiny, particularly when the model is generating its own answers without a fixed set of choices.

Technical Summary: Decomposing Wrong-Consensus Agreement in LLM Self-Consistency

Problem Statement

Sampling-based reasoning (Self-Consistency), where multiple LLM outputs are generated and aggregated via majority voting, is a standard technique for improving answer accuracy. However, empirical evidence indicates that this heuristic is not uniformly effective: agreement can be high even when the answer is wrong, and on difficult questions, majority voting can sometimes reduce accuracy relative to a single sample (a "backfire" effect).

Existing literature documents these failures but lacks a quantitative, counterfactual decomposition of wrong-consensus agreement. Specifically, it remains unclear how much of the observed agreement among wrong answers is due to:

Mechanical effects: The statistical inevitability of votes concentrating on a single option given a fixed per-case answer preference (e.g., a specific distractor is attractive to the model).
Correlated errors: Shared biases or systematic errors where the model's samples are not independent, causing them to latch onto the same incorrect answer beyond what mechanical preference would predict.

Methodology

The paper introduces a pluralistic agreement index ( $\Gamma$ ) and a counterfactual decomposition framework to distinguish between mechanical agreement and preference-unexplained residuals. The analysis is conducted entirely on existing per-run data (Ding [2]) for the GPT-4.1 family (including mini and nano variants) across two benchmarks: GPQA-Diamond (multiple-choice) and AIME (open-domain).

Key Definitions

Agreement Index ( $\Gamma$ ): Defined as the expected fraction of samples in a wrong run that agree with the plurality label, normalized by a reference scale $d = (1-p)/(C-1)$ , where $p$ is single-sample accuracy and $C$ is the number of options. This measures how "sticky" wrong runs are to a wrong consensus.
Mechanical Counterfactuals: The paper constructs a hierarchy of null models to simulate what agreement would look like if errors were independent but retained specific structural properties:
1. Uniform Null ( $\Gamma_{iid}$ ): Samples are drawn uniformly from wrong options. This represents a baseline where no specific option is preferred.
2. Leak-Free Per-Case Preference Null ( $\Gamma_{rival}$ ): This is the core contribution. For each case, the model's preference distribution over options is estimated using a leave-one-out scheme (using only other runs of the same case). Simulated runs are then drawn from this estimated preference distribution. Crucially, no run predicts its own agreement, preventing data leakage.

Decomposition Metric

The primary metric is Mechanical Coverage ( $\phi$ ):
$\phi = \frac{\Gamma_{rival}}{\Gamma^{(t)}_{emp}}$

$\Gamma^{(t)}_{emp}$ : The empirical agreement index on a held-out set of wrong runs.
$\Gamma_{rival}$ : The agreement index generated by the leak-free per-case preference null.
Interpretation:
- $\phi \approx 1$ : The observed agreement is almost entirely explained by the mechanical attraction to a specific (but potentially wrong) option. The "shared bias" account over-claims here.
- $\phi \ll 1$ : A significant preference-unexplained residual ( $\delta = 1 - \phi$ ) remains. This suggests that even after accounting for per-case preferences, the samples cluster on the wrong consensus more than independence would predict, consistent with correlated errors or shared training biases.

Key Results

1. Benchmark-Associated Direction in Mechanical Coverage

The decomposition reveals a distinct difference between constrained multiple-choice and open-domain tasks:

GPQA-Diamond (Multiple-Choice): The mechanical per-case preference explains 81–93% of the observed wrong-consensus agreement ( $\phi \in [0.806, 0.927]$ ). The residual is small. This suggests that on multiple-choice tasks, the "shared bias" often manifests as the model collectively latching onto a specific, attractive distractor, which is captured by the per-case preference channel.
AIME (Open-Domain): The mechanical preference explains only 59–78% of the agreement ( $\phi \in [0.586, 0.781]$ ). A substantial residual of 1.56–2.80 $\Gamma$ units survives. This residual is consistent with run-to-run preference variation (modeled via a Dirichlet-multinomial null) and suggests that on open-domain tasks, agreement involves a component not explained by fixed per-case preferences.

2. Reproduction of the "Backfire" Effect

The study reproduces the finding that majority voting can backfire on hard questions. In difficulty-binned analysis, the voting gap (consensus accuracy minus single-sample accuracy) becomes negative on the hardest bins (e.g., $-0.09$ for GPT-4.1-nano on GPQA).

3. Low Empirical Ceiling for High Agreement

High self-consistency is informative but not certifying.

The accuracy of the "highest-agreement bin" (top quintile of self-consistency) ranges from 0.42 to 0.83.
While this represents a 1.2–3.6 $\times$ lift over the base rate, it is far from 1.0. Agreement is graded evidence, not certification.

4. Champion Fragility

Even when two independent runs of the same case produce a high-agreement winner, the "champion" (plurality winner) is unstable.

Flip Rate: On AIME, the flip rate is 0.82; on GPQA, it is 0.40.
Even at very high self-consistency ( $\alpha \ge 0.98$ ), the flip rate remains 0.17 on GPQA and 0.78 on AIME.

5. Soft vs. Hard Agreement

Replacing hard majority voting with soft, probability-weighted aggregation (using the Jensen gap) yields no appreciable improvement in AUROC for predicting correctness. The "soft" agreement adds no signal over the "hard" plurality concentration. Furthermore, the Jensen gap is larger for wrong runs than correct ones, indicating that wrong runs have more dispersed distributions.

Significance and Claims

The paper positions itself as a descriptive, understanding contribution rather than a proposal for a new voting method or a discovery of new failure modes. Its significance lies in:

Quantitative Decomposition: It provides the first counterfactual hierarchy to decompose wrong-consensus agreement into mechanical per-case preferences and preference-unexplained residuals.
Reframing "Shared Bias": It challenges the blanket assumption that high agreement on wrong answers is solely due to correlated errors. On multiple-choice tasks, much of this agreement is mechanical (captured by per-case preferences). On open-domain tasks, a robust residual remains, consistent with systematic error patterns.
Model-Agnostic Framework: The decomposition is defined entirely in terms of observed run data and counterfactual simulations, making it applicable to any model where per-run data is available.
Empirical Boundaries: It establishes that self-consistency confidence is graded evidence, not certification, with empirical ceilings well below perfect accuracy even for high-agreement bins.

The authors explicitly state that the mechanism of the residual (e.g., whether it is strictly "shared training bias" or run-level heterogeneity) is not identified, only that a per-case preference-only model is insufficient to explain open-domain wrong-consensus agreement. The results are bounded by the i.i.d. assumption in the counterfactuals, though the direction of the findings (high mechanical coverage on GPQA, substantial residual on AIME) is robust to sampling correlation.