💬 NLP

DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles

DiscoUQ is a novel framework that enhances uncertainty quantification in multi-agent LLM systems by analyzing the structured linguistic and geometric properties of inter-agent disagreement, achieving superior calibration and performance over traditional voting-based methods.

Original authors: Bo Jiang

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bo Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very tricky riddle. Instead of asking just one person, you ask a team of five experts. They all think about it and give you their answers.

The Old Way (The "Head Count" Method):
In the past, if three experts said "Yes" and two said "No," the system would just say, "Okay, the answer is probably 'Yes' because 3 is bigger than 2." It would give you a confidence score of 60%.

But here's the problem: Not all disagreements are created equal.

Scenario A: The two "No" experts are just guessing and have no evidence. The three "Yes" experts are rock-solid.
Scenario B: The two "No" experts have discovered a brand-new, brilliant piece of evidence that the three "Yes" experts completely missed.

In both cases, the "Head Count" says 60% confidence. But in Scenario B, you should actually be very worried! The old method throws away all the interesting details about why they disagreed.

The New Way (DISCOUQ):
This paper introduces a new system called DISCOUQ. Think of DISCOUQ as a super-smart referee who doesn't just count votes; they watch the whole game to see how the players are arguing.

Instead of just looking at the final score, DISCOUQ looks at two things:

The "Story" of the Argument (Linguistic Structure):
The referee reads the notes the experts wrote.
- Did the minority team bring up new facts? (If yes, be careful!)
- Did they argue logically, or were they just shouting?
- Did they disagree right at the beginning, or only at the very end?
- Did the majority sound super confident, or were they hedging their bets?
The "Shape" of the Thoughts (Embedding Geometry):
Imagine every expert's thought process is a dot on a giant map.
- If the three "Yes" experts are huddled tightly together in one corner, and the two "No" experts are far away in another corner, that's one kind of disagreement.
- If everyone is scattered all over the map, that's a different kind of chaos.
- DISCOUQ measures how "clumped" or "spread out" these dots are to guess how reliable the group is.

How It Works in Practice:
The researchers built three versions of this referee:

The Simple Reader: Reads the arguments and uses a basic math formula to guess the confidence.
The Map Reader: Just looks at the "shape" of the thoughts on the map.
The Super-Brain: A complex computer brain that combines both the story and the map.

The Results:
When they tested this on four different types of hard questions (like science trivia, logic puzzles, and fact-checking), DISCOUQ won big time.

It was more accurate: It knew when the "majority vote" was actually wrong.
It was more honest: The old methods often said "I'm 90% sure!" when they were actually only 60% sure. DISCOUQ's confidence scores matched reality much better.
It was efficient: The best version only needed to ask the computer to read the arguments one extra time for tricky questions. It didn't need to call the computer thousands of times like other methods.

The Big Takeaway:
When a team of AI agents disagrees, the way they disagree tells you a lot more than just who won the vote. By listening to the details of the argument and looking at how their thoughts connect, we can build AI systems that know exactly when to trust themselves and when to say, "I'm not sure, let's double-check."

It's the difference between a teacher who just counts how many students raised their hands, and a teacher who actually listens to the students' explanations to see who really understands the lesson.

1. Problem Statement

Multi-agent Large Language Model (LLM) systems are increasingly used for complex reasoning, where multiple instances of an LLM independently answer a question, and their outputs are aggregated (typically via majority voting).

The Limitation: Current uncertainty quantification (UQ) methods rely on shallow voting statistics (e.g., a 3-to-2 vote yields 60% confidence). This approach discards rich semantic information contained in the agents' reasoning processes.
The Core Issue: Two scenarios with identical vote margins (e.g., 3-to-2) can have vastly different reliability implications. In one, the minority might offer weak arguments based on different evidence; in another, the minority might introduce compelling new information that the majority missed. Standard vote counting treats these identically, leading to poorly calibrated confidence estimates.

2. Methodology: The DISCOUQ Framework

The authors propose DISCOUQ (Disagreement-Structure Confidence for Uncertainty Quantification), a framework that extracts and leverages the internal structure of inter-agent disagreement to produce well-calibrated confidence estimates.

A. System Setup

Agents: A 5-agent system ( $K=5$ ) using role-specialized prompts (e.g., Analytical Reasoner, Devil's Advocate) to encourage diverse reasoning styles.
Base Model: Qwen3.5-27B.
Output: Each agent produces a reasoning chain ( $r_k$ ) and a final answer ( $\hat{y}_k$ ).

B. Feature Extraction

DISCOUQ extracts two complementary families of features from the agents' outputs:

Linguistic Structure Features (LLM-Extracted):
- An LLM (with temperature 0) analyzes the disagreement patterns to produce six scores:
  - Evidence Overlap: Degree to which majority/minority cite the same facts.
  - Minority New Information: Extent of genuinely new arguments by the minority.
  - Minority Argument Strength: Logical soundness of the minority's reasoning.
  - Majority Confidence Language: Certainty expressed (hedging vs. assertive).
  - Reasoning Complexity: Complexity of the overall reasoning.
  - Divergence Depth: At what stage reasoning diverges (early, middle, late).
- Total: 9 features (including vote confidence).
Embedding Geometry Features (Vector-Based):
- Reasoning texts are embedded using a sentence transformer (BGE-large).
- Seven geometric metrics are computed from the cosine-distance structure:
  - Overall dispersion, Majority/Minority cohesion, Cluster distance, Minority outlier degree, Majority centrality, and PCA variance ratio.
- Total: 8 features (including vote confidence).

C. Three Proposed Methods

The authors propose three classifiers of increasing complexity to predict the probability of correctness $P(\hat{y} = y^* | q)$ :

DISCOUQ-LLM (M1): Logistic regression on the 9 linguistic structure features. Highly interpretable.
DISCOUQ-Embed (M2): Logistic regression on the 8 embedding geometry features. Requires zero extra LLM calls beyond the initial agents.
DISCOUQ-Learn (M3): A two-layer MLP combining all 17 features (linguistic + geometric + verbalized confidence).

3. Key Contributions

Formalization of Disagreement Structure: Defined disagreement not just as a count, but as a set of linguistically and geometrically motivated features (evidence overlap, argument strength, embedding dispersion).
Novel Framework: Introduced DISCOUQ, which moves beyond vote counting to analyze the semantic content of disagreement.
Comprehensive Evaluation: Evaluated on four diverse benchmarks (StrategyQA, MMLU, TruthfulQA, ARC-Challenge) against nine baselines, demonstrating superior performance and calibration.

4. Experimental Results

The system was evaluated on 2,750 questions using a 5-agent setup.

Performance (AUROC):
- DISCOUQ-LLM (M1) achieved the highest average AUROC of 0.802, outperforming the strongest baseline, LLM Aggregator (B6), which scored 0.791.
- M1 significantly outperformed baselines on StrategyQA, MMLU, and TruthfulQA.
Calibration (ECE):
- DISCOUQ methods showed drastically better calibration. The average Expected Calibration Error (ECE) dropped from 0.084 (Vote Count baseline) and 0.098 (LLM Aggregator) to 0.036 (M1).
- This represents a 57–63% reduction in calibration error.
Performance by Disagreement Tier:
- Weak Disagreement (Low Vote Confidence): DISCOUQ provided the largest gains here, where simple vote counting fails. For example, on MMLU's weak tier, M1 improved AUROC by 6.5 percentage points over the baseline.
- Unanimous Votes: Even when agents agree (vote confidence = 1.0), DISCOUQ-LLM could still discriminate correct from incorrect answers (AUROC > 0.5) by analyzing verbalized confidence and structure features.
Generalization:
- Features learned on one benchmark generalized to others with near-zero performance degradation (Cross-Benchmark AUROC $\approx$ 0.802).
Cost-Efficiency:
- DISCOUQ-LLM required only 1,624 extra LLM calls (one per non-unanimous question) to achieve the best results.
- DISCOUQ-Embed required zero extra LLM calls (only embedding inference) and still outperformed most baselines.
- In contrast, the LLM Aggregator (B6) required 2,750 extra calls (one per question) but achieved lower AUROC.

5. Significance and Implications

Beyond Voting: The paper demonstrates that the nature of disagreement is more informative than the count of disagreement. Simple logistic regression on structured features captures signals that complex neural aggregators often miss.
Calibration: The framework solves the "overconfidence" problem common in LLMs. By mapping structured disagreement features through a sigmoid function, it produces probabilities that are statistically well-calibrated, enabling reliable selective prediction (abstaining on uncertain questions).
Practical Deployment: DISCOUQ-LLM offers the best trade-off between computational cost and performance. It provides state-of-the-art uncertainty estimation with minimal overhead, making it viable for production systems.
Heterogeneous Teams: Experiments showed that while mixing different models (e.g., Qwen and Llama) reduces the rate of disagreement, the disagreements that do occur are more informative, further validating the utility of analyzing disagreement structure.

In conclusion, DISCOUQ establishes that analyzing the semantic and geometric structure of agent disagreement is a superior method for uncertainty quantification compared to traditional voting or raw LLM confidence scores, offering a robust, calibrated, and cost-effective solution for multi-agent LLM systems.