💬 NLP

Council Mode: Mitigating Hallucination and Bias in LLMs via Multi-Agent Consensus

This paper introduces "Council Mode," a novel multi-agent consensus framework that mitigates hallucinations and biases in Large Language Models by routing queries through a triage classifier, generating parallel responses from diverse frontier models, and synthesizing them via a structured consensus mechanism, resulting in significant improvements in factual accuracy and reduced bias variance.

Original authors: Shuai Wu, Xue Li, Yanna Feng, Yufang Li, Zhijun Wang

Published 2026-04-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuai Wu, Xue Li, Yanna Feng, Yufang Li, Zhijun Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Council Mode": How to Stop AI from Lying (Without Getting a PhD)

Imagine you ask a super-smart AI a tricky question, like "Who invented the lightbulb?" or "What's the best treatment for a broken leg?" Sometimes, the AI gives you a confident, smooth-sounding answer that is completely made up. This is called a hallucination. It's like a student who didn't study for the test but writes a very convincing essay anyway.

This paper introduces a clever new system called Council Mode. Instead of asking just one AI for the answer, it asks a whole "council" of different AIs to debate it out, then picks the best version.

Here is how it works, broken down into simple steps with some fun analogies.

1. The Problem: The "Single Expert" Trap

Think of a single Large Language Model (LLM) like a genius but slightly arrogant professor.

They know a lot, but they have blind spots.
If they don't know the answer, they might just make something up to save face because they are so confident.
If they have a bias (e.g., they only read books from one country), their answers will be skewed.

The paper argues that relying on just one "professor" is risky.

2. The Solution: The "Council of Experts"

The Council Mode is like hiring a panel of judges instead of just one.

The Setup: You don't ask one AI. You ask three different AIs at the same time (e.g., one from OpenAI, one from Google, one from Anthropic).
The Analogy: Imagine you are trying to solve a mystery.
- AI #1 is a detective who loves technology.
- AI #2 is a historian who loves old books.
- AI #3 is a scientist who loves lab reports.
- If you ask them all, "Who invented the lightbulb?", they might all agree. But if they disagree, the system knows to be careful.

3. The Three-Step Process

Step 1: The Bouncer (Intelligent Triage)

Before the council even meets, a quick "bouncer" checks your question.

The Analogy: If you ask, "What's 2+2?", the bouncer says, "No need to call the whole council! I'll just tell you it's 4."
This saves time and money. Only the hard, tricky questions get sent to the full council.

Step 2: The Parallel Debate (Expert Generation)

For the hard questions, the system sends the prompt to three different AIs simultaneously.

The Analogy: It's like sending three different journalists to the same crime scene. They write their own reports independently.
Because they are different "species" of AI (trained on different data), they make different mistakes. If AI #1 makes a mistake, AI #2 might get it right.

Step 3: The Editor (Consensus Synthesis)

This is the magic part. A fourth AI (the "Editor") reads all three reports and writes the final story.

The Analogy: Imagine the Editor is a wise old librarian.
- Agreement: If all three say "The sky is blue," the Editor writes, "The sky is blue."
- Disagreement: If AI #1 says "The sky is green" and AI #2 says "The sky is blue," the Editor notices the conflict. It doesn't just pick a winner; it explains why they disagree and checks the facts.
- Unique Findings: If AI #3 says something no one else mentioned (like "The sky looks green at sunset"), the Editor keeps that interesting detail.
The final output isn't just a "majority vote." It's a nuanced, fact-checked summary that highlights where everyone agrees and where they fight.

Why Does This Work? (The Magic Math)

The paper uses a simple probability trick.

If one AI has a 20% chance of lying (hallucinating), that's bad.
But the chance that three different AIs all lie about the exact same thing at the exact same time is tiny (like 0.2% or less).
The Metaphor: It's very unlikely that three different people, from different backgrounds, would all accidentally tell the same lie. If they all say the same thing, it's probably true. If they disagree, the system catches the lie.

The Results: What Did They Find?

The researchers tested this against the best single AIs available (like GPT-5.4 and Claude Opus 4.6).

Less Lying: The Council Mode reduced "hallucinations" (lies) by 36%.
More Truth: It scored much higher on tests designed to see if the AI tells the truth.
Less Bias: Because it mixes different AIs, it doesn't lean too hard in one political or cultural direction. It's more balanced.
The Trade-off: It takes a little longer (about 8 seconds instead of 3) because it has to wait for three AIs to talk and then for the Editor to write. But for important questions, the extra time is worth it for the accuracy.

The Bottom Line

Council Mode is like moving from a "One-Man Show" to a "Town Hall Meeting."
Instead of trusting one voice that might be wrong or biased, it listens to a diverse group, checks their work, and synthesizes the best possible answer. It turns AI from a confident liar into a careful, fact-checking team.

In short: If you want the AI to be right, don't ask one expert. Ask a committee.

Based on the technical report provided, here is a detailed summary of the Council Mode framework.

1. Problem Statement

Large Language Models (LLMs), particularly those utilizing Mixture-of-Experts (MoE) architectures, face two critical failure modes:

Hallucinations: The generation of fluent but factually incorrect content. In MoE models, this is exacerbated by "expert collapse" (where specific experts over-specialize on incorrect correlations) and sparse routing, which limits the activation of the model's total parametric knowledge.
Systematic Bias: Routing mechanisms can introduce biases based on surface-level features rather than semantic content, leading to inconsistent performance across different domains.

Traditional mitigation strategies like Retrieval-Augmented Generation (RAG) or Reinforcement Learning from Human Feedback (RLHF) operate within the confines of a single model's parametric memory. The paper argues that these methods are insufficient because they lack the cognitive diversity required to cross-verify facts and correct idiosyncratic errors.

2. Methodology: The Council Architecture

The authors propose Council Mode, a structured multi-agent consensus framework that dispatches queries to multiple heterogeneous frontier LLMs in parallel and synthesizes their outputs. The pipeline operates in three distinct phases:

Phase 1: Intelligent Triage

To optimize computational resources, a lightweight classifier ( $T$ ) evaluates query complexity.

Mechanism: A two-stage filter (client-side regex and server-side lightweight LLM) identifies trivial queries (e.g., greetings, short inputs).
Action: Trivial queries bypass the full council for a direct answer. Complex queries are routed to the full consensus pipeline.

Phase 2: Parallel Expert Generation

Non-trivial queries are dispatched simultaneously to $N$ distinct expert models ( $E_1, E_2, ..., E_N$ ).

Diversity: The system utilizes models from different providers (OpenAI, Anthropic, Google) with different architectures and training data to maximize cognitive diversity.
Implementation: In the study, $N=3$ (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro).
Latency: Execution is asynchronous (using Promise.all()), ensuring total latency is bounded by the slowest expert rather than the sum of all experts.

Phase 3: Consensus Synthesis

A dedicated synthesis model ( $S$ ) aggregates the responses from the experts using a structured protocol containing 16 mandatory rules. Instead of simple majority voting, the output is decomposed into four sections:

Consensus Points ( $O_{consensus}$ ): Factual claims supported by all experts.
Disagreements ( $O_{disagree}$ ): Conflicting information identified between experts.
Unique Findings ( $O_{unique}$ ): Insights present in only one expert's response.
Comprehensive Analysis ( $O_{analysis}$ ): A final, fact-checked response that resolves disagreements and contextualizes unique findings.

3. Key Contributions

Framework Proposal: Introduction of a three-phase multi-agent consensus framework (Triage $\rightarrow$ Parallel Generation $\rightarrow$ Structured Synthesis) specifically designed to mitigate hallucinations and bias.
Mathematical Formalization: Formal definition of the consensus mechanism, including the triage function, parallel dispatch protocol, and the set-theoretic aggregation of expert claims.
Open-Source Implementation: Release of a complete, cloud-native implementation integrated into an open-source AI workspace, featuring real-time streaming of expert reasoning states via Server-Sent Events (SSE).
Empirical Validation: Comprehensive evaluation demonstrating significant improvements over single-model baselines, supported by ablation studies proving the necessity of model heterogeneity and structured synthesis.

4. Experimental Results

The Council Mode was evaluated against five individual frontier models (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, DeepSeek V3.2, Seed 2.0 Pro) across multiple benchmarks.

Hallucination Reduction: On the HaluEval benchmark, Council Mode achieved a 35.9% relative reduction in hallucination rates compared to the best-performing individual model (Claude Opus 4.6).
- Average Hallucination Rate: 10.7% (Council) vs. 16.7% (Best Single).
Truthfulness: On TruthfulQA, the framework improved the Truthful score by 7.8 percentage points (82.6% vs. 74.8%).
Bias Mitigation: The variance in performance across six domains (Science, History, Medicine, Law, Technology, Finance) was reduced by 85–89% ( $\sigma^2$ dropped from ~0.025 to 0.003), indicating highly consistent neutrality.
Complexity Scaling: As task complexity increased (up to 10 reasoning steps), Council Mode maintained significantly higher accuracy (71.2% at max complexity) compared to single models, which showed steep degradation.
Ablation Insights:
- Heterogeneity is Key: Using three instances of the same model (e.g., 3x GPT-5.4) yielded only an 18.3% reduction in hallucinations, whereas the heterogeneous Council achieved 35.9%.
- Synthesis vs. Voting: Replacing structured synthesis with simple majority voting increased hallucination rates by 32.7%, proving that deep semantic analysis of disagreements is critical.

5. Significance and Limitations

Significance:
The paper demonstrates that cognitive diversity is a more effective lever for reliability than simply scaling a single model. By treating LLMs as a "council" of experts rather than a monolithic oracle, the system can identify and filter out individual model failures. The structured synthesis approach preserves minority insights and reasoning chains that are typically lost in voting mechanisms, offering a new paradigm for building trustworthy AI systems.

Limitations:

Latency: The approach incurs a 2–3x increase in latency (average 8.4s vs. 2.8–5.6s for single models), making it less suitable for real-time, low-latency applications.
Cost & Availability: Reliance on multiple frontier model APIs increases operational costs and dependency on external providers.
Shared Misconceptions: If all experts share the same training data bias or misconception, the consensus mechanism cannot correct it.

Conclusion:
Council Mode represents a significant step forward in LLM reliability, shifting the focus from single-model alignment to multi-agent consensus. It offers a robust, open-source solution for applications where factual accuracy and bias reduction are prioritized over speed.