💬 NLP

Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

This paper introduces a three-source decomposition framework to reveal that multi-agent LLM debate convergence often stems from harmful social conformity and vacuous reasoning rather than genuine deliberation, demonstrating that while such conformity can be predicted and reduced, doing so without distinguishing between beneficial and harmful influence fails to improve overall accuracy.

Original authors: Xiqi Hao, Zengqing Wu, Yu-Xuan Qiu, Chuan Xiao, Ruiqi Xu, Shuyuan Zheng, Jianbin Qin

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiqi Hao, Zengqing Wu, Yu-Xuan Qiu, Chuan Xiao, Ruiqi Xu, Shuyuan Zheng, Jianbin Qin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of five brilliant, but slightly nervous, students sitting in a room to solve a difficult math problem. They each write down their own answer first. Then, they are allowed to talk to each other, share their answers, and explain their logic. The goal is for them to help each other get to the right answer.

This paper is like a detective story investigating what actually happens in that room. The researchers wanted to know: When the students change their answers after talking, are they actually learning, or are they just giving in to peer pressure?

Here is the breakdown of their findings using simple analogies:

1. The "Flip" Isn't Always a "Flip"

In the past, researchers counted how many times a student changed their answer after hearing others. They assumed every change was a result of the group discussion.

The authors say: Wait a minute.
Imagine asking a student to just look at their own answer again, alone in a quiet room, with no one else around. Surprisingly, 37% of the time, they change their mind anyway!

The Analogy: It's like you're sure you locked the front door, but when you double-check, you realize you didn't. You didn't need a neighbor to tell you; you just needed a moment of self-reflection.
The Finding: A huge chunk of the "conformity" we see is actually just students being unstable and changing their minds on their own.

2. The Three Reasons for Changing Answers

The researchers broke down every answer change into three distinct buckets:

Bucket A: The "Oops, I was wrong" (Spontaneous Instability)
The student changes their answer just because they re-read the question, even if no one else spoke. This happens a lot (37% of the time) and is usually bad—they often switch from the right answer to the wrong one just because they got confused by their own second thoughts.
Bucket B: The "I'll just go with the crowd" (Stance-Induced Conformity)
The student was actually confident and stable when alone. But then, they see that four other students picked a different answer. Even without hearing why those students picked it, the student changes their answer to match the majority.
- The Result: This is mostly harmful. About 64% of the time, a student who was originally correct switches to the wrong answer just to fit in.
Bucket C: The "You convinced me" (Reasoning-Induced Persuasion)
The student sees the other students' answers and their step-by-step logic. They change their answer because of the argument.
- The Twist: Even this "smart" change is dangerous. The researchers found that students often switch to the wrong answer just because the logic looked convincing, even if the logic was nonsense.

3. The "Fake Logic" Experiment

To test if students were actually thinking or just following the format, the researchers ran a special experiment. They gave the students:

No reasoning: Just the answer.
Fake reasoning: Text that looked like a logical argument (using words like "therefore," "because," and "step-by-step") but contained absolutely no real facts or logic. It was just "word salad."
Bad reasoning: Arguments that made sense but led to the wrong conclusion.

The Shocking Result:
When students were shown the Fake Reasoning (the word salad), 30% of the students who were originally correct changed their answer to the wrong one.

The Metaphor: It's like a student seeing a peer write a long, fancy essay with complex diagrams. Even if the essay says nothing of value, the student thinks, "Wow, that looks smart, I must be wrong," and changes their answer. The appearance of reasoning is enough to trick them.

4. Can We Fix This?

The researchers tried to predict which students were most likely to give in to peer pressure and then gave them a special "independence" prompt (telling them to stick to their own logic).

In a controlled lab setting (where they knew the right answers): They successfully identified the "at-risk" students and reduced the number of wrong changes by 13.6%.
In the real world (where they don't know the right answers): They tried to stop students from changing answers to match the group. While this stopped students from copying the wrong answers, it also stopped them from copying the right answers.
- The Lesson: You can't just tell students "don't listen to others." If you do, you stop the bad influence, but you also block the good help. To make the group smarter, you need a way to verify if the group is actually right, not just a way to stop them from talking.

Summary

The paper concludes that Multi-Agent Debate (groups of AI agents talking) isn't as magical as we thought.

Half the time, answer changes are just the AI getting confused by itself, not the group.
When they do listen to the group, they often follow the crowd blindly, even if the crowd is wrong.
The "look" of logic is just as persuasive as real logic. AI agents are easily tricked by the format of an argument, even if the argument is empty.

The takeaway? Just because a group of AIs agrees on an answer doesn't mean they are right. They might just be agreeing because they are unstable, easily impressed by fancy words, or afraid to stand alone.

Technical Summary: Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

Problem Statement

Multi-agent debate (MAD) is widely utilized to enhance Large Language Model (LLM) reasoning by having multiple agents challenge and refine each other's answers. However, a critical ambiguity remains: when agents converge on a shared answer, does this convergence reflect genuine deliberation and improved reasoning, or is it merely social compliance (conformity)?

Current literature often interprets any answer change following peer exposure as evidence of persuasion. This paper argues that such interpretations are flawed because they fail to distinguish between three distinct mechanisms driving stance changes:

Spontaneous Instability: The tendency of models to revise answers upon re-examination even without new information.
Stance-Induced Conformity: Changes driven solely by the presence of peer positions (normative influence).
Reasoning-Induced Persuasion: Changes driven by exposure to the logical content of peer reasoning (informational influence).

Without isolating these factors, the "flip rate" (the percentage of agents changing answers) conflates baseline model instability with genuine social influence, leading to an overestimation of conformity and a misunderstanding of MAD's efficacy.

Methodology

The authors propose a three-source decomposition framework operationalized through a controlled experimental design using counterfactual conditions.

Experimental Setup

Agents: Five agents (primarily GPT-4o, with robustness checks on DeepSeek-V4-Flash and Qwen-Plus) answer multiple-choice questions independently in Round 0.
Disagreement Filtering: Only questions where agents initially disagree proceed to Round 1.
Counterfactual Conditions: In Round 1, each agent-question pair is observed under three parallel, independent conditions:
1. Self-Reflection ( $a^{(sr)}$ ): The agent re-evaluates its initial answer with no peer information.
2. Stance-Only ( $a^{(so)}$ ): The agent sees peer answers but no reasoning.
3. Full Reasoning ( $a^{(r)}$ ): The agent sees peer answers and their reasoning traces.

Decomposition Logic

An observed answer change is attributed to a specific source based on the agent's behavior across these conditions:

Spontaneous Change: $a^{(sr)}_i \neq a^{(0)}_i$ . The agent changes without any peer input.
Strict Conformity (Stance-Induced): $a^{(sr)}_i = a^{(0)}_i$ (stable in isolation) AND $a^{(so)}_i \neq a^{(0)}_i$ AND $a^{(so)}_i \in \{a^{(0)}_j\}_{j \neq i}$ . The agent changes only when exposed to peer stances.
Reasoning-Induced Change: $a^{(sr)}_i = a^{(0)}_i$ , $a^{(so)}_i = a^{(0)}_i$ , AND $a^{(r)}_i \neq a^{(0)}_i$ . The agent resists both self-reflection and stance exposure but changes upon seeing peer reasoning.

Information Gradient Experiment

To disentangle the persuasive power of reasoning form from content, the authors conducted a controlled experiment on a high-risk subset (agents initially correct and stable). They introduced:

Invalid Reasoning: Vacuous text with reasoning structure but no domain content.
Wrong Reasoning: Plausible arguments leading to an incorrect answer.
This allowed measurement of error adoption rates attributable purely to the structural appearance of reasoning versus substantive logical errors.

Prediction and Intervention

The study also developed a prediction model to identify agents at risk of harmful conformity (correct $\to$ wrong) based on Round 0 features (e.g., peer disagreement structure, confidence). This model was used to test targeted interventions (prompts encouraging independence) in both diagnostic (with ground truth) and deployment (without ground truth) settings.

Key Results

1. Decomposition of Stance Convergence

Spontaneous Instability is Dominant: In the primary MMLU-Pro setting, 37% of agent-question observations changed under self-reflection alone. This indicates that a significant portion of "peer influence" is actually baseline model instability.
Strict Conformity is Lower but Harmful: After controlling for instability, the strict conformity rate is 29%. Crucially, this conformity is predominantly harmful: 63.6% of these changes involve a correct agent switching to a wrong answer.
Net Accuracy Impact: While individual accuracy improves slightly with peer information, at the aggregate level (majority vote), peer influence provides no measurable gain beyond self-reflection because harmful and beneficial conformity approximately cancel each other out.

2. The Role of Reasoning

Reasoning Often Increases Errors: Among agents resistant to self-reflection and stance-only exposure, exposure to full peer reasoning led to further changes. However, only 30% of these changes improved accuracy; the majority (70%) either degraded accuracy or failed to improve it.
Form Over Content: The information-gradient experiment revealed that reasoning-like presentation (even when vacuous) causes 29.7% of resistant agents to abandon correct answers. Adding substantive but wrong reasoning increased this to 54.5%. This suggests LLMs are susceptible to the structural appearance of reasoning, not just its logical validity.

3. Prediction and Mitigation

Predictability: Harmful conformity is predictable from Round 0 features (specifically peer disagreement structure) with an AUC of 0.79.
Diagnostic Intervention: In a setting with ground-truth labels, targeting the top 20% of at-risk agents with independence-promoting prompts reduced harmful conformity by 13.6 percentage points and improved accuracy by 14.6 pp.
Deployment Limitation: In a realistic deployment setting (without ground-truth labels), the same intervention reduced peer adoption but did not improve overall accuracy. This is because the intervention blocked both harmful adoptions (correct $\to$ wrong) and beneficial adoptions (wrong $\to$ correct), which are difficult to distinguish without correctness signals.

Significance and Claims

The paper claims to shift the understanding of reliability in multi-agent systems from a purely social influence problem to a dual problem involving individual reasoning instability and social conformity.

Re-evaluating MAD: The authors argue that stance convergence cannot be interpreted as a success signal based on flip rates alone. A substantial share of convergence is spontaneous degradation, and the remaining strict conformity is often accuracy-reducing.
Context-Induced Compliance: The findings suggest that LLMs treat the mere presence of peer reasoning as evidence, analogous to authority bias in humans. This implies that context design in multi-agent systems is a critical safety-relevant choice.
Limits of Prompt-Level Mitigation: The study highlights a boundary for prompt-based interventions. While targeted prompts can reduce harmful conformity when correctness signals are available, they cannot improve accuracy in blind deployment scenarios because they indiscriminately suppress both harmful and beneficial influence. Reliable deployment, therefore, requires runtime verification mechanisms or correctness signals in addition to risk scoring.

The work concludes that future MAD designs should move beyond uniform exposure to peer answers and incorporate verification, confidence calibration, or disagreement-aware gating to mitigate the risks of spontaneous instability and harmful conformity.