💬 NLP

Misinformation Propagation in Benign Multi-Agent Systems

This paper investigates how intent-based misinformation propagates through benign multi-agent systems, finding that while such misinformation degrades performance and persists across agent interactions, multi-agent debate generally offers greater robustness than single-agent prompting, with outcomes heavily dependent on group composition, decision protocols, and the underlying models.

Original authors: Jonas Becker, Jan Philip Wahle, Terry Ruas, Bela Gipp

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonas Becker, Jan Philip Wahle, Terry Ruas, Bela Gipp

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a team of expert detectives trying to solve a mystery. Usually, they work together, sharing clues and debating the facts to find the truth. This is how Multi-Agent Systems (MAS) work in the world of Artificial Intelligence: several AI "agents" (like advanced chatbots) talk to each other to solve complex problems, hoping that their combined wisdom is better than any single detective working alone.

But what happens if one of those detectives starts reading a fake newspaper?

This paper, titled "Misinformation Propagation in Benign Multi-Agent Systems," investigates exactly that scenario. The researchers wanted to see what happens when a team of AI agents is trying to solve a problem, but some of them are secretly fed false information. Crucially, these agents are "benign," meaning they aren't trying to trick anyone; they are just honest detectives who happen to be looking at the wrong clues.

Here is a breakdown of their findings using simple analogies:

1. The Setup: The "Fake Clue" Experiment

The researchers created a massive dataset of fake facts called MINT (Misinformation INTents). They took real questions (like "Which country borders Vietnam with code 855?") and wrote fake stories to go along with them. These stories came in different flavors, like:

Clickbait: "You won't believe the shocking truth!"
Conspiracy: "The government is hiding the real code."
Rumor: "I heard from a traveler that..."

They then fed these fake stories to AI agents and watched what happened.

2. The Solo Detective vs. The Team

The Solo Detective (Single-Agent):
When a single AI agent was given a fake clue, it often got confused. It's like a detective who finds a fake map and confidently draws the wrong route. The AI's performance dropped significantly, especially on tasks requiring specific knowledge or ethical judgment.

The Team of Detectives (Multi-Agent Debate):
Next, they put the agents in a room to debate. Some had the fake clues; others had the truth.

The Good News: The team was generally more robust than the solo detective. Even if one agent started with a fake clue, the group discussion often helped correct the mistake. The "self-reflection" of the debate acted like a safety net.
The Bad News: The fake information didn't just disappear. It often stuck around. If an agent said, "I think the answer is Laos because of this fake story," other agents often kept that idea in their minds, even after the debate. The misinformation "infects" the conversation, even if the final answer is sometimes corrected.

3. The Voting vs. The Consensus

The researchers tested two ways the team could make a final decision, like a jury:

Voting: Everyone raises their hand for their answer, and the majority wins.
Consensus: The last person to speak gets to decide the final answer based on what everyone else said.

The Findings:

Voting is usually more accurate when everyone is honest. However, if the "fake clue" agents become the majority, the whole team votes for the wrong answer. It's like a mob mentality; if most people believe a lie, the vote reflects that lie.
Consensus was more stable when the team was under "peer pressure" from fake information. The final decision-maker seemed better at filtering out the noise and sticking to the truth, even if the majority was confused.

4. The "Peer Pressure" Effect

One of the most interesting findings was about self-correction.

If an agent started with a fake clue but was surrounded by three or more honest agents, it was much more likely to change its mind and admit the truth.
However, if the honest agents were in the minority, the agent with the fake clue tended to stick to its guns.
The Model Matters: Not all AI brains react the same way. One model (Llama-3.3) was very sensitive to peer pressure and often changed its answer based on what others said. Another model (GLM-4.7) was much more stubborn and stayed accurate regardless of what the others said.

5. The Bottom Line

The paper concludes that while AI teams are generally better at spotting lies than a single AI, they aren't immune.

Misinformation spreads: Even in a friendly debate, false ideas can linger and influence the group.
Group size and rules matter: Having more honest agents helps, and the way the team decides (voting vs. consensus) changes how well they resist lies.
It depends on the AI: Some AI models are naturally more gullible to peer pressure than others.

In short, if you put a group of honest AI detectives in a room, they are usually smart enough to find the truth. But if you slip a fake clue into the mix, that lie can stick around and confuse the group, especially if the team isn't big enough or if the wrong decision-making rules are used. The system isn't perfect, but it's often better than a single detective working alone.

Technical Summary: Misinformation Propagation in Benign Multi-Agent Systems

Problem Statement

As Large Language Model (LLM) based Multi-Agent Systems (MAS) are increasingly deployed in high-stakes domains such as medical diagnosis, legal analysis, and forensic decision-making, their reliability faces a critical risk: the propagation of misinformation. While MAS are often touted for improving reasoning through debate and task decomposition, their robustness is uncertain when agents operate under incorrect information. This incorrectness may stem from retrieval-augmented generation errors, hallucinations, misinterpreted external sources, or adversarial inputs.

The core problem addressed is how contextual misinformation influences the outcomes of fully benign multi-agent debates. In this context, "benign" implies that agents follow prescribed debate protocols without intent to deceive; however, misinformation enters the system through the local context available to specific agents. The paper investigates whether multi-agent debate (MAD) amplifies or mitigates the performance degradation caused by such misinformation, and how group composition and decision protocols influence system robustness.

Methodology

Experimental Setup

The authors evaluate single-agent prompting against multi-agent debate (MAD) under both uninformed and misinformed conditions. The experiments vary three key factors:

Misinformation Relevance: Distinguishing between task-relevant and task-irrelevant misinformation.
Group Composition: Varying the number of misinformed versus uninformed agents within a group.
Decision Protocol: Comparing Consensus (where the final agent decides based on prior contributions) versus Voting (majority rule at the end of the debate).

Datasets and Models

Models: The study utilizes Llama-3.3-70B-Instruct and GLM-4.7-Flash. Notably, Llama-3.3 is used both to generate misinformation and to evaluate it, simulating a plausible real-world scenario where agents consume content from the same model family.
Tasks: Performance is measured across three dimensions:
- Reasoning: WinoGrande (pronoun resolution/commonsense).
- Knowledge: Complex Web Questions (CWQ) (fact-retrieval and combination).
- Alignment: Ethics benchmark (commonsense ethical judgment).
MINT Dataset: To support these experiments, the authors introduce MINT (Misinformation INTents), a dataset comprising 10,278 LLM-generated misinformation texts. These texts are aligned with 1,142 samples from the target datasets and categorized into nine intent-based types: Neutral, Clickbait, Hoax, Rumor, Satire, Propaganda, Framing, Conspiracy, and Other. A human audit confirmed that the generated texts faithfully represent their intended false facts and categories, though inter-annotator agreement was modest.

Procedure

In the Single-Agent setup, misinformation is appended directly to the prompt. In the Multi-Agent setup, a subset of agents receives the misinformed prompt while others remain uninformed. The agents engage in a five-turn debate. The study measures:

Task Accuracy: The correctness of the final decision.
Opinion Persistence: The probability that an answer proposed at turn $t$ is repeated by an agent at turn $t+1$ , indicating whether misinformation spreads or is corrected.
Self-Correction: The rate at which initially misinformed agents switch to the correct answer by the end of the debate.

Key Results

1. Impact on Single-Agent Performance

Exposure to relevant misinformation significantly degrades single-agent performance.

On the CWQ dataset, Llama-3.3 accuracy dropped by 26.75% (from 0.49 to 0.36).
Ethical alignment tasks (Ethics) and reasoning tasks (WinoGrande) also saw substantial declines (approx. 25% and 19%, respectively).
Irrelevant misinformation had a weaker effect, and irrelevant true information often improved performance, suggesting the degradation is due to semantic alignment and intent rather than mere token distraction.

2. Multi-Agent Robustness and Persistence

Multi-agent debate generally mitigates the performance degradation observed in single agents, though misinformation persists.

Mitigation: The accuracy drop in MAD was smaller (-2.2% to -10.3%) compared to single-agent setups (-12.9% to -17.2%).
Persistence: Despite the mitigation, misinformation introduced by one agent often persists across turns. On CWQ and WinoGrande, agents retained misinformed answers at higher rates than uninformed ones (average deltas of -10.4% and -7.7%, respectively).
Model Differences: Llama-3.3 showed stronger persistence of misinformed answers compared to GLM-4.7. The authors attribute this partly to the "self-preference" effect, where LLMs may favor responses generated by their own model family.

3. Influence of Group Composition and Decision Protocols

Voting vs. Consensus:
- Voting generally achieves higher absolute accuracy but is more sensitive to "misinformed peer pressure." As the number of misinformed agents increases, voting accuracy drops significantly.
- Consensus is more stable under misinformation exposure. For Llama-3.3, consensus maintained a relatively flat accuracy curve even as misinformed agents increased, whereas voting performance degraded.
Peer Pressure and Correction: A sharp increase in self-correction occurs only when uninformed agents form a majority. If misinformed agents are a minority, the system often corrects the error; if they are a majority, the error tends to dominate.
Fully Misinformed Groups: Even when all agents are misinformed, MAD still outperforms a misinformed single-agent setup, suggesting that the debate process itself offers some inherent resilience, though performance declines as misinformation prevalence rises.

Significance and Contributions

The paper makes four primary contributions:

Assessment of Vulnerability: It demonstrates that fully benign multi-agent systems are vulnerable to contextual misinformation, particularly in knowledge-intensive and ethical alignment tasks, though the vulnerability is task-dependent.
Analysis of Debate Dynamics: It reveals that while MAD can mitigate performance degradation compared to single agents, misinformation often persists through the debate. The system's ability to self-correct is heavily dependent on the majority status of uninformed agents.
Design Implications: The study identifies that robustness is not solely a function of the underlying model but is significantly influenced by group composition and decision protocols. Specifically, consensus-based decision-making appears more robust against misinformed peer pressure than voting in certain model configurations.
Resource Release: The authors release the MINT dataset and all experimental code to facilitate further research into misinformation in agentic systems.

Limitations

The authors acknowledge several limitations:

The study focuses on two specific open-weight models; results may not generalize to all model families.
The misinformation is machine-generated (MINT), which may differ from human-written or adversarially optimized misinformation.
The experimental setup uses fixed debate structures and turn counts, not covering dynamic role assignments, tool use, or long-term memory.
The human audit of MINT showed only "fair" inter-annotator agreement, indicating that fine-grained misinformation strategies can be ambiguous.

In conclusion, the work provides evidence that while multi-agent debate offers a degree of resilience against misinformation, it does not eliminate the risk. The reliability of such systems in high-stakes environments depends critically on how agents exchange information, the composition of the agent group, and the decision-making protocols employed.