🤖 AI

Insider Attacks in Multi-Agent LLM Consensus Systems

This paper investigates insider attacks in multi-agent LLM consensus systems by proposing a world-model-based framework that uses reinforcement learning to train a malicious agent to effectively delay or prevent agreement among benign agents.

Original authors: Xiaolin Sun, Zixuan Liu, Yibin Hu, Zizhan Zheng

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaolin Sun, Zixuan Liu, Yibin Hu, Zizhan Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of friends trying to decide where to go for dinner. They are all using a super-smart AI assistant to help them chat, argue, and eventually agree on a single restaurant. Usually, everyone is on the same page: they all want to eat together.

But what if one of the friends is actually a "secret saboteur"? They look like a normal friend, they use the same chat app, and they say nice things, but their hidden goal is to make sure the group never agrees on a place. They want the group to stay confused and split up forever.

This paper is about studying exactly that kind of "secret saboteur" inside a group of AI agents. Here is a simple breakdown of what the researchers did and found:

The Problem: The "Wolf in Sheep's Clothing"

In many AI systems, multiple agents talk to each other to solve problems. The researchers call this "consensus." They usually assume everyone is a good guy trying to solve the problem.

However, the paper points out that a malicious AI could sneak in as a regular member. Instead of crashing the system from the outside (like a hacker breaking a door), this "insider" stays inside the conversation. They use subtle tricks—like being vague, pretending to agree but then changing their mind, or confusing the others—to keep the group from ever reaching a decision.

The Solution: Teaching the Saboteur to "Think Ahead"

The researchers wanted to see how good a malicious AI could get at this game. They tried two different ways to build the "saboteur":

The "Blind" Saboteur (The Baseline): This is like a person who just has a bad prompt written into their brain: "Do whatever it takes to stop them from agreeing!" They try to be tricky, but they don't really understand how the other friends think. They just guess.
The "Strategic" Saboteur (The New Method): This is the main focus of the paper. The researchers built a system where the saboteur first learns a "World Model."
- The Analogy: Imagine the saboteur is playing a video game. Before they make a move, they build a mental simulation of how their friends will react. They learn that "Friend A is stubborn and hates moving," while "Friend B is easily swayed."
- Once the saboteur has this mental map, they use a technique called Reinforcement Learning (like training a dog with treats) to figure out the perfect sequence of confusing messages to say to keep the group arguing.

The Experiment: A Line of Numbers

To test this, the researchers created a simple game:

Imagine a line of numbers from 0 to 20.
Several AI agents are standing at different spots on this line.
Their goal is to all move to the same number.
One agent is the "saboteur."

They gave the "good" agents different personalities:

The Stubborn: Hard to move, likes to stay put.
The Suggestible: Easily swayed by others.
The Neutral: Just wants to cooperate.

What They Found

The results showed that the "Strategic Saboteur" (the one with the World Model) was much better at causing trouble than the "Blind Saboteur."

Success Rate: When the "Blind" saboteur tried to stop the group, the good agents still managed to agree most of the time (about 86-90% of the time, depending on the mix of personalities).
The Strategic Saboteur: When the "Strategic" saboteur used their mental simulation, the group agreed much less often (dropping to around 70-80% or lower in some cases).
Time Wasted: The Strategic Saboteur also made the arguments last longer. The group spent more rounds talking and less time actually finishing the task.

The "Guessing" Twist

The researchers also tested a version where the saboteur didn't know the personalities of the other agents at the start. They had to "guess" the personalities by watching the first round of conversation. Even with this handicap, the Strategic Saboteur performed almost as well as the one who knew the personalities beforehand. This shows the system is very good at learning on the fly.

The Bottom Line

The paper concludes that if you have a group of AI agents trying to agree on something, a malicious agent that learns how the others think (using a "World Model") and plans its moves carefully (using "Reinforcement Learning") is a much bigger threat than one that just tries to be annoying randomly.

Important Note: The researchers only tested this in a very simple, controlled game (the line of numbers). They did not test this on real-world complex tasks, medical advice, or financial systems. They are just showing that this specific type of "smart insider attack" works better than a "dumb" one in their specific experiment.

Technical Summary: Insider Attacks in Multi-Agent LLM Consensus Systems

Problem Definition

The paper addresses a critical vulnerability in multi-agent Large Language Model (LLM) systems: insider attacks within consensus formation processes. While existing frameworks often assume all participating agents are aligned with the system objective, this work posits that a malicious agent can infiltrate a group as a legitimate member. Unlike external attacks that perturb prompts or infrastructure, an insider attacker operates from within the deliberation process. The attacker's goal is to strategically disrupt, delay, or prevent the benign agents from reaching a shared decision (consensus) while maintaining a superficially cooperative demeanor. The challenge lies in the fact that in language-based systems, influence is exerted through subtle natural-language strategies—such as persuasion, framing, selective agreement, or ambiguity—rather than direct system compromise.

The authors formalize this as a sequential decision-making problem where a malicious agent seeks to maximize a reward function that penalizes benign consensus and rewards prolonged disagreement.

Methodology

To solve the optimization problem of finding an effective attack policy, the authors propose a world-model-based framework that combines latent dynamics modeling with Reinforcement Learning (RL). This approach is designed to overcome the sample inefficiency of directly optimizing against interacting LLMs, which requires expensive multi-agent rollouts.

1. Latent-Agent World Model

Instead of querying the full multi-agent LLM system at every step, the authors train a surrogate model to predict the evolution of the benign agents' states.

State Representation: The model takes as input the current decision positions of visible neighbors ( $y^t_{N_k}$ ) and their behavioral attributes ( $\psi_{N_k}$ ), along with the attacker's action (message and decision).
Heterogeneity Handling: Recognizing that agents with different personalities react differently to the same input, the model is a multi-head feed-forward neural network. It encodes agent positions and behavioral attributes, then uses personality-specific heads to predict the next decision position for each benign agent type.
Training: The model is trained via supervised regression on trajectory data collected from interactions involving benign agents with varying personality types (stubborn, suggestible, neutral) and a random malicious agent.

2. Behavioral Attribute Inference

In realistic scenarios, an attacker may not know the specific personality types of benign agents. To address this, the authors introduce a behavior-attribute classifier.

This classifier infers an agent's behavioral type (e.g., stubborn, suggestible) from their historical decision trajectories and broadcast messages.
It utilizes a recurrent trajectory encoder for position history and a BERT-based encoder for message semantics.
The inferred attributes are then fed into the world model and the attacker's policy, allowing the attack to adapt to the observed behavior of the group.

3. RL-Based Attacker Optimization

The insider attack is formulated as a Markov Decision Process (MDP) solved using Deep Q-Networks (DQN).

Policy: The attacker learns a policy $\pi_{adv}$ that maps the visible neighborhood state to an adversarial message-decision pair.
Reward Function: The reward is designed to encourage disagreement (e.g., $r_t = 1$ if consensus is not reached, $0$ otherwise).
Efficiency: By using the learned world model for rollouts rather than the actual LLMs, the training process becomes significantly more sample-efficient, allowing the attacker to reason about long-term cumulative effects.

Key Contributions

Formalization of Insider Manipulation: The paper casts insider manipulation in multi-agent LLM consensus as a sequential decision-making problem, moving beyond simple prompt-based adversarial examples to a dynamic, adaptive attack framework.
Model-Based Attack Framework: The authors develop a world-model-based approach that learns surrogate dynamics of heterogeneous benign agents. This enables efficient attacker optimization without the computational cost of repeated full-system simulations.
Empirical Validation: The framework is instantiated in a 1-D discrete consensus environment. The study provides preliminary evidence that a learned, model-based attacker is more effective at disrupting consensus than a baseline attacker relying solely on a static malicious prompt.

Experimental Results

The authors evaluated the framework in a setting with $N$ agents on a 1D line (positions 0–20), using GPT-4o as the underlying model. They compared four settings:

No Attacker: A clean baseline with an extra neutral agent.
LLM-Based Attacker: A baseline using a static malicious prompt without explicit dynamic modeling.
RL-Based Attacker (Known Attributes): Uses the DQN policy with ground-truth knowledge of benign agent personalities.
Guessed RL Attacker: Uses the same DQN policy but infers personality types from the first interaction round.

Key Findings:

Consensus Rate Reduction: Both the LLM-based and RL-based attackers reduced the consensus rate compared to the no-attacker baseline.
Superiority of RL: The RL-based attacker achieved the strongest disruption on average, resulting in the lowest consensus rate (e.g., dropping from 0.95 in the baseline to 0.83 overall) and the longest average episode duration (increasing from 4.32 to 4.91 rounds).
Effectiveness of Inference: The "Guessed RL Attacker" performed nearly as well as the "Known Attribute" variant, demonstrating that inferring behavioral attributes from a single initial round is sufficient to adapt the attack strategy effectively.
World Model Accuracy: The latent world model achieved a Mean Absolute Error (MAE) of 0.927 in predicting next positions, indicating it captures meaningful trends in benign agent movement despite the difficulty of exact integer prediction.

Significance and Limitations

Significance:
The paper argues that combining latent world models with reinforcement learning is a promising direction for adaptive insider attacks. It demonstrates that explicit modeling of benign-agent interaction dynamics allows an attacker to be more effective than relying solely on the intrinsic strategic capabilities of a prompted LLM. This highlights a specific vulnerability in multi-agent consensus systems where the "human-in-the-loop" or "agent-in-the-loop" assumption of alignment is fragile.

Limitations:
The authors explicitly note several constraints:

Simplified Environment: The experiments are conducted in a simplified 1-D consensus setting with a small number of agents.
Predefined Types: The study relies on predefined behavioral types (stubborn, suggestible, neutral) rather than open-ended personality spectra.
Restricted Protocol: The communication protocol is structured and limited compared to open-ended multi-agent conversations.
Future Work: The authors state that future research will need to address more open-ended protocols, larger agent populations, and the development of defense mechanisms against such insider manipulation.