💻 computer science

Latent Agents: A Post-Training Procedure for Internalized Multi-Agent Debate

This paper introduces "Latent Agents," a post-training framework that distills compute-intensive multi-agent debates into a single LLM, achieving comparable performance with up to 93% fewer tokens while revealing interpretable agent-specific activation subspaces that facilitate more efficient control over reasoning behaviors.

Original authors: John Seon Keun Yi, Aaron Mueller, Dokyun Lee

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: John Seon Keun Yi, Aaron Mueller, Dokyun Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but chatty student who is great at solving problems, but only when they are allowed to argue with two other students. This "Multi-Agent Debate" method works well: the students talk back and forth, correct each other's mistakes, and eventually agree on the right answer. However, this process is slow and expensive because it requires generating a huge amount of text (like a long transcript of a classroom argument) just to get one answer.

The authors of this paper asked: Can we teach a single student to hold that entire debate inside their own head, so they can give the right answer quickly without all the chatter?

They developed a method called IMAD (Internalized Multi-Agent Debate). Here is how it works, broken down into simple steps:

1. The Training Camp (Two-Stage Learning)

To teach the model this skill, they used a two-step training process:

Stage 1: Learning the Script (Supervised Fine-Tuning).
Imagine giving the student a stack of transcripts from those successful classroom debates. The student reads them over and over, not necessarily to learn the math answers, but to learn the structure of the argument. They learn how "Agent 1" speaks, how "Agent 2" critiques, and how they eventually reach a consensus. The model learns to mimic this entire conversation format.
Stage 2: The Silent Drill (Reinforcement Learning).
Now, the teacher changes the rules. The student is still asked to solve problems, but the teacher starts penalizing them for writing out the whole argument.
- First, the teacher says, "You can write the whole debate, but you must get the answer right."
- Then, the teacher says, "Okay, try to get the answer right, but write less and less of the debate."
- Finally, the teacher says, "Get the answer right, but you can only write the final answer. The debate must happen entirely in your head."
Through this pressure, the model learns to perform the complex reasoning steps internally (in its "latent space") and only output the final result.

2. The Results: Fast and Accurate

The paper tested this on math problems and logic puzzles.

The Magic: The new "Internalized" model performed just as well as (or sometimes better than) the slow, chatty multi-agent debate.
The Savings: It used up to 93% fewer words (tokens) to get the answer. It's like getting a detailed legal verdict without reading the 500-page trial transcript.

3. The "Ghost in the Machine" (Agent Subspaces)

Here is the most fascinating part. The researchers wondered: Did the model just memorize the answers, or did it actually keep the different "personalities" of the agents alive inside its brain?

To test this, they used a technique called Activation Steering. Think of the model's brain as a vast room with different "directions" or hallways.

They found that the model had created specific "hallways" for each agent's personality (e.g., a "Critical Thinking" hallway, a "Code-Like" hallway, a "Step-by-Step" hallway).
By pushing the model's internal state slightly toward one of these hallways, they could make the model act like that specific agent.
The Proof: When they steered the model, it didn't just give a generic answer; it adopted the specific style and reasoning patterns of the agent they targeted. This proves the model didn't just collapse into a single blob of knowledge; it preserved the distinct "voices" of the debate internally.

4. The Safety Test: Catching the "Bad Agent"

Finally, they tested if this structure helps with safety.

They trained a model where one of the internal agents was instructed to be malicious (to give harmful advice or make up fake facts).
Because the model had distinct "hallways" for each agent, the researchers could find the specific "hallway" for the malicious agent.
They then applied Negative Steering (pushing the model in the opposite direction of that hallway).
The Result: This successfully suppressed the bad behavior (the malicious advice or fake facts) much better than trying to steer a normal model. Crucially, this "surgery" removed the bad traits without breaking the model's ability to do math or logic. It's like removing a specific bad habit from a person without making them forget how to speak or walk.

Summary

The paper shows that we can take a slow, expensive process where multiple AI agents argue to solve a problem, and distill it into a single, fast AI that does the arguing inside its own head. Not only is this faster and cheaper, but it also leaves behind a "map" of the different reasoning styles, allowing us to selectively turn off bad behaviors (like lying or being harmful) without hurting the model's overall intelligence.

1. Problem Statement

Multi-agent debate (MAD) has proven effective for improving the reasoning, factual accuracy, and hallucination resistance of Large Language Models (LLMs). However, traditional MAD is computationally expensive and inefficient because it requires:

Multiple Inference Calls: Running several distinct LLM instances.
Verbose Transcripts: Generating long conversational histories across multiple rounds before reaching a consensus.
High Token Consumption: This leads to significant latency and cost, making it impractical for real-time or resource-constrained applications.

The core challenge is to distill the collaborative reasoning benefits of multi-agent debate into a single LLM that can perform the reasoning internally (in its latent space) without generating the verbose external debate transcript, thereby retaining performance while drastically reducing inference costs.

2. Methodology: Internalized Multi-Agent Debate (IMAD)

The authors propose IMAD, a two-stage fine-tuning pipeline designed to internalize the debate process. The framework consists of three main phases:

A. Dataset Construction

Source: A dataset of 944 debate traces was generated using GPT-3.5-turbo.
Task: Arithmetic problems (six randomly generated two-digit numbers) were chosen to focus on structural learning rather than complex domain knowledge.
Structure: The dataset includes $n=3$ agents and $m=2$ rounds. Crucially, structure tags (e.g., <|Agent 1|>, <|Round 1|>, <|Consensus|>) are added to the logs to explicitly teach the model the debate format.
Filtering: Debates without a final majority consensus are discarded.

B. Stage 1: Supervised Fine-Tuning (SFT) - Structure Learning

Goal: Teach the single LLM to replicate the format and structure of a multi-agent debate.
Process: The model is trained via standard autoregressive next-token prediction on the full debate traces (including all agent turns and the final consensus).
Outcome: The model learns to generate a structured sequence of "virtual agents" that critique and refine arguments, effectively mimicking the external debate dynamic within a single generation stream.

C. Stage 2: Reinforcement Learning (RL) - Internalization

Goal: Transition the model from generating explicit, verbose debates to performing the reasoning internally and outputting only the final answer.
Algorithm: Group Relative Policy Optimization (GRPO).
Dynamic Reward Scheduling: The reward function $r(x, y)$ $r (x, y)$ combines two components with dynamic weights:
1. Format Reward ( $R_{fmt}$ ): Initially positive if the output contains structural tags. The weight $w_{fmt}$ decays over training, removing the incentive to output verbose structures.
2. Correctness with Length Clipping ( $R(y; l)$ ): A reward of 1 is given only if the correct answer appears within a truncated prefix of length $l$ . The length limit $l$ undergoes annealing, starting leniently ( $l_0$ ) and shrinking to a target limit ( $l^*$ ) that only fits a concise answer.
Mechanism: As the format reward vanishes and the length limit shrinks, the model is forced to perform the multi-perspective analysis learned in SFT internally (in its latent space) to satisfy the correctness constraint without violating the length constraint.

3. Key Contributions

IMAD Framework: A novel two-stage pipeline that distills multi-agent debate into a single LLM, achieving competitive performance with up to 93% fewer tokens compared to explicit multi-agent debate.
Mechanistic Discovery (Agent Subspaces): Through activation steering analysis, the authors demonstrate that IMAD models do not collapse into a single reasoning mode. Instead, they develop distinct, linearly separable agent subspaces in the activation space. These subspaces correspond to specific agent perspectives (e.g., Chain-of-Thought, Self-Critique, Program-of-Thought).
Controllable Safety via Steering: The paper demonstrates that these internalized agent subspaces are controllable. By injecting a "malicious" agent during training and then applying negative steering (subtracting the malicious agent's steering vector), the model can suppress harmful traits (e.g., evil intent, hallucinations) more effectively than steering a base model, with less degradation to general task performance.

4. Experimental Results

The authors evaluated IMAD on three benchmarks: GSM8K (math), MMLU-Pro (multi-domain knowledge), and Big-Bench Hard (logical reasoning), using LLaMA-3.1 8B, Qwen 2.5 7B, and Mistral Nemo 12B.

Performance vs. Efficiency:
- IMAD matches or exceeds the accuracy of explicit multi-agent debate (Debate) across all models and benchmarks.
- Token Reduction: IMAD uses only 6.3% to 21.1% of the tokens required by explicit debate (a 5–16x efficiency gain).
- Generalization: Despite being trained only on arithmetic, IMAD generalizes well to complex reasoning tasks (MMLU-Pro, BBH) and open-ended summarization tasks.
Agent Subspace Analysis:
- Using Contrastive Activation Addition (CAA), the authors extracted steering vectors for specific agent personas.
- Faithfulness: Steered IMAD models showed significantly higher alignment (ROUGE scores) with target agent personas compared to steered base models (average improvement of 15.41% in AUC).
- This confirms that the collaborative structure is preserved as identifiable latent representations.
Malicious Agent Control:
- When trained with a malicious agent, IMAD models allowed for near-complete suppression of "evil" traits via negative steering (scores dropping to ~0), whereas base models retained residual malicious behavior.
- Stability: IMAD models maintained high task performance (GSM8K accuracy) and low perplexity under extreme steering coefficients, while base models suffered from performance collapse and incoherence.

5. Significance and Implications

Efficiency: IMAD offers a practical solution to the high cost of multi-agent systems, enabling the deployment of debate-like reasoning in production environments with single-model latency.
Interpretability: The discovery of agent subspaces provides a new mechanistic understanding of how LLMs internalize complex reasoning. It suggests that "internal debate" is not just a black-box optimization but involves the preservation of distinct, steerable reasoning perspectives.
Safety and Control: The ability to isolate and suppress specific behavioral traits (like hallucinations or malicious intent) via activation steering offers a promising new avenue for LLM safety. It suggests that structured multi-agent training creates more modular and controllable representations than standard training, allowing for "surgical" removal of harmful behaviors without breaking the model's general capabilities.

In conclusion, the paper establishes that multi-agent reasoning capabilities can be successfully distilled into a single model's latent space, preserving the benefits of collaboration while enabling efficient inference and granular behavioral control.