Technical Summary: SAT: Sequential Agent Tuning for Coordinator-Free Plug-and-Play Multi-LLM Training

1. Problem Statement

Large Language Models (LLMs) with massive parameter counts achieve strong performance but face prohibitive deployment costs regarding memory, computation, and energy. While recent approaches propose using teams of smaller, efficient LLMs to collectively match or exceed single large models, they face two critical challenges:

1. Coordination Complexity: Jointly updating multiple agents introduces compounding distribution shifts, making training unstable and difficult to coordinate without a central controller.
2. Theoretical Gaps: Existing multi-agent methods often rely on predefined role assignments (e.g., planner/solver) which constrain system potential, and they lack robust theoretical foundations guaranteeing that multi-agent systems will achieve strong, stable improvements.

The paper addresses the question: Can teams of small, efficient models collectively achieve or surpass the performance of one well-tuned large model, while ensuring training stability and allowing for modular upgrades?

2. Methodology: Sequential Agent Tuning (SAT)

The authors propose Sequential Agent Tuning (SAT), a coordinator-free training paradigm that treats the team as a factorized policy and employs block-coordinate updates.

Core Mechanisms

• Factorized Policy & Masked Activation: The team policy is defined as a product of individual agent policies: $\pi(a|s) = \prod_{i=1}^n \pi^{(i)}(a_i|s)$. To handle scenarios where only subsets of agents act at a given state (e.g., tool routers), the framework uses a "masked activation" view where inactive agents take a fixed no-op action.
• Sequential Block-Coordinate Updates: Instead of updating all agents simultaneously, agents are updated sequentially in a specific order $\sigma$. When updating agent $\sigma(i)$, the system conditions on an intermediate policy $\hat{\pi}_{i-1}$, which combines the newly updated policies of previous agents in the sequence with the current policies of the remaining agents.
• Sequence-Aware Advantage Estimator: To mitigate distribution shift caused by sequential updates, SAT employs an on-policy advantage estimator that conditions on the current intermediate occupancy ($d_{\hat{\pi}_{i-1}}$). This ensures the advantage is evaluated under the policy state at the moment of the update, rather than the initial team policy.
• Per-Agent KL Trust Regions: To isolate occupancy drift, each agent update is constrained by a per-state KL divergence trust region ($D_{KL}(\pi^{(i)}_{tar} || \pi^{(i)}_{cur}) \leq \delta_i$). This limits how much the state visitation distribution can shift during a single agent's update.
• Plug-and-Play Alignment: When an agent is replaced with a stronger pre-trained model, a "Stage-0" alignment step projects the new model onto the existing trust region around the current agent's policy. This ensures the new agent starts within the theoretical bounds required for monotonic improvement.

Algorithmic Implementation

The training loop (Algorithm 1) involves:

1. Collecting on-policy rollouts under the current team policy.
2. Iterating through agents in order.
3. Constructing the intermediate policy $\hat{\pi}_{i-1}$.
4. Computing sequence-level advantages (using Group Relative Policy Optimization principles like group normalization and clipping) conditioned on $\hat{\pi}_{i-1}$.
5. Optimizing the agent with a clipped objective and a KL penalty, enforcing the trust region constraint via a high-quantile monitor.

3. Theoretical Contributions

The paper establishes a theoretical framework providing three critical guarantees:

1. Monotonic Improvement: The framework provides a high-probability certificate that the team's performance $J(\bar{\pi})$ improves monotonically after a full stage of updates. The improvement bound is derived as:
   $$J(\bar{\pi}) - J(\pi_{cur}) \geq \sum (\text{Information-Geometric Gain}) - \sum (\text{Occupancy-Shift Penalty}) - \sum (\text{Estimator Bias}) - \sum (\text{Finite-Sample Error})$$
   The occupancy-shift penalty scales with $\sum \sqrt{\delta_i}$, justifying the use of small per-agent trust regions.

2. Sequence-Agnostic Guarantees: The improvement bounds hold regardless of the order in which agents are updated within a stage. This allows for flexible scheduling or learned update orders.

3. Plug-and-Play Invariance: The theory proves that agents can be upgraded to stronger models without retraining the rest of the team. If the new agent is initialized within the trust region (via Stage-0 alignment), the monotonic improvement guarantees remain valid. Furthermore, upgrades can "tighten" the performance certificate, either increasing the surrogate value at fixed radii or achieving the same gain with smaller radii (reducing cumulative penalty).

4. Experimental Results

The authors evaluated SAT across three domains: mathematical reasoning (AIME24/25, MATH-500, ZebraLogic), active reasoning (ARBench), and planning (AutoLogi, PlanBench).

• Performance vs. Large Models: A team of three 4B-parameter agents (12B total) trained with SAT surpassed the much larger Qwen3-32B model on AIME24/25 benchmarks by an average of 3.9%. A team of three 8B agents (24B total) achieved competitive or superior results against 30B–70B baselines and thinking-model baselines (e.g., OpenAI o3-mini) across multiple metrics.
• Heterogeneity: Experiments showed that mixing model sizes and families (e.g., 2x4B + 1x8B) yields performance gains, though with diminishing returns as capacity increases. Mixing "thinking" and "non-thinking" agents often degraded performance, suggesting the need for compatible sequence-level surrogates.
• Plug-and-Play Validation: The authors validated the plug-and-play theory by swapping agents in a trained team. Replacing two 4B agents with 8B agents in a trained team boosted the composite score by 10.4% without retraining the remaining agent. Full replacement to three 8B agents increased the score to 70.4% (from a baseline of 56.5%) with a low violation rate (2.5%).
• Trust Region Dynamics: Empirical analysis confirmed that the violation rate of the KL constraints scales approximately as $\sqrt{\delta}$, consistent with the theoretical occupancy-shift penalty.

5. Significance and Claims

The paper claims that SAT offers a practical and scalable path for deploying high-performance AI systems, particularly in resource-constrained environments. Its significance lies in:

• Decentralized Training: Eliminating the need for a central controller or predefined roles, allowing agents to learn a factorized policy through sequential updates.
• Theoretical Rigor: Providing the first provable guarantees for monotonic improvement and plug-and-play invariance in multi-LLM training, addressing the stability issues of joint updates.
• Efficiency: Demonstrating that small-model teams can outperform significantly larger monolithic models, challenging the notion that performance scales linearly only with model size.
• Modularity: Enabling the "plug-and-play" replacement of agents, allowing systems to evolve and upgrade components without the cost of full retraining.

The authors position this work as a shift from simply scaling monolithic models to strategically advancing teams of smaller, coordinated agents.