Technical Summary: STRONGER-MAS

Problem Statement

The paper addresses the underexplored challenge of applying on-policy Reinforcement Learning (RL) to Multi-Agent Systems (MAS) for Large Language Models (LLMs). While MAS improves performance through role-based orchestration and RL strengthens policies via environment rewards (e.g., Group Relative Policy Optimization, or GRPO), combining them presents two critical bottlenecks:

1. Algorithmic Mismatch: Standard GRPO assumes a single agent generating multiple responses to an identical prompt to form a comparison group. In MAS, prompts differ significantly across turns and roles due to evolving interaction histories and role-specific contexts. Applying standard grouping leads to invalid advantage estimates because candidates in a group do not share the same prompt, violating the fairness required for credit assignment.
2. Systemic Limitations: Existing on-policy RL frameworks (e.g., VERL, AReaL) are designed for single-model, single-policy training. They lack the infrastructure to support concurrent rollouts of multiple heterogeneous agents, manage diverse MAS workflows, and perform independent on-policy updates for multiple policy models simultaneously.

Methodology

The authors propose STRONGER-MAS, a framework comprising a novel algorithm (AT-GRPO) and a dedicated MAS Training System.

1. AT-GRPO Algorithm

Agent- and Turn-wise Grouped Relative Policy Optimization (AT-GRPO) adapts group-relative optimization for the multi-agent setting.

• Tree-Structured Sampling: Instead of sampling $K$ full trajectories from the initial state (which results in incomparable groups after the first turn), the system samples $K$ candidate actions at each turn $t$ for each agent $i$ from the current state.
• Agent- and Turn-wise Grouping: Experiences are grouped based on the acting agent and the specific turn number. A unique group key is generated for each agent-turn pair. This ensures that all $K$ candidates within a group share the identical prompt (observation context + role-specific instructions + interaction history), satisfying the requirement for fair advantage calculation.
• Greedy Execution: From the $K$ sampled candidates, the action with the highest reward is greedily selected to proceed to the next turn, concentrating exploration on coordination-critical decisions.
• Credit Assignment: The reward function combines a global team reward ($r_{team}$) and agent-specific local rewards ($r_{loc}$) via a mixture coefficient $\alpha$: $r_{t,i} = \alpha r_{team} + r_{loc}$. This balances shared objectives with role-specific incentives.

2. MAS Training System

To support AT-GRPO, the authors designed a novel training system that decouples resource pools for different policies:

• Independent Resource Pools: Each policy (whether shared or role-specific) is assigned an independent GPU resource pool containing a RolloutWorker (for inference) and an UpdateWorker (for optimization).
• Concurrent Rollouts: The system orchestrates diverse MAS workflows where multiple agents interact with a CPU-based environment pool.
• Dynamic Routing: Trajectory data generated by agents is routed to the specific UpdateWorker corresponding to their assigned policy, enabling true on-policy updates for multiple models simultaneously.

Key Contributions

1. AT-GRPO Algorithm: A new RL algorithm that enables valid advantage estimation in multi-turn, multi-agent environments by grouping experiences based on agent identity and turn number rather than just the initial prompt.
2. MAS Training System: A scalable infrastructure capable of executing rollouts for diverse MAS workflows and performing independent on-policy updates for multiple policies (both role-sharing and role-specialized).
3. Empirical Validation: Extensive experiments across four domains (Game, Planning, Coding, Math) demonstrating that AT-GRPO significantly outperforms single-agent RL and standard MAS baselines.
4. Policy Analysis: A systematic investigation into the trade-offs between role-sharing policies (one model for all agents) and role-specialized policies (distinct models per role), showing that the optimal choice depends on task characteristics (e.g., specialization benefits coding, while sharing may benefit math).

Results

The method was evaluated on Qwen3 models (1.7B and 8B) across various benchmarks:

• Long-Horizon Planning: AT-GRPO overcomes a key bottleneck of single-agent RL. On tasks like Sokoban and Plan-Path, it boosted accuracy from a baseline of 14.0–47.0% to 96.0–99.5%.
• Coding and Math: The approach yielded consistent gains.
  • Coding: Average improvements of 3.87–7.62% on benchmarks like LiveCodeBench and APPS.
  • Math: Average improvements of 9.0–17.93% on benchmarks like AIME and OlympiadBench.
• Role Specialization:
  • In coding tasks (Coder vs. Tester), role-specialized policies outperformed shared policies, improving accuracy by ~3 points (Qwen3-1.7B).
  • In math tasks, shared policies sometimes outperformed specialized ones, suggesting that functional overlap allows a single model to leverage data from both reasoning and tool-use roles.
• Scalability: The system demonstrated effective scaling with ensemble sizes (e.g., increasing from 1 to 7 agents), achieving continuous performance gains where baseline methods saturated.
• Comparison: AT-GRPO outperformed existing MARL frameworks like MAPoRL, MARFT, and CURE, particularly in multi-turn iterative alignment scenarios.

Significance and Claims

The paper claims that STRONGER-MAS provides a general solution for training cooperative LLM-based MAS. Its significance lies in:

• Bridging the Gap: It successfully integrates the strengths of MAS (collaboration, role specialization) with the performance gains of on-policy RL, a combination previously hindered by algorithmic and system limitations.
• Emergent Collaboration: The results suggest that on-policy RL within MAS fosters emergent collaboration, where agents learn to co-evolve specialized skills and align their actions more efficiently (reducing the number of turns to solve tasks).
• Task-Dependent Architecture: The work highlights that there is no "one-size-fits-all" policy architecture; the choice between shared and specialized policies must be determined by the specific characteristics of the task (e.g., functional overlap vs. distinct roles).
• Robustness: The method remains effective even when using sparse, outcome-only rewards, indicating that the algorithmic improvements of AT-GRPO are not solely dependent on complex reward shaping.

The authors conclude that their framework offers a scalable and faithful abstraction for training cooperative LLM agents, overcoming the coordination bottlenecks that limit current single-agent or naive multi-agent RL approaches.