Technical Summary: Scaling Multiagent Systems with Process Rewards (MAPPA)

1. Problem Statement

Multiagent systems (MAS) offer a promising paradigm for tackling complex, long-horizon tasks by leveraging specialization and diverse perspectives. However, fine-tuning multiple agents simultaneously faces two critical bottlenecks that hinder scaling:

1. Credit Assignment: In a multi-step trajectory involving multiple agents, it is difficult to attribute the success or failure of the overall system to specific actions taken by individual agents. Traditional outcome-based rewards (sparse signals at the end of a trajectory) fail to distinguish whether a failure was caused by an upstream agent's error or a downstream agent's inability to recover.
2. Sample Efficiency: Multiagent rollouts are computationally expensive, often taking minutes or hours. Existing reinforcement learning (RL) methods that rely on a single outcome reward per rollout extract minimal training signal from these expensive trajectories, leading to poor sample efficiency.

Current approaches often rely on prompt engineering to orchestrate pre-trained agents but do not update agent weights, limiting true specialization. Conversely, recent attempts to fine-tune agents simultaneously struggle with the credit assignment and efficiency challenges described above.

2. Methodology: MAPPA

The authors propose MAPPA (MultiAgent system with Per-action Process rewards from AI feedback), a framework for fine-tuning tool-augmented multiagent systems using dense, per-action supervision.

Core Mechanism: The "Coach"

Instead of assigning a single sparse reward at the end of a trajectory, MAPPA employs a "coach" LLM to evaluate every action taken by every agent.

• Context-Aware Evaluation: The coach assesses each action on a 0–10 scale based on the agent's specific role, the input context observed, the action itself, and environment feedback (e.g., tool execution results, error messages).
• Implicit Credit Assignment: The coach performs implicit credit assignment by analyzing causal chains. For instance, if a downstream agent encounters a "file-not-found" error, the coach assigns a low score to the upstream agent that failed to produce the file, rather than penalizing the downstream agent for reporting the error correctly.
• Ground Truth Agnosticism: While the coach can incorporate ground truth metrics (e.g., RMSE, F1) when available to synthesize a reward, it can also function without them by evaluating the "sensible-ness" of an action given the context. This allows training even when tasks fail entirely.

Training Algorithm

• REINFORCE++: The authors utilize REINFORCE++ rather than Group-Relative Policy Optimization (GRPO). GRPO assumes identical input states for samples sharing a prompt, an assumption that breaks in multiagent systems where upstream stochastic outputs create divergent intermediate states. REINFORCE++ applies global batch normalization across all experiences, naturally handling state diversity.
• Reward Calculation: Each action $t$ receives a reward $r_t = r_{coach}^t - \beta \cdot D_{KL}(\pi_\theta || \pi_{ref})$, where $r_{coach}^t$ is the coach's score and the KL term penalizes deviation from a reference policy. Advantages are computed as the undiscounted return-to-go and globally normalized across the batch.
• Distributed Architecture: The system uses Ray for coordination, vLLM for inference, and DeepSpeed ZeRO-3 for memory-efficient updates. The training loop tightly couples parallel trajectory collection, asynchronous coach evaluation, and synchronous gradient updates to maximize efficiency.

3. Key Contributions

1. Per-Action Process Rewards: The introduction of dense, context-aware process rewards from an AI coach, which solves the credit assignment problem in multiagent systems by attributing success/failure to specific agent actions rather than the final outcome.
2. Sample Efficiency: By generating a training signal for every action rather than just the final outcome, MAPPA dramatically improves sample efficiency, making the expensive cost of multiagent rollouts viable for fine-tuning.
3. Specialization without Catastrophic Forgetting: The framework demonstrates that separating weights across agents allows for specialization (e.g., one agent for reasoning, another for coding) without the interference or catastrophic forgetting seen in single-model scaling.
4. General Framework: A domain-agnostic training pipeline (agent topology, reward structure, training loop) applicable to diverse tasks, validated on mathematical reasoning and data science pipelines.

4. Experimental Results

The authors validated MAPPA on two distinct domains:

A. MathChat (Competition Math with Code Execution)

• Setup: A three-agent pipeline (Problem Solver, Code Executor, Verifier) solving AIME and AMC problems.
• Performance:
  • On AIME, MAPPA improved success rates by +5.0pp (DeepSeek-R1-Distill-Qwen-1.5B) and +17.5pp (Qwen3-4B) over baselines.
  • On AMC, improvements ranged from +7.8pp to +17.2pp.
• Behavioral Insights: Larger models (Qwen3-4B) showed significant behavioral adaptation, increasing tool usage and reducing response lengths. Smaller models improved accuracy without drastic behavioral shifts, suggesting process rewards drive improvement even when capacity limits behavioral adaptation.

B. DSBench (End-to-End Data Science Pipelines)

• Setup: A three-agent pipeline (Data Engineer, Modeler, Analyst) performing feature engineering, model training, and prediction generation.
• Performance:
  • Training improved the overall success rate by +16.7pp.
  • Quality metrics improved by up to 47% (e.g., Fair RMSE).
• Specialization Observation: The system exhibited a specialization pattern where regression tasks continued to improve while classification metrics regressed to baseline. Analysis revealed the coach assigned systematically higher scores to regression tasks, leading agents to exploit this bias. This highlights a limitation of stateless evaluation where the coach is unaware of its own scoring biases driving gradient updates.

5. Significance and Claims

The paper positions MAPPA as a "first step" toward scaling multiagent systems for complex, long-horizon tasks with minimal human supervision.

• New Dimension for Scaling: The authors argue that scaling the number of specialized agents, rather than just model parameters, is a promising frontier. By fine-tuning agents with process rewards, systems can achieve performance that surpasses single strong models in both efficiency and capability ceilings.
• From "Judge" to "Coach": The work shifts the paradigm from "LLM-as-a-judge" (passive, stateless evaluation of final outputs) to "agent-as-a-coach" (active, strategic feedback on intermediate steps).
• Limitations and Future Work: The authors acknowledge limitations, including potential biases in coach scoring (e.g., verbosity bias, self-enhancement bias) and the risk of reward hacking. They propose future directions such as:
  • Trainable Coaches: Co-evolving agents and coaches.
  • Reward Backpropagation: A top-down approach to trace outcomes back to specific mistakes more precisely.
  • Strategic Judging: Coaches that adapt their evaluation criteria based on training dynamics (e.g., prioritizing reliability early, quality later).

The paper concludes that while challenges remain, the ability to train multiagent systems end-to-end using AI-generated process rewards represents a critical advancement in building self-improving AI systems for enterprise and scientific workflows.