Technical Summary: SHARP – Shapley Credit-based Optimization for Multi-Agent Systems

1. Problem Statement

The integration of Large Language Models (LLMs) with external tools via Multi-Agent Systems (MAS) offers a promising paradigm for solving complex reasoning and planning tasks. However, training these systems is hindered by the credit assignment challenge. In collaborative MAS architectures (typically involving a planner and specialized worker agents), it is often unclear which specific agent is responsible for the success or failure of a decision trajectory.

Existing methods predominantly rely on sparse, globally broadcast rewards that treat the multi-agent team as a monolithic entity. This approach fails to capture individual contributions, leading to inefficient reinforcement learning (RL) updates where high-quality individual actions are obscured by overall team performance. Furthermore, current value-free optimization methods (e.g., DPO, GRPO) and specialized MARL frameworks often struggle to disentangle the marginal contributions of individual agents within a joint action space.

2. Methodology: SHARP Framework

To address these limitations, the authors propose SHARP (Shapley-based Hierarchical Attribution for Reinforcement Policy), a framework designed to stabilize multi-agent training through precise, fine-grained credit attribution.

2.1. System Architecture

SHARP adopts a self-play multi-agent modeling paradigm using a shared policy $\pi_\theta$.

• Roles: A single model instantiates heterogeneous roles (Planner and Workers) via role-specific system prompts.
• Execution: The planner generates high-level subtask queries, which trigger worker agents to interact with external tools (e.g., search, calculators, code interpreters).
• Optimization: The entire process is optimized end-to-end as a single trajectory under a shared policy, allowing for unified parameter updates.

2.2. Tripartite Decomposed Reward Mechanism

The core innovation of SHARP is a reward decomposition principle that moves beyond monolithic broadcast signals. The total surrogate reward $\bar{R}_{i,m}$ for agent $m$ on trajectory $i$ is a weighted combination of three components:

1. Global Broadcast-Accuracy Reward ($R^b$): A binary terminal signal ($0$ or $1$) indicating whether the final response aligns with the ground truth. This ensures global task alignment and is broadcast to all agents in the trajectory.
2. Shapley-Based Marginal-Credit Reward ($R^{mc}$): This component quantifies the specific contribution of each agent using Shapley values from cooperative game theory.
   • Counterfactual Masking: The system isolates an agent's causal impact by measuring the performance delta ($\Delta$) when that agent is ablated (masked) from the decision trajectory while preserving the environment and other agents.
   • Planner Credit: The planner's credit is defined as the mean of the positive marginal contributions of its worker pool, aligning the planner's objective with collective utility.
   • Worker Credit: Calculated as the difference in task success between the full trajectory and the trajectory with the specific worker removed.
3. Tool-Process Reward ($R^{tool}$): A process-level signal that evaluates the validity and executability of tool invocations, ensuring that agents do not merely hallucinate tool usage but execute them correctly.

2.3. Optimization Strategy

SHARP utilizes an improved Group Relative Policy Optimization (GRPO) approach.

• Advantage Normalization: Agent-specific advantages are normalized across trajectory groups by subtracting the mean and dividing by the standard deviation of rewards for that specific agent role.
• Clipped Surrogate Objective: The framework employs a clipped objective function to ensure stable policy updates, preventing large deviations that could destabilize the collaborative learning process.

3. Key Contributions

The paper outlines three primary contributions:

1. SHARP Framework: A novel RL framework that stabilizes multi-agent training by normalizing agent-specific advantages and providing precise credit attribution, offering a unified solution for aligning heterogeneous agents.
2. Flexible Reward Decomposition: The introduction of a tripartite reward mechanism (Global Alignment, Shapley-based Marginal Credit, and Process Validity) applicable to various LLM sizes and multi-agent structures (sequential chains, communication graphs, and hierarchical planner-worker paradigms).
3. Empirical Validation: Extensive experiments demonstrating that SHARP significantly outperforms state-of-the-art baselines, including single-agent RL and existing multi-agent frameworks.

4. Experimental Results

The authors evaluated SHARP across five real-world benchmarks: MuSiQue, GAIA-text, WebWalkerQA, FRAMES, and DocMath-Eval.

• Performance Gains: Using a Qwen3-8B backbone, SHARP achieved average match improvements of 23.66% over single-agent approaches and 14.05% over existing multi-agent baselines.
• Ablation Studies: Removing Shapley-based credit assignment resulted in significant performance drops, confirming that marginal credit modeling is a key driver of success, more so than architectural complexity alone.
• Scalability: SHARP demonstrated strong scaling laws, achieving a 14.41-point improvement over single-agent baselines on an 8B backbone. Performance gains widened as model size increased.
• Stability: Unlike baselines that exhibited fluctuations during training, SHARP showed monotonic improvement in accuracy over 180 optimization steps, indicating reduced reward variance.
• Coordination Analysis: SHARP reshaped agent interactions by increasing the proportion of "useful" subagents and reducing "harmful" subagent calls from 5.48% to 4.40%.

5. Significance and Claims

The paper claims that SHARP provides a mathematically grounded foundation for addressing the credit assignment bottleneck in collaborative multi-agent systems. By explicitly disentangling individual causal impacts from aggregate outcomes, SHARP enables:

• Stable Joint Optimization: Facilitating coherent coordination among heterogeneous agents.
• Interpretability: Offering insights into which agents contribute positively or negatively to a task.
• Generalization: Demonstrating robust performance across diverse task distributions and model scales.

The authors position SHARP not merely as a performance booster but as a principled approach to scalable and interpretable coordination in hierarchical MAS, suggesting that precise credit attribution is essential for the effective training of tool-augmented multi-agent systems.