Technical Summary: GRPO Does Not Close the Multi-Agent Coordination Gap

Problem Statement

The paper investigates the ability of Large Language Models (LLMs) to coordinate as multiple agents sharing a common resource, a critical capability for systems where LLMs act as autonomous workers, crawlers, or analysts. The authors utilize the Dining Philosophers problem (Dijkstra [1971]) as a clean testbed. In this setup, $N$ agents sit around a circular table with $N$ forks; each agent must hold two adjacent forks to eat. The group succeeds only if all agents eat without deadlock (a state where every agent holds one fork and waits indefinitely).

The study addresses two primary questions:

1. How well do current frontier and open-weight LLMs coordinate in this multi-agent setting?
2. Can Group Relative Policy Optimization (GRPO), fine-tuned on rollouts from the task itself, close the performance gap between weaker open-weight models and frontier systems?

Methodology

Experimental Setup

• Models Evaluated: The study evaluates seven models across 630 episodes (30 episodes per model at philosopher counts $N \in \{5, 10, 15\}$).
  • Frontier Closed-Source: Claude Opus 4.7, GPT-5.4, Gemini 3.1 Pro, Grok 4.2.
  • Open-Source Baseline: Mistral-Small 24B.
  • Target Models: Qwen3-14B (base) and Qwen3-14B fine-tuned with GRPO.
  • Supplementary: A smaller-scale experiment (270 episodes) involving Qwen3-8B, Qwen3-8B+GRPO, and DeepSeek-R1-Distill-Qwen-7B.
• Environment: Each philosopher is implemented as a LangGraph agent wrapping the LLM. Agents interact via six tools: pick-up-left, pick-up-right, put-down-left, put-down-right, eat, and think.
• Reward Function: A four-component scalar reward ($r$) is calculated per episode:
  $$r = \alpha \cdot r_{nd} + \beta \cdot r_{thru} - \gamma \cdot r_{fair} - \delta \cdot r_{idle}$$
  Where:
  • $r_{nd}$: Binary indicator for no deadlock (weight $\alpha=1.0$).
  • $r_{thru}$: Normalized throughput (meals eaten).
  • $r_{fair}$: Variance in meals per philosopher (penalty).
  • $r_{idle}$: Fraction of think (no-op) actions (penalty).
  • Note: The weights are $\alpha=1.0, \beta=0.5, \gamma=0.3, \delta=0.1$.

Training Procedure

• Algorithm: Group Relative Policy Optimization (GRPO) as implemented in OpenPipe ART.
• Configuration:
  • Base models: Qwen3-14B and Qwen3-8B.
  • Adapter: LoRA (rank 8).
  • Data: Rollouts collected at $N=5$ philosophers, 5 rounds per episode.
  • Training Steps: 16 steps total, with 2 groups of 3 episodes sampled per step.
  • Checkpointing: The default policy saves the adapter from the final step (Step 15).

Key Results

1. The Coordination Gap

There is a statistically significant performance gap between frontier models and the open-weight Qwen3-14B:

• Frontier Models: Achieve mean rewards between 0.45 and 0.87 at $N=5$. Deadlock rates range from 0.10 to 0.53.
• Mistral-Small 24B: Achieves a mean reward of 0.83 at $N=5$ (and up to 0.99 at higher $N$), outperforming several frontier models at larger scales.
• Qwen3-14B (Base): Achieves a mean reward of only 0.13 at $N=5$, with a deadlock rate of 0.87.

2. Ineffectiveness of Naive GRPO Fine-Tuning

Fine-tuning Qwen3-14B with GRPO on task rollouts did not close the gap:

• Statistical Significance: A Welch's t-test on per-episode rewards at $N=5$ yielded $p=0.66$ and Hedges' $g = -0.11$. The trained model (0.092) was numerically slightly worse than the base (0.128), but the difference was not statistically significant.
• Scaling: The lack of improvement persisted at $N=10$ and $N=15$.
• Conclusion: The bottleneck for the 14B model is not compute capacity but training methodology.

3. Training Dynamics and Checkpoint Issues

• Reward Collapse: Training reward trajectories for both 8B and 14B runs peaked at Step 9 and subsequently declined.
• Suboptimal Checkpoint: The default saved checkpoint (Step 15) was strictly worse on the training distribution than the peak performance at Step 9. The authors note that without a validation set, the default "save last step" policy selected a degraded model.

4. Reward Function Degeneracy

The paper identifies a critical flaw in the reward formulation:

• Degenerate Maximum: The reward formula allows a maximum score of 1.0 for an episode with zero meals and zero actions, provided no deadlock occurs (since $r_{nd}=1$ and all other terms are zero).
• Empirical Evidence:
  • DeepSeek-R1-Distill-Qwen-7B: Scored 1.0 across all configurations because it emitted no parseable tool calls (inaction), thus avoiding deadlock.
  • Mistral-Small 24B: At $N=5$, it achieved a mean reward of 0.83 with zero meals, indicating it took actions that failed to result in eating but avoided deadlock.
• Implication: The reward signal prioritizes "not failing" (no deadlock) over "making progress" (eating), which can incentivize inaction or trivial strategies.

Significance and Claims

The paper claims that capacity alone does not produce coordinated multi-agent behavior at the open-weight 14B scale. The primary contributions are:

1. Empirical Null Result: Demonstrating that standard GRPO fine-tuning on task rollouts fails to improve multi-agent coordination for Qwen3-14B, leaving it far behind frontier models and even a 24B open-source baseline.
2. Methodological Critique: Identifying that the "default recipe" for RL fine-tuning (specifically checkpoint selection and reward shaping) is insufficient. The authors argue that the bottleneck is not the model's inherent capability but the training methodology.
3. Three Interventions: The paper proposes three specific areas for future intervention to improve results:
   • Reward Shaping: Modifying the reward to prevent the degenerate maximum (e.g., requiring at least one meal for the no-deadlock bonus).
   • Checkpoint Discipline: Selecting the best checkpoint based on training reward or a held-out validation set rather than the final step.
   • Curriculum: Implementing a curriculum across problem scales (philosopher counts), which was not tested in this study.

The authors maintain a modest stance, noting that their findings are scoped to the specific "default-recipe" GRPO run on Qwen3-14B at $N=5$. They do not claim GRPO is fundamentally incapable of solving the problem, but rather that the specific implementation and hyperparameters used failed to bridge the coordination gap.