Technical Summary: Advantage Collapse in Group Relative Policy Optimization

1. Problem Definition: Advantage Collapse

The paper identifies a critical failure mode in Group Relative Policy Optimization (GRPO), a leading algorithm for Reinforcement Learning from Verifiable Rewards (RLVR) in Large Language Models (LLMs). While GRPO eliminates the need for a critic network by computing advantages through intra-group reward comparisons, it is susceptible to advantage collapse.

This phenomenon occurs when all responses within a sampled group receive identical rewards (e.g., all correct or all incorrect). In such cases:

• The within-group reward variance ($\sigma_R$) becomes zero.
• The advantage calculation $\hat{A}_i = (r_i - \mu_R) / (\sigma_R + \epsilon)$ collapses to near-zero for all samples.
• The resulting policy gradient vanishes, rendering the training batch ineffective regardless of the individual response quality.

The authors note that this failure is prevalent in mathematical reasoning tasks where binary verification often yields homogeneous outcomes. In experiments across five model scales, 28–45% of training batches exhibited complete advantage collapse. Crucially, this failure remains invisible to standard metrics like loss curves and accuracy, leading to wasted computational resources without immediate detection.

2. Methodology

The paper proposes a unified framework comprising a diagnostic metric and an algorithmic intervention.

2.1. Diagnostic Metric: Advantage Collapse Rate (ACR)

To quantify the severity of this issue, the authors introduce the Advantage Collapse Rate (ACR).

• Definition: ACR measures the proportion of training batches where the standard deviation of rewards ($\sigma_R$) falls below a small threshold $\tau$.
• Mechanism: It leverages reward statistics already computed during standard GRPO training, incurring zero computational overhead.
• Predictive Power: The authors demonstrate that early-stage ACR (measured over the first 100 steps) strongly predicts final model performance ($R^2 = 0.617$). A high ACR correlates with training stagnation, enabling practitioners to identify problematic configurations before significant compute is wasted.

2.2. Algorithmic Intervention: Adaptive Virtual Sample Policy Optimization (AVSPO)

To mitigate collapse without requiring additional model rollouts, the authors propose AVSPO, a plug-and-play extension of GRPO.

• Virtual Samples: When ACR indicates a collapse (homogeneous rewards), AVSPO injects virtual reward samples into the group. These are synthetic reward values, not actual model outputs.
• Stratified Assignment:
  • For all-correct groups, virtual rewards are assigned values slightly below the maximum observed reward to create variance.
  • For all-incorrect groups (common in early training), virtual rewards are assigned small positive values (anchors) to contrast with the zero rewards of actual samples.
• Advantage Recomputation: These virtual samples participate in the normalization statistics (mean and standard deviation) to ensure $\sigma_R' > 0$, thereby generating non-zero advantages for the real samples.
• Adaptive Triggering: The intervention is triggered dynamically based on real-time ACR monitoring and training progress. A threshold adapts to avoid over-intervention during stable training phases.
• Efficiency: AVSPO requires no additional LLM forward passes. The virtual samples do not contribute to the policy gradient (no $\nabla_\theta \log \pi$ term), serving solely to restore the normalization baseline.

3. Key Contributions

1. Diagnostic Metric (ACR): The first metric specifically designed to quantify gradient ineffectiveness in group-based policy optimization. It explains 62% of the variance in final performance, allowing for early detection of training stagnation.
2. Algorithmic Extension (AVSPO): A lightweight method that injects virtual samples to restore reward diversity in collapsed groups. It enables learning from batches that would otherwise be wasted, requiring no additional inference cost.
3. Empirical Validation: Comprehensive experiments across models ranging from 0.5B to 14B parameters on six mathematical reasoning benchmarks.

4. Experimental Results

The authors evaluated AVSPO against baselines including Vanilla GRPO, DCPO, INTUITOR, and RENT.

• Collapse Reduction: AVSPO reduced the Advantage Collapse Rate from 28–45% (GRPO) to 11–18%, representing a 58–63% relative reduction.
• Performance Gains: AVSPO yielded consistent accuracy improvements of 4–6 percentage points over GRPO across all model scales.
  • For example, on the MATH-500 benchmark, Qwen2.5-Math-1.5B improved from 58.6% (GRPO) to 67.2% (AVSPO).
• Generalization: The method maintained generalization on out-of-domain tasks (MMLU-Pro), indicating it does not lead to overfitting on specific reward structures.
• Efficiency: The computational overhead of AVSPO is negligible (<0.01% per step), as it only involves scalar arithmetic on reward statistics.
• Ablation Studies:
  • Stratified Reward Assignment was found superior to random or fixed partial credit strategies.
  • Adaptive Thresholding outperformed fixed thresholds, balancing intervention frequency with training stability.
  • Mechanism Isolation: Both "all-incorrect" and "all-correct" collapse scenarios require specific handling; the full AVSPO method addresses both.

5. Significance and Claims

The paper claims that advantage collapse is a fundamental bottleneck in RLVR training for LLMs, particularly in domains with binary, deterministic verifiers.

• Diagnostic Utility: The authors position ACR as a standard diagnostic tool that can be applied to any group-based policy optimization method, not just GRPO. It provides transparency into training efficiency, allowing practitioners to detect gradient ineffectiveness before accuracy degradation becomes visible.
• Algorithmic Efficiency: By recovering learning signals from collapsed groups without additional rollouts, AVSPO offers a more efficient use of generated trajectories compared to filtering or resampling strategies (which either discard data or increase compute costs).
• Broader Impact: The work aims to advance the field of RLVR by providing tools for more stable and efficient training. The authors suggest that ACR could reduce computational waste and its associated energy footprint, while improving the transparency of LLM alignment pipelines.

The paper concludes that while algorithmic improvements are valuable, real-time diagnostics are equally critical for optimizing the training process of reasoning models.