Technical Summary: Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a primary method for enhancing the reasoning capabilities of Large Language Models (LLMs) on tasks with objectively checkable outcomes. However, a persistent issue known as diversity collapse plagues this paradigm: while the model's single-sample accuracy (Pass@1) improves, its ability to find a correct solution across multiple attempts (high-k Pass@k) frequently degrades as training progresses.

Previous explanations attribute this to the self-reinforcing nature of on-policy updates, where trajectories already likely to succeed are sampled and reinforced more often, sharpening the policy distribution onto a shrinking subset of solutions. This paper argues that this "sharpening bias" is a universal property of RLVR updates but does not fully explain when an update expands the model's reasoning boundary versus when it merely reshapes probability mass among already reachable solutions. The authors propose viewing diversity collapse through the lens of overtraining: once a problem's contribution to a reference metric (specifically high-k Pass@k) has saturated, further updates cannot expand the reasoning boundary but continue to concentrate probability mass, leading to collapse.

Methodology

Theoretical Analysis: Overtraining and Saturation

The authors formalize the reasoning boundary using Pass@k, defined as $1 - (1-p)^k$ for a problem with success probability $p$. They demonstrate that under standard RLVR setups (typically $n=8$ rollouts per problem), the empirical success rate $\hat{p}$ is quantized (e.g., $0/8, 1/8, \dots$).

• Aggressive Saturation: For a large $k$ (e.g., $k=256$), even a single observed success ($\hat{p}=1/8$) yields a Pass@256 value of $\approx 1$.
• Overtraining Regime: Consequently, any problem that has produced at least one correct rollout is effectively in a "saturated" regime regarding the reasoning boundary. Updates on these problems (whether reinforcing correct trajectories or suppressing incorrect ones) provide negligible marginal gain to high-k Pass@k but continue to sharpen the policy, causing diversity collapse.
• The Zero-Success Bucket: The only bucket where updates can meaningfully expand the reasoning boundary is $B_{0/n}$, where no correct rollouts have been observed.

Empirical Investigation

The study employs two complementary approaches to validate that RLVR does expand the reasoning boundary, even if aggregate metrics suggest otherwise:

1. Observational Tracking: Tracking problems that are initially unsolvable (Pass@256 = 0) under the base model. The authors find that a non-trivial fraction of these problems become solvable during standard training, even as a larger number of previously solvable problems exit the boundary.
2. Interventional Restriction: Training variants where updates are restricted to specific rollout-success buckets ($B_{m/8}$). Restricting updates to $B_{0/8}$ (the only non-saturated bucket) results in Pass@256 improvements over the base model on difficult benchmarks, whereas updates on higher-success buckets lead to declines.

Proposed Solution: Bayesian Boundary Gating (BBG)

To mitigate overtraining, the authors propose Bayesian Boundary Gating (BBG), a problem-level method that redirects optimization away from saturated problems.

• Mechanism: BBG estimates the expected marginal contribution of a problem to the reasoning boundary using a Beta-Binomial Bayesian model. Given $m$ successes in $n$ rollouts, the posterior distribution of the true success probability $p$ is $Beta(m+1, n-m+1)$.
• Utility Calculation: The expected marginal utility $u(m; n, k)$ is calculated as the expectation of the derivative of the Pass@k objective: $E[k(1-p)^{k-1}]$.
• Gating Strategy: Problems with negligible boundary utility (where the estimated marginal contribution falls below a threshold $\tau$) are hard-gated and excluded from backpropagation. Only problems with significant estimated utility (primarily those in low-success buckets) receive updates.
• Implementation: BBG is applied as a coefficient on the problem-level loss, compatible with existing objective designs like W-RF (Weighted Reinforcement with Negative Reinforcement).

Key Results

Experiments were conducted using Qwen2.5-Math-7B trained on the MATH dataset, evaluated across four benchmarks: MATH-500, OlympiadBench, MinervaMath, and AIME 2025.

• Baseline Performance: Standard RLVR methods (REINFORCE, GRPO, W-RF) consistently improve Pass@1 but suffer significant degradation in high-k Pass@k (e.g., Pass@256) on intermediate-difficulty benchmarks, confirming the diversity collapse phenomenon.
• BBG Performance:
  • BBG achieves the highest average Pass@k across the range $k \in [1, 256]$ on all benchmarks.
  • Crucially, BBG is the only method that maintains or improves Pass@256 relative to the base model across all four benchmarks, effectively preventing the collapse observed in other methods.
  • On the most difficult benchmark (AIME 2025), BBG shows substantial gains across the entire evaluated range, whereas other methods degrade at high $k$.
• Ablation Insights: The authors show that simple interventions like early stopping (stopping when validation Pass@1 saturates) or bucket masking (excluding high-success buckets) can partially recover high-k performance, validating the overtraining hypothesis. However, BBG provides a more principled and effective solution by dynamically weighting updates based on estimated boundary utility.

Significance and Claims

The paper claims three primary contributions:

1. Formalization of Diversity Collapse: It reframes diversity collapse not merely as a sampling bias issue but as a consequence of overtraining relative to the reasoning boundary. It demonstrates that under standard rollout budgets, the vast majority of RLVR updates are overtraining for high-k Pass@k.
2. Reinterpretation of Aggregate Decline: The paper provides evidence that the widely reported aggregate decline in high-k Pass@k does not imply a failure to acquire new reasoning capabilities. Instead, it reflects a net outcome where gains on initially unsolvable problems are masked by overtraining-induced losses on already-solvable problems.
3. Effective Mitigation via BBG: By redirecting optimization toward problems with non-saturated boundary contributions, BBG successfully decouples the improvement of Pass@1 from the degradation of high-k Pass@k, improving the overall reasoning diversity of the model.

The authors maintain a modest tone regarding limitations, noting that their experiments are limited to 7B-scale models and a single training set. They acknowledge that diversity collapse may have other causes (e.g., token-level credit assignment) and that BBG's suppression of updates on high-success problems may reduce Pass@1 in scenarios where single-sample accuracy is the sole priority. However, the work establishes a new perspective on RLVR training dynamics, suggesting that the "sharpening" of the policy is often a byproduct of optimizing for a metric that has already been maximized for specific problems.