ReCo: Reweighting GRPO Against Distributional Concentration

Problem

Group Relative Policy Optimization (GRPO) has emerged as a standard reinforcement learning method for post-training language models, particularly in domains with verifiable rewards like mathematics and coding. While GRPO improves sampling efficiency for small sampling budgets (low $k$ in Pass@$k$), recent observations indicate it fails to surpass, and often underperforms, the base model when the sampling budget is large (high $k$). This suggests a reduction in the coverage of reasoning paths. The paper identifies that GRPO concentrates probability mass on responses that the base model already generates with high probability, thereby reducing the diversity of problem-solving trajectories and limiting the model's ability to discover correct solutions that lie in lower-probability regions of the base distribution.

Methodology

The authors propose ReCo (Reweighting GRPO Against Distributional Concentration), a method designed to mitigate this concentration by modifying the GRPO update at two levels:

1. Response-Level Reweighting:
   In standard GRPO, high-probability responses are sampled more frequently within a rollout group. Since the group advantage is averaged equally across sampled responses, these frequent trajectories accumulate a larger aggregate gradient mass simply due to repeated occurrence. ReCo addresses this by normalizing each response's gradient contribution by its expected occurrence count within the group.

   • The weight is defined as the inverse of the expected count: $w_i = \frac{1}{G \cdot \bar{\pi}_{\theta_{old}}(o_i | q)}$, where $\bar{\pi}$ is the length-normalized likelihood.
   • This prevents frequent responses from dominating the group-level gradient, ensuring that the update does not disproportionately reinforce trajectories that are already likely under the old policy.

2. Token-Level Reweighting:
   Standard GRPO uses an importance ratio ($\pi_\theta / \pi_{\theta_{old}}$) to scale token-level gradients. This creates a feedback loop where tokens that become more likely under the current policy receive larger updates, further reinforcing saturated decision points and reducing alternative paths.

   • ReCo replaces the standard importance ratio with a variance-based ratio: $r_{var} = \frac{p_\theta(1 - p_\theta)}{p_{old}(1 - p_{old})}$.
   • The term $p(1-p)$ represents the Bernoulli variance. This ratio assigns larger update scales to non-saturated decision points (where $p$ is neither close to 0 nor 1, meaning alternative token choices remain plausible) and downweights saturated choices (where $p \approx 1$).
   • This mechanism encourages updates at decision points where the policy has not yet collapsed to a single choice, thereby preserving reasoning diversity.

Key Contributions

• Analysis of GRPO Concentration: The paper traces the reduction in Pass@$k$ for large $k$ to two specific mechanisms in the GRPO update: the dominance of frequent responses at the group level and the amplification of saturated tokens at the token level.
• ReCo Algorithm: The introduction of a two-level reweighting scheme that normalizes response contributions by expected occurrence and replaces the token importance ratio with a variance-based metric.
• Empirical Validation: Extensive evaluation across different model families (Qwen2.5-Math, Llama-3.1) and scales (1.5B, 7B, 8B) on five mathematical reasoning benchmarks (AIME 2024/2025, AMC 2023, OlympiadBench, MATH 500).

Results

• Pass@$k$ Performance: ReCo consistently outperforms GRPO on Pass@$k$ for large values of $k$ (e.g., Pass@64), indicating improved coverage of correct reasoning trajectories. For small values of $k$, ReCo remains comparable to GRPO, maintaining sampling efficiency.
• Base Model Recovery: On the Llama-3.1-8B-Instruct backbone, where standard GRPO was observed to fall below the base model's performance on Pass@$k$, ReCo significantly reduces this degradation and even surpasses the base model on certain benchmarks (AIME 2024/2025).
• Diversity Metrics: Training dynamics analysis shows that while GRPO rapidly loses diversity (decreasing Distinct-2, increasing Self-BLEU), ReCo maintains rollout diversity closer to the base model throughout training.
• Token Variance: ReCo maintains substantially higher token-level Bernoulli variance in the early and middle segments of responses compared to GRPO, indicating that reasoning paths remain non-saturated for longer.
• Generalizability: The method also improves Pass@$k$ on code reasoning benchmarks (HumanEval, HumanEval+, LiveCodeBench), suggesting the concentration issue is not limited to mathematical reasoning.
• Compatibility: ReCo can be combined with other GRPO variants (e.g., DAPO, 80/20 Rule) to yield further improvements.

Significance

The paper claims that ReCo addresses the internal mechanisms of GRPO that lead to distributional concentration without requiring external diversity objectives or regularizers. By reweighting the update itself based on the likelihood of occurrence and the saturation of decision points, ReCo preserves a broader set of correct reasoning paths. This allows the model to retain the sampling efficiency gains of reinforcement learning while expanding its coverage of the solution space, effectively mitigating the "diversity collapse" observed in standard GRPO training. The work suggests that the limitations of RLVR in expanding reasoning capacity are partly due to the optimization dynamics of the policy update itself, which can be corrected through targeted reweighting.