EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimization

1. Problem Statement

The paper addresses critical stability and efficiency limitations in Reinforcement Learning with Verifiable Rewards (RLVR), specifically within the dominant Group Relative Policy Optimization (GRPO) framework used for training Large Language Models (LLMs) on reasoning tasks (e.g., math, code).

Two primary failure modes of standard GRPO are identified:

• High Estimator Variance: GRPO calculates the advantage function by normalizing rewards against the mean and standard deviation of a small group of sampled responses ($G$). When $G$ is small (due to computational constraints), the local group statistics are noisy, leading to unstable gradients.
• Vanishing Gradients in Saturated Regimes: In "saturated failure" scenarios (where a model fails all attempts for a difficult prompt, yielding all zero rewards), the local group mean becomes zero. Consequently, the relative advantage for all responses vanishes ($r_i - \mu_{group} = 0$), resulting in a null gradient and wasted training steps. Conversely, in saturated success regimes, no relative signal exists to drive improvement.

Existing solutions like DAPO attempt to mitigate this by filtering out unstable updates, which leads to data waste. The paper argues for a method that retains all data while stabilizing the estimation.

2. Methodology: Empirical Bayes Policy Optimization (EBPO)

The authors propose EBPO, a framework that reframes advantage estimation using Empirical Bayes (EB) inference. Instead of estimating baselines in isolation for each prompt, EBPO "borrows strength" from the policy's accumulated global statistics.

Core Mechanism: Shrinkage Estimation

EBPO replaces the purely local GRPO baseline with a shrinkage estimator that dynamically blends the noisy local group mean ($\mu_{group}$) with a stable global prior ($\mu_{glob}$).

• The Model: The latent success probability of a prompt $\theta_q$ is assumed to be drawn from a global distribution $\theta_q \sim \mathcal{N}(\mu_{glob}, \tau^2)$.
• The Baseline: The EBPO baseline $V_q^{EB}$ is defined as:
  $$V_q^{EB} = (1 - S_q)\mu_{group} + S_q\mu_{glob}$$
  Where $S_q$ is the shrinkage factor determined by the ratio of within-group variance ($\sigma^2$) to between-group variance ($\tau^2$):
  $$S_q = \frac{\sigma^2/G}{\sigma^2/G + \tau^2}$$
• Online Updates: To ensure scalability, global statistics ($\mu_{glob}, \sigma^2, \tau^2$) are updated dynamically using Welford's online algorithm, avoiding the noise of small-batch statistics.

Key Innovations

1. Non-Vanishing Gradients: Even when a model fails all attempts in a group ($r_i=0$), the EBPO baseline remains non-zero (driven by $\mu_{glob}$). This generates a negative advantage signal ($0 - V_q^{EB} < 0$), providing informative gradients to penalize failure on difficult tasks rather than stalling.
2. Difficulty-Aware Penalty: The penalty for failure scales with the global task difficulty. Failing an "easy" task (high $\mu_{glob}$) incurs a larger penalty than failing a "hard" task, allowing the model to distinguish between global difficulty and local noise.
3. Curriculum Learning via Clustering: The authors introduce Clustered Sampling (Topic or Difficulty-based). By training on coherent clusters rather than random shuffles, the online estimator adapts to consistent local distributions, reducing the estimation error of the prior and preventing "mismatch" between the prior and specific task domains.

3. Theoretical Guarantees

The paper provides rigorous theoretical proofs demonstrating EBPO's superiority over GRPO:

• MSE Reduction: The EBPO baseline achieves a strictly lower Mean Squared Error (MSE) in estimating the true difficulty $\theta_q$ compared to the unbiased sample mean used in GRPO. This is a classic result of shrinkage estimators (bias-variance trade-off).
• Non-Vanishing Gradients: Theoretically proven to provide non-zero gradient contributions in saturated failure regimes, provided the global success rate $\mu_{glob} > 0$.
• Entropy Conservation: The framework implicitly regularizes entropy decay. By dampening updates for low-variance groups via the global normalization constant, EBPO prevents premature policy collapse and maintains higher exploration capabilities compared to GRPO.

4. Experimental Results

The authors evaluated EBPO on diverse benchmarks (AIME-2024/2025, OlympiadBench, MATH-500, AMC23) using various models (LLaMA3-8B, Qwen3-8B, Qwen3-14B).

• Performance: EBPO consistently outperforms GRPO and other baselines (DAPO, Dr. GRPO, EntropyMech).
  • On Qwen3-8B, EBPO achieved an average Pass@1 of 64.39%, surpassing GRPO (58.72%) by over 5%.
  • In resource-constrained settings (Group Size $G=8$), EBPO outperformed GRPO by 11.28%, demonstrating superior sample efficiency.
• Training Stability:
  • Gradient Flow: Unlike GRPO, which suffers from vanishing gradients in saturated regimes, EBPO maintains robust gradient norms throughout training.
  • Update Stability: EBPO strictly bounds the per-step KL divergence, preventing the erratic update spikes and model collapse observed in GRPO.
  • Exploration: EBPO maintains higher policy entropy, indicating better exploration and generalization.
• Curriculum Impact: Difficulty-stratified curriculum learning (EBPO-diff) further enhanced performance, particularly on elite-level benchmarks like AIME and OlympiadBench, by isolating strong logical rationales early in training.

5. Significance and Contributions

• Algorithmic Framework: EBPO is the first framework to integrate Empirical Bayes inference directly into the online optimization loop of GRPO, utilizing shrinkage estimators to stabilize gradient variance.
• Theoretical Foundation: It provides formal guarantees for reduced variance, non-vanishing gradients, and bounded entropy decay, addressing the theoretical gaps in current RLVR methods.
• Practical Efficiency: EBPO enables high-performance training with significantly smaller group sizes ($G$), drastically reducing computational costs (memory and FLOPs) while maintaining or improving model reliability.
• Robustness: The method effectively handles the "sparse reward" problem inherent in mathematical reasoning, allowing models to learn from failure cases that would otherwise be discarded or ignored by standard GRPO.

In conclusion, EBPO represents a significant step forward in stabilizing RLVR for LLMs, offering a theoretically grounded, computationally efficient, and highly effective alternative to standard group-relative optimization.