Technical Summary: EP-GRPO

Problem Statement

The paper identifies three fundamental flaws in credit assignment inherent to Group Relative Policy Optimization (GRPO), a leading algorithm for Reinforcement Learning with Verifiable Rewards (RLVR) in Large Language Models (LLMs). These limitations impair the efficiency and effectiveness of reasoning tasks:

1. Uniform Token-Level Granularity: Standard GRPO assigns a uniform advantage value to all tokens within a generated sequence. This coarse granularity ignores the heterogeneous information value of tokens. Empirical analyses show that high-entropy tokens (representing decision pivots) are critical for reasoning, while low-entropy tokens often represent deterministic, redundant derivations. Equal treatment dilutes the gradient signal at critical decision points.
2. Uniform Polarity: GRPO assigns the same sign of advantage (positive or negative) to every token in a sequence based solely on the final outcome. This leads to "polarity misalignment," where correct intermediate steps in a failed trajectory are penalized, and erroneous steps in a successful trajectory are inadvertently rewarded. Existing solutions like Process Reward Models (PRMs) address this but require costly external annotations and struggle with generalization.
3. Zero-Variance Collapse: When all sampled answers in a group receive identical rewards (common in early training or on difficult tasks), the normalized advantage for all tokens becomes zero. This causes the outcome-driven gradient signal to vanish completely, halting learning. While increasing group size mitigates this, it is computationally expensive. Existing fixes often rely on additional sampling or heuristic reshaping that does not align with logical progress.

Methodology: EP-GRPO

The authors propose Entropy-Progress Aligned GRPO (EP-GRPO), a framework that transforms sparse, outcome-only feedback into dense, self-supervised token-level learning signals without external reward models. The method consists of three core components:

1. Entropy-Guided Outcome Advantage

To address granularity, EP-GRPO modulates the original sequence-wide advantage ($\hat{A}_i$) using a token-level weight derived from predictive entropy ($H_{i,t}$).

• Mechanism: The method calculates predictive entropy for each token. A sigmoid gate maps standardized entropy values to weights ($W_{i,t}$).
• Effect: High-entropy tokens (decision pivots) receive amplified weights, while low-entropy tokens (deterministic derivations) are suppressed. The modulated advantage is $\hat{A}^{outcome}_{i,t} = W_{i,t} \cdot \hat{A}_i$. This preserves the original direction but adjusts the magnitude based on information content.

2. Implicit Progress Advantage

To address polarity and zero-variance collapse, EP-GRPO introduces a directional signal derived from policy divergence.

• Signal Generation: The log-probability divergence between the current policy ($\pi_\theta$) and a frozen reference policy ($\pi_{ref}$) is calculated: $s_{i,t} = \lambda (\log \pi_\theta - \log \pi_{ref})$. This serves as a self-supervised process signal.
• Directional Anchor: Since divergence lacks an intrinsic direction (confidence can be misdirected), it is anchored to the outcome advantage.
  • If the variance of group rewards is non-zero, the anchor is $\text{sign}(\hat{A}_i)$.
  • If the variance is zero (collapse), the anchor degrades to the raw reward sign ($\text{sign}(r_i - \theta_{reward})$).
• Progress Alignment: To normalize these signals across asynchronous reasoning trajectories, tokens are mapped to "logical progress buckets" based on cumulative entropy ($\tau_{i,t}$). Normalization (Z-score) occurs within these buckets rather than by physical position, thereby comparing tokens in similar reasoning stages.
• Result: This yields a directed, dense reward ($\hat{A}^{progress}_{i,t}$) that remains active even when outcome advantages vanish.

3. Final Token-Level Advantage

The final advantage combines both signals:
$$\hat{A}^{final}_{i,t} = \hat{A}^{outcome}_{i,t} + \hat{A}^{progress}_{i,t}$$
A theoretical analysis proves that this gradient corresponds to a regularized GRPO objective with an entropy-weighted squared log-ratio term, effectively acting as precision-guided regularization that stabilizes exploration at critical decision points.

Main Contributions

1. Entropy-Guided Modulation: A mechanism to convert coarse sequence-wide advantages into token-level weights, prioritizing gradient updates at high-entropy decision pivots.
2. Anchored Implicit Process Signals: A method to transform policy divergence into directed process rewards by anchoring them to outcome advantages. This provides fine-grained feedback without external PRMs and maintains gradient flow during zero-variance collapse.
3. Logical Progress Alignment: A strategy for mapping cumulative entropy that projects asynchronous trajectories into a unified coordinate system, enabling progress-aligned normalization and ensuring meaningful signals even when outcome-based gradients vanish.

Experimental Results

The authors evaluated EP-GRPO on Qwen2.5-3B and Qwen2.5-7B models across five mathematical reasoning benchmarks: MATH500, AMC23, Minerva, AIME24, and AIME25.

• Performance Gains: EP-GRPO consistently outperformed standard GRPO and its variants (e.g., higher temperature, more rollouts).
  • For Qwen2.5-3B, average accuracy improved from 18.14% (GRPO) to 22.93% (+26.4%).
  • For Qwen2.5-7B, average accuracy improved from 27.11% (GRPO) to 30.34% (+11.9%).
  • Notably, the 3B EP-GRPO model surpassed the commercial DeepSeek-R1-671B on the AMC23 benchmark.
• Pass@k Performance: EP-GRPO showed significant advantages on highly difficult datasets (AIME24/25) as the number of samples ($k$) increased, indicating a superior ability to discover correct reasoning paths in complex scenarios.
• Ablation Studies: Removing a single component (entropy gating, implicit progress signals, or zero-variance degradation) led to performance drops, confirming that the three mechanisms are non-redundant and collectively necessary.
• Efficiency: Training wall-clock time was comparable to standard GRPO, as the additional calculations (entropy, divergence) are lightweight. However, EP-GRPO generated reasoning chains approximately 20% longer, suggesting deeper reasoning without temporal cost.

Significance

The paper claims that EP-GRPO addresses the fundamental limitations of GRPO in RLVR by introducing a self-supervised, dense credit assignment mechanism. By leveraging the model's intrinsic information flow (entropy) and policy dynamics (divergence), it eliminates the need for costly external Process Reward Models while simultaneously solving the zero-variance collapse that plagues standard group-based optimization. The results suggest that improved credit assignment, rather than merely increasing model size or sampling budget, is a critical lever for enhancing the reasoning capabilities of LLMs.