CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning

1. Problem Statement

Reinforcement Learning (RL) has become a dominant paradigm for fine-tuning Large Language Models (LLMs) for complex reasoning tasks (e.g., math and code). However, a critical challenge in this process is managing policy entropy, which dictates the balance between exploration (trying new actions) and exploitation (refining known good actions).

Existing methods, particularly Proximal Policy Optimization (PPO) and its variants (like GRPO and DAPO), rely on a clipping mechanism to ensure training stability. This mechanism discards gradient signals from tokens whose importance sampling ratios fall outside a specific interval $[1-\epsilon, 1+\epsilon]$. The authors identify that this clipping inadvertently causes two distinct failure modes:

1. Entropy Collapse: By clipping Positive-Advantage Low-Probability (PA&LP) tokens, the algorithm suppresses valuable exploratory behaviors. These tokens often represent high-entropy, critical decision points. Their removal leads to premature convergence and a loss of output diversity.
2. Entropy Explosion: By clipping Negative-Advantage Low-Probability (NA&LP) tokens, the algorithm prevents the model from reinforcing high-probability choices. This forces the model into excessive exploration, delaying convergence and causing instability.

Current solutions, such as DAPO's "clip-higher" strategy, attempt to mitigate entropy collapse by extending the upper bound but fail to address the root cause: the complete discarding of gradients from tokens outside the clipping interval.

2. Methodology: CE-GPPO

The authors propose CE-GPPO (Coordinating Entropy via Gradient-Preserving Policy Optimization), a novel algorithm that reintroduces gradients from clipped tokens in a controlled, bounded manner.

Core Mechanism

Instead of strictly zeroing out gradients for tokens outside the clipping interval, CE-GPPO uses a stop-gradient operation to preserve these signals and scales them using learnable coefficients ($\beta_1$ and $\beta_2$).

The objective function is defined as:
$$\ell^{(i)} = \begin{cases} \beta_1 \cdot \frac{1-\epsilon}{\text{sg}(\delta)} \delta \cdot \hat{A}_{i,t} & \text{if } \delta < 1-\epsilon \text{ and } \hat{A}_{i,t} < 0 \quad (\text{NA\&LP}) \\ \beta_2 \cdot \frac{1+\epsilon}{\text{sg}(\delta)} \delta \cdot \hat{A}_{i,t} & \text{if } \delta > 1+\epsilon \text{ and } \hat{A}_{i,t} > 0 \quad (\text{PA\&LP}) \\ \delta \cdot \hat{A}_{i,t} & \text{otherwise} \end{cases}$$

Where:

• $\delta$ is the importance sampling ratio.
• $\hat{A}_{i,t}$ is the estimated advantage.
• $\text{sg}(\cdot)$ denotes the stop-gradient operation (preventing the ratio itself from being updated).
• $\beta_1$ controls the weight of gradients from NA&LP tokens (encouraging exploitation/convergence).
• $\beta_2$ controls the weight of gradients from PA&LP tokens (encouraging exploration).

Theoretical Justification

• Entropy Dynamics: The authors prove that policy entropy evolution is governed by the covariance between log-probabilities and the advantage function. By reintroducing clipped gradients, CE-GPPO directly manipulates this covariance to stabilize entropy.
• Stability: Unlike methods that simply remove clipping, CE-GPPO restricts the magnitude of these "out-of-clip" gradients to $\beta \cdot (1 \pm \epsilon)$. Since $\beta$ is typically close to 1, the gradient magnitude remains bounded, ensuring the policy does not deviate excessively from the old policy (preserving the trust region concept).
• Complexity: The method adds no additional forward passes or auxiliary components, maintaining the same computational and memory complexity as standard PPO/GRPO.

3. Key Contributions

1. Mechanism Discovery: The paper systematically analyzes the intrinsic link between token probability, advantage functions, and entropy dynamics, identifying that clipped PA&LP and NA&LP tokens are the primary drivers of entropy instability.
2. Algorithm Proposal: CE-GPPO is introduced as a gradient-preserving alternative that decouples the forward and backward passes to allow fine-grained control over entropy via $\beta_1$ and $\beta_2$.
3. Empirical Validation: Extensive experiments demonstrate that CE-GPPO effectively mitigates both entropy collapse and explosion, outperforming strong baselines across various model scales and reasoning benchmarks.

4. Experimental Results

The authors evaluated CE-GPPO on DeepSeek-R1-Distill-Qwen models (1.5B and 7B parameters) across mathematical reasoning (AIME, HMMT, MATH), code reasoning (HumanEval, LCB), and instruction following benchmarks.

• Performance Gains: CE-GPPO consistently outperformed baselines (GRPO, DAPO, CISPO, GSPO).
  • On the 7B model, CE-GPPO achieved a 3-point improvement over the best baseline (DAPO) on average.
  • On the 1.5B model, it achieved a 2.5-point improvement.
  • Notable gains were observed on difficult tasks like AIME25 (66.0 vs 59.7 for DAPO on 7B) and HMMT25.
• Entropy Stability:
  • Native GRPO suffered from rapid entropy collapse.
  • DAPO showed entropy rebound (over-exploration) in later training stages.
  • CE-GPPO maintained a stable, high entropy throughout training, leading to a smoother convergence and better final performance.
• Hyperparameter Sensitivity: The method is robust. The setting $\beta_1 = 0.5, \beta_2 = 1$ (giving more weight to exploration/PA&LP tokens) was found to be generally effective across different model sizes.
• Comparison with SOTA: CE-GPPO outperformed CISPO (which suffered from model collapse due to unbalanced gradients) and GSPO (which suffers from entropy explosion).

5. Significance

This work addresses a fundamental limitation in current RLHF/RLVR pipelines: the trade-off between training stability (via clipping) and the preservation of exploration signals.

• Paradigm Shift: It challenges the notion that clipped tokens must be discarded, showing instead that they contain critical information for regulating entropy.
• Practical Impact: By providing a simple, computationally free modification (re-weighting gradients) that significantly boosts reasoning capabilities in LLMs, CE-GPPO offers a practical tool for improving model performance in complex reasoning domains without the instability often associated with entropy regularization.
• Dynamic Control: It introduces a mechanism to dynamically balance exploration and exploitation by tuning $\beta_1$ and $\beta_2$, suggesting a path toward more adaptive RL training schedules.