Technical Summary: Flexible Entropy Control in RLVR from a Gradient-Preserving Perspective

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR), particularly using Group Relative Policy Optimization (GRPO), has become a pivotal method for enhancing the reasoning capabilities of Large Language Models (LLMs). However, continuous training in this paradigm often leads to policy entropy collapse. This phenomenon is characterized by a rapid decay of policy entropy to near-zero values, resulting in premature overconfidence, reduced output diversity, and the vanishing of gradient norms, which ultimately inhibits further learning and degrades model performance.

A primary factor driving these dynamics is Gradient-Preserving Clipping, a mechanism inherited from Clipped Proximal Policy Optimization (PPO-Clip) that constrains the importance sampling ratio within a trust region. While existing mitigation strategies, such as the "Clip-Higher" approach, attempt to address this by statically adjusting clipping thresholds, they lack a theoretical framework connecting clipping mechanisms to precise entropy control. Consequently, current methods are often static and fail to provide flexible, stage-specific entropy regulation necessary for effective RL training.

Methodology

This paper proposes a novel approach to reshape entropy control in RLVR from a Gradient-Preserving Perspective. The methodology is built upon three core components:

1. Theoretical and Empirical Analysis of Entropy Dynamics

The authors theoretically analyze the relationship between the reinforcement learning update gradients and the global entropy gradients. By computing the inner product of the objective gradient ($\nabla_z L$) and the entropy gradient ($\nabla_z H$), they identify four distinct regions of the importance sampling ratio ($r_t$) that dictate whether entropy increases or decreases:

• E1 (High Gain): High probability tokens with positive advantage encourage further sharpening (entropy decrease).
• E2 (Low Gain): Low probability tokens with positive advantage encourage flattening (entropy increase).
• E3 (High Drop): High probability tokens with negative advantage encourage flattening (entropy increase).
• E4 (Low Drop): Low probability tokens with negative advantage encourage sharpening (entropy decrease).

Empirical verification confirms that selectively applying Gradient-Preserving Clipping to these specific regions directly influences the trajectory of policy entropy.

2. Dynamic Clipping Thresholds for Entropy Regulation

Based on the theoretical findings, the paper introduces a dynamic clipping threshold mechanism. Instead of using fixed scalar bounds ($\epsilon$), the clipping thresholds are formulated as functions of the current policy probability ($\pi_\theta$) with a stop-gradient operation:
$$\epsilon(\pi_\theta) := f(\bar{p}_t), \quad \bar{p}_t = \text{sg}[\pi_\theta(a_t|s_t)]$$
where $f(\bar{p}_t)$ is negatively correlated with the probability.

• For Positive Advantage: The upper clipping threshold is increased for low-probability tokens (to encourage exploration/E2) and decreased for high-probability tokens (to prevent over-concentration/E1).
• For Negative Advantage: The lower clipping threshold is decreased for high-probability tokens (to maintain stability) and relaxed for low-probability tokens (to suppress sub-optimal regions/E4).

This mechanism allows for precise, probability-dependent modulation of the trust region to either promote or suppress entropy changes.

3. Entropy Control Strategies

Leveraging the dynamic regulation mechanism, the authors design three distinct training strategies to manage entropy evolution over the training lifecycle:

• Increase-then-Decrease (ID): Designed for SFT-aligned models. It begins with a phase of entropy increase (encouraging diverse reasoning) followed by a phase of entropy decrease (converging to optimal performance).
• Decrease-Increase-Decrease (DID): Suitable for base or high-entropy models. It initially reduces entropy to stabilize optimization, allows a controlled increase to prevent collapse, and finally decreases entropy for convergence.
• Oscillatory Decay (OD): Designed for long training runs or scenarios prone to local minima. It employs a state machine that triggers entropy boosting or suppression based on whether the current entropy falls below a lower bound or exceeds a decaying upper bound, maintaining a dynamic balance throughout training.

Key Contributions

1. Theoretical Insight: The paper provides a theoretical derivation and empirical validation of how specific importance sampling ratio regions contribute to entropy growth and reduction, establishing a clear link between Gradient-Preserving Clipping and entropy dynamics.
2. Regulation Mechanism: It proposes a dynamic clipping threshold mechanism that modulates the upper and lower bounds based on token probability, enabling flexible and precise control over entropy fluctuations.
3. Strategy Design: It introduces and evaluates three specific entropy control strategies (ID, DID, OD) that adapt the training process to different model states and objectives, effectively mitigating entropy collapse.

Experimental Results

The proposed methods were evaluated on Qwen2.5-Math-7B, Qwen2.5-7B, and Phi-4-14B models using the DAPO-MATH dataset. Performance was benchmarked across multiple mathematical reasoning tasks (AIME24/25, AMC, MATH-500, GSM8k, Olympiad) and non-mathematical domains (GPQA-Diamond, LiveCodeBench, MMLU-Redux).

• Performance Gains: All three strategies (ID, DID, OD) consistently outperformed the baseline GRPO and other state-of-the-art methods (e.g., Clip-Higher, Entropy-Regularization, GSPO) across multiple benchmarks. For instance, on Qwen2.5-Math-7B, the Ours-ID strategy achieved a score of 33.1 on AIME24 compared to 24.4 for GRPO, and 48.3 on Olympiad compared to 46.2.
• Entropy Control: Training curves demonstrated that the proposed methods successfully prevented entropy collapse. The ID strategy showed a clear initial entropy rise followed by a controlled decay, while OD maintained oscillatory behavior that prevented premature convergence.
• Generalization: The methods demonstrated scalability to larger models (Phi-4-14B) and generalization to non-mathematical domains, showing consistent improvements in Pass@K metrics and reward scores.
• Efficiency: The dynamic clipping mechanism introduced minimal computational overhead, with training times comparable to the baseline GRPO.

Significance and Claims

The paper claims that by shifting the perspective of entropy control from static threshold tuning to a gradient-preserving, probability-dependent dynamic modulation, it is possible to precisely manage the trade-off between exploration and exploitation in RLVR.

The authors assert that their work addresses the critical issue of entropy collapse, which has been identified as a bottleneck in scaling RL for LLM reasoning. By providing a theoretical framework and practical strategies (ID, DID, OD), the paper offers a robust solution for maintaining output diversity and gradient flow during continuous training. The results suggest that flexible entropy control is not merely a regularization technique but a fundamental component for achieving superior reasoning performance in large language models. The work emphasizes that the effectiveness of these strategies lies in their ability to adapt the trust region constraints according to the specific probability states of tokens, rather than applying uniform adjustments.