Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) is a cornerstone technique for enhancing the reasoning capabilities of Large Language Models (LLMs) in tasks like mathematics and coding. However, RLVR training is frequently plagued by entropy collapse, a phenomenon where the policy's entropy drops rapidly during training.

This collapse causes two critical failures:

1. Restricted Exploration: The model generates homogeneous reasoning trajectories (rollouts), failing to discover diverse or correct solutions.
2. Optimization Disruption: In group-based algorithms like GRPO (Group Relative Policy Optimization), homogeneous rollouts lead to non-discriminative advantage scores, causing training to stagnate.

While recent works attempt to mitigate this via heuristic interventions (e.g., adjusting clipping bounds, re-weighting positive samples, or entropy-aware advantages), these methods lack a unified theoretical understanding of why they work or fail. Their effectiveness is often bounded because they rely on coarse-grained adjustments without a principled mechanism to control token-level entropy dynamics.

2. Methodology & Theoretical Analysis

The authors propose a shift from coarse-grained global entropy monitoring to fine-grained token-level entropy analysis.

A. Theoretical Derivation (Theorem 1)

The paper derives a tight analytical approximation for the token-level entropy change ($\Omega(s)$) at each update step. The change is shown to be governed by four key factors:

1. Clipping Indicator ($I_{clip}$): Whether the importance sampling ratio is clipped.
2. Advantage ($A$) & Old Policy ($\pi_{old}$): Weighting factors for the update.
3. Token Probability ($\pi_\theta$): The current probability of the token.
4. Conditional Entropy ($H(s)$): The entropy of the current state.

The derived formula for entropy change is:
$$\Omega(s) \approx -\frac{\eta}{L} \mathbb{E}_{\pi_\theta} \left[ I_{clip} \cdot A \cdot \frac{\pi_{old}}{\pi_\theta} (1 - \pi_\theta) (\log \pi_\theta + H(s)) \right]$$

B. The Four-Quadrant Framework

Based on the joint effect of Advantage ($A$) and Token Probability ($\pi_\theta$), the authors categorize token updates into four quadrants:

• Quadrant I (Exploitation): High prob, Correct ($A>0$). Entropy decreases (concentrates mass).
• Quadrant II (Exploration): Low prob, Correct ($A>0$). Entropy increases (diversifies policy).
• Quadrant III (Suppression): Low prob, Incorrect ($A<0$). Entropy decreases (penalizes rare errors).
• Quadrant IV (Error-Correction): High prob, Incorrect ($A<0$). Entropy increases (flattens distribution to seek alternatives).

Entropy Collapse is identified as a state where the entropy-decreasing forces (Quadrants I & III) consistently overwhelm the entropy-increasing forces (Quadrants II & IV).

C. Critique of Existing Methods

The paper analyzes existing heuristics (e.g., Clip-Higher, Unlikeliness, Entropy-Aware Advantage) and finds they are limited because:

• They only adjust one or two of the four governing factors.
• They apply coarse-grained adjustments (e.g., global clipping) rather than token-level control.
• Some methods (like Entropy-Aware Advantage) can inadvertently accelerate collapse by amplifying the entropy change of tokens that are already undergoing drastic shifts.

3. Proposed Solution: STEER

To address these limitations, the authors introduce STEER (Stabilizing Token-level Entropy-changE via Reweighting).

• Core Mechanism: STEER calculates the theoretically estimated entropy change $\Omega(s)$ for every token.
• Adaptive Reweighting: It applies a scaling weight $\lambda(s)$ to the token's gradient. Tokens with excessively large entropy changes (either increasing or decreasing) are down-weighted to stabilize the dynamics.
• Weight Function:
  $$\lambda(s) = \exp\left(-\alpha \frac{|\Omega(s)|}{\max_{s' \in B} |\Omega(s')|}\right)$$
  This ensures that tokens driving unstable entropy dynamics contribute less to the gradient, while stable tokens are preserved.
• Advantages:
  • Principled: Based on the derived theoretical approximation, not heuristics.
  • Fine-grained: Operates at the token level.
  • Efficient: Adds negligible computational cost (only requires calculating $\Omega(s)$).
  • Universal: Compatible with various RL algorithms (GRPO, RLOO, OPO) and model families.

4. Experimental Results

The authors evaluated STEER across six mathematical reasoning benchmarks (AIME, AMC, MATH, etc.) and three coding benchmarks (LiveCodeBench, Zeta, Internal).

• Performance: STEER consistently outperformed state-of-the-art baselines (including GRPO, OPO, Clip-High, Entro. Adv., etc.).
  • On Qwen2.5-Math-7B, STEER achieved an average score of 48.6%, surpassing the second-best method (OPO at 46.4%) by a significant margin.
  • On Qwen2.5-14B, it achieved 45.2% average, outperforming competitors.
• Entropy Control: STEER successfully maintained stable entropy dynamics throughout training, preventing the rapid collapse seen in baselines. In extreme scenarios (where clipping was disabled), STEER remained stable while other methods failed.
• Generalization:
  • Model Scale: Effective across 1.5B, 7B, and 14B models.
  • Model Families: Works on Qwen, Llama, and Mistral architectures.
  • Algorithms: Successfully applied to GRPO, RLOO, and OPO.
• Coding Tasks: STEER improved performance on code generation and editing tasks, demonstrating robustness beyond pure math reasoning.

5. Significance and Contributions

1. Theoretical Breakthrough: The paper provides the first tight analytical approximation for token-level entropy change in RLVR, demystifying the mechanisms behind existing heuristics and revealing their inherent limitations.
2. Unified Framework: It introduces a four-quadrant view of entropy dynamics, offering a clear conceptual map for understanding how different token types contribute to exploration vs. exploitation.
3. Effective Algorithm (STEER): It proposes a simple, theoretically grounded method that adaptively reweights tokens to stabilize training. STEER achieves superior performance without requiring complex architectural changes or heavy computational overhead.
4. Practical Impact: By solving the entropy collapse problem, STEER enables more stable and effective scaling of RLVR for advanced reasoning tasks, which is critical for the development of next-generation reasoning models.

In summary, this work moves the field from trial-and-error heuristics to a principled, theoretical understanding of entropy dynamics in RLVR, offering a robust solution (STEER) that significantly advances the state of the art in LLM reasoning.