Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control

1. Problem Statement

Reinforcement Learning (RL) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs). However, a critical bottleneck limits the scalability of RL: Performance Saturation. As training scales, model performance often plateaus earlier than expected, preventing further gains despite increased data and compute.

• Root Cause: The paper identifies the collapse of entropy as the primary mechanism behind this saturation. In standard RL, the model over-commits to a narrow region of solutions (exploitation) and ceases to explore alternative reasoning trajectories.
• Limitations of Existing Solutions: Current methods to prevent entropy collapse (e.g., loss regularization, clipping, or positive-negative decoupling) often result in unstable entropy curves. Entropy may drift too high (causing instability) or too low (causing saturation) over long training horizons. These methods lack the precision to treat entropy as a controllable hyperparameter, similar to learning rates.

2. Methodology: Entrocraft

The authors propose Entrocraft, a simple yet effective framework that enables precise, user-customized control over entropy curves during RL training.

A. Theoretical Foundation

The paper first establishes a theoretical link between advantage values and entropy changes under realistic LLM policy assumptions:

• Key Insight: Entropy change ($\Delta H$) is negatively correlated with the advantage ($\hat{A}$).
  • Positive Advantage: Updates that increase the probability of high-reward tokens tend to decrease entropy (model becomes more confident/narrow).
  • Negative Advantage: Updates tend to increase entropy.
• Implication: If training is dominated by positive advantages (which is common as the model learns), entropy naturally collapses. To maintain or increase entropy, one must selectively filter out positive-advantage samples or retain negative-advantage samples.

B. Core Mechanism: Rejection Sampling

Instead of modifying the RL objective function (e.g., adding entropy loss terms), Entrocraft uses a rejection-sampling filter applied during the rollout generation phase.

• Process:
  1. Generate a batch of rollouts and estimate advantages.
  2. Calculate the current batch entropy ($H$).
  3. Compare $H$ against a target entropy range $(h_{low}, h_{high})$.
  4. Dynamic Filtering:
     • If $H$ is too low: Reject high-advantage (positive) samples and retain low/negative-advantage samples to force entropy up.
     • If $H$ is too high: Retain high-advantage samples and reject negative ones to force entropy down.
• Advantages: This method is objective-agnostic (works with GRPO, GSPO, etc.), requires no auxiliary loss functions, and directly biases the advantage distribution to steer entropy.

C. Entropy Annealing Schedules

The paper introduces the concept of treating entropy as a tunable hyperparameter similar to learning rates.

• Linear Annealing: Through systematic study, the authors found that a linear decay schedule (starting with high entropy and gradually lowering it to a slightly lower target) performs best.
• Stability: This prevents the instability seen in constant-target methods, where the scarcity of negative-advantage samples in later training stages causes the entropy controller to fail.

3. Key Contributions

1. Theoretical Analysis: Provided rigorous proofs showing that entropy changes are negatively related to advantages and amplified by model confidence. This explains why standard RL leads to entropy collapse.
2. Entrocraft Algorithm: Designed a plug-and-play rejection-sampling filter that allows for precise crafting of entropy curves. It transforms entropy from a passive diagnostic metric into an active, controllable training hyperparameter.
3. Entropy Annealing: Demonstrated that linearly decaying entropy schedules outperform fixed targets, enabling stable long-term training.
4. Empirical Validation: Showed that Entrocraft significantly improves generalization, output diversity, and extends the effective training horizon.

4. Experimental Results

The method was evaluated on math reasoning benchmarks (MATH-500, AMC-23, AIME-24/25/26) using models like Qwen3-4B/8B/14B and Llama-3.1-8B.

• Performance Saturation: Entrocraft addresses the saturation problem, allowing models to sustain improvement for up to 4$\times$ longer before plateauing compared to standard GRPO.
• Model Efficiency: A 4B model trained with Entrocraft outperformed an 8B baseline trained with standard GRPO.
• Output Diversity: The method significantly increased output diversity, raising pass@K by 50% over the baseline.
• Benchmark Scores:
  • On MATH-500 (mean@32), Entrocraft achieved 79.0 (vs. 75.3 for GRPO).
  • On AIME-25 (mean@32), it achieved 16.5 (vs. 8.9 for GRPO).
• Comparison: Entrocraft consistently outperformed other entropy-preserving baselines (Entropy Loss, Clip-Higher, Clip-Cov, W-Reinforce, EntroPIC) across all metrics and model sizes.

5. Significance

• Solving Scalability: Entrocraft provides a solution to the "scaling law" bottleneck in LLM RL, proving that performance saturation is not inevitable but a result of uncontrolled entropy dynamics.
• Paradigm Shift: It shifts the perspective of entropy from a side effect to be managed via regularization to a primary control knob that can be scheduled and annealed.
• General Applicability: As a drop-in replacement requiring no changes to the underlying RL objective or estimator, it is highly applicable to existing and future policy-gradient algorithms.
• Long-Term Training: It enables stable, long-horizon training, which is crucial for developing complex reasoning capabilities in LLMs without the risk of "mode collapse" or instability.

In summary, Entrocraft offers a theoretically grounded, empirically superior method to control exploration-exploitation trade-offs in LLMs, unlocking sustained performance gains that were previously inaccessible due to entropy collapse.