Technical Summary: Group Entropy-Controlled Policy Optimization (GEPO)

1. Problem Statement

Reinforcement Learning (RL) has become a dominant paradigm for post-training Large Language Models (LLMs) to enhance alignment and reasoning. However, balancing the exploration-exploitation trade-off remains a critical challenge, particularly when training on heterogeneous task mixtures (e.g., combining mathematics, coding, physics, and instruction following).

Current entropy-controlled RL methods operate primarily at two granularities:

1. Policy-level: Calculating entropy over the entire batch and applying a uniform regulation signal (Figure 1a).
2. Token-level: Calculating token covariance and regulating individual tokens (Figure 1b).

The paper identifies a fundamental flaw in these approaches when applied to Group Relative Policy Optimization (GRPO): Entropy Heterogeneity. Different task domains exhibit distinct entropy regimes under the same policy. For instance, physics prompts may naturally concentrate around low entropy, while instruction-following prompts exhibit high entropy.

This heterogeneity induces two specific pathologies in standard GRPO:

• Entropy-Dependent Bias: The normalization of advantages within groups creates statistically non-comparable signals across groups with different entropy levels. Low-entropy groups (often high-accuracy tasks) produce strong, consistent gradient signals, while high-entropy groups (often low-accuracy tasks) produce weak, noisy signals.
• Optimization Imbalance: The batch-level optimization becomes dominated by low-entropy tasks, causing high-entropy tasks to receive diminishing learning signals. This leads to a self-reinforcing cycle where low-entropy tasks converge prematurely, while high-entropy tasks fail to explore effectively or suffer from "runaway entropy" leading to degenerate outputs.

2. Methodology: GEPO

The authors propose Group Entropy-Controlled Policy Optimization (GEPO), a lightweight, model-agnostic extension to GRPO that performs entropy-conditioned asymmetric advantage shaping at the group level.

Core Mechanism

GEPO utilizes Group Entropy ($\mathcal{H}_g$), estimated from the existing grouped samples of a prompt, as a diagnostic signal. Instead of applying a global entropy target, GEPO shapes the advantage signal ($A_i$) for each response based on the group's entropy state:

$$\hat{A}_i = \omega(A_i, \mathcal{H}_g) \cdot A_i$$

The shaping function applies asymmetric scaling coefficients ($\alpha_{low}, \alpha_{high} \in (0, 1)$):

• Low-Entropy Groups ($\mathcal{H}_g < \mathcal{H}_{low}$): Positive advantages are attenuated by $\alpha_{low}$. This prevents over-exploitation and premature convergence in tasks where the model is already confident.
• High-Entropy Groups ($\mathcal{H}_g > \mathcal{H}_{high}$): Negative advantages are attenuated by $\alpha_{high}$. This prevents the premature suppression of exploration in tasks where the model is uncertain, allowing the policy to discover correct reasoning paths.
• Otherwise: The advantage remains unchanged.

Key Design Choices

1. Asymmetric Shaping ($\alpha_{high} < \alpha_{low}$): The authors empirically observe that low-entropy groups are more fragile; aggressive penalization of negative advantages in these groups can trigger "length collapse" (where the model shortens responses to reduce uncertainty). Therefore, GEPO applies a gentler nudge ($\alpha_{low}$ is higher) to low-entropy groups compared to the stronger attenuation ($\alpha_{high}$ is lower) applied to high-entropy groups.
2. Adaptive Thresholds: Fixed entropy thresholds are brittle across different base models and training stages. GEPO derives thresholds ($\mathcal{H}_{low}, \mathcal{H}_{high}$) dynamically from the batch's entropy distribution (mean and standard deviation) and smooths them using an Exponential Moving Average (EMA). This allows the method to automatically calibrate to the specific entropy scale of the base model without task-specific tuning.

3. Key Contributions

• Theoretical Insight: The paper provides a theoretical analysis (Propositions 2.1 and 2.2) demonstrating that normalized advantages in GRPO are structurally biased by group entropy. It proves that the discrepancy between policy-weighted and reference reward distributions is entropy-dependent, leading to non-comparable advantage signals across heterogeneous tasks.
• Algorithmic Innovation: GEPO introduces the first group-level entropy control mechanism that performs asymmetric advantage shaping. Unlike previous methods that treat entropy as a global or token-level property, GEPO treats it as a prompt-group diagnostic to rebalance optimization pressures.
• Zero-Cost Implementation: The method requires no additional sampling or value function estimation. It leverages the existing grouped samples in GRPO, adding negligible computational overhead.

4. Experimental Results

The authors evaluated GEPO on two base models (Intern-S1-mini and Qwen3.5-9B) across thirteen benchmarks covering mathematics, physics, science, code generation, and instruction following.

• Performance: GEPO consistently outperformed GRPO and recent entropy-controlled baselines (AEPO, Clip-Cov, KL-Cov).
  • On Intern-S1-mini, GEPO achieved the best average score (54.2) and won on 7 out of 13 benchmarks, including significant gains on AIME25, IMO-Bench, and GPQA.
  • On Qwen3.5-9B, GEPO achieved the highest average score (71.2), surpassing strong baselines like Clip-Cov (70.9) and KL-Cov (69.9).
• Stability: GEPO demonstrated superior training stability. While GRPO suffered from catastrophic performance collapse (e.g., on Qwen3.5-9B around step 170) due to unbounded entropy growth, and AEPO exhibited persistent oscillations, GEPO maintained smooth, monotonically improving validation curves.
• Entropy Dynamics: Analysis of training dynamics showed that GEPO preserves differentiated exploration levels across tasks. Unlike AEPO, which forces a uniform entropy pattern, GEPO allows tasks requiring broad exploration to maintain higher entropy while deterministic tasks settle at lower entropy, preventing both premature collapse and runaway entropy.

5. Significance and Claims

The paper claims that GEPO addresses a critical gap in multi-task RL post-training: the inability of current methods to handle the structural bias introduced by entropy heterogeneity across diverse tasks.

The authors assert that:

1. Task-Specific Regulation is Essential: Driving heterogeneous tasks toward a uniform entropy pattern is suboptimal. Effective multi-task learning requires preserving task-specific exploration regimes.
2. Asymmetry is Crucial: The fragility of low-entropy groups necessitates an asymmetric approach to advantage shaping to avoid length collapse while still encouraging exploration.
3. Scalability: By relying on group entropy estimated from existing rollouts and adaptive thresholds, GEPO offers a scalable, model-agnostic solution that improves both average performance and inter-task balance without requiring explicit task annotations or additional sampling costs.

In conclusion, the paper positions GEPO as a robust framework for stabilizing RL training in complex, multi-domain LLM post-training scenarios, ensuring that the optimization process respects the inherent uncertainty and diversity of different task types.