Technical Summary: GTPO and GRPO-S

1. Problem Statement

Current Reinforcement Learning (RL) algorithms for Large Language Models (LLMs), particularly Group Relative Policy Optimization (GRPO) and its variants like DAPO, suffer from coarse-grained credit assignment. In these frameworks, a uniform reward (e.g., +1 for a correct final answer, 0 or -1 for an incorrect one) is assigned to every token within a generated sequence. This paradigm creates two critical limitations:

1. Penalization of Correct Reasoning: A sequence containing dozens of correct logical steps receives zero reward if the final answer is incorrect, effectively discarding valuable intermediate reasoning.
2. Rewarding Flawed Paths: Conversely, a sequence that reaches a correct answer through guessed or flawed intermediate steps receives full credit, reinforcing incorrect reasoning patterns.
3. Length Bias: Standard GRPO designs implicitly favor longer sequences for correct answers due to the aggregation of uniform advantages, which is statistically suboptimal.

While recent works have utilized policy entropy as a regularization constraint or a binary mask to filter tokens, they fail to actively reshape the reward signal to distinguish between high-value exploration in correct solutions and confident errors in incorrect ones.

2. Methodology

The authors propose Dynamic Entropy Weighting, a framework that repurposes policy entropy as an intrinsic anchor for fine-grained credit assignment. The core hypothesis is that high policy entropy within a reasoning sequence correlates with pivotal decision points (cognitive effort). The framework redistributes rewards based on this entropy, operating through two complementary algorithms:

A. Group Token Policy Optimization (GTPO)

GTPO operates at the token level, assigning a unique, entropy-weighted reward to every token.

• Successful Sequences ($O^+$): Tokens with high entropy (indicating valuable exploration or uncertainty) receive a reward bonus. The reward is shaped as:
  $$\tilde{r}^+_{i,t} = \alpha_1 r_i + \alpha_2 \frac{H_{i,t}}{\sum_{k} H_{k,t}} \cdot d_t$$
  where $H_{i,t}$ is the generation entropy of the token, and $d_t$ is the count of successful sequences active at step $t$. This reinforces uncertain but correct steps.
• Unsuccessful Sequences ($O^-$): Tokens with low entropy (indicating high confidence) receive heavier penalties. The reward is shaped using inverse entropy:
  $$\tilde{r}^-_{j,t} = \alpha_1 (-1) + \alpha_2 \frac{1/H_{j,t}}{\sum_{k} (1/H_{k,t})} \cdot h_t \cdot (-1)$$
  This penalizes confident mistakes, encouraging the model to break out of incorrect reasoning modes.
• Advantage Calculation: Separate advantage functions are computed for positive and negative sets, normalized over all tokens in the batch to ensure consistent scaling.

B. Sequence-Level GRPO (GRPO-S)

GRPO-S is a lightweight, sequence-level variant designed for stability in long Chain-of-Thought (CoT) tasks.

• Instead of token-level rewards, it modulates the global reward for an entire sequence based on its average entropy ($\hat{H}_k$).
• Successful sequences with higher average exploration receive bonuses, while incorrect sequences with high confidence (low average entropy) receive penalties.
• This approach offers a trade-off between the micro-level precision of GTPO and macro-level sequence stability.

Theoretical Guarantees

The paper provides theoretical analysis proving:

1. Reward Mass Conservation: Under specific hyperparameter constraints ($\alpha_1 + \alpha_2 = 1$), the sum of shaped positive rewards equals the sum of original binary rewards, ensuring the total learning signal remains unbiased relative to the ground truth.
2. Asymptotic Consistency: As training progresses and entropy variance among successful sequences diminishes, the gradient bias introduced by entropy weighting vanishes, ensuring convergence to the same global optimum as standard GRPO/DAPO.
3. Variance Reduction: The token-level normalization used in GTPO is statistically superior to the two-stage averaging in standard GRPO, reducing gradient variance.

3. Key Contributions

1. Identification of Coarse-Grained Limitations: The paper fundamentally optimizes the GRPO framework by addressing its inability to assign credit to specific reasoning steps.
2. GTPO Algorithm: Introduces the first true, fine-grained, per-token credit assignment within the efficient GRPO framework using dynamic entropy weighting.
3. GRPO-S Algorithm: Proposes a sequence-level variant that captures the exploratory value of entire sequences, offering superior stability for long CoT reasoning.
4. Theoretical Analysis: Provides proofs for the unbiasedness and convergence of the proposed objective functions.
5. Empirical Validation: Demonstrates significant performance gains on challenging mathematical reasoning benchmarks.

4. Experimental Results

Experiments were conducted on Qwen2.5-7B and Qwen2.5-32B models across AIME 2024, AIME 2025, and MATH 500 benchmarks.

• Performance: Both GTPO and GRPO-S established new state-of-the-art results, consistently outperforming GRPO, DAPO, and the advanced "DAPO w/ Forking Tokens" baseline.
  • On AIME 2024 (Qwen2.5-7B), GTPO achieved a Pass@32 of 64.89%, a massive improvement over DAPO's 35.51%.
  • On AIME 2025, GRPO-S achieved a Pass@32 of 23.33% (7B) and 34.33% (32B), outperforming all baselines.
• Exploration vs. Stability: Unlike DAPO, which often plateaus due to premature policy collapse (entropy collapse), GTPO and GRPO-S exhibited an "entropy rebound" phenomenon. The models maintained higher exploration, leading to longer, more diverse reasoning chains and higher final reward ceilings.
• Sample Efficiency: The methods converged within approximately 210 training steps, indicating that the performance gains were not due to longer training but to more efficient learning signals.

5. Significance and Claims

The paper claims that harnessing and directing model uncertainty is a promising frontier for developing reliable AI systems. By treating policy entropy not merely as a regularization term but as a dynamic signal for active reward shaping, the proposed methods enable:

• Precise Credit Assignment: Distinguishing between valuable exploration in correct paths and confident errors in incorrect paths without requiring an explicit value network.
• Robust Reasoning: Moving LLMs beyond pattern imitation toward verifiable, step-by-step reasoning.
• Scalability: Providing a principled trade-off between granular precision (GTPO) and sequence stability (GRPO-S) that scales effectively across different model sizes and task difficulties.

The authors conclude that their framework resolves the stability-exploration trade-off inherent in current RLHF methods, allowing models to explore complex reasoning spaces more effectively while avoiding the pitfalls of overconfident, incorrect reasoning.