💻 computer science

Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models

This paper introduces Selective-Advantage Entropy-Adaptive Horizon GRPO (SA-AH-GRPO), a reinforcement learning algorithm that asymmetrically applies entropy-based discounting to negative-advantage rollouts while preserving full gradient signals for successful trajectories, thereby significantly stabilizing training and improving performance on mathematical reasoning benchmarks compared to standard GRPO.

Original authors: Chirag Chawla, Rohan Charudatt Salvi, Madhav S. Baidya

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chirag Chawla, Rohan Charudatt Salvi, Madhav S. Baidya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a student (an AI language model) how to solve math problems. The student is trying to learn by practicing, getting feedback, and adjusting their study habits. This paper introduces a smarter way to give that feedback, making the learning process faster, more stable, and less confusing.

Here is the breakdown of the paper's ideas using simple analogies:

The Problem: The "One-Size-Fits-All" Tutor

The current standard method for teaching these AI models is called GRPO. Think of GRPO as a strict tutor who treats every single word the student writes the same way.

The Issue: If the student is confident and writes a clear, correct sentence, the tutor gives them a high-five. But if the student is confused, stammers, or guesses wildly (high "entropy" or uncertainty), the tutor gives them the exact same amount of attention—sometimes even punishing them just as hard as if they were confidently wrong.
The Result: This confuses the student. When they are already guessing, getting a harsh penalty can make them panic and learn the wrong things. Conversely, if the student gets a problem right, but made a few shaky guesses along the way, the tutor might accidentally punish those shaky guesses, weakening the lesson that "this path leads to the right answer."

The Solution: Two New Teaching Strategies

The authors propose two upgrades to this tutoring system, which they call AH-GRPO and SA-AH-GRPO.

1. The "Uncertainty Discount" (AH-GRPO)

Imagine the tutor has a special rule: "If the student is clearly confused, we turn down the volume on the feedback."

How it works: The tutor checks how unsure the student is at every word. If the student is guessing wildly (high entropy), the tutor says, "Okay, this part is messy, so I won't count it as heavily against you."
The Benefit: This prevents the student from getting overwhelmed by noise when they are struggling. It shortens the "lesson horizon," focusing only on the parts of the answer that are clear.

2. The "Selective Advantage" Rule (SA-AH-GRPO) — The Star of the Show

This is the paper's main innovation. It adds a crucial twist to the rule above: "Only turn down the volume if the student got the whole answer wrong."

Scenario A: The Student Gets it Right (Positive Advantage)
Even if the student stumbled or guessed a few words along the way, if they eventually solved the problem correctly, the tutor says: "Great job! Every word you wrote, even the shaky ones, helped you get there. We count all of them!"
- Why? Because the final result was a success. We want to reinforce the entire path that led to the win.
Scenario B: The Student Gets it Wrong (Negative Advantage)
If the student fails to solve the problem, the tutor says: "You got it wrong. Now, let's look at where you were confused. We will ignore the parts where you were just guessing wildly, so we don't accidentally teach you the wrong lesson from those guesses. We only focus on the clear mistakes."
- Why? When you are wrong and also confused, your guesses are just noise. Punishing the noise too hard creates a chaotic learning signal.

The Results: A Smoother Ride

The authors tested this new method (SA-AH-GRPO) on math problems using two different sizes of AI models (a smaller 1.5B model and a larger 3B model).

For the Larger Model (3B): The new method didn't necessarily make the model smarter than the old one (it was already quite smart), but it made the training much more stable. Imagine driving a car: the old method was like driving on a bumpy road where the car swerved left and right. The new method smoothed out the road. The "swerving" (variance) was reduced by 3.6 times. The car reached the same destination but with a much smoother ride.
For the Smaller Model (1.5B): The new method actually helped the model learn better. It improved the final score by nearly 5 percentage points compared to just starting from scratch. It seems the smaller model needed that extra stability to learn effectively.

The "Secret Sauce": Why It Works

The paper argues that this approach respects the difference between exploration and exploitation:

When a model is exploring (guessing), it's okay to be uncertain.
When a model succeeds, we should reward the whole journey, even the uncertain parts.
When a model fails, we should be careful not to punish the "noise" of uncertainty too harshly, or we confuse the learning signal.

Summary

Think of SA-AH-GRPO as a wise coach who knows when to be strict and when to be lenient.

If you win the game, the coach cheers for every move you made, even the risky ones.
If you lose the game, the coach ignores the moments where you were just guessing and focuses only on the clear errors, so you don't get discouraged or confused.

This simple change in how feedback is weighted makes the AI training process faster, more stable, and more effective, especially for smaller models that need extra help to find their footing.

Technical Summary: Selective-Advantage Entropy-Adaptive Horizon GRPO

Problem Statement

Group Relative Policy Optimisation (GRPO) has emerged as a dominant algorithm for Reinforcement Learning from Verifiable Rewards (RLVR) in language models, particularly for reasoning tasks. However, standard GRPO suffers from two primary limitations regarding gradient weighting:

Symmetric Token Weighting: It treats every token position equally, regardless of the model's local uncertainty (entropy). At high-entropy positions, where the model is exploring multiple plausible continuations, the gradient signal is inherently noisy. Applying uniform weights to these positions can destabilize training by treating noisy updates with the same magnitude as confident, low-entropy updates.
Symmetric Rollout Treatment: It applies identical gradient weighting to both successful (positive-advantage) and failed (negative-advantage) trajectories. Penalizing uncertain token positions in a successful trajectory is counterproductive, as those tokens contributed to a correct solution and should receive full credit. Conversely, aggressive penalization of uncertain positions in failed trajectories may introduce conflicting gradient signals.

Existing approaches to token-level weighting often rely on auxiliary reward models or fixed heuristics. This paper proposes a method that derives weights directly from the model's own predictive uncertainty without additional supervision.

Methodology

The authors introduce two hierarchical extensions to GRPO: Adaptive-Horizon GRPO (AH-GRPO) and Selective-Advantage AH-GRPO (SA-AH-GRPO).

1. Entropy-Adaptive Horizon Discount

The core mechanism is a per-token weight derived from the cumulative normalized entropy of the trajectory up to that point.

Normalized Entropy: For a token $t$ in rollout $i$ , the normalized entropy $\tilde{H}^{(i)}_t$ is calculated as the Shannon entropy of the policy's distribution divided by $\log V$ (vocabulary size), estimated using the top- $K$ logits ( $K=500$ ) for efficiency.
Discount Factor: A per-step discount $\gamma^{(i)}_t = \exp(-\alpha \tilde{H}^{(i)}_t)$ is applied.
Cumulative Weight: The weight for token $t$ is the product of all preceding discounts: $w^{(i)}_t = \prod_{s=1}^t \gamma^{(i)}_s = \exp(-\alpha \sum_{s=1}^t \tilde{H}^{(i)}_s)$ .
Effect: When $\alpha > 0$ , high-entropy prefixes result in stronger discounting, effectively shortening the "effective horizon" over which gradients propagate through uncertain sequences.

2. AH-GRPO (Uniform Application)

In AH-GRPO, the entropy-based weight $w^{(i)}_t$ is applied uniformly to all tokens in all rollouts, regardless of whether the trajectory is successful or failed. This reduces the influence of noisy gradients at uncertain positions but risks attenuating the signal on correct solutions that happen to contain uncertain steps.

3. SA-AH-GRPO (Selective Application)

SA-AH-GRPO introduces an asymmetry based on the group-normalized advantage $\hat{A}_i$ :

Positive Advantage ( $\hat{A}_i \geq 0$ ): The trajectory is deemed successful. All tokens, including those at high-entropy positions, receive a weight of $1$ (no discount). This preserves the full gradient signal for correct solutions.
Negative Advantage ( $\hat{A}_i < 0$ ): The trajectory is deemed failed. The entropy-adaptive weight $w^{(i)}_t$ is applied. This suppresses gradient updates at structurally ambiguous (high-entropy) positions within failed trajectories, preventing the model from over-correcting based on noisy signals.

The loss function is modified to normalize by the effective (weighted) token count rather than the raw token count.

Experimental Setup

Models: Qwen 2.5-1.5B-Instruct and Qwen 2.5-3B-Instruct.
Dataset: GSM8K (mathematical reasoning), using 800 training and 500 evaluation examples.
Fine-tuning: Low-Rank Adaptation (LoRA) with rank $r=16$ .
Reward Function: A composite reward ( $r_{max}=7.5$ ) combining correctness, format adherence (specific XML tags), presence of solution tags, and reasoning step density.
Baselines: Standard GRPO ( $\alpha=0$ ) and AH-GRPO ( $\alpha=0.5$ ).
Ablation: A sweep of $\alpha \in \{-0.25, 0.0, 0.10, 0.25, 0.50\}$ on the 1.5B model.

Key Results

1. Accuracy and Stability (3B Model)

On the Qwen 2.5-3B-Instruct model, the zero-shot baseline is strong (Pass@1 $\approx$ 0.831).

Peak Accuracy: SA-AH-GRPO achieved a peak Pass@1 of 0.858 (step 30), matching the peak performance of standard GRPO (0.852) and AH-GRPO (0.862).
Training Variance: SA-AH-GRPO significantly reduced training variance to 0.0246, representing a 3.6 $\times$ reduction compared to standard GRPO (0.0885) and a 2.6 $\times$ reduction compared to AH-GRPO (0.0630).
Conclusion: At the 3B scale, the primary benefit is substantial stabilization of training without sacrificing peak accuracy.

2. Accuracy and Stability (1.5B Model)

On the Qwen 2.5-1.5B-Instruct model, the zero-shot baseline is lower (Pass@1 $\approx$ 0.637).

Peak Accuracy: SA-AH-GRPO achieved a peak and final Pass@1 of 0.686, a +4.9 percentage point improvement over the zero-shot baseline. This outperformed standard GRPO (peak 0.668) and AH-GRPO (peak 0.676).
Variance: Interestingly, the 1.5B model showed higher training variance (0.1718) with SA-AH-GRPO compared to other methods. The authors attribute this to the model having more "headroom" to improve, leading to greater reward variation during the learning process, while still benefiting from the unrestricted gradient signal on positive rollouts.

3. Ablation Study ( $\alpha$ )

The sweep on the 1.5B model confirmed that positive $\alpha$ values (entropy discounting) consistently outperform negative $\alpha$ values (entropy amplification).

$\alpha = -0.25$ degraded performance below the baseline.
Among positive values, $\alpha=0.10$ yielded the highest peak but lower final stability, while $\alpha=0.50$ offered a better balance.
Crucially, SA-AH-GRPO at $\alpha=0.50$ (0.686) significantly outperformed symmetric AH-GRPO at the same $\alpha$ (0.676), validating the selective application strategy.

4. Training Dynamics

Implicit Curriculum: As training progresses, the mean normalized entropy decreases (from ~0.029 to ~0.009 on the 3B model). Consequently, the discount factor naturally approaches 1, acting as an implicit annealing schedule where the model relies less on the entropy filter as it becomes more confident.
Response Length: Mean generation length decreased over training (e.g., 243 to 171 tokens on 3B), suggesting the method encourages more concise reasoning chains.

Significance and Claims

The paper claims to be the first work to introduce per-token, per-rollout entropy-adaptive discounting in an RLVR setting, and the first to apply it asymmetrically based on the sign of the advantage.

The authors argue that SA-AH-GRPO provides a principled inductive bias for RLVR on structured generation tasks by:

Preserving Gradient Signals: It ensures that correct solutions, even those containing uncertain steps, receive full credit, preventing the suppression of valid learning signals.
Stabilizing Training: By selectively dampening gradients only on failed trajectories at uncertain positions, it reduces conflicting gradient directions and training variance.
Eliminating Auxiliary Models: The weighting mechanism relies solely on the policy's own predictive uncertainty, requiring no additional reward models or fixed heuristics.

The results suggest that asymmetric entropy discounting is a practical improvement for GRPO-based fine-tuning, offering superior stability on larger models and consistent accuracy gains on smaller models. Future work is proposed to evaluate these methods at larger scales (7B, 70B) and across different task domains.