Technical Summary: Hysteretic Policy Optimization (HPO) for Sparse-Reward LLM Training

1. Problem Statement

The paper investigates a specific failure mode in Group Relative Policy Optimization (GRPO) when applied to Large Language Models (LLMs) under sparse verifiable reward regimes (e.g., reasoning tasks where only a small fraction of sampled responses are correct). The authors identify two interdependent biases that destabilize training in this setting:

1. Sign Imbalance (Negative Dominance): In sparse-reward scenarios, the initial policy often generates mostly incorrect responses. Consequently, the majority of sampled trajectories fall below the group mean, resulting in a high frequency of negative advantages ($p_- > p_+$). Symmetric treatment of these abundant negative samples can overwhelm the rare positive signals, suppressing useful but noisy reasoning trajectories.
2. Response-Level Length Bias: Standard GRPO normalizes the loss by the length of each individual response ($1/|\tau_i|$). This creates an inverse relationship between response length and update magnitude:
   • Short correct responses receive disproportionately large positive gradients, biasing the model toward brevity.
   • Long incorrect responses receive disproportionately small negative gradients, failing to adequately penalize verbose failures.
   • This interaction makes optimization pressure dependent on both reward and length, which is particularly problematic for reasoning tasks where length correlates with search depth or explanation.

2. Methodology

The authors propose Hysteretic Policy Optimization (HPO), a minimal modification to the GRPO objective designed to address these biases through two mechanisms:

2.1 Fixed HPO

HPO introduces two structural changes to the GRPO objective:

• Mean-Length Normalization: Instead of normalizing each response by its own length ($|\tau_i|$), HPO normalizes the entire batch by the mean response length ($|\bar{\tau}|$). This removes the inverse scaling of update magnitude with response length, ensuring that long incorrect responses are penalized appropriately and short correct responses do not dominate the gradient.
• Asymmetric Hysteretic Weighting: The policy gradient updates are scaled asymmetrically based on the sign of the advantage ($\hat{A}_i$):
  • Positive advantages ($\hat{A}_i \geq 0$) retain full weight (factor = 1).
  • Negative advantages ($\hat{A}_i < 0$) are down-weighted by a hysteretic factor $\alpha \in [0, 1]$.
  • The objective function becomes:
    $$J_{HPO}(\theta) = \mathbb{E}_X \left[ \frac{1}{N|\bar{\tau}|} \sum_{i=1}^N \sum_{j=1}^{|\tau_i|} w_i \ell_{i,j}(\theta) \right]$$
    where $w_i = 1$ if $\hat{A}_i \geq 0$, and $w_i = \alpha$ otherwise.

2.2 Adaptive HPO (A-HPO)

Recognizing that the optimal $\alpha$ depends on the training regime (requiring less hysteresis as the policy improves), the authors introduce A-HPO. This variant dynamically adjusts $\alpha$ based on the sign statistics of the current batch:

• It estimates the ratio of positive to negative advantage frequencies ($\hat{p}_+ / \hat{p}_-$).
• The adaptive weight is set as:
  $$\alpha_{adaptive} = \text{clip}\left( \frac{\hat{p}_+}{\hat{p}_- + \epsilon}, \alpha_{min}, 1 \right)$$
• Mechanism: When negative samples dominate (sparse regime), $\hat{p}_+ / \hat{p}_-$ is small, causing $\alpha$ to decrease and dampen negative updates. As the policy improves and the distribution balances, $\alpha$ approaches 1, recovering a symmetric update. This avoids manual tuning of a fixed $\alpha$.

3. Key Contributions

1. Identification of Failure Modes: The paper explicitly links GRPO's instability in sparse-reward regimes to the combination of negative-advantage dominance and response-level length normalization.
2. HPO Algorithm: A one-parameter interpolation between "positive-only" learning (which discards negative feedback) and fully symmetric updates. It combines mean-length normalization with asymmetric weighting.
3. A-HPO: A lightweight, adaptive rule that automatically tunes the hysteretic weight based on batch-level sign statistics, removing the need for manual hyperparameter search.
4. Empirical Validation: Ablation studies demonstrate that the optimal behavior lies between the extremes of positive-only ($\alpha=0$) and symmetric ($\alpha=1$) updates, validating the necessity of intermediate hysteresis.

4. Experimental Results

The authors evaluated HPO and A-HPO on two tasks: TeleLogs (5G network root-cause analysis) and Countdown (arithmetic construction), using models ranging from 1.5B to 7B parameters.

• TeleLogs (Qwen2.5-1.5B):
  • A-HPO achieved a final reward of 0.84, outperforming GRPO (0.73, +15%), GSPO (0.76, +11%), and SAPO (0.80, +5%).
  • A-HPO reached high reward levels significantly faster (sample efficiency) while maintaining a comparable response length to baselines, avoiding the "length collapse" seen in positive-only methods.
• Countdown (1.5B–7B models):
  • Gains were most pronounced in early training and harder configurations (Countdown-4 with 4 numbers) where rewards are sparse.
  • For example, on Llama3.1-3B with Countdown-4, A-HPO improved the 200-step reward from 0.40 (GRPO) to 0.54.
  • In saturated regimes (e.g., 7B models on easier Countdown-3), the performance gap narrowed, indicating A-HPO is most critical when the policy is still learning.
• Ablation Studies:
  • Fixed HPO with $\alpha=0$ (positive-only) collapsed to very short outputs (16 tokens) and lower rewards (0.71).
  • Fixed HPO with $\alpha=1$ (symmetric) underperformed (0.72) due to negative dominance.
  • The best fixed performance was at $\alpha \approx 0.6$, which A-HPO matched automatically without tuning.
  • Analysis of the "surrogate contribution balance" confirmed that A-HPO successfully balances the magnitude of positive and negative update contributions, whereas GRPO remains noisy and negative-dominated.

5. Significance and Claims

The paper claims that HPO and A-HPO offer a practical, low-overhead modification to GRPO that significantly improves stability and sample efficiency in sparse-reward RL for LLMs.

• Core Principle: The authors argue that in sparse-reward regimes, negative-advantage updates should be utilized but not trusted symmetrically when they are overwhelmingly frequent.
• Balance: The method successfully prevents the suppression of useful reasoning trajectories by abundant failures while avoiding the degenerate behavior of discarding negative evidence entirely.
• Adaptivity: A-HPO is presented as a robust solution that automatically tracks the intermediate regime between positive-only and symmetric updates, adapting as the training distribution shifts from sparse to dense rewards.

The work positions HPO as a necessary refinement for compute-efficient LLM fine-tuning when dense process rewards or critic feedback are unavailable.