Technical Summary: Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for eliciting long-chain reasoning in Large Language Models (LLMs), primarily utilizing Group Relative Policy Optimization (GRPO). However, the reliance on binary outcome rewards (correct/incorrect) induces two structural failure modes that hinder training efficiency and reasoning quality:

1. Zero-Advantage Collapse: In a group of rollouts, if all trajectories share the same outcome (e.g., all correct or all incorrect), the group-relative advantage becomes zero ($\sigma_R = 0$). This results in a vanishing gradient, wasting the computational effort of the entire group. This regime becomes increasingly frequent as training progresses and prompts either saturate (easy prompts) or remain unsolved (hard prompts).
2. Hallucinated Certainty: In late-stage training, the model's policy entropy decreases even on incorrect rollouts at high-stakes decision tokens. The model becomes "confidently wrong," ossifying erroneous reasoning patterns. Binary rewards cannot distinguish between a confident error and a hesitant one, failing to penalize this specific pathology.

The root cause of both modes is that a binary outcome reward indicates whether a trajectory succeeded but provides no information on how it arrived at the answer, leaving the policy without process-level signals.

Methodology: ISPO (Intrinsic Signal Policy Optimization)

The authors propose ISPO, a reward-level augmentation to GRPO that densifies the sparse outcome reward using two intrinsic signals computed entirely from the policy's own conditional probabilities. This approach requires no external reward model or process annotation.

1. Sequence-Level Signal: Conditional IFD

To address Zero-Advantage Collapse, ISPO introduces a sequence-level signal measuring how much the thinking trajectory ($T$) informs the final answer ($A$).

• Definition: The Conditional Instruction Following Difficulty (IFD) is defined as the log-probability gap between predicting the answer with the thinking trace versus without it:
  $$\text{IFD}_{\text{cond}}(\tau) = \frac{1}{|A|} [\log p_\theta(A|Q, T) - \log p_\theta(A|Q)]$$
• Theoretical Guarantee: The authors prove that the expected value of this term equals the conditional Kullback-Leibler (KL) divergence $D_{KL}(p_\theta(\cdot|Q, T) \parallel p_\theta(\cdot|Q))$. This identity formally guarantees that even when the outcome reward is homogeneous ($\sigma_{\text{outcome}}=0$), the variance of the composite reward remains non-zero ($\sigma_R > 0$) as long as the thinking traces differ, thereby preserving the gradient.

2. Token-Level Signal: Directional Shaping

To address Hallucinated Certainty, ISPO adds a fine-grained, outcome-conditioned reward at the token level, restricted to a subset of "critical tokens."

• Critical Token Selection: Tokens are selected via the intersection of two filters:
  1. Entropy Filter ($\mathcal{A}$): Selects tokens with high local uncertainty (top-$k\%$ entropy).
  2. Drift Filter ($\mathcal{C}_\Delta$): Selects tokens where the current policy has significantly drifted from the frozen base policy.
     This intersection isolates tokens that are both uncertain and actively shaped by RL, representing high-leverage decision points.
• Directional Reward:
  • Correct Rollouts: Reward low perplexity (high probability) at critical tokens to reinforce fluent execution.
  • Incorrect Rollouts: Reward high entropy. Crucially, a hinge penalty $\max(0, \tau_H - H_t)$ is applied to low-entropy tokens. This asymmetrically penalizes "confidently-wrong" predictions (low entropy on incorrect paths) while leaving uncertain incorrect tokens relatively unaffected, directly targeting the hallucinated certainty regime.

3. Composite Reward

The final reward $R(\tau)$ combines the binary outcome $R_o$ with the intrinsic signals:
$$R(\tau) = R_o + \lambda_1 (2R_o - 1) \text{IFD}_{\text{cond}}(\tau) + \lambda_2 R_{\text{token}}(\tau)$$
The sign factor $(2R_o - 1)$ ensures that strong thinking-answer dependence is rewarded for correct answers but penalized for incorrect ones.

Key Contributions

1. Formalization of Failure Modes: The paper formally identifies and analyzes Zero-Advantage Collapse and Hallucinated Certainty as structural failures of binary-outcome RLVR.
2. Intrinsic Signal Design: It proposes ISPO, which integrates a sequence-level Conditional IFD (with an exact conditional-KL identity) and a token-level directional reward with a hallucinated-certainty hinge.
3. Theoretical Gradient Preservation: It provides a proposition proving that ISPO preserves non-zero gradients in zero-advantage batches, effectively "unlocking" training data that standard GRPO discards.
4. Empirical Superiority: Extensive experiments demonstrate consistent performance gains across multiple model scales and benchmarks.

Experimental Results

The authors evaluated ISPO on three Qwen2.5 base models (1.5B, 7B Math, and 7B General) across five mathematical reasoning benchmarks: AIME24, AIME25, AMC23, MATH-500, and OlympiadBench.

• Performance: ISPO consistently outperformed competitive baselines (including Vanilla GRPO, Dr. GRPO, PACR, Scaf-GRPO, and PREPO).
  • On Qwen2.5-Math-7B, ISPO achieved a Pass@1 of 55.4% (average across benchmarks), surpassing the strongest baseline (Scaf-GRPO at 51.3%) by 4.1 points.
  • The largest gains were observed on the hardest benchmarks (AIME24/AIME25), where Zero-Advantage Collapse is most frequent. For instance, on AIME24 with the 7B model, ISPO reached 48.9% compared to 43.3% for Scaf-GRPO.
• Efficiency: ISPO recovered gradients in batches that would otherwise be discarded by GRPO. While Vanilla GRPO discarded ~24.7% of rollouts due to zero advantage, ISPO maintained 100% useful rollouts with only a ~4.4% wall-clock overhead (due to one extra forward pass for answer-only probability).
• Multi-Sample Evaluation: In pass@32 evaluations, ISPO showed significant improvements (e.g., +13.7% on AIME24 pass@32 over ProGRPO), indicating that the method successfully preserves sample-level diversity and prevents the collapse into confidently-wrong reasoning.
• Ablation Studies: Removing either the Conditional IFD or the directional token shaping resulted in performance drops, confirming that both components are necessary. The intersection of entropy and drift filters was shown to be superior to using either filter alone.

Significance and Claims

The paper claims that ISPO addresses the fundamental limitation of sparse binary rewards in RLVR by leveraging the model's own probability surface as a dense source of information. The authors characterize the dense intrinsic signals as a "momentum for reasoning" that keeps learning active where outcome gradients vanish.

The significance of the work lies in its ability to:

1. Stabilize Training: By preventing Zero-Advantage Collapse, ISPO ensures that difficult prompts (which often result in all-wrong groups) continue to provide learning signals rather than being discarded.
2. Correct Pathologies: By explicitly penalizing confident errors, ISPO mitigates the "Hallucinated Certainty" failure mode, leading to more robust reasoning trajectories.
3. Orthogonal Integration: The method composes orthogonally with existing optimizer-level refinements and data-curation strategies, offering a generalizable improvement to the RLVR pipeline without requiring changes to the rollout generation process or external reward models.

The authors note limitations, specifically that the approach is currently restricted to tasks with verifiable outcomes and has been validated primarily on mathematical reasoning tasks with Qwen2.5 models.