Technical Summary: VeriGate (Verifier-Gated Step-Level Supervision for GRPO)

1. Problem Statement

Group Relative Policy Optimization (GRPO) has emerged as an effective method for training Large Reasoning Models (LRMs) using verifier-based outcome rewards. However, the authors identify two critical limitations in standard GRPO that hinder performance on complex reasoning tasks:

1. Zero-Gradient Collapse (Reward Degeneracy): GRPO relies on group-relative advantages. When all sampled trajectories for a given prompt receive the same verifier reward (e.g., all incorrect, receiving a reward of 0), the group-relative advantage collapses to zero. In these "degenerate" batches, the reward-driven learning signal vanishes, leaving only the KL-regularization term. This causes learning to stall precisely on hard prompts where the model needs guidance most.
2. Lack of Step-Level Credit Assignment: Outcome-only rewards assign a single scalar to the entire reasoning trajectory. Consequently, every token in a correct solution receives the same positive advantage, and every token in an incorrect solution receives the same negative advantage. This prevents the model from distinguishing between genuinely useful reasoning steps and incidental or spurious ones, limiting exploration and the ability to learn robust reasoning processes.

While Process Reward Models (PRMs) offer denser, step-level supervision, naively integrating them into GRPO introduces new risks:

• Reward Hacking: PRMs are learned models with inherent biases. Directly optimizing aggregated PRM scores (treating them as outcome rewards) can lead the policy to exploit patterns the PRM favors without improving actual correctness.
• Brittle Aggregation: Collapsing step scores into a single trajectory scalar (e.g., via product or minimum) can distort the effective reward if a single step is mis-scored, discarding fine-grained credit assignment.
• Loss of Verifier Authority: Replacing trustworthy verifier signals with imperfect PRM signals can degrade performance on tasks where the verifier is reliable.

2. Methodology: VeriGate

The authors propose VeriGate (Verifier-Gated Step-Level GRPO), a method that integrates process supervision into GRPO only when the verifier signal is uninformative. The approach is built on three core design choices:

S1: Verifier-Gated Supervision

VeriGate implements a dynamic switching mechanism based on the informativeness of the verifier rewards within a sampled group:

• Informative Verifier: If the sampled trajectories for a prompt yield mixed verifier rewards (some correct, some incorrect), VeriGate operates exactly as standard GRPO, relying solely on the verifier signal.
• Degenerate Verifier: If all sampled trajectories receive the same verifier reward (specifically, all zeros), the standard GRPO advantage collapses. In this specific case, VeriGate activates PRM-based token-level supervision as a fallback mechanism.
• Rationale: This preserves the reliability of the verifier whenever it can distinguish correctness, restricting the use of the imperfect PRM to only those hard prompts where the verifier provides no learning signal.

S2: Future-Cumulated Token Rewards (FCTR)

Instead of aggregating PRM step scores into a single trajectory-level scalar (which causes brittle credit assignment), VeriGate converts step scores into future-cumulated rewards.

• For a trajectory with steps $j=1 \dots S$, the reward for step $j$ is defined as the sum of the PRM scores from step $j$ to the end of the trajectory: $c_{i,j} = \sum_{k=j}^{S} r_{i,k}$.
• This design assigns credit to a token not just for the quality of the immediate step, but for the quality of the continuation it enables. It smooths out local PRM noise and ensures that early tokens are rewarded for enabling downstream correct reasoning.

S3: Group-Normalized Token-Level Advantages

To integrate these dense rewards into the GRPO framework without reintroducing absolute reward hacking, VeriGate transforms the future-cumulated rewards into group-normalized token-level advantages.

• The advantage for a token in step $j$ of trajectory $i$ is calculated as: $A_{i,j} = \frac{c_{i,j} - \bar{c}}{\sigma(c)}$, where $\bar{c}$ is the mean and $\sigma(c)$ is the standard deviation of all future-cumulated rewards in the sampled group.
• Robustness Property: The authors prove (Proposition 3.1) that these advantages are invariant to prompt-wise positive affine transformations of the PRM rewards ($c' = a \cdot c + b$). This means that if a policy learns to uniformly shift or rescale PRM scores (a common form of reward hacking), the normalized advantages remain unchanged, preventing the policy from exploiting such artifacts.

3. Key Contributions

1. Verifier-Gated Process Supervision: A simple extension to GRPO that preserves standard verifier updates when informative and falls back to PRM-based token-level supervision only for all-zero verifier groups.
2. Continuation-Aware Credit Assignment: A novel mechanism using future-cumulated rewards and group-normalized advantages to provide dense, step-level learning signals without collapsing step scores into a single scalar.
3. Reduced Susceptibility to Reward Hacking: Theoretical and empirical evidence showing that VeriGate's normalized advantage formulation is invariant to affine transformations of PRM rewards, limiting over-optimization of imperfect PRM signals compared to methods that optimize aggregated PRM scores.
4. Empirical Gains: Demonstrated improvements in reasoning accuracy, reduction in zero-gradient failures, and improved reasoning quality across multiple benchmarks.

4. Experimental Results

The authors evaluated VeriGate using Qwen2.5-Instruct models (1.5B and 7B parameters) trained on MATH and GSM8K datasets, with evaluation on six reasoning benchmarks (GSM8K, MATH-500, AMC, AIME-24, MinervaMath, OlympiadBench).

• Accuracy Improvements: VeriGate improved average accuracy by approximately 20% for the 1.5B model and 12% for the 7B model compared to outcome-only GRPO and PRM-as-outcome baselines.
• Zero-Gradient Mitigation: The method substantially reduced the fraction of prompts receiving zero verifier rewards after training, indicating better exploration and escape from degenerate learning regimes.
• Reward Hacking Resistance:
  • Quantitative: Unlike PRM-as-ORM, where PRM scores rose while verifier scores remained flat (indicating hacking), VeriGate showed aligned growth in both verifier and PRM rewards.
  • Qualitative: Analysis of "hacked" responses (high PRM score, low verifier score) showed that PRM-as-ORM models often produced verbose, structurally plausible but logically flawed reasoning. VeriGate produced coherent reasoning trajectories that led to correct answers.
  • Cross-PRM Verification: When evaluated with an external PRM (Math-Shepherd) different from the training PRM, VeriGate maintained high performance, whereas PRM-as-ORM performance dropped significantly, suggesting VeriGate learns genuine reasoning rather than overfitting to specific PRM idiosyncrasies.
• Ablations: Removing the verifier gate or the future-cumulation mechanism resulted in significant performance drops, validating both design choices.

5. Significance and Claims

The paper claims that VeriGate offers a "simple drop-in modification" to GRPO that addresses its fundamental weaknesses on hard reasoning problems without sacrificing the reliability of verifier-based rewards.

• Reliability: By gating PRM usage, the method ensures that the trustworthy verifier drives learning whenever possible, using PRMs only as a recovery mechanism for degenerate cases.
• Robustness: The use of centered, normalized advantages derived from future-cumulated rewards provides a mathematically grounded defense against a common class of PRM reward hacking (affine transformations).
• Reasoning Quality: The results suggest that VeriGate improves the quality of reasoning (faithful step-by-step derivation) rather than just optimizing for a specific reward model's preferences.

The authors acknowledge limitations, noting that the method still depends on the availability and quality of PRMs and that poor step segmentation by the model could weaken credit assignment. They also note that current experiments are focused on mathematical reasoning, and broader evaluation is needed for other domains like code generation or tool use. However, the work establishes a framework for integrating dense process supervision into RLVR in a way that is theoretically robust and empirically effective.