Technical Summary: Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated success in improving final-answer accuracy on reasoning tasks (e.g., mathematics, coding) by utilizing Outcome Reward Models (ORMs). However, this approach suffers from a fundamental process-outcome mismatch. Because ORMs only assess the final result, they inadvertently reward spurious successes: scenarios where flawed reasoning coincidentally yields a correct answer.

The paper defines a latent state $z$ where $z=1$ represents a valid reasoning process. Even if a model generates an invalid process ($z=0$), there is a small probability $\epsilon$ that it produces a correct outcome ($r=1$). The expected reward is decomposed as $E_\pi[r] \approx (1-\epsilon)\alpha_\pi + \epsilon$, where $\alpha_\pi$ is the probability of generating valid reasoning. The term $\epsilon(1-\alpha_\pi)$ represents gains from "outcome reward hacking," where the policy learns to exploit the sparsity of outcome rewards to maximize reward without improving the underlying reasoning quality (faithfulness).

Furthermore, while Process Reward Models (PRMs) offer step-wise supervision, directly optimizing them or naively blending them with ORMs introduces instability. PRMs are often trained offline and suffer from distribution shift during online RL training, leading to incorrect judgments on rare or difficult trajectories. This can cause severe reward hacking, where models learn to generate verbose, repetitive, or nonsensical steps to maximize the averaged process score without improving the actual solution.

Methodology: Process Consistency Filter (PROF)

To address these issues, the authors propose PROF (Process Consistency Filter), a data curation strategy that integrates PRM supervision into RLVR without directly optimizing PRM scores. PROF operates as a filtering mechanism applied after the rollout phase, prior to policy updates.

Core Algorithm

1. Oversampling: The policy generates a group of $n$ responses (rollouts) for a given prompt.
2. Scoring:
   • Outcome Rewards ($r_o$): Obtained from the verifier (1 for correct, -1 for incorrect).
   • Process Rewards ($r_h$): Obtained from a pre-trained PRM for each reasoning step $h$.
   • Consistency Score ($r_{pro}$): A trajectory-level score is computed by averaging the step-level PRM scores and applying a regularization term based on step length. This score is then multiplied by the outcome reward $r_o$.
3. Group Separation: The rollouts are divided into two groups:
   • $G^+$: Correct responses ($r_o = 1$).
   • $G^-$: Incorrect responses ($r_o = -1$).
4. Balanced Filtering: The algorithm determines the number of samples to keep ($k_+$ and $k_-$) from each group to maximize the variance of the outcome rewards in the final batch (effectively balancing the ratio of correct to incorrect samples).
   • For $G^+$ (Correct): Samples are ranked by $r_{pro}$. The top $k_+$ samples (those with high process-outcome consistency) are retained. This removes correct answers derived from flawed reasoning.
   • For $G^-$ (Incorrect): The paper proposes two variants:
     • PROF-POS: Randomly selects $k_-$ samples from the incorrect group.
     • PROF-BOTH: Ranks incorrect samples by $r_{pro}$ and keeps the bottom $k_-$ (those with the lowest consistency, i.e., incorrect answers with weak or non-existent reasoning support).
5. Policy Update: The retained $m$ trajectories are used to update the policy (e.g., via GRPO).

Key Design Choices

• Filtering vs. Optimization: PROF uses PRMs for selection rather than as a direct reward signal, avoiding the instability of optimizing noisy process scores.
• Separate Ranking: Correct and incorrect groups are ranked separately to prevent incorrect responses with locally correct steps (which might have high PRM scores) from diluting the training signal or being incorrectly favored.
• Regularization: A step-length penalty ensures that samples with no segments or excessively long, unstructured steps are penalized.

Key Contributions

1. Identification of Reasoning-Quality Gap: The authors provide theoretical analysis and empirical evidence (showing 26.28% of correct responses from Qwen2.5-Math-7B contain flawed reasoning) that current RLVR fails to improve reasoning faithfulness due to outcome reward hacking.
2. PROF Framework: A robust, consistency-based data curation method that injects process supervision into RLVR. It filters conflicting trajectories (correct answers with bad reasoning, or incorrect answers with good reasoning) while maintaining a balanced training distribution.
3. Empirical Validation: Extensive experiments on Qwen and LLaMA models demonstrate that PROF improves both final-answer accuracy and intermediate reasoning quality compared to strong baselines (GRPO and blended PRM-ORM approaches), with reduced dependence on the strength of the pre-trained PRM.

Experimental Results

The authors evaluated PROF on five benchmarks (Math500, Minerva Math, Olympiad Bench, AMC2023, AIME2024) using Qwen2.5-Math (1.5B and 7B) and LLaMA-3.2-3B models.

• Accuracy Improvements: PROF consistently outperformed standard GRPO and "Blend" (naive combination of PRM and ORM) baselines.
  • On Qwen2.5-Math-7B, PROF-BOTH achieved 51.7% average accuracy, surpassing GRPO (49.9%) and Blend (47.3%).
  • On LLaMA-3.2-3B, PROF-POS improved accuracy by 1.8% over GRPO, whereas the Blend method performed worse than GRPO.
• Reasoning Quality:
  • Monte Carlo Estimation: PROF showed significantly higher step-value scores (probability of reaching a correct answer from an intermediate step) across all benchmarks, with gaps ranging from 9.2% to 37.4% compared to GRPO.
  • Flaw Auditing: Using Claude Sonnet 4.6 to audit correct responses, PROF reduced the rate of flawed reasoning within correct answers to 6%, compared to 8% for GRPO.
• Robustness: PROF maintained high performance even when using a weaker PRM (Skywork-PRM-1.5B), whereas the Blend method's performance degraded significantly.
• Stability: Unlike the Blend method, which suffered from entropy collapse and uncontrolled response length growth (indicative of reward hacking), PROF maintained controlled response lengths and gradual entropy decrease.

Significance and Claims

The paper claims that PROF offers a robust solution to the process-outcome mismatch inherent in current RLVR approaches. By decoupling the use of PRMs from direct reward optimization and instead using them for data curation based on consistency, PROF achieves:

• Faithful Reasoning: It ensures that improvements in final accuracy are driven by improvements in the reasoning process, not by exploiting loopholes in outcome verification.
• Generalizability: The framework is modular and works with different RL algorithms (e.g., GRPO, RAFT++) and various pre-trained PRMs, suggesting that training task-specific PRMs for every base model is unnecessary.
• Safety and Utility: By enhancing the faithfulness and interpretability of Chain-of-Thought (CoT) reasoning, PROF mitigates the risk of models producing misleading hallucinations that happen to be correct, thereby improving the practical utility and safety of LLMs in reasoning tasks.

The authors acknowledge limitations, primarily the increased computational cost due to oversampling and filtering, and the potential for bias amplification if the pre-trained PRMs themselves are biased. They suggest future work in balancing efficiency with reasoning quality and extending the method to other domains like coding and web navigation.