ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning

Technical Summary of ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning

Problem Statement
Reinforcement Learning with Verifiable Rewards (RLVR) has become a dominant paradigm for enhancing the mathematical reasoning capabilities of Large Language Models (LLMs). In this setting, models generate long reasoning trajectories, and an automatic verifier assigns a binary outcome reward based on the final answer. While critic-free methods like GRPO, DAPO, and GSPO avoid value estimation errors by assigning uniform response-level advantages, they lack the ability to distinguish between correct and incorrect intermediate reasoning steps. Conversely, Proximal Policy Optimization (PPO) offers the potential for token-level credit assignment via a learned critic. However, in mathematical reasoning characterized by long horizons and sparse terminal rewards, standard critics struggle to accurately evaluate intermediate states. Without access to the reference answer during the evaluation of partial trajectories, standard critics often produce noisy advantage estimates, leading to suboptimal policy updates. Furthermore, existing PPO-based RLVR systems exhibit an asymmetry: while reference answers determine the terminal reward, they remain unavailable to the critic responsible for token-level credit assignment.

Methodology: ReDiPPO
The authors propose ReDiPPO (Reference-guided and Discrepancy-aware PPO), a framework designed to improve token-level credit assignment without altering the inference-time policy interface. The method operates through two primary mechanisms:

1. Reference-Guided Value Calibration:
   ReDiPPO employs a dual-critic architecture during training.

   • Standard Critic ($V^{std}$): Evaluates the partial reasoning state (prompt + partial response) without access to the reference answer, mirroring the conditions of the policy model.
   • Reference-Guided Critic ($V^{ref}$): Conditions on the prompt, partial response, and the reference answer (provided as a training-time privileged signal). This allows the critic to assess whether a partial trajectory is compatible with the correct solution more accurately.
     The reference-guided critic provides the baseline for computing the actor's advantage ($\hat{A}^{ref}_t = G - V^{ref}_t$), while the policy itself remains prompt-only, ensuring no privileged information leaks into the generation process.

2. Discrepancy-Aware Token Reweighting:
   The framework leverages the disagreement between the two critics as a signal for reliability.

   • Discrepancy Calculation: The absolute difference between the standard and reference-guided value estimates ($e_{i,t} = |v^{ref}_{i,t} - v^{std}_{i,t}|$) is computed for each token. A large discrepancy indicates a state where the reference-blind critic is uncertain or likely to err.
   • Reweighting: This discrepancy is normalized and clipped to produce a token-level weight ($w_{i,t}$). The reference-guided advantages are then scaled by these weights ($\tilde{A}_{i,t} = \text{Whiten}(w_{i,t} \cdot \hat{A}^{ref}_{i,t})$) before being used in the PPO objective. This mechanism emphasizes updates for tokens in states where the reference information significantly alters the value assessment, effectively down-weighting noisy estimates from the standard critic.

Key Contributions

• Reference-Guided Value Calibration: The introduction of a critic that utilizes reference answers as privileged information during training to construct a more informative baseline for token-level advantages, addressing the difficulty of evaluating partial solutions under sparse rewards.
• Discrepancy-Aware Token Reweighting: A novel mechanism that converts the disagreement between reference-guided and standard critics into bounded token-level weights, allowing the optimization process to adaptively focus on reasoning states where standard value estimation is unreliable.
• Empirical Validation: The method is evaluated across six mathematical reasoning benchmarks (AIME 2024/2025/2026, HMMT 2025, Minerva Math, OlympiadBench) and three distinct policy backbones (Qwen3-4B-Instruct, Qwen3-4B-Thinking, and OLMo3-7B-Instruct).

Experimental Results
ReDiPPO consistently outperforms strong baselines, including critic-free methods (DAPO, GSPO) and vanilla PPO.

• Performance Gains: Across all three backbones, ReDiPPO achieved the highest average accuracy. Specifically, it improved upon vanilla PPO by 1.19 to 2.37 percentage points. For instance, on the Qwen3-4B-Instruct backbone, the average score increased from 53.66 (PPO) to 56.03 (ReDiPPO).
• Critic Quality: Analysis of the training dynamics shows that the reference-guided critic achieves significantly higher Path Selection Accuracy (PSA) and Explained Variance (EV) compared to the standard critic.
• Discrepancy Analysis: The study reveals a strong correlation between high reference-standard discrepancy and difficult reasoning trajectories (longer lengths and lower accuracy). Furthermore, the value estimation gains from reference conditioning are most pronounced in the later stages of the reasoning trajectory, justifying the token-level reweighting approach over a single global difficulty score.

Significance and Claims
The paper claims that ReDiPPO addresses the central challenge of reliable token-level credit assignment in long-horizon mathematical reasoning by effectively utilizing reference answers as training-time privileged information without compromising the policy's inference capabilities. The authors assert that their approach provides a more informative critic baseline and a robust mechanism for identifying and reweighting unreliable credit assignments. The work suggests that leveraging reference-guided value learning is a promising direction for PPO-based RLVR, particularly in domains where outcome verification is possible but intermediate step evaluation is ambiguous. The authors note that a remaining open question is whether this approach transfers to domains lacking unique, rule-verifiable answers.