Technical Summary: Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR), particularly methods like Group Relative Policy Optimization (GRPO), has advanced reasoning capabilities in domains such as mathematics and programming. However, existing approaches face two fundamental limitations:

1. Token-Level Neglect: Binary correctness rewards (correct/incorrect) applied at the trajectory level fail to distinguish the distinct contributions of individual tokens within a reasoning chain.
2. Zero-Advantage Inefficiency: In groups where all sampled responses are incorrect (or all correct), the trajectory-level advantage is zero, rendering vast amounts of training data useless for policy optimization.

Recent attempts to address this have utilized entropy as a signal for advantage shaping to encourage exploration or refine token importance. However, the authors argue that entropy is fundamentally misaligned with correctness. High entropy can indicate productive exploration of alternative solutions or detrimental confusion caused by incorrect reasoning paths. Consequently, entropy-based methods often lead to contradictory design choices (e.g., rewarding high-entropy tokens in some contexts while suppressing them in others) because entropy quantifies uncertainty without assessing the quality of the underlying reasoning.

Methodology: Contrastive Policy Optimization (CPO)

The paper proposes Contrastive Policy Optimization (CPO), a framework that replaces entropy with a token-level contrastive disagreement signal for correctness-aware advantage shaping.

Core Mechanism

CPO leverages the intuition that correctness emerges through comparison. Just as a student identifies mistakes by comparing their work to a reference answer, an LLM can identify erroneous tokens by contrasting its vanilla generation distribution ($\pi$) with a reference-guided posterior distribution ($\pi_{post}$).

1. Contrastive Disagreement ($\delta_t$):
   The method defines a token-level signal as the log-ratio between the probability of a token under a reference-guided prompt and its probability under the vanilla prompt:
   $$\delta_t(x, y) = \log \frac{\pi_{post}(y_t | x, y^\star, y_{<t})}{\pi(y_t | x, y_{<t})}$$
   Here, $y^\star$ is a reference answer (oracle). Theoretically, this ratio correlates monotonically with the probability that a specific token leads to a correct completion ($g(x, y_{<t}, y_t)$).

   • Positive $\delta_t$: Indicates the token aligns with the reference and is likely correct.
   • Negative $\delta_t$: Indicates the token contradicts the reference and is likely incorrect.

2. Advantage Shaping:
   CPO integrates this token-level signal into the trajectory-level advantage ($A_i$) derived from GRPO. The shaped advantage $\tilde{A}_{i,t}$ is computed as:
   $$\tilde{A}_{i,t} = A_i + \text{clip}(\alpha \cdot \delta_{i,t}, \text{bounds})$$

   • Clipping: To ensure stability, the magnitude of the token-level signal is clipped so it cannot reverse the direction of the trajectory-level advantage (e.g., a correct trajectory cannot be penalized by a single token's disagreement).
   • Zero-Advantage Resolution: For trajectories with zero advantage ($A_i = 0$), the trajectory-level signal provides no gradient. CPO utilizes $\delta_t$ to provide meaningful learning signals even in these cases, enabling optimization on data previously considered useless.

3. Theoretical Unification:
   The authors demonstrate that On-Policy Distillation (OPD) is a special case of CPO. In OPD, an external teacher model provides the posterior distribution. CPO generalizes this by allowing the posterior to be instantiated by any distribution more informed about correctness than the current policy, including self-conditioning on reference answers.

Key Contributions

1. Contrastive Disagreement as a Signal: The paper introduces and validates (theoretically and empirically) that contrastive disagreement is a more reliable token-level correctness signal than entropy. It distinguishes between useful exploration and detrimental confusion.
2. Theoretical Foundation for OPD: The framework provides a principled grounding for On-Policy Distillation, unifying diverse variants (external teachers, reference-guided self-teachers, critic-guided policies) under a single correctness-driven objective.
3. CPO Framework: A computationally efficient method for advantage shaping that resolves the zero-advantage problem in RLVR, enabling more informative gradients and effective exploration.
4. Empirical Validation: Extensive experiments showing CPO outperforms entropy-based RLVR methods on both in-domain math reasoning and out-of-domain generalization tasks.

Experimental Results

Experiments were conducted on Qwen2.5-Math-7B and Qwen3-Base-4B using the MATH dataset for training.

• Performance Gains: CPO substantially outperforms GRPO and other entropy-based baselines (e.g., Entropy-Tokens, Entropy-Adv, RL-ZVP).
  • On Qwen2.5-Math-7B, CPO improved average performance by 7.7% over GRPO.
  • On Qwen3-Base-4B, CPO improved average performance by 8.5% over GRPO.
• Generalization: Unlike many baselines that suffer from out-of-domain (OOD) degradation, CPO preserved or improved OOD performance on benchmarks like GPQA and MMLU-Pro.
• Ablation Studies:
  • Data Effectiveness: The study found that incorrect responses primarily drive in-domain improvements (exploitation), while correct responses are crucial for maintaining OOD generalization (exploration). A balanced ratio of $\alpha_+ : \alpha_-$ (1:1) yielded the best results.
  • Token-Level vs. Trajectory-Level: Token-level shaping proved superior to averaging disagreement at the trajectory level, as it better preserves crucial variations in credit assignment.
  • Entropy Analysis: CPO maintains higher training entropy and longer response lengths compared to GRPO, indicating sustained exploration. Word cloud analysis showed CPO focuses on task-critical tokens (math symbols, code constructs), whereas entropy focuses on linguistic variability.

Significance and Claims

The paper claims that CPO represents a shift from heuristic entropy-based interventions to a correctness-aware optimization strategy. By utilizing the contrast between self-generated and reference-guided distributions, CPO provides a reliable signal that aligns policy learning with task success.

The authors emphasize that the distinct roles of response types are critical: correct responses allow the model to explore novel solution paths (preventing overfitting), while incorrect responses allow the model to exploit and refine the most promising paths. Balancing these through the contrastive disagreement signal leads to superior performance without sacrificing generalization. Furthermore, the framework's ability to resolve the zero-advantage problem suggests a more efficient use of training data in RLVR settings.