Technical Summary: CEPO – RLVR Self-Distillation using Contrastive Evidence Policy Optimization

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for post-training large language models (LLMs) on reasoning tasks. The standard approach, Group Relative Policy Optimization (GRPO), simplifies training by eliminating value networks and normalizing rewards within groups of sampled responses. However, GRPO suffers from a fundamental credit assignment bottleneck: it assigns a uniform sequence-level advantage to every token in a correct trajectory and a uniform negative signal to every token in an incorrect one.

This uniformity is inefficient for mathematical reasoning, where a single decisive inferential step or a minor arithmetic error determines the outcome, while the majority of tokens are grammatical fillers or boilerplate. Uniform credit assignment wastes gradient signal on filler tokens and fails to distinguish decisive reasoning steps from non-informative content, leading to slow convergence and noisy updates.

Prior attempts to fix this via on-policy self-distillation (conditioning the model on the correct answer as a teacher) have faced two critical issues:

1. Information Leakage: Methods like OPSD and SDPO minimize per-token divergence between the student and a privileged teacher. Theoretically, this introduces a vocabulary-wide sum of $r$-conditioned weights into the gradient. As training progresses, this "harmful deviation" dominates the benign signal, causing the model to encode spurious correlations between the input and the answer ($x \to r^+$), degrading generalization.
2. Weak Signal Quality: The recent RLSD method resolved leakage by using a stop-gradient on the evidence ratio $P^+_T(y_t)/P_S(y_t)$ to modulate the GRPO advantage. However, RLSD has three signal limitations:
   • Fluency Confound: The denominator $P_S(y_t)$ reflects base-rate fluency, suppressing the ratio for common tokens regardless of their semantic relevance.
   • Asymmetric Negative Signal: For wrong trajectories, the penalty is indirect and lacks explicit grounding in what the wrong answer predicts.
   • One-Sided Evidence: RLSD cannot distinguish a filler token (supported equally by correct and wrong answers) from a decisive reasoning step (supported by the correct answer but disfavored by the wrong one), as both yield the same ratio.

2. Methodology: Contrastive Evidence Policy Optimization (CEPO)

The authors propose CEPO, which replaces the single-reference evidence ratio with a contrastive ratio between a correct-answer teacher and a wrong-answer teacher.

Core Mechanism

Instead of comparing the student prior $P_S$ against the correct teacher $P^+_T$, CEPO compares the correct teacher $P^+_T$ against a wrong teacher $P^-_T$. The wrong teacher is constructed by conditioning the model on a rejected answer ($r^-$) drawn from the same training batch (specifically, the lowest-reward rejected rollout), incurring no additional sampling cost.

The Contrastive Evidence Delta is defined as:
$$\Delta^{CE}_t = \text{sg}\left( \log \frac{P^+_T(y_t)}{P^-_T(y_t)} \right)$$
Where $\text{sg}(\cdot)$ is the stop-gradient operator.

Bayesian Interpretation

The authors derive a clean Bayesian interpretation for $\Delta^{CE}_t$ as a differential belief update. It measures how much a token $y_t$ simultaneously strengthens the posterior belief in the correct answer $r^+$ while weakening the belief in the wrong answer $r^-$.

• Decisive Steps: Receive a large positive $\Delta^{CE}_t$ (strongly favored by $r^+$, disfavored by $r^-$).
• Filler Tokens: Receive $\Delta^{CE}_t \approx 0$ (supported equally by both $r^+$ and $r^-$).

Training Objective

The token-level advantage is modulated by this contrastive weight:
$$\hat{A}^{(i)}_t = A^{(i)} \cdot \left[ (1-\lambda) + \lambda \cdot \text{clip}(w^{CE}_t, 1-\epsilon_w, 1+\epsilon_w) \right]$$
where $w^{CE}_t = \exp(\text{sign}(A) \cdot \Delta^{CE}_t)$. The policy is updated using a standard PPO-clipped surrogate objective. The parameter $\lambda$ decays linearly from an initial value to 0 over a warmup period.

3. Key Contributions & Theoretical Guarantees

The paper establishes that CEPO inherits the structural safety of RLSD while strictly generalizing its signal quality:

1. Structural Safety: CEPO satisfies Direction Anchoring (privileged information cannot flip the sign of the update) and Leakage-Free Gradients (no vocabulary-wide $r$-conditioned sum). The wrong-answer teacher $r^-$ enters only as a stop-gradiented scalar.
2. RLSD Containment: If the wrong-answer teacher carries no information ($P^-_T = P_S$), CEPO reduces exactly to RLSD. Thus, RLSD is a degenerate case of CEPO.
3. Discriminative Sharpness: The authors prove that CEPO assigns strictly sharper credit than RLSD at a token $y_t$ if and only if the wrong-answer teacher disfavors that token relative to the student prior ($P^-_T(y_t) < P_S(y_t)$). This condition empirically concentrates on arithmetically and inferentially decisive positions, while filler tokens receive neutral weights ($w^{CE}_t \approx 1$), avoiding the introduction of spurious signals.

4. Experimental Results

The authors evaluated CEPO on Qwen3-VL models (2B and 4B parameters) trained on the Geo3k geometry dataset and tested on five multimodal mathematical reasoning benchmarks: DynaMath, LogicVista, MathVisionmini, MMMU, and WeMath.

• Performance Gains:
  • 2B Scale: CEPO achieved 43.43% average accuracy, outperforming GRPO (41.17%) by +2.26 percentage points.
  • 4B Scale: CEPO achieved 60.56%, outperforming GRPO (57.43%) by +3.13 percentage points.
  • Gains were most significant on benchmarks requiring fine-grained multi-step reasoning (e.g., LogicVista, MathVisionmini) and minimal on knowledge-retrieval benchmarks (MMMU).
• Failure of Baselines:
  • OPSD and SDPO degraded below the untrained baseline (e.g., 34.96% vs. 39.73% for 2B), empirically confirming the theoretical prediction of information leakage.
  • RLSD showed marginal improvement over GRPO but was outperformed by CEPO.
• Efficiency: CEPO adds only one additional teacher forward pass per trajectory compared to GRPO (similar overhead to RLSD), with no extra sampling cost.

5. Significance and Claims

The paper claims that CEPO offers a principled and practical solution to the credit assignment problem in RLVR. Its significance lies in:

• Resolving the Trade-off: It achieves the "best of both worlds": the structural safety required to prevent information leakage (a prerequisite for stable training) and the signal quality necessary to distinguish decisive reasoning steps from filler.
• Empirical Validation of Theory: The collapse of leakage-prone methods (OPSD/SDPO) below the baseline serves as empirical confirmation of the theoretical leakage analysis, while CEPO's success validates the contrastive approach.
• Interpretability: Analysis of token weights shows CEPO successfully localizes credit to critical algebraic derivations and suppresses fluent but uninformative prose, a behavior not observed in RLSD.

The authors conclude that CEPO provides a robust building block for the next generation of credit-aware RLVR training, with future work suggested for extending the method to larger models, text-only reasoning, and code generation.