Technical Summary: RLCSD – Reinforcement Learning with Contrastive On-Policy Self-Distillation

1. Problem Statement: Privilege-Induced Style Drift

The paper identifies a critical pathology in On-Policy Self-Distillation (OPSD), a paradigm where a single model acts as both teacher and student, with the teacher granted access to "privileged context" (e.g., a verified solution) unavailable to the student.

While existing OPSD methods (e.g., OPSD, SDPO, SRPO, RLSD) aim to provide dense, token-level supervision by minimizing the distributional gap between the student and the privileged teacher, the authors demonstrate that this learning signal is fundamentally flawed. They term this flaw privilege-induced style drift.

• The Mechanism: Conditioning the teacher on a verified solution (the privileged context) does sharpen its confidence on task-bearing tokens (e.g., mathematical operators). However, it simultaneously induces a systematic, correctness-orthogonal shift in generative behavior. The privileged teacher tends to produce shorter, more assertive outputs, suppresses exploratory hedges, and reallocates probability mass toward formatting and discourse tokens (e.g., "Wait," "Therefore," "\n\n").
• The Consequence: The resulting learning signal ($e_{c,t}$) is dominated by these style tokens rather than the task-bearing tokens that determine the answer.
• Observed Pathologies: This signal skew leads to two distinct failure modes in training:
  1. Entropy Explosion and Instability: Methods like OPSD and SDPO suffer from unbounded entropy growth, leading to response-length explosions and training collapse.
  2. Premature Length Shrinkage: Methods like RLSD exhibit a drastic reduction in response length, limiting the model's ability to perform long-horizon reasoning.

2. Methodology: RLCSD

To address privilege-induced style drift, the authors propose RLCSD (Reinforcement Learning with Contrastive On-Policy Self-Distillation). The core insight is that the stylistic shifts induced by the privileged context are shared across both correct and incorrect hints when presented under identical prompt templates. By contrasting a correct hint against an incorrect hint, the shared stylistic component can be cancelled out, leaving a signal concentrated on task correctness.

The RLCSD pipeline consists of three stages:

Stage 1: Rollout Sampling and Partitioning

For a given query $x$, the student policy $\pi_S$ samples a group of $G$ rollouts. A rule-based verifier partitions these into a correct subset ($\mathcal{G}^+$) and an incorrect subset ($\mathcal{G}^-$). Groups with uniform outcomes (all correct or all incorrect) are discarded as they yield zero advantage under standard GRPO normalization.

Stage 2: Constructing the Contrastive Token-Level Signal

Instead of using a single privileged hint, RLCSD constructs a contrastive signal $e_{ctr,t}$:

1. Positive Hint ($y_c^*$): Sampled from $\mathcal{G}^+$.
2. Negative Hints ($y_{w,k}^*$): $K$ hints sampled from $\mathcal{G}^-$.
3. Symmetric Prompting: Both hints are wrapped in an identical prompt template (Problem + Reference Solution + Answer). This ensures the teacher treats both as ground-truth references, making the induced stylistic shifts identical.
4. The Signal: The contrastive signal is computed as the difference between the log-probability gap under the correct hint and the average log-probability gap under the $K$ negative hints:
   $$e_{ctr,t} = \log \pi_T(y_t | x, y_c^*, y_{<t}) - \log \left( \frac{1}{K} \sum_{k=1}^K \pi_T(y_t | x, y_{w,k}^*, y_{<t}) \right)$$
   Refinements: The method employs K-marginalization over negative hints to stabilize the signal against heterogeneous error types and excludes the target rollout from the hint pool to prevent the teacher from becoming over-confident in its own generation.

Stage 3: Verifier-Anchored Modulation and Loss

The contrastive signal $e_{ctr,t}$ is not used as a standalone reward but as a modulation of the outcome-level advantage ($A_{ORM}$) derived from GRPO.

• Modulation: $e_{ctr,t}$ is soft-normalized (via $\tanh$) and rescaled to produce a token-level modulation term $r_t$.
• Sign-Preserving Clamp: The modulation is added to $A_{ORM}$, but the result is clamped to ensure the update direction never contradicts the verifier's outcome signal.
• Two-Path Loss: The final objective aggregates a two-path PPO-style clipped loss:
  1. An unmodulated path for tokens where the contrastive signal is weak (noise).
  2. A modulated path for tokens where the signal is strong.
     Crucially, these paths are normalized independently within each rollout to prevent the modulated signal from being diluted by the majority of unmodulated tokens.

3. Key Contributions

1. Characterization of Privilege-Induced Style Drift: The paper identifies and quantifies a systematic bias in OPSD where privileged conditioning shifts the learning signal toward stylistic tokens, causing training instability or premature length shrinkage.
2. RLCSD Algorithm: A novel method that mitigates this drift through symmetric contrastive cancellation between correct and incorrect privileged hints. It integrates this cleaned signal into Reinforcement Learning with Verifiable Rewards (RLVR) as a verifier-anchored token-level modulation.
3. Generalizability of the Contrastive Principle: The authors demonstrate that the contrastive construction is a method-agnostic component. It can be plugged into existing OPSD baselines (like OPSD and RLSD) to improve their stability and performance, suggesting the principle applies broadly to on-policy distillation.
4. Extension to Cross-Model Distillation: The paper extends the insight to cross-model On-Policy Distillation (OPD), showing that a similar decomposition of task-bearing vs. surface-style tokens explains why stylistically aligned but weaker teachers can sometimes outperform stronger, stylistically distant teachers.

4. Experimental Results

Experiments were conducted on Qwen3 (1.7B, 4B, 8B) and Olmo-3-7B-Think across mathematical reasoning (AMC23, AIME24, AIME25) and logical reasoning (Knights & Knaves).

• Performance: RLCSD consistently outperforms GRPO and prior OPSD methods (OPSD, SDPO, SRPO, RLSD) across all benchmarks.
  • On Qwen3-8B, RLCSD achieved a mean@12 of 90.8 on AMC23 (vs. 90.1 for GRPO) and 77.5 on AIME24.
  • On Logical Reasoning (OOD), RLCSD showed significant gains, reaching 81.8 pass@1 on Knights & Knaves (4-8 roles) and 79.0 on the 9-role OOD split, outperforming GRPO by a wide margin.
• Training Stability: Unlike competing OPSD methods which exhibited entropy explosion or response-length shrinkage, RLCSD maintained stable entropy and response lengths throughout training, comparable to GRPO.
• Ablation Studies:
  • Removing the contrastive mechanism (using only one-sided hints) caused performance drops of 3.0–5.4 points, confirming the necessity of the contrast.
  • Removing the K-marginalization or allowing self-rollouts in the hint pool degraded performance, validating the design refinements.
  • Removing the two-path loss aggregation (using a single global average) significantly reduced performance, proving the need to isolate the modulated tokens.

5. Significance and Claims

The paper claims that RLCSD offers a robust solution to the instability plaguing on-policy self-distillation. By isolating the "task-bearing" signal from the "style" noise inherent in privileged conditioning, RLCSD enables stable, high-performance training of reasoning models without the memory overhead of maintaining a separate teacher model.

The authors modestly claim that the contrastive principle is a general insight for on-policy distillation. It not only improves the specific RLCSD algorithm but also serves as a diagnostic tool for selecting teachers in cross-model distillation settings, where the alignment of reasoning styles may be more critical than raw capability. The work suggests that future distillation methods must account for and mitigate the parasitic stylistic shifts induced by privileged context.