Self-Distilled RLVR

1. Problem Statement

The paper addresses a critical instability in On-Policy Self-Distillation (OPSD), a popular training paradigm for Large Language Models (LLMs).

• Context: In OPSD, a single model acts as both the student and the teacher. The "teacher" version is conditioned on privileged information ($r$), such as a verified reasoning trace or the correct answer, while the "student" only sees the input query ($x$). The goal is to use the teacher's dense, token-level signals to guide the student's learning.
• The Failure Mode: Despite initial performance gains, OPSD suffers from severe privileged information leakage and training instability. The model learns to implicitly "cheat" by encoding correlations between the input ($x$) and the privileged information ($r$) into its parameters. During inference (where $r$ is unavailable), the model explicitly references these invisible references, leading to a rapid peak in performance followed by systematic degradation.
• Root Cause: The authors argue that standard distribution matching in OPSD is ill-posed under information asymmetry. The student is forced to match a conditional distribution $P(y|x, r)$ that depends on $r$, which it cannot observe. This creates an irreducible mutual information gap that drives the model to learn spurious $x \to r$ correlations rather than genuine reasoning.

2. Methodology: RLSD (RLVR with Self-Distillation)

The authors propose RLSD, a new training paradigm that decouples the direction of updates from the magnitude of updates, leveraging the strengths of both Reinforcement Learning with Verifiable Rewards (RLVR) and Self-Distillation.

Core Concept

Instead of using the teacher as a generative target for distribution matching (which causes leakage), RLSD repurposes the teacher as a token-level credit evaluator within a policy gradient framework.

Algorithmic Steps

1. Sequence-Level Direction (RLVR):

   • The model generates a trajectory $y$.
   • A verifier provides a binary reward $R(x, y)$ (0 or 1).
   • A sequence-level advantage $A$ is computed (e.g., via Group Relative Policy Optimization, GRPO). This determines the direction of the update: positive for correct trajectories, negative for incorrect ones. This anchors the optimization to the environment, ensuring reliability.

2. Token-Level Magnitude (Self-Distillation):

   • The model computes the privileged information gain ($\Delta_t$) for each token $y_t$:
     $$\Delta_t = \text{sg}(\log P_T(y_t | x, r) - \log P_S(y_t | x))$$
     where $P_T$ is the teacher (conditioned on $r$) and $P_S$ is the student (conditioned only on $x$). The stop-gradient ($\text{sg}$) ensures $\Delta_t$ is a static weight.
   • This gain represents how much the privileged information revises the model's belief about the token.
   • A token-level weight $w_t$ is constructed:
     $$w_t = \exp(\text{sign}(A) \cdot \Delta_t) = \left( \frac{P_T(y_t)}{P_S(y_t)} \right)^{\text{sign}(A)}$$
   • Interpretation:
     • If the trajectory is correct ($A > 0$), tokens supported by the privileged info ($P_T > P_S$) receive higher positive credit.
     • If the trajectory is incorrect ($A < 0$), tokens disfavored by the privileged info ($P_T < P_S$) receive higher negative credit (blame).

3. Clipped Credit Assignment:

   • The final token-level advantage is $\hat{A}_t = A \cdot \text{clip}(w_t, 1-\epsilon, 1+\epsilon)$.
   • This is used to update the policy, replacing the uniform advantage used in standard GRPO.

Key Design Principles

• Direction Anchoring: The update direction is strictly determined by the environment reward ($A$), preventing the privileged information from flipping the gradient sign.
• Magnitude Modulation: The privileged information only scales the intensity of the update for specific tokens, allowing for fine-grained discrimination without altering the fundamental learning objective.
• No Auxiliary Loss: RLSD requires no external teacher model or auxiliary loss functions; it only needs one additional forward pass to compute teacher logits.

3. Key Contributions

1. Theoretical Diagnosis of OPSD:

   • Proved that OPSD's objective contains an irreducible mutual information gap ($I(Y_t; R | X, Y_{<t}) > 0$) due to information asymmetry.
   • Demonstrated that this gap causes the gradient to contain a path-dependent deviation term that drives the model to encode $x \to r$ correlations, leading to leakage and performance collapse.
   • Established an Impossibility Trilemma for shared-parameter self-distillation: one cannot simultaneously achieve objective stability, sustained improvement, and leakage-free training under distribution matching.

2. Proposal of RLSD:

   • Introduced a paradigm that unifies RLVR and Self-Distillation by using the teacher solely for credit assignment magnitude rather than distribution matching.
   • Provided a formal proof that RLSD is structurally immune to privileged information leakage because the gradient direction is decoupled from the privileged signal.

3. Empirical Validation:

   • Demonstrated that RLSD achieves higher convergence ceilings and superior stability compared to GRPO and OPSD.
   • Showed that RLSD outperforms baselines on complex multimodal reasoning tasks.

4. Experimental Results

The authors evaluated RLSD on Qwen3-VL-8B-Instruct across five multimodal reasoning benchmarks (MMMU, MathVista, MathVision, ZeroBench, WeMath).

• Performance: RLSD achieved the highest average accuracy (56.18%), outperforming the base LLM by 4.69% and standard GRPO by 2.32%.
• Convergence Speed: RLSD surpassed GRPO trained for twice as many steps in just 200 steps, demonstrating faster convergence.
• Stability: Unlike OPSD, which peaked early and degraded, RLSD maintained stable training dynamics and avoided the "performance collapse" phase.
• Credit Assignment Analysis: Visualizations (heatmaps) confirmed that RLSD correctly assigned high credit to decisive reasoning steps (e.g., calculation steps) and high blame to errors (e.g., misreading a graph), whereas GRPO applied uniform credit and OPSD showed signs of leakage.

5. Significance

• Solving the Leakage Problem: The paper provides a rigorous theoretical and practical solution to the long-standing issue of information leakage in self-distillation, enabling the safe use of privileged information (like reference answers) during training without compromising inference integrity.
• Efficiency: RLSD offers a "drop-in" replacement for standard GRPO that requires no external teacher models, significantly reducing computational overhead while providing dense, token-level supervision.
• Paradigm Shift: It challenges the prevailing view that self-distillation must rely on distribution matching. By reframing the teacher's role as a magnitude modulator rather than a generative target, the authors unlock a new path for stable, high-performance reasoning model training.
• Generalizability: While validated on multimodal reasoning, the framework is applicable to any RLVR setting where verifiable outcomes exist, offering a robust alternative to sparse-reward RL and unstable self-distillation.