Technical Summary: Direction-Adaptive Self-Distillation for LLM Reasoning

1. Problem Statement

The paper addresses a critical failure mode in On-Policy Self-Distillation (OPSD), an emerging post-training paradigm for Large Language Models (LLMs). In OPSD, a model acts as both student and teacher: the "student" generates rollouts, while a "privileged" version of the same model (conditioned on reference traces, hints, or feedback) serves as a teacher to provide dense token-level supervision.

While OPSD aims to combine the outcome supervision of Reinforcement Learning from Verifiable Rewards (RLVR) with fine-grained guidance, recent studies indicate it degrades complex reasoning. The authors diagnose this degradation as a directional mismatch:

• Uniform Attraction: Standard OPSD applies a uniform "attractive" force, pulling the student toward the privileged teacher's distribution.
• The Consequence: This suppresses predictive uncertainty (epistemic markers like "wait," "hmm," or "alternatively") which are crucial for exploration, hypothesis revision, and self-correction.
• The Paradox: Simply reversing the direction (pushing the student away from the teacher) to restore uncertainty fails because it corrupts low-entropy scaffolding tokens (routine execution steps), leading to execution errors.

The core problem is that a single supervision direction (either attraction or repulsion) cannot simultaneously preserve exploration at high-entropy decision points and stabilize execution at low-entropy routine steps.

2. Methodology: Direction-Adaptive Self-Distillation (DASD)

The authors propose Direction-Adaptive Self-Distillation (DASD), which reframes privileged self-distillation from uniform imitation into entropy-routed directional supervision. The method dynamically switches the direction of the teacher's pressure based on the student's predictive entropy ($H_t$) at each token.

Core Mechanism

DASD introduces a signed coefficient, $\omega_t \in [-1, 1]$, to the self-distillation objective. This coefficient determines whether the student is pulled toward (attractive) or pushed away from (repulsive) the privileged teacher.

1. Entropy Routing:

   • High-Entropy Tokens (Forks): At decision points where the student is uncertain ($H_t$ is high), the privileged teacher's guidance is treated as a constraint that might prematurely collapse alternative reasoning paths. DASD applies repulsive pressure (negative $\omega_t$) to push the student away from the teacher, preserving exploration and the ability to revise hypotheses.
   • Low-Entropy Tokens (Scaffolding): At routine execution steps where the student is confident ($H_t$ is low), the privileged teacher provides stable, correct guidance. DASD applies attractive pressure (positive $\omega_t$) to pull the student toward the teacher, stabilizing step-level execution and preventing local errors.

2. Mathematical Formulation:
   The objective combines a global verifier anchor with a local, signed distillation term. The signed loss for a token $t$ is:
   $$\tilde{D}_t^{(i)}(\theta) = \omega_t^{(i)} \cdot \text{KL}\left( \pi_\theta(\cdot | x, o_{<t}^{(i)}) \parallel \text{sg}[\pi_\theta(\cdot | x, r, o_{<t}^{(i)})] \right)$$
   Where:

   • $\omega_t^{(i)}$ is computed using a tanh router based on the deviation of the current token's entropy from the trajectory's entropy quantile.
   • A gap-gate (sigmoid function) modulates the magnitude based on the log-evidence gap between the student and the privileged teacher, ensuring only reliable disagreements influence the update.
   • The global verifier (e.g., a programmatic reward) remains the anchor for trajectory-level correctness, while the self-teacher provides the local directional signal.

3. Training Dynamics:
   The method uses Group Relative Policy Optimization (GRPO) as the backbone. The advantage term is adjusted to include the directional correction: $\hat{A}_t = A_G + \beta \omega_t \bar{\delta}_t$. This allows the model to learn from the privileged trace during training without requiring that trace at inference time.

3. Key Contributions

The paper makes three primary contributions:

1. Diagnosis of Token-Level Directional Mismatch: The authors identify that OPSD degradation stems from applying a uniform supervision direction to heterogeneous token populations. They demonstrate via probes that conformity harms high-entropy forks (suppressing exploration), while deviation harms low-entropy scaffolding (corrupting execution).
2. Proposal of DASD: They introduce an entropy-routed objective that dynamically routes the privileged self-teacher signal: repulsive for high-entropy forks to preserve search, and attractive for low-entropy scaffolding to ensure stability.
3. Empirical Validation: Through extensive experiments, they show that DASD improves reasoning performance by maintaining the "health" of the reasoning process—simultaneously achieving rigorous step execution and flexible self-correction.

4. Experimental Results

The authors evaluated DASD on six mathematical reasoning benchmarks (MATH500, Minerva, HMMT25, OlympiadBench, AIME24, AIME25) across three model scales (Qwen3-1.7B, 4B, and 8B).

• Performance: DASD achieved the best macro Avg@16 across all scales and benchmarks. The gains were most pronounced on harder, competition-style benchmarks (e.g., HMMT25, AIME25) that require maintaining multiple plausible solution paths.
• Comparison to Baselines: DASD outperformed strong baselines including GRPO, HEPO, and various self-distillation methods (OPSD, SDPO, RLSD, SRPO). Notably, while standard OPSD often performed close to or worse than the base model, DASD consistently improved upon the RLVR baseline.
• Pass@k Scaling: DASD demonstrated superior Pass@k scaling, indicating that the method expands the set of reachable solution paths (diversity) rather than just sharpening a single deterministic prediction.
• Reasoning Health: Diagnostic metrics confirmed that DASD preserves both Step Accuracy (execution quality) and Exploration Density (frequency of epistemic markers and revision rates), whereas OPSD collapsed exploration and GRPO often lacked sufficient self-correction.
• Generalization: The method showed consistent improvements in cross-domain (code, scientific reasoning, tool use) and cross-family (Olmo, Llama) evaluations, suggesting the mechanism is not specific to the Qwen3 architecture or math domain alone.

5. Significance and Claims

The paper claims that the productive lever for self-distilled reasoning is the direction of supervision, not merely its strength or magnitude.

• Mechanism Insight: The work establishes that entropy is not just a magnitude knob for weighting updates (as in prior entropy-guided methods) but a sign selector for privileged teacher pressure.
• Paradigm Shift: It challenges the assumption that self-distillation must be purely attractive. By allowing the model to "push away" from a privileged teacher at critical decision points, DASD resolves the train-inference mismatch where the model learns a solution-conditioned style that fails when privileged information is absent.
• Practical Impact: DASD offers a way to leverage privileged information (like verified traces) during training to improve reasoning robustness without requiring external teachers or sacrificing the exploratory behavior necessary for solving complex problems.

The authors conclude that while the method inherits the requirement for verified traces during training (a property of the privileged self-distillation paradigm), it successfully decouples the need for privileged context at inference time while significantly enhancing reasoning capabilities.