Technical Summary: Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning (ATESD)

1. Problem Statement: Teacher-Side Exposure Mismatch

While On-Policy Self-Distillation (OPSD) has emerged as a dominant method for improving Large Language Model (LLM) reasoning by aligning supervision with the student's own rollouts, this paper identifies a critical, previously unexamined bottleneck: teacher-side exposure mismatch.

In standard OPSD, the "teacher" (a frozen copy of the model) conditions on the full reference solution (privileged Chain-of-Thought) when providing token-level supervision. The authors argue that this default assumption—that more privileged information is always better supervision—is flawed.

• The Mismatch: When the full reference reasoning far exceeds the student's current competence (common in hard problems), the resulting token targets become too strong to absorb. The teacher's predictive distribution becomes overly sharp and concentrated on tokens the student cannot yet generate, creating a supervision signal that is informative but unlearnable.
• Empirical Evidence: Controlled sweeps of fixed exposure levels ($\alpha \in [0, 1]$) on AIME 2024 reveal two key patterns:
  1. Suboptimality of Full Exposure: Intermediate exposure levels (e.g., $\alpha=0.5$) often outperform full exposure ($\alpha=1.0$).
  2. Monotonic Mismatch Growth: The teacher-student distribution mismatch (measured by on-policy KL divergence and top-1 disagreement) grows monotonically as the teacher sees more privileged reasoning.
  3. Regime Dependency: The optimal exposure level varies by problem difficulty (easy problems prefer full exposure; hard problems prefer lower exposure), suggesting a single fixed default is insufficient.

2. Methodology: ATESD

To address this, the authors propose Adaptive Teacher Exposure for Self-Distillation (ATESD). This framework treats teacher exposure not as a fixed hyperparameter, but as a learnable, training-state-conditioned control variable.

Core Components

1. Exposure-Modulated Teacher:

   • Instead of conditioning on the full reference $y^*$, the teacher conditions on a truncated prefix $truncate(y^*, \alpha_t)$, where $\alpha_t \in [0, 1]$ is the exposure ratio.
   • The truncation applies only to the reasoning prefix; the final boxed answer is always preserved to maintain the task constraint.
   • The student rollout protocol remains unchanged (on-policy sampling from the problem-only prompt), ensuring the comparison isolates the effect of the teacher's context.

2. Beta Exposure Controller ($\pi_\phi$):

   • A lightweight Multi-Layer Perceptron (MLP) maps compact training-state statistics (e.g., recent loss, mismatch EMAs, student self-confidence) to the concentration parameters $(a_t, b_t)$ of a Beta distribution.
   • The controller samples a global exposure ratio $\alpha_t$ for a hold window of $H$ student updates. This ensures a single decision governs a short episode, allowing for stable credit assignment.
   • The Beta distribution is constrained to be unimodal ($a, b > 1$), allowing the policy to learn a preferred exposure level with varying confidence.

3. Closed-Loop Training with Delayed Credit:

   • Challenge: Exposure decisions affect the student's future learning progress, not just the immediate loss. A high-exposure decision might be beneficial long-term even if it causes a temporary loss spike.
   • Solution: The controller is trained via REINFORCE using a discounted learning-progress reward.
   • Reward Signal ($R$): Composed of two parts:
     • $G_{lp}$: Discounted sum of positive loss improvements over a lookahead window ($L$ steps).
     • $G_{gt}$: A teacher-grounded credit score based on the log-probability of verified reference tokens, ensuring the selected exposure still aligns with the ground truth.
   • This approach addresses the delayed credit assignment problem inherent in on-policy distillation, avoiding the myopic bias of one-step loss proxies.

3. Key Contributions

1. Identification of Teacher-Side Exposure Mismatch: The paper formalizes the observation that full privileged reasoning is not universally optimal and that mismatch grows with exposure, motivating the shift from fixed defaults to adaptive control.
2. ATESD Framework: A novel method that models exposure as a continuous, learnable variable via a Beta-policy controller, trained with a two-timescale schedule (fast student updates, slow controller updates) and delayed reward signals.
3. Empirical Validation: Comprehensive experiments demonstrating that adaptive exposure consistently outperforms fixed exposure and standard baselines across model scales and reasoning benchmarks.

4. Experimental Results

The method was evaluated on AIME 2024, AIME 2025, and HMMT 2025 using Qwen3 models (1.7B, 4B, and 8B).

• Performance Gains: ATESD consistently outperformed the OPSD baseline and other RL/distillation baselines (SFT, GRPO).
  • Qwen3-1.7B: +0.95 Average@12 points over OPSD.
  • Qwen3-4B: +2.05 Average@12 points over OPSD (reaching 65.65 Average@12).
  • Qwen3-8B: +2.33 Average@12 points over OPSD.
• Ablation Findings:
  • Delayed Credit: Removing the delayed lookahead reward (using immediate one-step feedback) significantly degraded performance (52.22 vs. 59.17), confirming the necessity of long-horizon credit assignment.
  • Learned vs. Fixed: The learned policy (59.17) outperformed the best fixed exposure setting (57.44) and uncontrolled stochastic exposure (54.94), proving that the controller learns a dynamic strategy rather than simply selecting a static optimal constant.
  • Mismatch Reduction: On fixed positive trajectories, adaptive exposure reduced the token-level teacher-student KL divergence (from 0.0136 to 0.0061), demonstrating that it prevents the teacher from over-constraining the student.

5. Significance and Claims

The paper claims that adaptive teacher exposure is an effective new axis for reasoning self-distillation. By treating the teacher's access to privileged information as a controllable variable rather than a fixed assumption, ATESD resolves the supervision-side analogue of the rollout mismatch.

The authors emphasize that their contribution is not merely a new curriculum or a different student training distribution, but a fundamental shift in how the teacher conditions on reference solutions. The method establishes that:

• Full exposure is a suboptimal default.
• Exposure must be matched to the student's current learning regime.
• Learning this exposure requires delayed credit assignment mechanisms similar to those used in Reinforcement Learning.

The paper concludes that ATESD provides a robust, scalable improvement for post-training reasoning models, particularly as model capacity increases, while noting future work could explore per-sample difficulty conditioning and application to broader reasoning domains beyond mathematical contests.