Technical Summary: Self-Distillation Enables Continual Learning

Problem Statement

Foundation models currently face a fundamental limitation: they remain static after deployment. While they can adapt behavior at inference time via prompting or retrieval, they do not update their parameters to acquire new skills or internalize new knowledge. The core challenge is continual learning—enabling models to learn and improve over time without degrading existing capabilities.

Existing approaches to learning from expert demonstrations, the primary alternative to explicit reward functions, rely on Supervised Fine-Tuning (SFT). However, SFT is inherently off-policy, training on a fixed, offline data distribution. This leads to catastrophic forgetting, where adapting to new tasks causes severe regression in prior general capabilities. Conversely, on-policy learning (training on trajectories generated by the model itself) has been shown to reduce forgetting and improve generalization, but it typically requires explicit reward functions (Reinforcement Learning), which are often unavailable in real-world settings.

The central question addressed by this work is: How can we obtain the benefits of on-policy learning when only expert demonstrations are available?

Methodology: Self-Distillation Fine-Tuning (SDFT)

The authors propose Self-Distillation Fine-Tuning (SDFT), a method that enables on-policy learning directly from demonstrations by leveraging the in-context learning (ICL) capabilities of large pretrained models.

Core Mechanism

SDFT utilizes a single model in two distinct roles during training:

1. Teacher: The model conditioned on both the task input ($x$) and an expert demonstration ($c$). This creates a demonstration-aware distribution $Q = \pi(\cdot|x, c)$. The prompt structure encourages the model to understand the intent behind the demonstration rather than verbatim copying.
2. Student: The base model conditioned only on the task input ($x$), producing distribution $P = \pi(\cdot|x)$.

Training Objective

The student is trained to match the teacher's behavior on trajectories generated by the student itself (on-policy). The objective minimizes the reverse Kullback-Leibler (KL) divergence between the student and teacher distributions:
$$\mathcal{L}(\theta) = D_{KL} (\pi_\theta(\cdot|x) \parallel \pi(\cdot|x, c)) = \mathbb{E}_{y \sim \pi_\theta(\cdot|x)} \left[ \log \frac{\pi_\theta(y|x)}{\pi(y|x, c)} \right]$$

Theoretical Interpretation

The paper interprets SDFT through two lenses:

1. Distillation: A student-teacher framework where the teacher is a "wiser," demonstration-aware version of the student.
2. Inverse Reinforcement Learning (IRL): The authors mathematically show that SDFT is equivalent to maximizing an implicit reward function. By assuming the demonstration-conditioned policy approximates the optimal policy ($\pi^* \approx \pi(\cdot|x, c)$), the log-ratio between the teacher and student distributions acts as an intrinsic reward signal. This allows the model to perform on-policy updates without an explicit reward function.

Practical Implementation

• Teacher Parameterization: The teacher's parameters are typically an Exponential Moving Average (EMA) of the student's parameters.
• Loss Function: While the theoretical derivation suggests Reverse KL, the authors found that Forward KL at the logit level yields better practical performance.
• Gradient Estimation: The loss is decomposed into a token-level loss, treating the teacher distribution as fixed during gradient computation.

Key Contributions

1. Novel Framework: Introduction of SDFT, a simple method to convert expert demonstrations into on-policy learning signals without explicit reward inference.
2. Theoretical Equivalence: Demonstration that self-distillation from demonstrations is mathematically equivalent to maximizing an implicit reward function derived from the model's own in-context learning capabilities.
3. Validation of ICL Assumption: Empirical evidence that demonstration-conditioned models act as high-quality teachers (producing optimal outputs) while remaining close to the base policy (minimizing deviation), satisfying the requirements for trust-region regularization.
4. Robustness to Data Limitations: Demonstration that SDFT can train reasoning models effectively even when supervision data lacks explicit chain-of-thought traces, preventing the "reasoning collapse" often seen with standard SFT on answer-only data.

Experimental Results

The authors evaluated SDFT in two settings: Skill Learning (acquiring new tasks) and Knowledge Acquisition (integrating new factual information).

1. Skill Learning (Sequential Learning)

• Tasks: Science Q&A, Tool Use, and Medical reasoning.
• Findings:
  • Catastrophic Forgetting: In sequential training experiments (learning three skills one after another), SDFT enabled a single model to accumulate skills without performance regression on prior tasks. In contrast, SFT exhibited severe interference, where learning a new task caused rapid degradation of previously learned skills.
  • Generalization: SDFT achieved higher accuracy on new tasks compared to SFT, indicating better in-distribution generalization.
  • Pareto Efficiency: SDFT consistently achieved superior trade-offs between new-task accuracy and retention of prior capabilities compared to baselines (SFT, DFT, and Re-invoke).

2. Knowledge Acquisition

• Setup: Integrating factual knowledge about 2025 natural disasters (post-training cutoff) into a model.
• Findings:
  • SDFT significantly outperformed SFT and Continual Pre-Training (CPT) on both strict accuracy and out-of-distribution (indirect) questions.
  • SDFT achieved near-perfect accuracy on indirect questions, suggesting the new knowledge was properly integrated into the model's internal memory rather than just memorized as specific answer patterns.

3. Scaling and Model Size

• ICL Dependency: The effectiveness of SDFT is tightly coupled with model size. Smaller models (e.g., 3B) with weak in-context learning failed to provide meaningful teacher signals, performing worse than SFT. Larger models (7B, 14B) showed monotonic improvement, with the 14B model achieving a 6.9-point gap over SFT.

4. Reasoning Models

• Answer-Only Supervision: When fine-tuning a reasoning model (Olmo-3-7B-Think) on data containing only final answers (no chain-of-thought), standard SFT caused a collapse in reasoning depth (shorter responses) and reduced accuracy. SDFT preserved the model's long chain-of-thought behavior while improving accuracy.

Significance and Claims

The paper claims that SDFT establishes on-policy distillation as a practical path to continual learning from demonstrations.

• Overcoming the Off-Policy Limitation: The work demonstrates that the benefits of on-policy learning (reduced forgetting, better generalization) can be realized even without explicit reward functions, provided the model possesses sufficient in-context learning capabilities.
• Practical Viability: While SDFT incurs a higher computational cost (approx. 2.5x FLOPs and 4x wall-clock time compared to SFT due to on-policy rollouts), the authors argue this is offset by the elimination of multi-stage training pipelines (e.g., SFT followed by Re-invoke) required by other methods to mitigate forgetting.
• Complementary to RL: The authors position SDFT not as a replacement for on-policy RL, but as a complementary regime. It can serve as an effective initialization for subsequent reward-based fine-tuning.

The paper concludes that SDFT enables foundation models to keep learning and improving over time, moving closer to the human-like ability to accumulate knowledge and refine skills throughout a lifetime.