Technical Summary: Strong Teacher Not Needed? On Distillation in LLM Pretraining

Problem Statement

Knowledge distillation (KD) in large language model (LLM) pretraining has long operated under the assumption that a "strong-to-weak" relationship is necessary: a teacher model must be significantly more capable (via larger architecture or more training tokens) than the student to yield performance gains. This paper challenges the completeness of this assumption. It investigates whether weaker teachers can improve stronger students and whether continuously strengthening a teacher (via scaling parameters or tokens) monotonically improves student performance. The authors argue that the prevailing view overlooks the critical role of teacher-student compatibility and the specific nature of the knowledge being transferred.

Methodology

The authors conduct a systematic empirical study varying teacher and student configurations across multiple dimensions:

• Architectures: Four model sizes based on the Llama3 family (0.7B, 1.7B, 3.8B, and 8.0B parameters).
• Training Budgets: Teachers are pretrained on token budgets ranging from 10B to 300B. Students are consistently trained on 50B tokens.
• Relationships: This setup creates diverse teacher-student relationships:
  • Strong-to-weak: Teacher is larger or more trained than the student.
  • Same-level: Teacher and student share architecture and token budget (differing only in random seed).
  • Weak-to-strong: Teacher is smaller or less trained than the student.
• Training Objective: The student is trained using a mixed loss function combining standard autoregressive language modeling (LM) and knowledge distillation (KD):
  $$\mathcal{L} = (1 - \alpha)\mathcal{L}_{lm} + \alpha\mathcal{L}_{kd}$$
  The authors systematically search for the optimal loss mixing coefficient $\alpha \in \{0.2, 0.4, 0.5, 0.6, 0.8, 1.0\}$ for each configuration.
• Evaluation: Performance is measured across three dimensions:
  1. In-domain Perplexity: Fit to the training distribution (FineWeb-Edu).
  2. Out-of-Distribution (OOD) Perplexity: Generalization across 11 diverse corpora (e.g., math, code, legal, biomedical).
  3. Downstream Accuracy: Performance on 15 benchmarks covering reasoning, commonsense, and factual knowledge.

Key Findings

1. Weak-to-Strong and Same-Level Distillation are Effective

Contrary to the strong-to-weak dogma, the study finds that teachers weaker than the student (smaller architecture or fewer training tokens) can still improve student performance, provided the loss mixing coefficient $\alpha$ is appropriately tuned (typically lower $\alpha$).

• Even a 0.7B teacher trained on 30B tokens improved a 1.7B student's downstream accuracy by 1.3%.
• Same-level distillation (identical architecture and token budget, different seed) consistently yields positive gains across all metrics, suggesting distillation transfers knowledge beyond what architecture and data volume alone capture.

2. Stronger Teachers Do Not Always Yield Better Students

The relationship between teacher strength and student performance is non-monotonic.

• Saturation and Reversal: At high token budgets, larger teachers can underperform smaller ones. For example, with 300B tokens, a 1.7B teacher yielded a +4.3% downstream accuracy improvement, while an 8.0B teacher yielded only +2.8%.
• Overtraining: For large teachers (e.g., 8.0B), increasing training tokens beyond a certain point (e.g., from 30B to 300B) degrades the distillation gains. The optimal $\alpha$ for these overtrained teachers also decreases, indicating that the student must rely less on the teacher's signal to avoid performance degradation.
• Compatibility: The results suggest that "compatibility" (matching the student's capacity and training regime) is more critical than absolute teacher scale.

3. Generalization Improves More Readily Than In-Domain Fitting

Distillation pretraining shows a distinct pattern across evaluation types:

• Generalization: OOD perplexity and downstream accuracy exhibit larger and more consistent gains than in-domain perplexity. Some configurations that degrade in-domain performance still improve OOD and downstream metrics.
• Mechanism: This suggests distillation primarily transfers useful knowledge for generalization rather than simply fitting the training distribution.
• Token Difficulty Analysis: Mechanistic studies reveal that distillation benefits "hard tokens" (high entropy) significantly more than easy tokens. This refutes the hypothesis that KD acts merely as a uniform regularizer (like label smoothing); instead, it acts as a targeted transfer of knowledge where the baseline model is most uncertain.

4. The Role of Loss Mixing ($\alpha$)

The optimal $\alpha$ is highly dependent on teacher strength:

• Weak Teachers: Require lower $\alpha$ (e.g., 0.2–0.4) to balance the unreliable teacher signal with ground-truth supervision. Pure distillation ($\alpha=1.0$) with weak teachers causes significant performance degradation.
• Strong Teachers: Favor higher $\alpha$ (up to 1.0), though even strong teachers show diminishing returns if overtrained.
• Non-Monotonicity: For very large teachers trained on massive token budgets, the optimal $\alpha$ can decrease, further supporting the finding that "stronger" is not always "better."

Significance and Claims

The paper claims to challenge the common belief that LLM pretraining via distillation strictly requires a strong teacher. Its significance lies in:

1. Redefining Teacher Selection: It demonstrates that in scenarios where frontier models are trained on previous-generation models (which may be smaller or less trained), distillation remains a viable and beneficial strategy.
2. Efficiency Implications: It suggests that investing in increasingly larger or more trained teachers may yield diminishing returns or even negative returns, encouraging a focus on teacher-student compatibility and loss mixing strategies instead.
3. Generalization Focus: It highlights that distillation is particularly effective for improving generalization (OOD and downstream tasks) rather than just in-domain fitting, offering a new perspective on the utility of KD in pretraining.

The authors conclude that the "strong-to-weak" framing is incomplete and that effective distillation depends on the compatibility of the teacher and student, the appropriate mixing of losses, and the specific generalization goals of the training process.