💬 NLP

Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

This paper systematically evaluates Knowledge Distillation in the post-training of large language models, revealing that while its advantage over supervised fine-tuning diminishes with abundant data, it remains highly effective when distilling from stronger teachers or when employing a two-stage strategy combining synthetic and human-annotated data for low-resource scenarios.

Original authors: Xin Liu, Simin Ma, Shujian Liu, Song Wang, Sathish Reddy Indurthi, Haoyun Deng, Lu Wang, Kaiqiang Song

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xin Liu, Simin Ma, Shujian Liu, Song Wang, Sathish Reddy Indurthi, Haoyun Deng, Lu Wang, Kaiqiang Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Small Student from a Big Teacher

Imagine you have a brilliant, overworked professor (the Teacher) and a bright but small student (the Student). The professor knows everything, but they are too big and slow to carry around in a backpack (like on a phone or a small laptop). The student is small and fast, but they don't know as much yet.

Knowledge Distillation (KD) is the process of the professor teaching the student. The goal is to make the student smart enough to do the professor's job, but without needing the professor's massive brain.

This paper asks a simple question: When does this teaching method actually work, and when does it fail?

1. The "Textbook" Experiment: When Data is Scarce vs. Abundant

The researchers tested this by giving the student different amounts of homework (training data).

The Scenario: They used a massive library of questions and answers (called the Tulu 3 dataset).
The Finding:
- Low Data (The "Cramming" Phase): When the student only had a few pages of homework (small dataset), the teaching method (KD) was a huge success. The student learned much faster and did better than if they just tried to study the textbook alone. It's like having a tutor explain the logic behind the answers, which helps when you don't have enough practice problems.
- High Data (The "Marathon" Phase): When the student had the entire library of homework, the advantage of the tutor disappeared. If the student has enough books to read on their own, they can figure out the answers just by studying the textbook (Supervised Fine-Tuning). The tutor didn't add much extra value because the student could already learn everything from the books.

The Analogy: If you are learning to cook with only one recipe, a chef showing you their secret techniques helps a lot. But if you have a library of 10,000 cookbooks, you can probably figure out the techniques yourself just by reading them all.

2. The "Super-Tutor" Twist

The researchers realized that in the "High Data" scenario, the teacher and the student were both studying from the exact same books. Of course, the student didn't need help; they had the same source material.

So, they tried a new experiment: They brought in a Super-Tutor.

This new teacher had studied a much larger, more diverse set of books (trained with Reinforcement Learning from Human Feedback).
The Result: Even when the student had a huge library of homework, the Super-Tutor still helped! The student learned things they couldn't have found in their own books.

The Takeaway: Knowledge Distillation only stops working if the teacher and student are reading the same limited material. If the teacher knows things the student can't find in the available data, the teaching method works great, no matter how much data you have.

3. The "Two-Stage" Strategy for Specialized Jobs

The paper also looked at real-world jobs where data is very hard to find, like translating a rare language or summarizing medical notes. In these cases, you might only have a handful of examples.

They proposed a clever Two-Stage Strategy:

Stage 1: The "Fake" Homework (Synthetic Data).
The teacher generates new, made-up practice problems based on the few real examples they have. The student practices on these "fake" problems first. This is like a coach simulating game scenarios so the player gets used to the style of play before seeing a real opponent.
Stage 2: The "Real" Homework (Human Data).
After the student has warmed up on the fake data, they are refined using the small amount of real, high-quality human-annotated data.

The Result: This combination consistently made the small student perform better than just using the real data alone. It's like warming up your muscles before a race; it helps you perform better when the real race starts.

Summary of Findings

Small Data = Big Help: Knowledge Distillation is a superpower when you don't have much data. It helps small models learn big lessons quickly.
Big Data = Diminishing Returns: If you have tons of data, the student can learn on their own, so the teacher doesn't add much value unless the teacher is significantly smarter and has different knowledge.
The "Super-Teacher" Rule: To get the most out of distillation, the teacher needs to know things the student cannot learn from the data alone.
The Two-Step Hack: For specialized tasks with very little data, use the teacher to generate practice problems first, then fine-tune on the real data. This builds a stronger student.

In short: Teaching a small model works best when the teacher has secrets the student can't find in the library, or when the student doesn't have enough books to read on their own.

Technical Summary: Understanding Knowledge Distillation in Post-Training

Problem Statement

Large Language Models (LLMs) achieve state-of-the-art performance but face significant deployment barriers in resource-constrained environments due to high computational and memory demands. While Knowledge Distillation (KD) offers a practical solution by transferring knowledge from large teacher models to smaller student models, existing research has primarily focused on task-specific or small-scale settings. There is a lack of systematic understanding regarding KD in the post-training stage for building general instruction-following models. Specifically, it remains unclear how KD scales with training data size, whether it offers advantages over standard Supervised Fine-Tuning (SFT) in large-data regimes, and how to effectively apply KD in domain-specific, low-resource scenarios where high-quality labeled data is scarce.

Methodology

The authors conduct a systematic study using the large-scale Tulu 3 dataset (939k instruction-response pairs). The experimental framework involves:

Training Paradigms:
- Teacher: A large model (Llama3.1-70B) fine-tuned via SFT on a specific dataset subset.
- Student: Smaller models (Llama3.1-8B, Llama3.2-1B, Llama3.2-3B) trained either via direct SFT or via KD from the teacher.
- Initialization: The study compares initializing the student from a raw pre-trained checkpoint (Base-S) versus an SFT-adapted checkpoint (SFT-S).
Distillation Objective:
- The paper adopts Generalized Knowledge Distillation (GKD) (Agarwal et al., 2024), which addresses train-inference mismatch by incorporating on-policy (student-generated) sequences.
- The objective combines supervised loss on ground-truth data and distillation loss on student-generated data, weighted by a hyperparameter $\lambda$ .
- The divergence measure uses Generalized Jensen-Shannon Divergence (JSD) with $\beta=0.5$ to balance mode-covering and mode-seeking behaviors.
Scaling and Teacher Strength Experiments:
- Data Scaling: Experiments vary training set sizes from 10k to the full 939k samples to observe KD performance relative to SFT.
- Teacher Strength: To test if KD benefits persist in large-data regimes, the authors replace the standard teacher with a stronger, instruction-tuned model (Llama3.3-70B-Instruct) trained via Reinforcement Learning from Human Feedback (RLHF) on a broader corpus.
Domain-Specific and Low-Resource Strategy:
- The study evaluates KD on low-resource tasks: Assamese-English translation, Dialogue Summarization, and ARC-Challenge.
- A Two-Stage KD Strategy is proposed:
  1. Synthetic Data Generation: Using a teacher model to generate pseudo-labeled data from unlabeled inputs (in-context learning).
  2. Refinement: Training the student on a mixture of synthetic teacher-labeled data followed by refinement on a small set of high-quality human annotations.

Key Results

Data Scaling: In low-data regimes (e.g., <80k samples), KD consistently outperforms SFT, with gains up to 5% absolute accuracy at 10k samples. However, as data size increases, the performance gap narrows significantly. In large-data settings, KD offers little additional gain over SFT when the teacher and student are trained on the same dataset, suggesting the student can recover most knowledge through direct supervision.
Teacher Strength: When distilling from a stronger, instruction-tuned teacher (Llama3.3-70B-Instruct) that possesses knowledge beyond the training data, KD restores substantial performance gains (approx. 4% average improvement) even in large-data settings. This indicates KD remains effective when the teacher provides novel patterns or reasoning strategies not present in the student's training data.
Initialization: Initializing the student with an SFT-adapted model (SFT-S) consistently outperforms initializing from a raw base model (Base-S), particularly in low-data regimes, suggesting that prior instruction adaptation provides a stronger foundation for distillation.
Model Size: The benefits of KD are most pronounced for smaller models (1B and 3B parameters). As student model size increases (e.g., 8B), the marginal gain from KD diminishes, as larger models can learn more directly from the data.
Domain Adaptation: In low-resource, domain-specific tasks, KD consistently improves performance over SFT. The proposed two-stage strategy (synthetic data warm-up + human annotation refinement) yields consistent improvements across translation, summarization, and reasoning tasks.

Contributions

Systematic Scaling Analysis: The paper identifies specific data regimes where KD is most effective compared to SFT, demonstrating that KD's advantage saturates in large-data settings unless the teacher possesses superior knowledge.
Teacher Strength Investigation: The study reveals that the saturation of KD in large-data settings is due to data overlap. It shows that distilling from a stronger, instruction-tuned teacher breaks this limitation, maintaining KD's efficacy even with abundant data.
Synthetic Data Strategy: The authors propose and validate a two-stage KD framework that leverages synthetic teacher-labeled data followed by human annotation refinement. This approach effectively addresses the scarcity of high-quality labeled data in domain-specific applications.

Significance

The paper provides practical guidance for building compact, efficient LLMs. It establishes that while KD is a powerful tool for low-data scenarios, its utility in large-data general instruction tuning depends heavily on the strength and diversity of the teacher model relative to the training data. Furthermore, the proposed two-stage strategy offers a viable pathway for deploying high-performance small models in specialized domains where human annotation is expensive or unavailable. The findings suggest that for data-scarce environments, carefully designed KD pipelines utilizing synthetic data and stronger teachers are essential for maximizing student model performance.