Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10

This systematic study on CIFAR-10 demonstrates that student capacity is a critical moderator of knowledge distillation effectiveness, revealing that larger students (R34) benefit significantly more than smaller ones (R18), while also highlighting the necessity of fixing implementation bugs and input-resolution mismatches to achieve optimal distillation performance.

The Gist

Imagine you are trying to teach a young apprentice (the Student) how to be a master chef. You have a famous, highly skilled chef (the Teacher) who knows everything about cooking. The goal of this research is to figure out the best way for the apprentice to learn from the master so they can cook great meals without needing the master's entire kitchen or years of experience.

In the world of Artificial Intelligence, this process is called Knowledge Distillation. The paper investigates three main things: how big the student is, how the teacher teaches, and whether the kitchen itself is set up correctly.

Here is what the study found, explained simply:

1. The Size of the Student Matters Most

The researchers tried teaching three different "sizes" of students using the same masters.

• The Tiny Apprentice (ResNet-18): This student is small and has a limited brain. Even when the teacher was very smart, this tiny student struggled to learn much new information.
• The Medium Apprentice (ResNet-34): This student is bigger and has more capacity. Even when the gap between the teacher's skill and the student's skill was the same as the tiny student's, the medium student learned much more.

The Analogy: Imagine trying to teach a toddler (Tiny Student) and a teenager (Medium Student) how to solve a complex puzzle. Even if the teacher explains it perfectly to both, the teenager will understand and retain the logic much better simply because they have a bigger "mental workspace." The study found that a bigger student can absorb more of the teacher's "secret knowledge" (called dark knowledge), regardless of how much better the teacher is than the student.

2. The "Bug" in the Teaching Method

There are two main ways to teach the student:

• Logit-KD (The Final Answer): The teacher shows the student the final probabilities of what the answer might be (e.g., "80% chance it's a cat, 20% dog").
• Feature-KD (The Middle Steps): The teacher shows the student how they think about the image in the middle of the process (e.g., "Look at these edges and shapes first").

The Discovery: The researchers found that in many previous studies, the "Middle Steps" method (Feature-KD) seemed to fail or perform worse than the "Final Answer" method. They discovered this wasn't because the method was bad, but because of a glitch in the code.

The Analogy: Imagine the teacher is trying to guide the student's hand while they draw. In the old, buggy version, the teacher was accidentally holding the student's hand too loosely, letting it shake wildly. The student couldn't learn the technique. Once the researchers fixed the "hand-holding" (a technical fix called gradient clipping), the "Middle Steps" method suddenly became just as good, and sometimes even better, than the "Final Answer" method.

3. Fixing the Kitchen Before Teaching

Before they even started teaching, the researchers noticed the "kitchen" (the computer architecture) was set up for a giant banquet hall (high-resolution images like 224x224), but they were trying to cook on a tiny counter (small images like 32x32).

The Discovery: The standard setup was crushing the small images, making them unrecognizable before the teacher even started. When they fixed the kitchen setup to fit the small counter, the teacher's own performance jumped by a massive 5 percentage points.

The Analogy: It's like trying to teach someone to drive a car, but the steering wheel is broken and the brakes are stuck. No matter how good the driving instructor is, the student can't learn. Fixing the car (the architecture) improved the results ten times more than any fancy teaching technique could.

Summary of the Findings

1. Bigger Students Learn Better: A medium-sized student learns significantly more from a teacher than a tiny student, even if the teacher is equally "smart" relative to both.
2. Don't Blame the Method: The "Middle Steps" teaching method works great, but only if the code is written correctly. A small coding bug had been hiding its success.
3. Fix the Basics First: Before trying advanced teaching tricks, you must ensure the computer model is built correctly for the size of the images it is processing. If the foundation is wrong, no amount of teaching will help.

The paper concludes that to get the best results, you need a student with enough brainpower to learn, a bug-free teaching method, and a correctly built computer model.

Technical Summary

Technical Summary: Student Capacity Moderates Knowledge Distillation Effectiveness

Problem Statement

Knowledge Distillation (KD) is a widely used strategy for compressing deep neural networks by training a smaller "student" model to mimic the soft output distributions or intermediate features of a larger "teacher" model. Despite its prevalence, the relative effectiveness of different KD paradigms (Logit-based vs. Feature-based) remains context-dependent. A critical, underexplored question is whether a stronger teacher always yields a better student, and specifically, how the capacity relationship between teacher and student modulates the effectiveness of distillation. Prior work suggests that excessive capacity mismatch can hinder transfer, but systematic evidence across multiple teacher-student pairs and KD strategies on controlled benchmarks has been limited. Furthermore, discrepancies in existing literature regarding the performance of Feature-KD versus Logit-KD may stem from implementation artifacts rather than fundamental algorithmic limitations.

Methodology

The authors conducted a systematic ablation study on the CIFAR-10 dataset (32×32 images, 10 classes) using ResNet-based architectures. The study focused on three specific teacher-student capacity configurations:

1. R50→R18: A large Bottleneck-based teacher (23.5M params) to a smaller BasicBlock student (11.2M params).
2. R34→R18: A medium BasicBlock teacher (21.8M params) to the same BasicBlock student (11.2M params).
3. R50→R34: The large Bottleneck teacher (23.5M params) to a larger BasicBlock student (21.8M params).

Experimental Controls and Corrections:

• Architecture: The authors corrected the standard ResNet stem for 32×32 inputs. They replaced the standard 7×7 convolution (stride 2) and MaxPool with a 3×3 convolution (stride 1) and Identity mapping. This modification preserves spatial resolution, which is critical for CIFAR-10, and was applied consistently to all models.
• Implementation Rigor: The study identified and corrected a critical bug in Feature-KD implementations: the exclusion of projection layer parameters from gradient clipping. This omission caused optimization instability (unclipped gradients up to 4.65) that suppressed Feature-KD performance.
• Protocol: Experiments were run with three random seeds (0, 1, 2) to report mean ± standard deviation. Hyperparameters for Logit-KD ($\alpha \in \{0.3, 0.5, 0.7\}$, $T \in \{2, 3, 4\}$) and Feature-KD ($\alpha \in \{0.3, 0.5, 0.7\}$, $\beta=0.5$) were systematically ablated.
• Loss Functions: The study compared Logit-KD (minimizing KL divergence between temperature-scaled distributions) and Feature-KD (aligning intermediate feature maps via MSE and cosine similarity after 1×1 projection).

Key Contributions

1. Student Capacity as a Moderating Factor: The study provides evidence that student capacity is a primary determinant of KD gain. R34 students consistently benefited more from distillation than R18 students, even when the teacher-student accuracy gaps were comparable.
2. Implementation Correctness in Feature-KD: The authors demonstrated that a specific gradient clipping bug (excluding projection layers) artificially suppressed Feature-KD performance, leading to misleading comparisons where Logit-KD appeared superior. Correcting this bug revealed that Feature-KD is competitive with or superior to Logit-KD in specific capacity configurations.
3. Architectural Prerequisites: The study highlights that input-resolution-aware architecture is a prerequisite for effective distillation. Correcting the ResNet stem for 32×32 inputs increased teacher accuracy by over 5 percentage points (pp), an effect an order of magnitude larger than any KD gain.
4. Systematic Ablation: The paper offers a reproducible benchmark comparing Logit-KD and Feature-KD across three distinct capacity pairs under controlled conditions, isolating the effects of capacity gaps from implementation noise.

Results

• Capacity Modulation:
  • R50→R34: Feature-KD achieved the highest gain of +0.30 pp (95.55% vs. 95.25% baseline), outperforming Logit-KD (+0.21 pp).
  • R34→R18: Feature-KD yielded a gain of +0.18 pp, while Logit-KD showed 0.00 pp improvement.
  • R50→R18: Logit-KD outperformed Feature-KD (+0.21 pp vs. +0.08 pp). The authors attribute the lower Feature-KD performance here to the R18 student's limited capacity rather than a flaw in feature-based distillation.
• Impact of Implementation Bugs: In the R50→R18 pair, the "bugged" Feature-KD (no projection clipping) showed a misleading gain of +0.26 pp (single seed). After correction and averaging over three seeds, the gain dropped to +0.08 pp, revealing the true performance gap relative to Logit-KD.
• Architectural Impact: The stem correction raised the ResNet-50 teacher accuracy from a lower baseline to 95.81% and the ResNet-34 teacher to 95.70%, demonstrating that architectural alignment with input resolution is more impactful than the distillation process itself.

Significance and Claims

The paper concludes that student capacity is a key moderating factor in KD effectiveness. A larger student (R34) appears capable of extracting more "dark knowledge" from a teacher than a smaller student (R18), regardless of the raw accuracy gap between them. This suggests that the magnitude of the teacher-student gap alone is an insufficient predictor of distillation success.

The authors emphasize that implementation correctness is critical, particularly for Feature-KD, where additional trainable components (projection layers) require careful handling (e.g., gradient clipping) to avoid optimization instability. The study argues that previous reports of Feature-KD underperformance may have been artifacts of such bugs rather than fundamental limitations of the approach.

Finally, the paper asserts that architectural correctness precedes distillation. Without proper adaptation of the network stem to the input resolution (32×32), KD experiments yield misleading results, as the baseline performance is severely compromised.

Limitations: The authors note that these findings are specific to CIFAR-10 and a limited set of ResNet pairs. While the results are directional and suggestive, stronger causal claims regarding student capacity effects would require replication across larger datasets (e.g., ImageNet) and more diverse architectures. The study uses three seeds, which is standard for pre-prints but falls short of the five-seed protocols increasingly expected for formal statistical significance.