Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective

1. Problem Statement

Chain-of-Thought (CoT) distillation aims to transfer reasoning capabilities from a large "teacher" model to a smaller, more efficient "student" model. A prevailing hypothesis in recent literature (e.g., Li et al., 2025; Chen et al., 2025) suggests the existence of a "capacity gap": when the capability mismatch between the teacher and student is too large, distillation fails or yields suboptimal results. This implies that practitioners should avoid the strongest available teachers in favor of "matched" ones, necessitating a costly hyperparameter search.

The authors argue that existing evaluations of this phenomenon suffer from methodological pitfalls that obscure practical reality. Specifically, prior work often:

1. Fails to compare distilled performance against the student's pre-distillation baseline (ignoring that distillation often degrades performance).
2. Uses cross-teacher data filtering, restricting training data to examples where both teachers are correct, which artificially neutralizes the advantage of stronger teachers.
3. Includes impractical settings (e.g., student larger than teacher) that contradict the efficiency goals of distillation.

2. Methodology

The authors propose a revised, practical evaluation protocol to re-examine the capacity gap under realistic deployment conditions.

A. Proposed Evaluation Protocol

1. Task Selection: Instead of using arbitrary benchmarks, they select tasks where CoT distillation is likely to yield genuine improvements. They use the performance gap in Few-Shot In-Context Learning (ICL) between teacher and student groups as a proxy. Tasks with a large ICL gap (>30 percentage points) on BIG-Bench Hard (BBH) are selected, as these represent scenarios where the student lacks specific task knowledge and needs the teacher.
2. Removal of Cross-Teacher Filtering: Training data is not filtered to the intersection of correct answers. Instead, each teacher model generates CoT rationales for all examples it solves correctly. This allows stronger teachers to provide more training data (due to higher accuracy) and cover harder cases, reflecting real-world usage.
3. Efficiency-Motivated Settings: Experiments are restricted to cases where the student model is strictly smaller than the teacher.
4. Baseline Comparison: Crucially, all distilled models are compared against the pre-distillation baseline (the student model with few-shot prompting but no fine-tuning) to ensure distillation actually improves performance.

B. Experimental Setup

• Datasets: 15 tasks from BIG-Bench Hard (BBH) selected based on the ICL gap criterion.
• Models:
  • Teachers: Qwen2.5-14B/72B (Small/Large size comparison) and Qwen2.5-32B/ QwQ-32B (Short/Long CoT comparison). Cross-family validation uses Gemma-2-9B/27B.
  • Students: Qwen2.5 models ranging from 0.5B to 7B (and up to 32B in some settings).
• Training: Full fine-tuning for smaller students; LoRA for larger students.
• Metrics: Accuracy on BBH tasks.

3. Key Contributions

1. Identification of Evaluation Pitfalls: The paper exposes that prior conclusions about the capacity gap are often artifacts of flawed experimental designs, specifically the lack of baseline comparison and the use of cross-teacher filtering.
2. Empirical Re-evaluation: Under the new protocol, the authors demonstrate that the capacity gap does not consistently dominate across tasks. In many cases, distillation degrades performance compared to the baseline, regardless of the teacher.
3. Practical Guidelines: The study establishes that when distillation is effective, the benefits of using a stronger teacher (higher accuracy, more training data) generally outweigh the negative effects of the capacity gap, provided the performance gap between candidate teachers is substantial.

4. Key Results

A. Distillation Often Degrades Performance

In the initial re-evaluation of standard math benchmarks (MATH, GSM8K, etc.) using prior protocols, the authors found that CoT distillation frequently degraded performance compared to the pre-distillation baseline. This suggests that recent instruction-tuned LLMs already possess strong reasoning capabilities, and fine-tuning on teacher rationales can cause catastrophic forgetting of pre-trained knowledge.

B. The Capacity Gap is Not Universal

Under the practical protocol (using BBH tasks where distillation is beneficial):

• Variable Impact: Capacity gap effects (where smaller teachers yield better small students) were observed in some tasks (e.g., Temporal Sequences, Tracking Shuffled Objects) but were absent in others (e.g., Dyck Languages, Word Sorting).
• Dominance of Teacher Strength: In tasks where a substantial performance gap existed between candidate teachers (e.g., Dyck Languages, Word Sorting), the stronger teacher consistently produced better or comparable students, even for very small student models.
• Data Quantity vs. Quality: The advantage of stronger teachers is largely driven by increased data availability. Stronger teachers solve more examples correctly, providing a larger training set. An ablation study confirmed that when data sizes are equalized (via filtering), the advantage of the stronger teacher diminishes significantly.

C. Guidelines for Practitioners

The authors propose two actionable guidelines:

1. Verify Efficacy: Always compare the distilled model against the pre-distillation baseline. If distillation does not improve performance, it should not be used.
2. Prefer Stronger Teachers: If distillation is effective and candidate teachers differ substantially in performance, select the higher-performing teacher. The benefits of their superior reasoning and larger dataset outweigh the theoretical capacity gap.

5. Significance

• Methodological Rigor: The paper highlights the critical importance of evaluation design in AI research. It demonstrates that a widely cited phenomenon (the capacity gap) can be an artifact of specific experimental constraints (filtering, lack of baselines) rather than a fundamental limitation.
• Cost Reduction: By advising practitioners to simply use the strongest available teacher (once efficacy is verified) rather than searching for a "matched" teacher, the paper eliminates a costly and unnecessary hyperparameter search process.
• Realistic Deployment: The findings shift the focus from theoretical "matching" to practical data availability and teacher performance, offering a more robust framework for deploying efficient reasoning models in production.

In conclusion, the paper argues that the "capacity gap" is not a universal barrier to CoT distillation. Instead, the primary challenge is ensuring the task is suitable for distillation, and once that is established, the strongest available teacher is typically the optimal choice.