Technical Summary: Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning

1. Problem Statement

The paper addresses the limitations of the current prevailing paradigm for enhancing reasoning capabilities in small language models: Sequence-Level Distillation via Supervised Fine-Tuning (SFT) on teacher-generated responses. While recent works (e.g., DeepSeek-R1 distillation) have demonstrated that SFT on high-quality teacher outputs can yield strong reasoning performance, the authors argue that this approach is fundamentally grounded in an SFT perspective rather than the core principles of knowledge distillation.

The authors identify three critical limitations in existing practices:

1. Inadequate Representation of Teacher Distribution: Current methods often rely on random sampling with heuristic filtering, which fails to adequately cover the full support of the teacher's sequence-level output distribution. This leads to poor mode coverage or the inclusion of noisy, low-probability sequences that are difficult for smaller student models to learn.
2. Misalignment of Learning Capacity: Standard SFT increases the likelihood of ground-truth tokens but can generate misleading gradients. Specifically, if a student assigns high probability to a token that the teacher assigns low probability, SFT pushes the student further away from the teacher's distribution. Existing methods lack a mechanism to identify target distributions that better align with the student's learning capacity.
3. Exposure Bias: Models trained with teacher forcing (exposed to teacher prefixes) suffer from distributional shifts during autoregressive inference, where they must rely on their own predictions. This mismatch leads to error accumulation and compounding deviations, particularly in long Chain-of-Thought (CoT) reasoning tasks.

2. Methodology

The authors propose DASD (Distribution-Aligned Sequence Distillation), a training pipeline designed to explicitly enforce teacher-student interaction and address the three limitations above. The pipeline consists of three core innovations:

A. Temperature-Scheduled Learning

To address the trade-off between learning stability and distribution coverage, the authors propose a two-stage training schedule:

• Stage 1 (Low Temperature): The student is first trained on high-confidence, low-temperature samples (e.g., $T=0.6$). These samples exhibit consistent patterns and allow for rapid, stable convergence.
• Stage 2 (High Temperature): The training continues with high-temperature samples (e.g., $T=1.0$). While these samples are harder to learn due to higher diversity and noise, they cover a broader range of the teacher's output modes.
• Rationale: This "easy-to-hard" curriculum (or reverse temperature annealing) allows the student to first grasp consistent patterns before attempting to learn the richer, more diverse latent information present in the teacher's full distribution.

B. Divergence-Aware Sampling (DAS)

To identify a target sequence-level distribution that supports effective learning, the authors introduce a distribution decomposition framework. They analyze the probability discrepancies between the teacher ($p_T$), the pre-distillation student ($p_S$), and the post-distillation student ($p_D$) across sentence-level segments of responses.

• Classification: Responses are categorized into four types: Teacher Sentences ($p_T \gg p_S$), Student Sentences ($p_S \gg p_T$), Shared Sentences (similar probabilities), and Boosted Sentences (similar $p_T, p_S$ but higher $p_D$).
• Selection Strategy: Empirical analysis reveals that Teacher Sentences (where the teacher assigns high confidence and the student assigns low probability) correlate most strongly with improved test-set performance. These instances provide clear, non-misleading gradients.
• Implementation: The sampling strategy prioritizes examples rich in Teacher Sentences. Unlike logit distillation, this method only requires token-level probabilities from the teacher and student, making it compatible with models having different tokenizers and architectures.

C. Mixed-Policy Distillation

To mitigate exposure bias, the authors introduce a lightweight mixed-policy stage following the initial off-policy SFT:

• Process: The trained student model generates responses to a subset of training questions. For responses that diverge significantly from the teacher (e.g., truncated or incorrect), the student's generation is truncated at a random point, and the teacher model is prompted to complete the sequence.
• Training: Only the teacher's continuations that pass quality filters are retained for fine-tuning. This creates a hybrid dataset containing both teacher-forced prefixes and student-generated prefixes, allowing the model to learn to correct its own errors and reducing the distributional shift between training and inference.

3. Key Contributions

• DASD-4B-Thinking Model: A fully open-source, 4B-parameter reasoning model that achieves state-of-the-art (SOTA) performance among open-source models of comparable scale.
• Methodological Innovations: The proposal of a three-component pipeline (Temperature-Scheduled Learning, Divergence-Aware Sampling, and Mixed-Policy Distillation) that redefines sequence-level distillation by focusing on distribution alignment rather than simple SFT.
• Data Efficiency: The model achieves its performance using only 448K training samples, an order of magnitude fewer than most existing open-source efforts (which often use millions of samples).
• Cross-Architecture Validation: The pipeline is demonstrated to be effective across diverse model families (e.g., distilling from gpt-oss-120b to Qwen3-4B) and scales (validated on a 30B MoE variant, DASD-30B-A3B-Thinking-Preview).

4. Experimental Results

The model was evaluated on five challenging benchmarks: AIME24, AIME25, LiveCodeBench (v5/v6), and GPQA-Diamond.

• Mathematical Reasoning: DASD-4B-Thinking achieved 88.5 on AIME24 and 83.3 on AIME25. These scores surpass significantly larger models, including the 32B-scale Qwen3-32B (81.4/72.9) and GLM-Z1-32B (80.8/63.6), as well as the 8B DeepSeek-R1-0528-Qwen3-8B (86.0/76.3).
• Code Generation: On LiveCodeBench v5, the model scored 69.3, outperforming Qwen3-32B (65.7) and NVIDIA-OpenReasoning-Nemotron-7B (63.9).
• Scientific Reasoning: On GPQA-Diamond, it achieved 68.4, matching the performance of the 32B Qwen3-32B and approaching the 253B NVIDIA-Nemotron-Ultra-253B (76.0).
• Ablation Studies: The results confirm that each component contributes incrementally. Low-temperature training establishes a strong baseline, high-temperature training expands capability, and mixed-policy distillation provides final robustness against exposure bias.

5. Significance and Claims

The paper claims that DASD-4B-Thinking demonstrates that complex reasoning capabilities can be efficiently transferred from large teacher models to lightweight students through a refined distillation pipeline. The authors emphasize that their approach:

• Redefines Distillation: It moves beyond the "SFT perspective" to explicitly address the core principle of distillation—learning the teacher's full output distribution.
• Challenges Scaling Laws: It achieves performance comparable to or better than models 8 to 60 times larger, suggesting that data quality and distillation strategy are more critical than sheer parameter count for reasoning tasks.
• Promotes Reproducibility: By open-sourcing the model, the training dataset, and the methodology, the authors aim to foster community research into efficient, transparent reasoning models.

The authors conclude that their work provides a new perspective for developing compact, high-performing reasoning models, though they acknowledge future work is needed to further refine distribution-aware reweighting and integrate agentic capabilities.