How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data

1. Problem Statement

The paper addresses a critical challenge in the fine-tuning of Reasoning Large Language Models (LLMs) (e.g., Qwen3-8B, DeepSeek-R1). While using synthetic data generated by a stronger "Teacher" model for Supervised Fine-Tuning (SFT) is a standard strategy for enhancing smaller "Student" models, this approach often fails or degrades performance when applied to reasoning models.

• The Core Issue: Directly fine-tuning a reasoning student model on data generated entirely by a powerful teacher model leads to catastrophic forgetting and performance drops.
• Root Cause: The authors identify stylistic divergence as the primary culprit. Reasoning models have distinct internal "thinking" styles (e.g., specific transitional phrases, discourse markers like "Okay, let's see," or "Hmm") developed during their pre-training and initial post-training. When a student model is forced to learn from a teacher with a different style, it incurs a distributional conflict. The model struggles to adapt its inherent reasoning style to the teacher's, disrupting the learning of actual reasoning capabilities (the "capability tokens").
• Consequence: Experiments showed that fine-tuning Qwen3-8B on data generated by GPT-OSS-120B resulted in performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench.

2. Methodology: TESSY Framework

To resolve the conflict between preserving teacher-level reasoning capabilities and maintaining student-level stylistic consistency, the authors propose TESSY (Teacher–Student Cooperation Data Synthesis).

Core Concept

TESSY decomposes the output sequence into two types of tokens:

1. Capability Tokens ($\mathcal{T}_{Cap}$): Task-solving content (e.g., code, mathematical derivations, logic steps).
2. Style Tokens ($\mathcal{T}_{Sty}$): Discourse markers, transitional phrases, and tone-setting words (e.g., "Let's see," "Wait," "However").

The framework aims to synthesize data where Capability Tokens are generated by the Teacher (to ensure high-quality reasoning) and Style Tokens are generated by the Student (to ensure distributional consistency).

Key Components

1. Alternating Generation:

   • The framework generates responses in an iterative, alternating manner.
   • The Student generates the initial span (usually a style token like "Okay, let's see").
   • The Teacher then generates the next span (capability tokens like the core logic).
   • The Student resumes to generate the next style transition, and so on.
   • The Final Answer is always generated by the Student to maintain stylistic consistency in the conclusion.

2. Generate-Then-Rollback Strategy (Boundary Control):

   • A key challenge is determining exactly where a "style" segment ends and a "capability" segment begins.
   • Mechanism: The model generates a fixed block of $k$ tokens (e.g., 20 tokens).
   • Boundary Predictors: Two specialized, lightweight models (trained via token-level sequence labeling) analyze the generated block:
     • Style Predictor: Identifies the last token that belongs to the "style" category.
     • Capability Predictor: Identifies the last token that belongs to the "capability" category.
   • Rollback: The system truncates the generated block at the predicted boundary. If the block contained mixed types, the "wrong" type tokens are discarded (rolled back), and the generation role switches to the other model for the next span.

3. Training Data Construction:

   • The process creates synthetic sequences $y = [s_1, t_1, s_2, t_2, \dots]$, where $s$ spans are from the Student and $t$ spans are from the Teacher.
   • This ensures the final training data has the reasoning depth of the Teacher but the linguistic surface of the Student.

3. Key Contributions

• Identification of Stylistic Divergence: The paper provides empirical evidence that stylistic mismatch, not just capability gaps, is a primary cause of SFT failure in reasoning models.
• TESSY Framework: A novel data synthesis pipeline that interleaves teacher and student models to create "student-consistent" data enriched with "teacher-level" reasoning.
• Boundary Prediction Mechanism: A practical "generate-then-rollback" approach using lightweight predictors to enforce strict separation between style and capability tokens during synthesis.
• Open Resources: The authors released the TESSY-Code-80K dataset and the code repository, facilitating further research in reasoning model fine-tuning.

4. Experimental Results

The authors evaluated TESSY on code generation tasks using GPT-OSS-120B as the Teacher and Qwen3-8B as the Student.

• Performance Gains:
  • LiveCodeBench-Pro: TESSY improved Qwen3-8B by 11.25% (from 25.35% to 36.69%). In contrast, direct Teacher-only fine-tuning dropped performance to 22.10%.
  • OJBench: TESSY improved performance by 6.68% (from 18.75% to 25.43%). Teacher-only fine-tuning caused a massive 10.02% drop (down to 8.73%).
• Generalization:
  • TESSY showed consistent improvements across different student models (including the larger MoE model Qwen3-30B-A3B) and different teacher models (including DeepSeek-R1 and Qwen3-235B).
  • It also demonstrated robustness in out-of-domain tasks (Math and Science benchmarks like AIME and GPQA), whereas Teacher-only methods often degraded performance.
• Efficiency:
  • TESSY-generated responses were significantly shorter (fewer tokens) than pure Teacher-generated responses, as the student's tendency to terminate thinking earlier guided the teacher to be more concise.
• Ablation Studies:
  • Using the Teacher to generate the final answer (even with TESSY) caused performance drops, confirming the necessity of student-generated conclusions.
  • Fine-tuning a Base model (Qwen3-8B-Base) with TESSY was less effective than fine-tuning the Reasoning model (Qwen3-8B), suggesting that the pre-existing reasoning knowledge in the reasoning model is a crucial starting point.

5. Significance and Impact

• Paradigm Shift in SFT: The paper challenges the assumption that "stronger teacher data is always better." It argues that for reasoning models, distributional alignment (style) is as critical as capability alignment.
• Solving Catastrophic Forgetting: By preserving the student's stylistic distribution, TESSY mitigates the catastrophic forgetting often seen when forcing a reasoning model to adopt a foreign thinking style.
• Practical Utility: The framework offers a scalable, offline solution for enhancing reasoning models without requiring expensive online reinforcement learning or retraining of massive teacher models.
• Future Direction: It highlights the importance of "on-policy" data generation and suggests that future reasoning model training must carefully balance the injection of new knowledge with the preservation of the model's intrinsic reasoning "personality."

In summary, TESSY demonstrates that to successfully fine-tune a reasoning model, one must not just teach it what to think (via the teacher), but also allow it to express how it thinks (via the student), effectively bridging the gap between capability and style.