💻 computer science

On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning

This paper reveals that supervised fine-tuning on long Chain-of-Thought trajectories from DeepSeek-R1 yields lower training loss but worse generalization than on gpt-oss-120b data due to inefficient, divergent reasoning patterns, and proposes filtering out frequently branching trajectories to significantly improve reasoning performance.

Original authors: Zhaoyi Li, Xiangyu Xi, Zhengyu Chen, Wei Wang, Gangwei Jiang, Ranran Shen, Linqi Song, Ying Wei, Defu Lian

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhaoyi Li, Xiangyu Xi, Zhengyu Chen, Wei Wang, Gangwei Jiang, Ranran Shen, Linqi Song, Ying Wei, Defu Lian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Perfect Student" Paradox

Imagine you are trying to teach a new student (an AI model) how to solve incredibly difficult math problems. You have two famous teachers: Teacher A (DeepSeek-R1) and Teacher B (gpt-oss).

Both teachers are geniuses. They both solve the exact same set of 500,000 math problems correctly. You record their step-by-step thinking processes (called "Chain-of-Thought") and use them to train your new student.

The Shocking Discovery:
You would expect that if Teacher A's notes are easier to read or if the student learns them faster (lower "training loss"), the student would become a better problem-solver.

Teacher A's notes were actually easier for the student to memorize. The student could recite them perfectly with very little effort.
Teacher B's notes were harder to memorize. The student struggled more during practice.

The Twist:
When you put the students to the test on new problems they've never seen before:

The student who studied Teacher A (the "easy" notes) failed miserably.
The student who studied Teacher B (the "hard" notes) became a math wizard.

Why? Because Teacher A was a "chatterbox" who went down every possible rabbit hole, while Teacher B was a "laser-focused" detective who went straight to the answer.

The Core Problem: "Exploring the Forest" vs. "Walking the Path"

The paper digs into how these two teachers think. They use two very different "reasoning patterns."

1. Teacher A (DeepSeek-R1): The "Branching Explorer"

Imagine Teacher A is walking through a dense forest. When they see a path, they don't just walk it. They stop, think, "What if I go left?" Then they go left, realize it's a dead end, come back, and think, "What if I go right?" Then they go right, realize that's a dead end too.

The Style: They constantly branch out. They propose many ideas, check them, discard them, and propose new ones.
The Result: Their notes are full of "What ifs," "Perhaps," and "Let's try this other way."
The Trap: The student learns to mimic this behavior. When the student faces a new problem, they get stuck in the forest, wasting time exploring 100 dead-end paths before finally finding the right one (or getting lost entirely). They inherit the habit of inefficient exploration.

2. Teacher B (gpt-oss): The "Deductive Detective"

Imagine Teacher B is walking through the same forest. They look at the map, identify the straight path to the exit, and walk it. They don't stop to check every bush. If they hit a wall, they turn around and take the one logical alternative path immediately.

The Style: They deduce. They move forward linearly, connecting one fact to the next like a chain.
The Result: Their notes are direct: "Because X is true, Y must be true. Therefore, Z."
The Benefit: The student learns to focus. They learn to build a strong, straight line of logic without getting distracted by every shiny object in the forest.

The "Training Loss" Illusion

Why did the student learn Teacher A's notes so easily?

Think of Training Loss as a measure of "how bored the student is."

Teacher A's notes are full of simple, repetitive calculations and obvious "What if" statements. These are easy to predict. The student gets bored quickly because the answers are obvious, so their "loss" (error rate) drops to near zero.
Teacher B's notes are dense with deep logical jumps. The student has to really think to understand the connection between steps. The "loss" stays higher because the student is working harder.

The Lesson: A low "boredom score" (low loss) doesn't mean the student is learning the right skills. It just means they are good at memorizing the chatter.

The Solution: Pruning the Tree

The researchers realized that Teacher A's "exploration" was actually structural redundancy. It was like a tree with too many dead branches.

They asked: What if we cut off the dead branches before teaching the student?

They took Teacher A's massive dataset and filtered out the problems where the teacher went on the most wild, branching tangents. They kept only the problems where the teacher stayed focused and logical.

The Result:
When they retrained the student using this "pruned" version of Teacher A's notes:

The student's performance jumped up significantly.
On tough math competitions (like AIME), the student improved by 5% to 5.5%.
The student stopped getting lost in the forest and started walking the straight path.

Summary Analogy

Imagine you are learning to drive.

Teacher A is an instructor who says, "Okay, let's try turning left. Oh, that's a dead end. Let's try right. No, that's a wall. Let's try going backward. Let's try driving on the sidewalk just to see what happens." You learn all the wrong moves, but you memorize the instructor's voice perfectly.
Teacher B is an instructor who says, "Turn the wheel left, check the mirror, accelerate." Simple, direct, effective.

The paper proves that memorizing the "chatter" of a genius (Teacher A) makes you a worse driver than learning the "direct logic" of a slightly less flashy genius (Teacher B). By filtering out the "chatter" (the branching paths), you can turn a good teacher into a great one.

1. Problem Statement

The paper addresses a critical paradox in the training of Large Reasoning Models (LRMs). While Supervised Fine-Tuning (SFT) on long Chain-of-Thought (CoT) trajectories is a standard method for distilling reasoning capabilities, the choice of the "teacher" model generating these trajectories significantly impacts the "student" model's generalization.

The authors identify a specific discrepancy:

Observation: When student models are trained on reasoning trajectories generated by DeepSeek-R1-0528, they achieve significantly lower training loss (indicating a better fit to the training distribution) compared to models trained on trajectories from gpt-oss-120b.
Paradox: Despite the lower training loss, models trained on DeepSeek-R1 data exhibit significantly worse generalization performance on downstream mathematical reasoning benchmarks (e.g., AIME, MATH) compared to those trained on gpt-oss-120b data.
Core Question: What intrinsic properties of the CoT trajectories (specifically reasoning patterns) cause this divergence between low training loss and poor generalization, given that both datasets contain verified, correct answers to identical problems?

2. Methodology

The authors conducted a controlled comparative study using two open-source LRMs (DeepSeek-R1-0528 and gpt-oss-120b) to generate CoT trajectories for ~500k identical mathematical problems. They employed a multi-faceted analysis approach:

A. Controlled Experimental Setup

Data: Verified correct trajectories from both teachers for the same ~500k prompts.
Student Models: Four base models of varying scales and families (Qwen2.5-7B/32B, Llama3.1-8B, Qwen3-8B).
Training: SFT performed with identical hyperparameters and training steps.
Evaluation: Tested on five benchmarks: MATH500, AIME24, AIME25, BeyondAIME, and HMMT25.

B. Multi-Faceted Analysis

Token-Level Loss Deconstruction:
- Analyzed the distribution of token-level loss.
- Identified that DeepSeek-R1 data has a "dilution effect": a massive proportion of near-zero-loss tokens (routine calculations) drags down the average loss, masking the fact that key logical transition tokens have similar loss values to gpt-oss data.
- High-loss tokens in DeepSeek-R1 were semantically linked to divergent exploration (e.g., "Perhaps," "Another"), whereas gpt-oss high-loss tokens were linked to convergent deduction (e.g., "Thus," "Yields").
Reasoning Behavior Analysis (Structural Level):
- Used DeepSeek-V3.2 to annotate reasoning steps into four discrete behaviors: Propose (branching/new idea), Deduce (linear inference), Verify, and Backtrack.
- Constructed Markov transition matrices to analyze step-to-step transitions.
- Finding: DeepSeek-R1 exhibits a high frequency of Propose steps (33.3% vs. 22.5%) and frequent Propose $\to$ Propose transitions (0.53 vs. 0.34), indicating a pattern of shallow, repetitive branching without deep deduction. In contrast, gpt-oss favors Deduce steps (74.6% vs. 62.0%), forming dense, convergent chains.
Intervention Experiments:
- Random Step Deletion: Randomly removed 10-30% of reasoning steps from training trajectories.
  - Result: Models trained on pruned gpt-oss data suffered severe performance drops (breaking the logical chain). Models trained on pruned DeepSeek-R1 data showed minimal degradation or even performance gains, confirming the presence of redundant, non-essential branches in the R1 data.
- Data Filtering: Proposed and tested a strategy to filter out trajectories with high branching frequencies.

3. Key Contributions

Discovery of the Generalization Discrepancy: The paper empirically demonstrates that lower SFT training loss does not guarantee better generalization in Long CoT SFT. It reveals that DeepSeek-R1 trajectories induce lower loss but worse downstream performance compared to gpt-oss-120b trajectories.
Identification of Reasoning Patterns as the Root Cause: The authors attribute this discrepancy to structural differences in reasoning patterns:
- DeepSeek-R1: Divergent, branch-heavy exploration with shallow deduction (structural redundancy).
- gpt-oss-120b: Convergent, deep deductive chains.
- Crucially, these patterns are transferred to student models during SFT, causing them to get "stuck" in redundant exploratory loops.
Proposed Remediation Strategy: The paper introduces a simple, effective data curation method: filtering out frequently branching trajectories from the teacher dataset.
- They propose two proxy metrics for filtering without full annotation: (1) Difference in step count compared to a reference model, and (2) Proportion of steps containing branching keywords (e.g., "Perhaps").

4. Key Results

Performance Gap: Models trained on gpt-oss-120b data consistently outperformed those trained on DeepSeek-R1 data across all base models.
- Example: On Llama3.1-8B, gpt-oss achieved 50.5% average accuracy vs. 29.5% for DeepSeek-R1, despite the latter having much lower training loss.
Filtering Efficacy: Removing the top 10-20% of most frequently branching trajectories from the DeepSeek-R1 dataset significantly improved generalization:
- AIME25: +5.1% improvement (Qwen3-8B).
- BeyondAIME: +5.5% improvement.
- Average across 5 benchmarks: +3.6% improvement.
Validation of Redundancy: The "Random Deletion" experiment confirmed that DeepSeek-R1 trajectories contain significant structural redundancy, as removing steps did not harm (and sometimes helped) performance, whereas it severely hurt gpt-oss-trained models.

5. Significance

Redefining Data Quality: The paper challenges the assumption that "correctness" and "low training loss" are the primary metrics for selecting SFT data. It argues that the structural efficiency and reasoning pattern of the data are more critical for generalization.
Mechanism of Failure: It provides a mechanistic explanation for why some SFT data fails: student models overfit to inefficient, redundant exploration patterns (shallow branching) rather than learning robust deductive logic.
Practical Guidance: The proposed filtering strategy offers a low-cost, high-impact recipe for improving Long CoT SFT. By curating datasets to remove "noisy" exploratory branches, practitioners can significantly boost the reasoning capabilities of distilled models without needing more powerful teacher models or complex RL training.
Broader Implication: This work suggests that for reasoning tasks, convergent, deep thinking is more valuable for generalization than divergent, trial-and-error exploration, even if the latter appears more "human-like" or comprehensive in the training data.