🤖 AI

Base Models Know How to Reason, Thinking Models Learn When

This paper reveals that while base models already possess the underlying mechanisms for reasoning, "thinking" models trained with Reinforcement Learning primarily learn the heuristics to orchestrate these pre-existing capabilities, whereas those trained via SFT-distillation acquire entirely new reasoning mechanisms.

Original authors: Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, Neel Nanda

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, Neel Nanda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question

Imagine you have a very smart, well-read student (the Base Model) who knows a lot of facts and can do basic math. Then, you train this student to become a "Thinking Model" (like the new AI models that take extra time to solve hard problems).

The big mystery this paper tries to solve is: What exactly did the student learn during that training?

Did they learn new ways of thinking they didn't have before?
Or did they just learn when to use the smart thinking skills they already had?

The Detective Work: "Constructive Model Diffing"

To answer this, the researchers built a special "detective tool" called Constructive Model Diffing. Think of it like taking apart a complex machine to see how it works, then trying to rebuild it using only two specific parts:

The "How" (Reasoning Mechanisms): These are the actual skills, like "doing long division," "checking your work," or "backtracking when you make a mistake." The researchers found these by looking at the Base Model's brain and identifying specific "switches" (vectors) that can force it to do these things.
The "When" (Reasoning Heuristics): This is the decision-making part. It's a little internal manager that says, "Okay, the problem is hard, now is the time to check your work," or "We are stuck, now is the time to backtrack."

The researchers tried to rebuild the "Thinking Model" by taking the Base Model (who has the skills) and giving it the Thinking Model's Manager (who knows when to use the skills).

The Two Types of Training

The paper looked at two different ways these Thinking Models were trained:

1. The "Reinforcement Learning" (RL) Group

These models were trained by playing a game where they got points for getting the right answer and penalties for wrong ones. They had to figure out the strategy themselves.

The Result: When the researchers rebuilt these models using the "Base Model + Manager" recipe, it worked amazingly well. They recovered about 76% of the performance gap.
The Analogy: Imagine a talented pianist who already knows how to play every note perfectly. The RL training was like hiring a conductor who taught them when to play fast, when to slow down, and when to pause for effect. The pianist didn't need to learn new notes; they just needed to learn the timing.
Conclusion: RL teaches the model how to orchestrate the reasoning skills it already possesses.

2. The "SFT-Distillation" Group

These models were trained by copying a "teacher" model. They were shown examples of how a smart AI solves a problem and told to mimic it.

The Result: When the researchers tried to rebuild these models using the same "Base Model + Manager" recipe, it failed. They only recovered about 11% of the performance gap.
The Analogy: Imagine a student who is taught to copy a teacher's handwriting. But the teacher is using a pen the student has never held before. The student learns to copy the shape of the letters, but they haven't actually learned how to hold the pen or the muscle memory required to write. The "Manager" (the timing) is there, but the "Base Model" (the student's hand) doesn't actually know how to perform the specific moves the teacher is doing.
Conclusion: SFT-distillation forces the model to learn new reasoning mechanisms (new ways of thinking) that the base model didn't have. You can't just tell the base model when to do them; you have to teach it how to do them first.

The "Aha!" Moment

The paper found a striking difference:

RL Models are like a master chef who already knows how to chop, sauté, and bake. The training just taught them when to use which technique to make a perfect dish.
Distilled Models are like a chef who was shown a video of a master chef making a dish using a secret spice the chef didn't know. The student learned to copy the actions of using the spice, but they had to fundamentally change their cooking style to include this new ingredient.

Why Does This Matter?

The paper concludes that Reinforcement Learning (RL) is a very efficient way to get reasoning out of a model because it unlocks what is already there. It teaches the model to be a better conductor of its own orchestra.

In contrast, Distillation (copying a teacher) often tries to teach the model entirely new skills. If the teacher uses a reasoning style the base model doesn't naturally have, the base model struggles to learn it just by watching, leading to a much lower success rate when trying to "steer" it back to its original state.

Summary in One Sentence

Base models already know how to reason; Reinforcement Learning teaches them when to use those skills, while Distillation tries to teach them new skills that the base model didn't know it could do.

Technical Summary: Base Models Know How to Reason, Thinking Models Learn When

Problem Statement

Large Language Models (LLMs) have demonstrated significant improvements in reasoning capabilities when trained to "think" (e.g., via Chain-of-Thought) before answering. However, a fundamental question remains: What exactly do these "thinking models" learn that their base counterparts lack? Prior hypotheses suggest they may acquire entirely new capabilities, learn to structure reasoning more effectively, repurpose existing representations, or simply benefit from increased inference-time computation. This paper seeks to empirically determine the nature of the knowledge gap between base models and their "thinking" fine-tuned versions, specifically comparing models trained via Reinforcement Learning (RL) versus those trained via Supervised Fine-Tuning (SFT) distillation.

Methodology

The authors introduce a two-part methodology to analyze and reconstruct the differences between base and thinking models:

1. Unsupervised Discovery of Reasoning Taxonomies

To avoid subjective manual inspection, the authors employ Sparse Autoencoders (SAEs) trained on sentence-level activations extracted from reasoning traces.

Architecture: They use Top-K SAEs with a deliberately small dictionary size (5–50 categories) and low sparsity ( $k=3$ ) to force the model to learn fundamental axes of reasoning variation rather than incidental linguistic details.
Training: SAEs are trained on sentence-level averaged activations from reasoning rollouts generated by thinking models on MMLU-Pro prompts.
Interpretation: An LLM-based interpretability pipeline generates human-readable titles and descriptions for each cluster (reasoning mechanism) based on top-activating exemplar sentences.
Evaluation: Taxonomies are scored on Consistency (F1 score of classification), Completeness (confidence in assignment), and Independence (semantic orthogonality between categories).

2. Constructive Model Diffing

The core contribution is a framework to explicitly reconstruct the performance gap between a base model and a thinking model using two interpretable components:

Reasoning Mechanisms (The "What"): Category vectors (steering vectors) optimized within the base model to induce specific reasoning behaviors identified by the SAE taxonomy. These represent the cognitive operations (e.g., backtracking, arithmetic).
Reasoning Heuristics (The "When"): A classifier extracted from the thinking model that determines when to activate a specific mechanism. This is derived from SAE activations and disagreement points between the base and thinking model's token predictions.

The Hybrid Model: The authors construct a "hybrid" model that combines the base model's generation capabilities with the thinking model's heuristics and the base model's optimized category vectors. If this hybrid recovers the thinking model's performance, it implies the thinking model primarily learned heuristics for orchestrating pre-existing base capabilities. If it fails to recover performance, it suggests the thinking model learned new mechanisms not present in the base model.

Key Contributions

Unsupervised Reasoning Taxonomy: A method to discover interpretable, human-understandable reasoning mechanisms (e.g., "Backtracking," "Arithmetic Computation," "Metacognitive Self-Prompts") without manual labeling, validated across nine different model configurations.
Constructive Model Diffing Framework: A novel approach to decompose the base-to-fine-tuned difference into "mechanisms" (vectors) and "heuristics" (timing classifiers), allowing for a causal reconstruction of the fine-tuned model's behavior.
Empirical Distinction Between Training Paradigms: The application of this framework reveals a fundamental divergence in what RL-trained and SFT-distilled models learn.

Results

The study evaluated nine base/thinking model pairs (four RL-trained, four SFT-distilled, one mixed) across GSM8K, MATH500, and a held-out Hendrycks-MATH set.

RL-Trained Models (Open-Reasoner-Zero): The hybrid models constructed from RL-trained pairs recovered approximately 76% of the base-to-thinking performance gap on average. In some cases (e.g., ORZ-0.5B), recovery exceeded 100% of the gap.
SFT-Distilled Models (DeepSeek-R1-Distill): The hybrid models recovered significantly less, averaging only 11% of the gap. Recovery was near zero for smaller models and improved to roughly 15–30% for larger models (14B/32B), but remained far below the RL performance.
Mixed Models (QwQ-32B): Models trained with SFT followed by RL recovered roughly 20% of the gap, behaving more similarly to the SFT-distilled models than the pure RL models.
Category Vector Convergence: Category vectors trained on RL pairs converged to significantly lower cross-entropy loss than those trained on SFT pairs, indicating that RL models utilize reasoning behaviors that already exist (or are easily accessible) in the base model's activation space.
Steering Sparsity: The successful RL hybrids required steering only 5–12% of tokens, whereas SFT hybrids often required higher intervention rates, yet still failed to recover performance.

Significance and Claims

The paper claims that the difference in performance recovery stems from the nature of the training paradigm:

RL Teaches Heuristics: Reinforcement Learning primarily teaches the model when to deploy reasoning mechanisms that the base model already possesses. The base model "knows how to reason," and RL optimizes the orchestration of these pre-existing capabilities. This is why the constructive diff (base mechanisms + thinking heuristics) successfully reconstructs RL model behavior.
SFT Distills New Mechanisms: Supervised Fine-Tuning (distillation) often forces the student model to adopt reasoning behaviors or mechanisms that the base model does not naturally possess or cannot easily access. Consequently, simply steering the base model with the thinking model's heuristics is insufficient to replicate the SFT model's performance, as the underlying "mechanisms" (vectors) are fundamentally different or missing.
Scale Dependence: The paper notes that larger base models (e.g., 32B) recover more of the SFT gap than smaller ones, suggesting that larger models inherently contain a broader repertoire of latent reasoning mechanisms that SFT can activate, whereas smaller models lack these mechanisms entirely.

Limitations: The authors acknowledge that their taxonomies are currently biased toward mathematical reasoning due to the training data (MMLU-Pro) and that the lower recovery for SFT models could potentially reflect limitations in their category vector optimization rather than a fundamental lack of mechanisms in the base model.

Conclusion: The paper provides a new lens on training paradigms, suggesting that efficient reasoning model development may benefit from RL-based approaches that orchestrate latent base capabilities, rather than SFT approaches that attempt to install entirely new reasoning patterns.