💬 NLP

Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation

This paper demonstrates that, under a fixed compute budget, standard instruction fine-tuning generally outperforms or matches reasoning distillation across most model scales and tasks, with the latter only becoming advantageous for large models on open-ended problems when combined with a cost-effective curriculum mixing 25–50% reasoning data.

Original authors: Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad, Céline Hudelot, Pierre Colombo

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad, Céline Hudelot, Pierre Colombo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a young apprentice (a small AI model) how to solve problems. You have a limited amount of money (compute budget) to spend on their training. You have two main ways to teach them:

The "Direct Answer" Method (IFT): The master teacher gives the apprentice a question and the correct, concise answer. "What is 2+2? It is 4."
The "Show Your Work" Method (Reasoning Distillation): The master teacher gives the question, but also writes out a long, detailed diary of their thought process before giving the answer. "Okay, let's see. 2 plus 2... well, if I have two apples and add two more, I count them one by one... that makes four. So the answer is 4."

The big question this paper asks is: Is it worth spending all that extra money to teach the apprentice the long "Show Your Work" method, or should we just buy a bigger, smarter apprentice and teach them the "Direct Answer" method?

Here is what the researchers found, broken down into simple analogies:

1. The "Long Walk" Problem

The "Show Your Work" method is incredibly expensive. The paper found that reasoning traces are 5 to 20 times longer than direct answers.

The Analogy: Imagine you are paying a taxi driver by the mile.
- Direct Answer: The driver takes a shortcut and drops you off in 1 mile.
- Reasoning: The driver decides to drive through every neighborhood, explaining the history of every street, before dropping you off. It takes 20 miles.
- The Cost: If you have a fixed budget for gas, choosing the "Reasoning" taxi means you can only afford a tiny, slow car. If you choose the "Direct Answer" taxi, you can afford a massive, powerful limousine.

2. The Surprising Winner: The Big Limousine

When the researchers compared the two methods while keeping the "gas budget" (compute) exactly the same, the Direct Answer (IFT) method almost always won.

The Finding: It was almost always more efficient to train a larger model using short, direct answers than to train a smaller model using long, detailed reasoning.
The Exception: The "Show Your Work" method only became the winner in two specific situations:
1. The Task: The question was open-ended (like writing a story or solving a complex math problem where there isn't just one right answer).
2. The Size: The apprentice was already quite large (at least 7 billion "brain cells" or parameters).
The Small Models: For tiny models, the "Show Your Work" method was a disaster. They would get stuck in their own thoughts, writing long, rambling diaries that were full of mistakes. They spent all their money driving in circles, never actually getting the right answer.

3. The "Goldilocks" Solution: A Mixed Diet

The paper discovered a clever middle ground that saves money while keeping the benefits.

Sequential Training: You can teach the apprentice mostly with "Direct Answers" (75% of the time) and only switch to "Show Your Work" for the last 25% of their training.
- The Result: This captures almost all the benefits of the expensive reasoning method but costs a fraction of the price. It's like giving the apprentice a standard diet but adding a special "reasoning supplement" at the end to sharpen their skills.
Mixed Training: If you mix the two types of data together but keep the "reasoning" portion low (under 50%), the model gets smarter at math and logic without learning to write long, expensive answers.
- The Result: You get a smarter model that still speaks concisely. It's like teaching someone to think deeply but forcing them to give short, punchy answers.

4. Why Small Models Fail at Reasoning

The researchers looked closely at why small models struggle with the "Show Your Work" method.

The Discovery: When a small model gets an answer wrong, it tends to write a longer explanation than when it gets it right. It's like a student who doesn't know the answer and keeps writing more and more sentences, hoping to stumble upon the right one.
The Consequence: This makes small reasoning models incredibly inefficient. They burn through their "gas budget" writing long, incorrect stories. As the models get bigger, they get better at knowing when to stop talking, making the "Show Your Work" method viable only for the larger, more capable students.

The Bottom Line

If you are building an AI and have a limited budget:

Don't automatically assume "Reasoning" (long thinking traces) is better.
Do consider training a larger model with direct answers first.
If you really need reasoning, use a hybrid approach: train mostly on direct answers, then add a small amount of reasoning data at the end. This gives you the best of both worlds without breaking the bank.

The paper essentially argues that we shouldn't blindly chase the "long thinking" trend; sometimes, a bigger brain with a shorter answer is the smarter, more efficient choice.

Technical Summary: Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation

Problem Statement

The prevailing paradigm for building capable small language models (SLMs) involves distilling reasoning traces from strong teacher models. However, reasoning traces are typically 5–20× longer than standard Instruction Fine-Tuning (IFT) outputs. This length disparity implies a significant compute overhead: a practitioner choosing reasoning distillation implicitly forgoes the option of training a larger IFT model on the same compute budget. While standard scaling laws suggest that allocating additional compute to model capacity is highly efficient, the specific trade-off between "scaling up" (larger IFT models) and "scaling reasoning" (distilling traces) remains unaddressed. This paper investigates whether the compute overhead of reasoning distillation is justified compared to scaling standard IFT, given fixed training and inference budgets.

Methodology

To isolate the effect of supervision format from other variables (such as teacher quality or prompt distribution), the authors employ a controlled experimental design:

Controlled Pairing: A single teacher model (Qwen3-235B-A22B) generates paired outputs for identical prompts: standard IFT outputs (reasoning mode disabled) and reasoning outputs (reasoning mode enabled). This ensures that supervision format is the sole independent variable.
Student Scales: Qwen2.5 base models are trained at five parameter scales: 0.5B, 1.5B, 3B, 7B, and 14B.
Training Regimes:
- Corpora: Experiments utilize a general-purpose corpus (1.3M pairs from Infinity-Instruct) and a math-centric corpus (300K pairs).
- Hybrid Strategies: The study evaluates sequential training (IFT followed by reasoning) and mixed training (shuffled IFT and reasoning data) across reasoning proportions ( $\rho$ ) of 0%, 25%, 50%, 75%, and 100%.
Evaluation: Models are evaluated on 18 benchmarks spanning four task families: General-MC, General-OE, Math-MC, and Math-OE.
Compute Matching: The core analysis compares models not just by parameter count, but by matched training and inference FLOPs. The authors track the exact FLOP overhead caused by the increased sequence length of reasoning traces.
Robustness: Key findings are replicated using a second teacher-student pair (Nemotron-Super-49B with Gemma-3) to confirm results are not architecture-specific.

Key Findings

1. Scaling IFT is Compute-Efficient by Default

When comparing models at matched training and inference FLOPs, standard IFT models lie on or near the Pareto frontier across the majority of configurations. Reasoning distillation only approaches or reaches the Pareto frontier under specific conditions:

Task Format: Reasoning benefits are primarily observed in open-ended (OE) tasks. On multiple-choice (MC) tasks, reasoning models consistently fall behind IFT models in efficiency due to the high overhead of generating long, often incorrect, traces.
Model Capacity: Reasoning distillation becomes computationally viable only at larger scales. Models with 7B parameters or more are required for reasoning to capture the Pareto frontier. Below this threshold (e.g., 0.5B–3B), the inefficiency of small models generating long, failure-correlated traces makes reasoning significantly more expensive per unit of accuracy gained.

2. The Role of Output Format and Capacity

The paper identifies that output format, rather than knowledge domain, is the primary predictor of reasoning efficiency.

Format Dictates Value: Open-ended tasks consistently benefit from reasoning distillation, whereas multiple-choice tasks do not, regardless of the domain (general, math, law, code, or physics).
Capacity Threshold: For open-ended tasks, reasoning models at 7B+ parameters match or exceed IFT accuracy while maintaining optimal compute efficiency. Below 7B, the "length penalty" is severe; incorrect reasoning traces are 1.5–2× longer than correct ones, and smaller models frequently produce these long, incorrect traces, degrading the accuracy-to-cost ratio.

3. Hybrid Training Strategies

The study demonstrates that committing fully to 100% reasoning data is unnecessary and often suboptimal.

Sequential Training: A curriculum mixing just 25–50% reasoning data (following IFT) captures most of the accuracy benefits of full reasoning distillation at a fraction of the training cost. This strategy allows models to rest on the Pareto frontier for open-ended tasks.
Mixed Training: Training with a mix of IFT and reasoning data (below the 50% threshold) allows models to improve accuracy on mathematical tasks while preserving the concise output style of standard IFT. This avoids the inference cost penalty associated with full reasoning generation. However, once the reasoning proportion exceeds 50%, models tend to switch entirely to reasoning-style generation, erasing inference efficiency gains.

Significance and Claims

The paper provides a systematic answer to the compute allocation question in reasoning distillation. Its primary contribution is challenging the assumption that reasoning distillation is universally superior for small models.

Re-evaluating the "Reasoning" Trend: The authors argue that for the vast majority of configurations, standard IFT is the more compute-efficient strategy. Reasoning distillation is justified only when two conditions are met simultaneously: the task requires an open-ended output format, and the model has at least 7B parameters.
Cost-Aware Model Design: The work emphasizes that practitioners should weigh accuracy gains against computational and environmental costs. It suggests that simpler approaches (IFT) or carefully designed hybrid curricula (25–50% reasoning) can often match or approach the performance of expensive, full-reasoning training regimes.
Practical Guidance: The findings offer a practical framework for deployment:
- If training compute is the constraint, use sequential mixing with low reasoning ratios.
- If inference cost is the constraint, use mixed training below the 50% threshold to retain concise outputs.
- Full reasoning distillation is recommended only for large-scale (7B+) models tackling open-ended tasks where the efficiency frontier is reached.

The authors conclude that while reasoning models boost raw performance, the structural conditions under which they are more compute-efficient than standard IFT are narrower than commonly assumed, urging a shift toward more targeted model development aligned with practical constraints.