💻 computer science

Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills

This paper proposes amortizing the high token cost of reasoning models by distilling compact natural-language skills from a small corpus of existing trajectories into non-reasoning models, which achieves comparable or superior performance on agentic benchmarks while emitting significantly fewer tokens by trading deep, per-instance search for wide, one-time corpus distillation.

Original authors: Agamdeep Singh, Srishti Gautam, Priyanshu Gupta, Nikita Mehrotra, Tanmay Bakshi, Sumit Gulwani

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Agamdeep Singh, Srishti Gautam, Priyanshu Gupta, Nikita Mehrotra, Tanmay Bakshi, Sumit Gulwani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Token Tax and the Magic Reference Sheet

Imagine you are trying to teach a super-smart robot how to do chores, like organizing a messy room or solving a tricky math problem. In the world of artificial intelligence, there's a popular trick called "reasoning." It's like telling the robot to "think out loud" before it acts. Instead of just grabbing a sock, the robot pauses and says, "Okay, I need to find the sock, check if it's clean, and then put it in the hamper." This "thinking" part makes the robot much better at hard tasks, but it comes with a heavy price tag: it uses up a massive amount of digital energy (called "tokens") every single time it tries. It's like paying a toll every time you drive down a familiar road, even if you've driven that road a thousand times before. The robot keeps re-deriving the same simple rules over and over, wasting energy on things it should already know.

The big question researchers are asking is: Can we teach the robot these rules once, so it doesn't have to "think" about them every single time? If we could give the robot a permanent "reference sheet" of the best ways to do things, could it become just as smart as the "thinking" robot, but without the expensive energy bill? This is the heart of the new research from the COLM 2026 workshop, which explores a clever way to save energy without losing smarts.

The Paper's Big Idea: "Reason Wide, Not Deep"

The researchers behind this study, working with Microsoft, noticed something funny about how these "reasoning" robots work. When they tackle a series of similar tasks—like helping a customer in a store or organizing a spreadsheet—they spend a huge chunk of their time re-inventing the wheel. They might reason through, "I shouldn't call the customer's account tool until they give me an email," over and over again, even though that rule never changes. It's like a student who, every time they take a math test, re-discovers that $2+2=4$ from scratch instead of just remembering it.

The team proposed a solution they call Passive Skill Distillation. Instead of making the robot "think" deeply every single time (which is like doing a deep search in a library for every single book), they decided to do a "wide search" across many past attempts to find the patterns.

Here is how they did it:

The Collection: They gathered a small pile of past attempts (about 35 to 50 tasks) where a robot tried to solve problems. These could be attempts where the robot was "thinking" or just "acting" without thinking.
The Analyst: They handed this pile of data to a different, very smart coding robot (an "agent"). They asked this analyst to look through the logs, find where the robots failed, and figure out why.
The Reference Sheet: The analyst robot wrote a short, simple set of rules in plain English (about 40 to 130 lines of text). For example, it might write: "Before you try to find a user's ID, make sure the email address is actually in the message, or you'll get an error."
The Injection: They took this short "skill" and pasted it into the system instructions of the non-thinking robot. Now, the robot doesn't need to "think" to remember the rule; it just reads the reference sheet and acts.

What They Found: Smarter, Faster, and Cheaper

The results were surprisingly good. When they tested these "reference sheet" robots on four different challenging benchmarks (like organizing a virtual house, fixing spreadsheets, and handling customer service calls), the robots with the skills performed almost as well as, or sometimes even better than, the expensive "thinking" robots.

The Score: On some tests, the non-thinking robot with the skill actually beat the thinking robot. For example, in a task called ALFWorld (organizing a virtual house), the skill-enhanced robot scored 0.787, while the thinking robot only scored 0.713.
The Savings: The thinking robots were using 3 to 6 times more output tokens than the skill-enhanced robots. In some cases, the skill-enhanced robot used 2.7 to 6 times fewer tokens.
The Cost: Creating this reference sheet was incredibly cheap. It cost only $1.28 to $2.44 per domain to generate the skill, whereas the thinking robots pay that "tax" every single time they run a task.

The Twist: You Don't Need the "Thinking" Logs

One of the most interesting findings was that the researchers didn't actually need the "thinking" logs to make a good reference sheet. They tried making skills using only the logs from robots that didn't think (the ones that just acted and often failed). Surprisingly, these skills were just as good as the ones made from the "thinking" logs.

In fact, for one specific test (SpreadsheetBench), the skill made from the "non-thinking" failures was 10 points better than the one made from the thinking logs. The researchers suggest this is because the "thinking" logs are full of the robot's internal monologue about what it thought was true, while the "non-thinking" logs show exactly what actually went wrong in the real world. It's like learning to drive: reading a driver's manual (the thinking log) is helpful, but watching a video of a car hitting a curb (the failure log) teaches you exactly what not to do.

Where the Reference Sheet Doesn't Work

The researchers were careful to note that this isn't a magic bullet for everything. The reference sheet works best for rules that are always the same, like "always check the email before calling the tool." However, on tasks that require unique, step-by-step logic for every single problem—like a complex spreadsheet where every cell depends on a different variable, or a customer service call with a very specific, weird history—the reference sheet couldn't help as much. In those cases, the "deep search" of thinking is still necessary.

The Bottom Line

This paper suggests a new way to look at artificial intelligence. Instead of forcing every robot to "think" deeply for every single task, we can teach them the common patterns once and for all. By doing a "wide search" across many past mistakes to find the rules, we can give robots a permanent skill set that makes them faster, cheaper, and sometimes even smarter than their "thinking" counterparts. It turns out that for many tasks, you don't need to re-invent the wheel every time; you just need a good map.

Technical Summary: Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills

1. Problem Statement

Large language models (LLMs) equipped with test-time reasoning modes (e.g., Chain-of-Thought) consistently outperform non-reasoning counterparts on complex, multi-step agentic tasks. However, this performance gain comes with a significant "reasoning premium": reasoning modes generate 3–6× more output tokens per episode. This cost is incurred repeatedly for every single task instance because the model re-derives procedural logic from scratch each time.

Analysis of reasoning traces reveals that much of this deliberation is not instance-specific problem-solving but rather the re-derivation of episode-invariant procedures (e.g., "do not call the account-lookup tool until an email is provided" or "heat X is an atomic command"). Non-reasoning models often fail precisely where this procedural knowledge is missing, leading to recurring errors. The paper posits that this recurring cost can be amortized: instead of paying the reasoning premium on every episode, the procedural knowledge can be extracted once, offline, and injected into the non-reasoning model's system prompt.

2. Methodology: Passive Skill Distillation

The authors propose a simple, "passive" distillation pipeline that converts a small corpus of existing trajectories into a compact, natural-language skill. The process involves three steps:

Corpus Collection: A small dataset of trajectories (35–50 tasks) is collected from a training split. This can be done using either non-reasoning rollouts alone or a paired corpus containing both reasoning and non-reasoning trajectories. Crucially, no new rollouts are generated specifically for the distillation process; the method utilizes logs that production systems already store.
Distillation via Coding Agent: An off-the-shelf coding agent (specifically Claude Code with Claude Sonnet 5) is tasked with analyzing the corpus. The agent executes code to compute corpus-level statistics, including:
- Failure-mode frequencies.
- Action n-grams and loop detection.
- Win/loss contrasts between successful and failing trajectories.
- (In paired conditions) Contrasts between non-reasoning failures and reasoning successes on the same tasks.
The agent compiles these findings into a compact natural-language skill (40–130 lines of markdown). These rules are concrete and traceable to specific transcript evidence (e.g., "Before calling find_user_id_by_email, check that the customer's message actually contains a real email address... this bug appeared in 13 of 22 rollouts").
Deployment: The distilled skill is appended verbatim to the system prompt of the non-reasoning model. No model weights are updated, and no per-instance search is performed at deployment. The skill acts as a cacheable prefix, adding negligible marginal cost.

3. Key Contributions

A Low-Cost Distillation Pipeline: A method requiring only existing rollouts and approximately $1–$ 3 of coding-agent time per domain to generate effective skills.
Amortization of the Reasoning Premium: Demonstration that distilled skills can recover 55%–100%+ of the performance gap between non-reasoning and reasoning models on held-out tasks, often exceeding the reasoning mode outright while emitting 2.7–6× fewer output tokens and zero reasoning tokens.
Source Independence: Evidence that reasoning traces are not a prerequisite for effective distillation. Skills distilled from non-reasoning trajectories alone are competitive with, and in some cases superior to, those distilled from paired reasoning/non-reasoning corpora.
Cost-Efficiency vs. Prompt Optimization: A comparison showing that distilled skills outperform state-of-the-art reflective prompt optimizers (GEPA) on specific benchmarks while costing significantly less to produce (4.1× lower cost).

4. Experimental Results

The method was evaluated on four agentic benchmarks using GPT-5.4-mini and Qwen3.6-27B: ALFWorld, SpreadsheetBench-Verified (SSB-Verified), τ2-bench Telecom, and τ2-bench Retail.

Performance Recovery: For GPT-5.4-mini, skills recovered the reasoning gap across all benchmarks. Notably, on ALFWorld and τ2-Retail, the non-reasoning model with a distilled skill outperformed the reasoning mode (e.g., 0.787 vs. 0.713 on ALFWorld).
Token Efficiency: The skill-enhanced non-reasoning models emitted 2.7–6× fewer output tokens than the reasoning mode. Specifically, GPT-5.4-mini showed a reduction of 2.9–4.5×, while Qwen3.6-27B achieved reductions up to 6×. On ALFWorld and SSB, the skill even reduced token counts compared to the plain non-reasoning baseline by preventing "flailing" retries and loops.
Model Agnosticism: Similar trends were observed with Qwen3.6-27B, where skills helped on three of four benchmarks, reaching near-ceiling performance on ALFWorld (0.980) and matching the reasoning mode on Telecom (0.933) with 5.9× fewer tokens.
Distillation Source Ablation:
- ALFWorld & Telecom: Skills from non-reasoning-only and paired corpora were statistically close.
- Retail: Paired corpora (including reasoning traces) produced slightly better skills.
- SSB-Verified: Non-reasoning-only skills significantly outperformed paired skills (0.560 vs. 0.460), suggesting that verbose reasoning narratives can sometimes anchor the distiller on model beliefs rather than ground-truth failure evidence.
Comparison with GEPA: On τ2 domains, distilled skills achieved higher pass rates than GEPA-optimized prompts (e.g., 45.8% vs. 39.2% on Retail) at a fraction of the production cost ($3.72 vs. $15.28, a 4.1× reduction).

5. Significance and Interpretation

The paper interprets these results through a search lens:

Test-time reasoning is characterized as deep search within a single episode. It re-discovers procedural knowledge from scratch for every instance, incurring a recurring cost.
Corpus distillation is characterized as wide search across many episodes. It extracts regularities in the failure distribution once, allowing the knowledge to be reused for free.

The authors argue that for procedural, domain-level knowledge (e.g., atomic commands, authentication workflows), "width over cheap trajectories" is the superior economic choice. The residual performance gaps observed in domains like Telecom and SpreadsheetBench are attributed to tasks requiring instance-specific deep search (e.g., long dependency chains, unique spreadsheet logic) that cannot be captured by a fixed prompt. In these cases, reasoning and distillation act as complements: the skill handles the invariants, while reasoning handles the unique instances.

The work suggests that the "reasoning premium" is often paying for the re-derivation of latent capabilities that can be surfaced more efficiently via prompt engineering based on empirical failure analysis, rather than through expensive per-instance computation.