🤖 AI

The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents

This paper reveals that adding procedural skills to LLM agents often causes significant performance regressions due to description osmosis, grounding displacement, and verification displacement, arguing that reliability depends more on improving grounding and verification than on optimizing procedural guidance.

Original authors: Darshan Tank, Baran Nama

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Darshan Tank, Baran Nama

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot assistant how to do your homework. You give it a set of "skills"—little cheat sheets or rulebooks that tell it how to solve specific problems, like "how to find a number in a spreadsheet" or "how to read a financial report." In the world of Artificial Intelligence, these are called LLM Agents (Large Language Model agents), and the goal is to make them better at real-world jobs. For a long time, scientists measured success simply by counting how many more tasks the robot got right after you gave it these skills. It was a "more is better" game. But just like giving a human a new, complicated instruction manual can sometimes confuse them and make them drop the ball on things they used to know, giving an AI too many skills can backfire. The big question researchers are asking is: When we add these skills, are we actually helping, or are we accidentally breaking things that were already working?

This paper, titled "The Regression Tax," dives deep into that exact problem. The authors, Darshan Tank and Baran Nama, ran a massive experiment with nearly 6,000 different attempts at solving office tasks (like reading financial documents or fixing spreadsheets) using three different AI models. They compared how the robots did with their new skills versus how they did with no skills at all.

Here is the twist they found: Adding skills doesn't just add wins; it also steals wins.

Think of it like a video game. You find a new "power-up" (a skill) that helps you beat a boss you were struggling with. Great! But maybe that same power-up makes you trip over a rock you used to jump over easily. The authors call this the "Regression Tax." They discovered that for every 100 new tasks the robots solved because of the skills, they actually broke about 59 tasks that they would have solved perfectly fine on their own. So, the "net" improvement is much smaller than it looks if you only count the new wins.

The paper identifies three sneaky ways these skills cause the robots to mess up:

Skill-Description Osmosis (The "Ghost" Effect): Imagine you have a rulebook sitting on your desk. Even if you never open it or read the instructions, just seeing the title "How to Calculate Taxes" might make you nervous and change how you do your math. The authors found that simply having the description of a skill in the robot's memory (even if the robot never actually uses the skill) can change its behavior and cause it to fail. It's like the robot is "osmosing" the vibe of the skill and getting confused just by its presence.
Grounding Displacement (The "Wrong Map" Problem): This happens when the skill tells the robot how to do something, but in doing so, it makes the robot look at the wrong part of the problem. It's like giving a driver a perfect set of driving instructions, but the instructions make them look at the wrong exit sign. The robot follows the procedure perfectly but ends up in the wrong place because the skill "displaced" its ability to find the right starting point.
Verification Displacement (The "No-Check" Zone): This is when the skill tells the robot to skip a safety check. Imagine a robot that usually double-checks its work before handing it in. A new skill might say, "Just do it fast!" and the robot forgets to check its answer. The math might be right, but because it didn't verify the result, it fails.

The authors also found that the current way we build these skills is a bit off-kilter. Most skill creators focus on writing the "middle" part—the step-by-step procedure. But the data shows that the real trouble spots are at the beginning (finding the right information, or "grounding") and the end (checking the answer, or "verification"). The skills are great at telling the robot what to do, but they often make the robot worse at knowing where to look or if it's right.

In short, the paper suggests that we shouldn't just count how many tasks an AI gets better at. We need to look at the "tax" it pays by breaking things it used to know. To make AI truly reliable, we need to stop just writing more procedures and start helping the robots get better at reading the map and checking their work. The authors suggest that if we fix these specific "grounding" and "verification" issues, we could recover a huge number of the tasks that are currently failing. It's not about having more skills; it's about having the right kind of help that doesn't trip the robot up.

Technical Summary: The Regression Tax: Decomposing Why Skills Help — and Hurt — LLM Agents

Problem Statement

Current evaluation of Large Language Model (LLM) agent skills relies primarily on average improvement in task success rates. This aggregate metric is incomplete because it obscures a critical cost: the addition of procedural skills can cause regressions, where tasks previously solved by a "bare" agent (without skills) are subsequently failed after skills are introduced.

Existing methods for managing skills focus on detecting and removing harmful skills only when they are retrieved or invoked. However, these methods fail to account for presence-only effects, where a skill's description in the system prompt alters agent behavior even if the skill body is never executed. Furthermore, current literature often treats skill-induced failures as diagnostic evidence rather than investigating the specific mechanisms (e.g., grounding errors vs. procedural errors) that cause them.

Methodology

The authors conducted a controlled, large-scale empirical study to decompose the net effect of adding skills into gains (tasks solved only with skills) and regressions (tasks failed only with skills).

Experimental Setup

Scale: 5,832 task-condition runs across nearly 6,000 trials.
Benchmarks: Two office-automation tasks:
1. OfficeQA-Pro: Complex question answering over U.S. Treasury financial documents (PDFs), requiring multi-step reasoning and table extraction.
2. SpreadsheetBench: Spreadsheet manipulation tasks on real Excel forum problems, involving formulas, filtering, and cell updates.
Model-Harness Stacks: Three distinct combinations were tested to ensure robustness:
1. OpenCode + MiniMax-M2.7
2. Codex + GPT-5.4-mini
3. Claude Code + Sonnet 4.6
Skill Libraries: For each stack, three distinct libraries were generated from the same set of failure signals using different creation pipelines:
1. Anthropic: Measurement-driven, iterative refinement.
2. OpenAI: Single-pass structural generation.
3. Ours (Sentient): Harness-agnostic, self-critique with reuse.
Conditions: Each task was run under four conditions: no skills (baseline), and the three distinct skill libraries.
Metrics: The study utilized the McNemar test on discordant pairs to determine statistical significance, explicitly tracking the counts of Gains, Regressions, Residual Failures (failed in both), and Retained tasks.

Key Contributions

1. Decomposition of Net Effect

The paper introduces a paired decomposition of pass-rate changes. Instead of reporting a single aggregate improvement, the authors report the specific counts of tasks gained versus tasks broken.

Finding: Across all conditions, 324 regression transitions offset 59% of the 553 gross gain transitions.
Implication: The "best-performing" skills often succeed not by solving more new tasks, but by regressing fewer existing ones. Libraries with identical net gains can have vastly different reliability profiles based on their regression counts.

2. Identification of Three Regression Mechanisms

Through trajectory analysis, the authors identified three distinct mechanisms by which skills cause regressions:

Skill-Description Osmosis: A skill changes agent behavior simply by being present in the context (system prompt), even when the skill body is never invoked. This effect is invisible to retrieval-based suppression methods.
Grounding Displacement: The skill's prescribed procedure overrides the agent's correct interpretation of inputs (e.g., selecting the wrong table, date, or entity). This is the dominant cause of regression in OfficeQA-Pro.
Verification Displacement: The skill's procedure suppresses or replaces checks the agent would otherwise perform on its outputs. This is a critical failure mode in SpreadsheetBench, where correct formulas are generated but fail due to missing validation logic.

3. Re-evaluation of Residual Failures

The study analyzed tasks that failed both with and without skills ("residual failures").

Finding: These failures are concentrated at the grounding (input) and verification (output) stages, not the procedural (method) stage.
Correction: By re-grading SpreadsheetBench failures using a full spreadsheet engine (rather than a value-only grader), the authors recovered 226 tasks where the agent had generated correct formulas that were previously marked as failures. This highlights that existing skills over-emphasize procedural guidance while under-supporting grounding and verification.

Results

Quantitative Findings

Regression Tax: In the OfficeQA-Pro benchmark, regressions cancelled out 66% of gross gains; in SpreadsheetBench, they cancelled 56%.
Statistical Significance: After applying a Bonferroni correction for multiple comparisons, only the results from the Claude Code · Sonnet 4.6 stack on SpreadsheetBench remained statistically significant ( $p < 0.001$ ). This suggests that many apparent improvements in other configurations may be indistinguishable from noise once regressions are accounted for.
Osmosis Impact: On the OpenCode · MiniMax stack, where skill bodies were invoked in only 5–14% of tasks, 84–93% of gains were invocation-free, demonstrating that the presence of descriptions alone drives significant performance changes (both positive and negative).

Mechanism Distribution

OfficeQA-Pro: Regressions were dominated by Grounding Displacement (72.8%), followed by Osmosis (17.3%).
SpreadsheetBench: Regressions were more mixed, with Osmosis (28.8%) and "Body Engaged" errors (18.9%) being prominent, alongside a significant number of grader artifacts.

Significance and Claims

The paper argues that the field must shift from evaluating skills based on aggregate improvement to evaluating them based on decomposed net effects (gains minus regressions).

Reliability over Raw Gain: The primary differentiator between skill libraries is not how many new tasks they solve, but how few existing tasks they break.
Beyond Invocation: Evaluation methodologies must separate the effects of skill description presence from skill body invocation. Current selection-based methods fail to measure the "osmosis" effect.
Design Shift: Existing skills over-emphasize the "method" stage (procedural guidance), which is rarely the source of failure. The authors claim that reliability depends more on improving grounding (input interpretation) and verification (output checking) than on refining procedural steps.
Evaluation Protocol: Future evaluations should report paired gain/regression counts and test the "presence-only" channel via controlled ablations (e.g., descriptions only vs. descriptions + bodies).

The authors conclude that while skills can improve agent performance, they impose a "regression tax" that is currently hidden by aggregate metrics. Addressing this requires a focus on grounding and verification rather than purely procedural optimization.