🤖 AI

Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents

This paper presents a comprehensive empirical study and a differential analysis framework demonstrating that seemingly relevant agent skills often induce functional failures and efficiency regressions by forcing unnecessary validation and implementation steps, thereby highlighting the need for safer and more cost-aware skill reuse mechanisms.

Original authors: Gen Dong, Yanjie Gao, Liqun Li, Tianyin Xu, Yu Hua, Fan Yang

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Gen Dong, Yanjie Gao, Liqun Li, Tianyin Xu, Yu Hua, Fan Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot assistant that can write code, solve math problems, or plan your weekend. To make this robot even better, you give it a "skill book"—a collection of reference sheets, recipes, and step-by-step guides for specific jobs. If the robot needs to build a website, it opens the "Web Dev" skill book; if it needs to bake a cake, it opens the "Baking" one. This is how modern AI agents work: they use these pre-written "skills" to guide their thinking and actions. It's like giving a student a textbook for every exam. But here's the twist: what if the textbook is actually wrong? What if the recipe tells you to add salt to a dessert, or the coding guide makes you build a bridge when you needed a boat? Sometimes, following the guide perfectly leads to disaster, or at least makes the robot waste a huge amount of time and energy. This is the puzzle researchers are trying to solve: when does a helpful guide become a harmful trap?

A new study titled "Agent Skills Can Be Harmful" dives deep into this problem. The researchers, a team from universities and Microsoft, set out to figure out exactly how and why these skill books cause AI agents to fail or become inefficient. They didn't just look at whether the robot succeeded or failed; they compared a robot using a specific skill against the same robot doing the exact same task without that skill (or with a different, similar skill). Think of it like a race where one runner wears a new pair of "magic" shoes and the other runs barefoot. If the runner with the shoes trips, was it the shoes, or were they just clumsy? By running these side-by-side "differential" tests, the team could pinpoint exactly which skill caused the trouble.

The study analyzed hundreds of these head-to-head races using two major testing grounds for AI skills. They found 307 specific instances where a skill caused a problem. These problems fell into two main buckets: Functional Failures (where the task simply didn't get done) and Efficiency Regressions (where the task got done, but the robot used way more energy and time than necessary).

Here is what they discovered, broken down into the story of the "Bad Guide":

1. The "Smart" Mistake (Functional Failures)
You might think the robot fails because it picks a skill that has nothing to do with the job—like using a "Car Repair" guide to fix a toaster. Surprisingly, the study found this is rare. Only 2 out of 125 failures happened because the skill was totally irrelevant.

Instead, the failures usually happened because the skill was too relevant. The guide seemed perfect, but it tricked the robot into doing the job the wrong way.

The "Wrong Recipe" (68.8% of failures): The most common issue was that the skill led the robot to implement a required step incorrectly or skip it entirely. For example, a skill might say, "Calculate the percentage," but the robot's guide accidentally told it to calculate the raw number instead. Or, a guide might show how to configure 3 settings but forget to mention the 4th one that was actually required. The robot followed the guide faithfully, but the guide was incomplete or misleading.
The "Wrong Address" (19.2% of failures): Sometimes the robot built the right thing but put it in the wrong place. If the task said "Save the file in the 'Kitchen' folder," but the skill guide said "Save files in the 'Living Room' folder," the robot would follow the guide and fail the test.
The "Wrong World" (10.4% of failures): Occasionally, the skill told the robot to change its environment in a way that broke the test. For instance, the skill might say, "Install this new version of a tool," but the test was designed to work with the old version. The robot followed the instruction, broke the setup, and failed.

2. The "Over-Engineered" Mistake (Efficiency Regressions)
In 182 cases, the robot actually finished the task, but it was incredibly wasteful. It used more than double the "tokens" (the digital currency AI uses to think) and took much longer than a robot without the skill.

The "Over-Check" (62.6% of failures): The biggest culprit here was "Excessive Procedure." The skill guides often turned simple checklists into mandatory, heavy-duty rituals. Instead of just checking if the code works, the guide might force the robot to run a full suite of 50 extra tests, or re-verify the same thing three times. It's like a student who, instead of just answering the math question, writes a 10-page essay on the history of numbers before writing the final answer.
The "Heavy Backpack" (25.3% of failures): Another major cause was "Context Bloat." The skill guide itself was just too long. Every time the robot thought, it had to read the entire massive guide again, which slowed it down and cost more "tokens." It's like trying to read a menu while someone is shouting a 50-page biography of the chef in your ear.

The Takeaway
The researchers built a tool called SKILLTRIAGE to help sort through these messes. Their main conclusion is a bit of a warning: just because a skill looks relevant doesn't mean it's safe. In fact, the most dangerous skills are the ones that look helpful but contain subtle errors, missing steps, or overly strict rules that force the AI to waste time or build the wrong thing.

The study suggests that for AI agents to be truly reliable, we need to stop treating these skill guides as perfect instructions. Instead, we need to treat them as suggestions that must be checked against the specific task, ensuring the robot doesn't blindly follow a recipe that tells it to put salt in the cake. The paper doesn't claim to have fixed the problem forever, but it provides a clear map of where the traps are hidden, helping developers build safer and smarter AI assistants in the future.

Technical Summary: Agent Skills Can Be Harmful

Problem Statement

While "skills" (instruction packages encoding task-specific procedures, examples, and constraints) are the de facto mechanism for extending Large Language Model (LLM) agents, their impact is inconsistent. Prior work indicates that while some skills improve success rates, others have no effect, increase token consumption and execution time, or even reduce success rates.

The core challenge addressed by this paper is the difficulty in isolating and attributing these negative outcomes specifically to the loaded skill. Existing evaluations often fail to distinguish skill-induced harm from base-agent limitations, task difficulty, or environmental artifacts. Furthermore, current benchmarks focus on measuring utility (pass rates) rather than uncovering the specific mechanisms of failure, leaving a gap in understanding how a specific skill document causes functional failures or hidden efficiency regressions.

Methodology

The authors propose a differential analysis framework inspired by differential testing to attribute failures to specific skills. The methodology involves constructing paired executions where the task, verifier, agent framework, model, and environment are held constant, while the skill setup varies.

Contrastive Setup:
- Target Run: An execution using the audited skill.
- Reference Run: An execution of the same task using either "no skill" or a "semantically matched" skill (retrieved from public repositories like smithery.ai and skillsmp.com based on semantic similarity).
- The reference run acts as a pseudo-oracle to demonstrate that the task is solvable or can be solved more cheaply under the same conditions without the specific audited skill.
Failure Definitions:
- Functional Failure: The target run fails the verifier while the reference run passes.
- Efficiency Regression: Both runs pass, but the target run exhibits substantially higher token use or execution time (defined as both metrics increasing relative to the reference, with at least one exceeding a threshold $T=2.0$ ).
Data Collection & Labeling:
- The study augments two existing benchmarks, SkillsBench (84 tasks) and SWE-Skills-Bench (490 tasks), with semantically matched public skills, expanding the comparison space from 826 to 20,664 potential pairs.
- Executions were performed using OpenCode 1.15.1 and Claude Opus 4.6.
- A total of 307 confirmed skill-induced failures were identified: 125 functional failures and 182 high-confidence efficiency regressions.
Tooling:
- The authors developed SKILLTRIAGE, a taxonomy-guided attribution tool that normalizes paired cases, extracts differential evidence, and produces triage reports to automate root-cause analysis.

Key Contributions

Contrastive Dataset: Construction of a dataset of 307 confirmed skill-induced failures, validated by no-skill or semantically matched-skill reference runs.
Empirical Study: The first comprehensive analysis categorizing and analyzing the root causes of functional failures and efficiency regressions in LLM agents.
Attribution Tool: Development and evaluation of SKILLTRIAGE, which successfully matches manually assigned root causes for 88.8% of functional failures and 72.5% of efficiency regressions.

Key Results and Findings

1. Root Causes of Functional Failures

The study found that functional failures are rarely caused by obviously irrelevant skills. Instead, Task-Implementation Faults (TIF) are the dominant cause, accounting for 68.8% (86/125) of failures.

Incorrect Required-Element Fill (36.8%): The agent implements a required element (e.g., an API call, calculation, or output format) incorrectly due to misleading guidance in a seemingly relevant skill.
Required-Element Omission (28.8%): The agent omits a required element because the skill's examples or templates did not include it, leading the agent to assume it was optional.
Artifact Misplacement (19.2%): The agent creates a plausible artifact but places it in the wrong directory or integration point, often following repository conventions rather than task specifications.
Environment Mismatch (10.4%): Skills induce runtime states (e.g., package versions, working directories) that conflict with the verifier's environment.
Applicability Mismatch (1.6%): Rare cases where the skill metadata incorrectly signals applicability.

Finding: Seemingly relevant skills often cause agents to misinterpret reusable guidance as mandatory task requirements, leading to incorrect implementations or omissions.

2. Root Causes of Efficiency Regressions

Efficiency regressions are not solely explained by prompt length. The dominant cause is Excessive Procedure (EP), accounting for 62.6% (114/182) of cases.

Excessive Verification (67 cases): Skills turn validation checklists into mandatory, redundant work, causing the agent to run unnecessary tests or loops.
Heavy Implementation Pipelines (30 cases): Skills prescribe overly complex workflows or construction recipes that increase token usage and time.
Context Bloat (25.3%): While less dominant than EP, this occurs when the skill body text itself adds significant overhead to every model call (43 of 46 cases in this category).

Finding: Skills often transform optional validation steps or complex recipes into mandatory procedures, driving up costs even when the task is successfully completed.

Significance and Claims

The paper claims to provide the first systematic, empirical evidence that skill reuse in LLM agents carries significant risks of functional failure and cost regression, even when skills are topically relevant.

Mechanism Identification: The study moves beyond aggregate metrics to identify specific failure mechanisms, such as "Task-Implementation Faults" and "Excessive Procedure," which were previously obscured in standard benchmark evaluations.
Practical Utility: The development of SKILLTRIAGE demonstrates that automated attribution of skill-induced failures is feasible, offering a pathway for screening skills in marketplaces and platforms.
Implications for Design: The findings suggest that skill authors must clearly distinguish between mandatory task requirements and optional examples/templates. Similarly, agent platforms should treat task-specified paths and environment states as guarded constraints to prevent skills from inducing mismatches.

The authors conclude that for safer and more cost-aware skill reuse, future research and tooling must address these specific root causes, particularly the tendency of skills to induce incorrect implementations and unnecessary procedural overhead.