💬 NLP

Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?

The paper introduces Skill-Use, a benchmark evaluating whether LLM agents can independently recognize and correctly apply structured skills in isolated environments, revealing that reliable skill use remains a significant challenge heavily dependent on the specific agent harness rather than being an inherent model capability.

Original authors: Jinyi Han, Yuanjian Xu, Ying Liao, Xinyi Wang, Zishang Jiang, Zixiang Di, Fanyang Lu, Zhichao Hu, Yanghua Xiao

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinyi Han, Yuanjian Xu, Ying Liao, Xinyi Wang, Zishang Jiang, Zixiang Di, Fanyang Lu, Zhichao Hu, Yanghua Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a super-smart, hyper-creative intern who can write code, analyze data, and draft emails. You give them a massive library of "Skill Books." Each book is a recipe for a specific job, like "How to fix a broken server" or "How to write a legal contract." The catch? You don't hand them the whole book at once. Instead, you only show them the title and a tiny blurb on the back cover. If they think the book is relevant, they have to ask for it, find the full instructions, and then follow those instructions exactly without skipping steps or doing anything forbidden. This is the world of AI "agents" trying to use "skills." For a while, people hoped that if you just gave these AI interns the right books, they would instantly become perfect workers. But nobody really knew if the AI could actually figure out which book to grab, or if it would just guess and hope for the best.

A team of researchers from Tencent Hunyuan and several top universities decided to put this idea to the test. They built a giant playground called SKILL-USE, a benchmark designed to see if AI agents can truly "read the room" and use these skill books correctly. They didn't just ask, "Did the AI finish the task?" They asked three much harder questions: Trigger (Did the AI even know to grab the right book?), Compliance (Did it follow the recipe step-by-step?), and Boundary (Did it avoid doing the forbidden things listed in the book?). They tested 177 different real-world tasks, from fixing software bugs to writing business reports, using 79 real skill documents and eight of the smartest AI models available today.

The results were a bit of a reality check. The researchers found that even the best AI setups only managed to get a "Skill-Use" score of about 0.613 (on a scale of 0 to 1). That means reliable skill use is still out of reach. The biggest problem wasn't that the AI couldn't follow the rules once it had the book; it was that the AI often didn't even know to pick the book up in the first place. It's like having a genius chef who can follow a recipe perfectly, but who keeps forgetting to open the fridge to get the ingredients. Furthermore, the researchers discovered that an AI's performance wasn't just about how "smart" the model was; it depended heavily on the "harness" or the software wrapper around it. Changing the wrapper was like changing the kitchen layout: sometimes the same chef became a star, and other times they stumbled.

In short, the paper suggests that giving AI agents a library of skills doesn't automatically make them better workers. The gap between "having a skill" and "using a skill" is huge. The AI often fails to recognize when a skill applies, or it gets distracted by the wrong skills when there are too many choices. While the AI is getting better at avoiding forbidden moves, it still struggles to follow complex procedures faithfully. The authors conclude that we can't just assume these agents will magically learn to use their tools; we need to build systems that help them recognize when to use a skill and how to stick to the plan, because right now, that's the hardest part of the puzzle.

Technical Summary: SKILL-USE

Problem Statement

Large Language Model (LLM) agents increasingly rely on skills—structured documents specifying when to act, which procedures to follow, and which tools are permitted. While existing evaluations assess the intrinsic quality of skills or their contribution to final task success, they fail to examine whether an agent can autonomously recognize a relevant skill from a partial description and apply it correctly. Current benchmarks either preload full skill documents (bypassing retrieval) or focus on surface-level instruction following (format/length) rather than domain-specific procedural adherence under progressive disclosure (where the agent sees only a skill's name and short description initially).

This gap leaves unexamined the full lifecycle of skill use: the agent must judge applicability, retrieve the full procedure, and then faithfully execute it while avoiding forbidden operations.

Methodology

The SKILL-USE Benchmark

The authors introduce SKILL-USE, a benchmark designed to evaluate skill use under progressive disclosure. It pairs 79 real-world skills (curated from community repositories like GitHub, SkillsBench, and SkillLearnBench) with 177 executable tasks across nine domains (e.g., Frontend & Mobile Dev, DevOps, Data Science, Security).

Evaluation Protocol:

Progressive Disclosure: The agent receives a user task and a skill metadata object (name, short description, file path) but not the full procedure. The agent must infer relevance, retrieve the file, and then execute.
Environment: Tasks run in isolated Docker sandboxes with real files and full tool access.
Scoring Dimensions: The benchmark decomposes skill use into three distinct facets:
- Trigger: Whether the agent retrieves the relevant skill based on the name/description alone.
- Compliance: Adherence to the prescribed procedure (weighted sum of required steps).
- Boundary: Avoidance of forbidden operations (weighted sum of prohibited actions).
Skill-Use (SU) Score: A gated aggregate score defined as $SU = g(y; s) \cdot [\alpha C(y) + (1-\alpha) B(y)]$ , where $g$ is the Trigger indicator, $C$ is Compliance, $B$ is Boundary, and $\alpha=0.7$ . Execution credit is granted only if the skill is triggered.

Construction Pipeline:
The dataset was built via a three-stage pipeline:

Curation: Skills were filtered for "skill exclusivity" (requirements not inferable from generic practice) and "process observability" (verifiable traces in the execution trajectory).
Rubric Design: Trajectory-based rubrics were created for each task, ensuring every scoring item traces to a specific skill requirement.
Validation: End-to-end validation included multi-agent adversarial reviews to ensure task scope, rubric verifiability, and the absence of generic requirements that strong models could solve without the skill.

Experimental Setup

The authors evaluated eight state-of-the-art LLMs (including Claude Opus 4.7/4.8, GPT-5.5, DeepSeek-V4-Pro, etc.) under two agent harnesses: Claude Code and Codex. All models were tested on the same 177 tasks within isolated sandboxes.

Key Results

1. Reliable Skill Use Remains Out of Reach

Even the strongest configuration (GPT-5.5 under the Claude Code harness) achieved a maximum SU score of only 0.613.

Dual Bottlenecks: Failure occurs at two independent stages. Models that successfully trigger a skill often fail to comply with its procedure (Compliance† maxes at ~0.638). Conversely, models capable of compliance sometimes fail to trigger the skill at all.
Compliance vs. Boundary: Models generally avoid forbidden operations (Boundary) more reliably than they complete full prescribed workflows (Compliance).

2. Skill Use is Harness-Dependent

Skill use behaves as a capability conditioned on the surrounding harness, not a fixed property of the model.

Ranking Instability: Model rankings shift significantly between harnesses. For instance, GPT-5.5 leads under Claude Code, while Claude Opus 4.8 leads under Codex.
Differential Stress: The Claude Code harness stresses triggering (retrieval), while Codex stresses procedural execution. Moving between harnesses compresses the spread of trigger rates but widens the spread of conditional compliance.

3. Impact of Skill Injection and Library Size

Retrieval is the Primary Bottleneck: Preloading the full skill document (bypassing progressive disclosure) significantly improves SU scores, primarily by boosting Trigger rates. Once the skill is triggered, preloading has negligible effect on execution quality, indicating that the decision to retrieve is the critical failure point.
Library Size: Increasing the skill library size from 1 to 30 skills causes a performance drop, but this is driven almost entirely by missed triggers (failing to select any skill) rather than selecting the wrong skill.

4. Generalization and Task Completion

Restraint Failure: High skill-use performance within scope does not correlate with the ability to refrain from using a skill when it is out of scope. In fact, the correlation is negative; models best at following skills are often worst at suppressing unnecessary invocations.
Task Completion: Following a skill's procedure improves task completion only when the SU score exceeds ~0.5. Below this threshold, partial or incorrect execution of a skill harms performance more than having no skill at all, as the model commits to a specific toolchain without completing it.

Significance and Contributions

The paper makes three primary contributions:

Formalization of Skill Use: It defines skill use as a distinct evaluation target, decomposing it into Trigger, Compliance, and Boundary to capture the nuances of recognition, adherence, and restraint under progressive disclosure.
SKILL-USE Benchmark: It releases a broad-coverage benchmark (9 domains, 79 skills, 177 tasks) with trajectory-based rubrics and a reusable three-stage construction pipeline, moving beyond artifact quality to process-level evaluation.
Empirical Insights: It provides evidence that reliable skill use is currently an open problem. The findings suggest that current agents struggle with the retrieval phase of progressive disclosure and that skill-use capability is highly dependent on the agent harness architecture rather than the base model alone.

The authors conclude that for skill libraries to be effective, models must not only be able to execute procedures but also reliably identify when those procedures are necessary, a capability that current frontier models have not yet mastered.