🤖 AI

Skill Coverage: A Test Adequacy Metric for Agent Skills

This paper introduces "skill coverage," a test adequacy metric that evaluates how thoroughly agent skills are exercised by measuring the extent to which documented behavior constraints are observed in agent trajectories, revealing that current benchmarks cover less than 44% of these constraints despite task success.

Original authors: Boyin Tan, Xiaowei Huang, Youcheng Sun

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Boyin Tan, Xiaowei Huang, Youcheng Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a highly skilled chef (the AI Agent) to cook a complex meal using a specific, reusable recipe card (the Agent Skill).

In the past, to see if the chef was good, we only looked at the final dish. If the meal tasted delicious and the customer was happy, we assumed the chef had followed the recipe perfectly. We thought, "Great job! The recipe worked!"

But this new paper asks a different question: Just because the meal was good, did the chef actually use every single instruction on the recipe card?

Maybe the recipe said, "Always chop onions with a sharp knife," but the chef used a dull one and still made a tasty meal. Or maybe it said, "Let the sauce simmer for 20 minutes," but the chef only simmered it for 5 minutes and it still tasted fine. We wouldn't know these things just by tasting the food.

The Problem: The "Good Meal" Trap

The authors argue that current tests for AI agents are like only tasting the final dish. They tell us if the task was completed successfully, but they don't tell us which parts of the skill were actually tested and which parts were ignored.

If an AI agent completes a task successfully, it doesn't mean it exercised every rule, warning, or step written in its skill document. It might have just gotten lucky or found a shortcut.

The Solution: "Skill Coverage"

The paper introduces a new way to measure testing called Skill Coverage. Instead of just checking the final result, they treat the recipe card itself as the thing being tested.

Here is how they do it, using a simple analogy:

Breaking the Recipe into Rules: First, they take the messy, long recipe card and break it down into specific, checkable rules.
- Example: Instead of the whole paragraph about "cooking pasta," they extract a specific rule: "When boiling water, the agent must add salt before the pasta."
- They call these Skill Behavior Constraints. They ignore the fluff (like "This recipe was written by Chef John in 2026") and focus only on the actual instructions the agent must follow.
The "Covered" vs. "Uncovered" Check: Next, they watch the agent do the task. They ask two questions for every single rule:
- Did the situation happen? (e.g., Did the agent actually try to boil water?)
- Can we see the proof? (e.g., Did the agent's log show it adding salt, or did it just say "I added salt" without proof?)
If the answer to both is "Yes," the rule is Covered. If the agent never faced that situation, or if they did but left no traceable proof, the rule is Uncovered.

Crucially: A rule can be Covered even if the agent did it wrong. The point isn't to see if they passed; it's to see if they tried the specific instruction in a way we can verify.

What They Found

The researchers tested this on a popular set of AI tasks called SkillsBench. They looked at how well different AI models (like Gemini, Claude, and Codex) followed their recipe cards.

The results were surprising:

Even when the AI successfully completed the task, it only "exercised" or "covered" about 40% of the rules on the recipe card.
This means 60% of the instructions were left untested. The AI might have skipped them, or the task didn't force the AI to use them, or the AI did them but didn't leave enough evidence for us to see.

The Big Takeaway

The paper concludes that Success $\neq$ Thorough Testing.

Just because an AI agent finishes a job successfully doesn't mean it has been rigorously tested on all the skills it claims to have. It's like a driver who gets to work on time but never actually used their turn signal or checked their blind spot during the trip. We know they arrived, but we don't know if they followed all the safety rules.

Skill Coverage is a new tool that tells us exactly how much of the "recipe" was actually used and verified, giving us a much clearer picture of how well an AI agent is truly trained, rather than just whether it got the job done.

Technical Summary: Skill Coverage: A Test Adequacy Metric for Agent Skills

Problem Statement

Current evaluations of Large Language Model (LLM) agents primarily rely on task-level outcomes (success/failure) to assess the utility of "agent skills"—reusable procedural knowledge artifacts encoded in natural language. However, a successful task completion does not reveal which specific instructions, expectations, or action patterns within a skill were actually exercised during execution. This creates a significant gap in test adequacy: users lack evidence regarding which parts of a documented skill have been tested and which remain unobserved under specific model-task settings. Consequently, it is difficult to determine if a skill is sufficiently validated for reuse, adaptation, or improvement.

Methodology

The paper introduces Skill Coverage, a test adequacy metric that treats the skill artifact itself as the object under test, rather than focusing solely on the final task outcome. The methodology consists of two primary stages:

1. Extraction of Skill Behavior Constraints (SBCs)

The authors define a conservative abstraction called a Skill Behavior Constraint (SBC) to formalize the testable portion of a skill document.

Definition: An SBC is a source-grounded, observable, EARS-style (Essentialize, Analyze, Refine, Simplify) normalized condition-action rule derived from natural-language skill documents.
Criteria: To be retained, a constraint must be:
1. Source-grounded: Supported by a concrete span in the skill document.
2. Constraint-bearing: Imposing an obligation, prohibition, usage pattern, workflow step, or deliverable property.
3. Observable: Judgeable from execution evidence (e.g., tool calls, code, file diffs, logs).
4. Atomic: Expressing one independent behavioral obligation.
Process: An LLM extracts candidate SBCs from skill documents, which are then audited and repaired by human reviewers to filter out background prose, metadata, and incidental implementation details. The resulting set $C(S)$ serves as the denominator for coverage analysis.

2. Coverage Judging in Agent Trajectories

The framework evaluates whether an agent trajectory $\tau$ provides sufficient evidence to exercise a specific constraint $c = \langle C_c, B_c, J_c \rangle$ .

Binary Judgment: A constraint is marked as Covered only if two conditions are met:
1. Applicability: The observable evidence $Obs(\tau)$ instantiates the applicability condition $C_c$ .
2. Adjudicable Evidence: The trajectory contains enough observable evidence to judge whether the constrained behavior $B_c$ was satisfied or violated.
Distinction: Coverage does not require the agent to succeed in the behavior; it only requires that the scenario was instantiated and the outcome was judgeable. If a constraint is applicable but the evidence is insufficient to judge the behavior, it remains Uncovered.
Metric: Skill coverage is calculated as the fraction of audited constraints in $C(S)$ that are covered by at least one trajectory in a test suite $T$ .

Validation Protocol

To ensure reliability at scale, the authors employ a validation-before-scale protocol. An LLM judge (DeepSeek V4 Flash) is used to assign coverage labels, but its performance is first validated against human auditors on a sample of 350 constraint-trajectory pairs. The judge achieved 93.71% exact agreement and a Cohen's $\kappa$ of 0.874 with human labels before being applied to the full benchmark.

Key Results

The methodology was applied to SkillsBench, evaluating 78 source tasks across seven agent-model configurations (including Gemini, Codex, and Claude variants).

Low Coverage Rates: Existing benchmark executions cover only 39.90% to 43.98% of the extracted skill behavior constraints on average. This indicates that more than half of the documented skill guidance remains untested or unjudgeable in current evaluations.
Task Success vs. Coverage: While successful task executions generally cover more constraints than unsuccessful ones (with a coverage gap of 2.25–13.50 percentage points), task success does not imply adequate testing. Even successful executions leave 38.96% to 47.14% of the skill denominator untested.
Distribution: Across configurations, only 8–22 out of 162 skills reached 90–100% coverage, while 41–52 skills remained in the 0–30% coverage bucket.

Significance and Claims

The paper claims that skill coverage provides a complementary signal to task-level outcomes, addressing a critical blind spot in agent evaluation.

Artifact-Level Adequacy: The work shifts the focus from "did the agent solve the task?" to "which parts of the reusable skill were actually exercised with judgeable evidence?"
Actionable Diagnostics: Uncovered constraints are not necessarily violations; rather, they identify scenarios or evidence gaps where the benchmark fails to instantiate the skill's conditions or capture sufficient trace data. This provides actionable feedback for:
- Benchmark Designers: To add tasks that exercise specific scenarios or improve trace capture for oracle-checkable evidence.
- Skill Authors: To identify which parts of a skill have not been validated under realistic agent executions.
- Users: To make informed decisions about reusing skills based on the extent of their testing.

The authors conclude that task success and skill coverage answer different evaluation questions and should be reported separately to provide a complete picture of agent capability and skill adequacy.