Technical Summary: A Framework for Evaluating Agentic Skills at Scale

Problem Statement

While "agent skills"—structured, reusable knowledge artifacts designed to augment Large Language Model (LLM) capabilities with domain-specific workflows, API patterns, and coding conventions—are rapidly adopted in industry, their rigorous evaluation remains under-studied. Existing benchmarks primarily measure general task-solving, tool use, or coding ability but fail to address how specific skills alter agent behavior across diverse models. Crucially, current methodologies rely on small, fixed sets of hand-authored tasks, making it impossible for skill authors to evaluate arbitrary, newly created skills or to determine if a skill actually improves performance on its intended tasks. There is a lack of a reusable methodology to measure the marginal utility of a skill, specifically regarding instruction adherence and goal completion.

Methodology

The authors propose a scalable, automated framework that transforms skills into executable evaluation tasks. The pipeline operates through the following stages:

1. Skill Analysis & Intent Inference: The system analyzes skill content (e.g., SKILL.md and supporting files) to infer realistic usage scenarios. In fully autonomous settings, a quorum of agents infers usage; in hybrid settings, user intent (e.g., Jira tickets, GitHub issues) guides the process.
2. Environment Engineering: An agent determines the execution environment requirements (e.g., CLI tools, API tokens, specific runtimes, database access). The framework categorizes dependencies (Table 1) and attempts to satisfy missing requirements or discards skills requiring hard-to-reproduce states (e.g., pre-populated external services) in autonomous modes.
3. Task Generation: The system generates realistic task proposals derived from skill content. It synthesizes inputs (e.g., code files, data) and constructs executable task specifications. Crucially, it generates two hidden scoring rubrics:
   • Goal-Completion Rubrics: Assess if the final artifact is correct and outputs are produced.
   • Instruction-Following Rubrics: Assess adherence to preferences encoded in the skill (e.g., library choices, naming conventions, prohibited patterns).
4. Validation: A quality-assurance agent verifies executability, checks for rubric leakage (ensuring the task description doesn't reveal the solution steps), and removes ambiguous tasks.
5. Evaluation Protocol: Each task is solved twice by a solver agent:
   • Without Skill: The agent receives the task and environment but no skill context.
   • With Skill: The agent is explicitly informed that the relevant skill is installed and available.
   • Judging: A separate judge agent (Sonnet 4.6 in this study) scores solutions against the hidden rubrics. The "Skill Delta" is calculated as the performance gain between the two conditions.

Key Contributions

• Scalable Evaluation Framework: A method to automatically synthesize realistic, executable evaluation tasks and custom rubrics from any given skill, allowing for the assessment of individual skills in isolation.
• Large-Scale Empirical Study: The authors applied this framework to a corpus of 500 real-world open-source skills (sourced from ~100 organizations) to generate 1,000 evaluation tasks.
• Comprehensive Model Benchmarking: The study evaluated 19 agent–model configurations, spanning proprietary frontier models (Anthropic, OpenAI, Google) and open-weights models (GLM, DeepSeek, Qwen, Nemotron, etc.), measuring instruction-following and goal-completion metrics.
• Open Dataset: Release of a dataset of realistic coding tasks and evaluation results to support future research on agent skills.

Results

1. Instruction Following vs. Goal Completion

• Goal Completion: Access to skills pushed goal completion scores close to saturation (often >90%) for most frontier models, suggesting that current synthetic tasks are not sufficiently challenging to discriminate between models on pure task success.
• Instruction Following: This metric showed substantial variance. Models varied widely in how closely they adhered to the specific workflows and conventions encoded in skills.
  • Top Performers: Opus 4.8 (88.0) and Opus 4.7 (87.7) achieved the highest instruction-following scores.
  • Open-Source Parity: GLM 5.1 reached a comparable 85.0 at a fraction of the cost.
  • Lagging Models: Kimi K2.6, MiniMax 2.7, Qwen3-Coder-Next, and Gemini 3.1 Flash Lite clustered around 57–60, roughly 25–30 points below the frontier. The Nemotron family lagged significantly (25.2–46.8).

2. Skill Utility and Model Behavior

• Substantial Gains: Access to a relevant skill yielded relative performance gains ranging from 5.5 to 22 points, driven primarily by improvements in instruction following.
• Closing the Gap: Skills significantly narrowed the performance gap between smaller/cheaper models and their flagship counterparts.
  • Example: With a skill, GPT-5.4 mini (84.5) became competitive with GPT-5.4 (88.2).
  • Example: GLM 5.1 (91.1) matched Sonnet 4.6 (91.5) and trailed Opus 4.8 (92.7) by only ~1.6 points.
  • Cost Implication: Achieving frontier-quality scores with skills on cheaper models (e.g., GLM 5.1 at ~$0.89/scenario) offered comparable quality to expensive models (Opus 4.8 at ~$3.26/scenario) at 3–4x lower cost.

3. Domain Impact

• Workflow-Heavy Domains: The largest gains were observed in domains where skills encode specific, procedural workflows (e.g., Media & File Processing +38.1 uplift, Security & Compliance +30.3).
• Reasoning-Heavy Domains: Domains relying on general guidelines or best practices (e.g., Testing, Data Processing) showed smaller gains, suggesting skills are most effective when they capture concrete procedures rather than abstract principles.

4. Diagnostic Capabilities

The framework allows for granular diagnosis of individual skills. By inspecting specific rubric criteria where the "with-skill" and "without-skill" solutions diverge, authors can identify exactly which behaviors a skill induces (e.g., switching from huggingface-cli to hf) and where it fails to change agent behavior.

Significance and Claims

The paper positions "skill utility" as a shift in evaluation focus: from "can the agent solve the task?" to "does the agent solve the task the way I want it to?"

• For Skill Authors: The framework provides a concrete tool to locate weak spots in a skill, validate that it induces the intended behavioral changes, and improve the artifact before deployment.
• For Deployment: The results suggest that for workloads where relevant skills are known in advance, pairing a smaller, cheaper model with a skill is a viable and cost-effective alternative to using larger, more expensive frontier models.
• Limitations: The authors acknowledge limitations, including the assumption that the agent is explicitly told a skill is available (ignoring skill-selection failures), reliance on a single LLM-as-judge (potential bias), and the exclusion of skills requiring complex, hard-to-reproduce environments (databases, multi-turn interactions).

The work concludes that while skills significantly enhance agent behavior, their effectiveness is highly model-dependent, and the proposed framework offers the first scalable methodology to rigorously measure this utility across the heterogeneous landscape of real-world skills.