How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings

1. Problem Statement

While Large Language Model (LLM) agents increasingly utilize "skills" (reusable, domain-specific knowledge artifacts) to extend their capabilities, existing benchmarks suffer from overly idealized evaluation conditions.

• The Gap: Current benchmarks (e.g., SKILLSBENCH) provide agents with hand-crafted, task-specific skills that are directly injected into the context. This bypasses the real-world challenges of skill selection (identifying useful skills among noise), skill retrieval (searching large repositories), and skill adaptation (generalizing non-task-specific skills to new queries).
• The Core Question: Do skills remain effective when agents must autonomously retrieve them from a large, noisy collection of 34,000+ real-world skills, and when the retrieved skills are not perfectly tailored to the specific task?

2. Methodology

A. Infrastructure and Dataset

• Skill Collection: The authors assembled a dataset of 34,198 real-world skills from open-source repositories (GitHub), filtered for permissive licenses (MIT/Apache 2.0), quality, and deduplication.
• Search Engine: A retrieval system was built using:
  • Indexing: Dense embeddings (Qwen3-Embedding-4B) for metadata and full content, plus sparse BM25 for keyword matching.
  • Search Strategies: Compared Direct Search (single query) vs. Agentic Search (iterative query formulation, candidate evaluation, and refinement).
  • Best Performer: Agentic Hybrid Search (combining keyword, semantic, and full-content indexing) achieved the highest Recall@k.

B. Progressive Evaluation Framework

The study evaluated agent performance on SKILLSBENCH (84 tasks) and TERMINAL-BENCH 2.0 (89 tasks) across six progressively realistic settings:

1. Curated + Forced Load: Skills provided and forced to load (Upper Bound).
2. Curated: Skills provided, agent decides to load them.
3. Curated + Distractors: Curated skills mixed with irrelevant retrieved skills.
4. Retrieved (w/ Curated): Agent retrieves top-5 skills from the pool containing curated skills.
5. Retrieved (w/o Curated): Agent retrieves from the pool without curated skills (pure general-purpose skills).
6. No Skills: Baseline.

Models Tested: Claude Opus 4.6, Kimi K2.5, and Qwen3.5-397B.

C. Skill Refinement Strategies

To address performance degradation, two refinement strategies were tested:

1. Query-Agnostic Refinement: Offline improvement of skills without knowledge of the specific target task (using synthetic test queries).
2. Query-Specific Refinement: The agent attempts the task using retrieved skills, self-evaluates the trajectory, and synthesizes a new, task-tailored skill set by merging relevant parts of retrieved skills and discarding noise.

3. Key Results

A. Performance Degradation in Realistic Settings

• Fragility of Skills: Performance gains from skills are highly fragile. As settings become more realistic, pass rates drop significantly.
  • Example (Claude Opus 4.6 on SKILLSBENCH):
    • Curated + Forced: 55.4%
    • Curated (Autonomous): 51.2%
    • Curated + Distractors: 43.5%
    • Retrieved (w/ Curated): 40.1%
    • Retrieved (w/o Curated): 38.4% (approaching the No-Skill Baseline of 35.4%).
• Bottlenecks Identified:
  1. Selection Failure: Agents often fail to load relevant skills even when available (e.g., loading rate dropped from 62% to 31% when distractors were added).
  2. Adaptation Failure: Without task-specific curation, agents struggle to extract useful info from general skills. In some cases (Kimi, Qwen), irrelevant retrieved skills actively hurt performance below the no-skill baseline.

B. Retrieval Performance

• Agentic Search Superiority: Agentic search significantly outperformed direct search.
  • Recall@3: Agentic Semantic (56.8%) vs. Direct Semantic (38.1%).
  • The ability to iteratively refine queries and inspect candidates is crucial for navigating large skill pools.

C. Effectiveness of Refinement

• Query-Specific Refinement: This strategy substantially recovered lost performance only when the initial retrieved skills had reasonable relevance.
  • SKILLSBENCH (w/ Curated): Improved Claude from 40.1% to 48.2%.
  • TERMINAL-BENCH 2.0: Improved Claude from 61.4% to 65.5%.
  • Mechanism: It allows the agent to synthesize information across multiple skills (e.g., combining weight sharding from one skill with autograd patterns from another).
• Query-Agnostic Refinement: Provided only marginal gains (e.g., +1.9% on SKILLSBENCH) because it cannot adapt to the specific nuances of the target task.
• Dependency on Initial Quality: Refinement acts as a "multiplier" on existing skill quality. When initial retrieval coverage is low (e.g., SKILLSBENCH w/o curated), refinement fails to generate new knowledge, and performance remains near baseline.

4. Key Contributions

1. Realistic Evaluation Framework: Introduced a progressive benchmarking suite that moves beyond idealized "hand-crafted" setups to test retrieval, selection, and adaptation in noisy, large-scale environments.
2. Empirical Evidence of Fragility: Demonstrated that skill benefits degrade consistently as conditions become realistic, often vanishing entirely when curated skills are removed.
3. Agentic Search Validation: Proved that iterative, agentic hybrid search is essential for effective skill retrieval, outperforming static semantic search.
4. Refinement Strategy Analysis: Showed that query-specific refinement is a viable path to recovering performance, but its success is strictly conditional on the initial relevance of the retrieved skills.

5. Significance and Implications

• Limitations of Current Skill Ecosystems: The paper highlights that simply having a large library of skills is insufficient; the current bottleneck is the agent's ability to select and adapt them.
• Future Directions: The results suggest that future agent systems must prioritize:
  • Better retrieval mechanisms (agentic search).
  • Dynamic, task-aware skill refinement (query-specific).
  • Skill ecosystems designed with the limitations of current LLMs in mind (e.g., ensuring skills are robust enough to be adapted without perfect curation).
• Generalizability: The findings hold across proprietary (Claude) and open-weight (Kimi, Qwen) models, indicating a systemic challenge in the field of agentic AI.

In conclusion, while skills hold promise for extending LLM capabilities, their utility is currently fragile in the wild. Without significant improvements in retrieval accuracy and task-aware refinement, agents often revert to baseline performance or are misled by noisy data.