Technical Summary: From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills

Problem Statement

Agent Skills are emerging as a critical mechanism for equipping Large Language Model (LLM) agents with domain-specific, procedural knowledge without requiring model retraining. Defined by a mandatory SKILL.md file containing metadata (YAML frontmatter) and unstructured Markdown instructions, these artifacts allow agents to retrieve context on demand. However, the unconstrained nature of the SKILL.md body creates significant variability in quality. Poorly authored skills can introduce irrelevant information, omit critical guidance, or present instructions in ways that degrade agent performance, increase context window consumption, and reduce reasoning effectiveness.

Despite rapid adoption, prior research has focused on the effectiveness and security of skills, leaving their authoring practices largely unexplored. The authors posit that, similar to how unstructured commit messages accumulated quality issues in traditional software engineering, SKILL.md files represent an emerging software quality concern. Without clear guidelines, poor authoring practices risk becoming entrenched, potentially propagating through training data and generation pipelines.

Methodology

The study employs a mixed-methods approach involving qualitative analysis, literature review, and automated detection, structured around four Research Questions (RQs).

1. Dataset Curation
The authors constructed a sample from the skills.sh marketplace, initially containing over 133,000 skills. Through a multi-step filtering process (removing non-English skills, retaining only those with significant adoption signals like weekly downloads and repository stars, and deduplicating), they arrived at a final sample of 238 real-world SKILL.md files.

2. RQ1: Semantic Component Analysis (Qualitative)
To understand the "anatomy" of these files, the authors conducted a two-stage qualitative content analysis on the H2-heading level of the Markdown bodies. Two authors independently coded a sample of 128 files, resolving disagreements through negotiated agreement to derive a hierarchical taxonomy of semantic components.

3. RQ2: Best Practices and Skill Smells (Multivocal Literature Review)
The authors conducted a Multivocal Literature Review (MLR) of 29 online sources (blogs, documentation, guides) to identify best practices for authoring SKILL.md files. They mapped these practices to the semantic components identified in RQ1. "Skill smells" were then defined as violations of these best practices (anti-patterns).

4. RQ3: Prevalence Detection (Automated)
To measure the prevalence of the 26 identified skill smells, the authors developed the Skill Smell Detector (SSD).

• Ground Truth: 53 files were manually labeled by two authors to create a validation dataset.
• Static Detection: Five smells (e.g., file length, character counts) were detected via Python scripts.
• Semantic Detection: The remaining 21 smells, requiring contextual reasoning, were detected using an open-source LLM (Qwen3.6-27B with 4-bit AWQ quantization). The model was prompted with definitions, positive examples, and negative examples to perform binary classification.
• Evaluation: The detector achieved a weighted F1-score of 0.78 on the validation set.

5. RQ4: Evolution Analysis (Longitudinal)
The authors mined the commit history of the 238 skills (yielding 1,295 commits, reduced to 142 files with multiple commits for analysis). They tracked the presence of skill smells over a 35-week timeline (October 2025 – June 2026) using a "last-observation-carried-forward" scheme to determine if smells are corrected or persist as skills evolve.

Key Contributions

1. Taxonomy of Semantic Components: The first systematic characterization of SKILL.md files beyond frontmatter, identifying 13 higher-level and 44 lower-level semantic components (e.g., Task, Introduction, References, Principles).
2. Mapping of Best Practices: A systematic mapping of existing authoring guidance to semantic components, revealing that current best practices cover only 7 of the 13 higher-level components.
3. Concept of Skill Smells: The introduction of "skill smells" as a new category of software quality issues for agentic systems, comprising 26 distinct violations categorized into groups like Under-Specified Guidance, Over-Prescribed Guidance, Context Bloat, and Security Hazards.
4. Automated Detection Tool: The development of the Skill Smell Detector (SSD), a hybrid tool combining static analysis and LLM-based semantic detection.
5. Empirical Evidence of Pervasiveness and Persistence: Large-scale evidence demonstrating that skill smells are ubiquitous and rarely corrected over time.

Results

Semantic Structure (RQ1)
The analysis revealed that while SKILL.md files are free-form, they follow a structural backbone. The three most common higher-level components are Task (74.4%), Introduction (63.5%), and References (57.1%). However, components like Evaluation, Tools, and Agent Architecture are frequently present but lack explicit authoring guidance.

Best Practices and Smells (RQ2)
The review identified 26 best practices. Crucially, only 7 of the 13 higher-level semantic components are governed by explicit best practices. The remaining six components lack documented recommendations, creating a gap between practice and guidance. This led to the definition of 26 skill smells.

Prevalence (RQ3)

• Ubiquity: Skill smells are pervasive. Over 99% of the analyzed SKILL.md files contain at least one smell.
• Density: The average file contains 10.5 skill smells. Only one file in the entire sample was completely free of smells.
• Most Common: The most prevalent smell is Rationalization Loophole (RL) (94%), where skills fail to discourage agents from skipping steps. Other common smells include Buried Gotchas (81%) and Execute Without a Plan (78%).
• Least Common: Lengthy Skill Name and Lengthy Skill Description were virtually non-existent in the sample.

Evolution (RQ4)
Longitudinal analysis showed no evidence that skill smells are corrected over time. Instead, their prevalence tends to stabilize or worsen as skills evolve. Once introduced, skill smells rarely disappear, suggesting they become entrenched during the development lifecycle.

Significance and Claims

The paper claims that SKILL.md quality is a critical, yet overlooked, software engineering concern for agentic systems. The authors argue that:

• Skill smells are not isolated documentation issues but fundamental quality defects that can impair agent performance, efficiency, and safety.
• The gap between recommended and actual practices is substantial, with nearly half of the semantic structure lacking authoring guidelines.
• Manual quality control is insufficient given the high prevalence and persistence of these issues.
• Early intervention is possible: Unlike traditional code smells which have decades of history, Agent Skills are in early adoption. The community has a unique opportunity to establish automated quality assurance tools (linters, IDE integrations) and improve authoring practices before poor habits become deeply embedded in the ecosystem.

The study concludes that treating SKILL.md files as executable software artifacts rather than passive documentation is essential for building trustworthy and maintainable agent ecosystems.