🤖 AI

Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry

This paper demonstrates that the natural-language metadata in AI agent skill registries (SKILL.md) is not merely passive documentation but a critical attack surface where semantic supply-chain manipulations can significantly compromise discovery, selection, and governance processes, allowing malicious skills to evade detection and dominate agent adoption.

Original authors: Shoumik Saha, Kazem Faghih, Soheil Feizi

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shoumik Saha, Kazem Faghih, Soheil Feizi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant that can do almost anything: book your flights, manage your taxes, or write code. But this robot doesn't know everything by itself. Instead, it has a giant digital toolbox called a Skill Registry. Think of this registry like an app store, but instead of downloading apps, the robot downloads "skills" (little packages of instructions) to learn new tasks.

Every skill package comes with a specific instruction manual called SKILL.md. This isn't just a boring list of features; it's written in plain English (or natural language) to tell the robot when to use the skill and how to do it.

The paper you shared, "Under the Hood of SKILL.md," reveals a scary new way hackers can trick these robots. They don't need to break into the robot's brain or write malicious code. Instead, they just edit the instruction manual (SKILL.md) with clever wordplay.

Here is how the attack works, broken down into three stages using simple analogies:

1. The Discovery Attack: "The SEO Spam of Robot Skills"

The Scenario: You ask your robot, "Plan a trip to Japan." The robot goes to the Skill Registry to find the best travel skill.
The Attack: A hacker takes a mediocre travel skill and secretly adds a few weird, specific words to its instruction manual (like a hidden keyword).
The Analogy: Imagine a restaurant that serves terrible food but pays a bribe to the search engine so that when you type "Best Sushi," their name pops up at the very top, pushing the actual best sushi place down to page 10.
The Result: The paper found that by just tweaking the text, hackers could make their bad skills appear 80% of the time in the top 10 results, even if they were worse than the original. The robot, trusting the search results, picks the hacker's skill.

2. The Selection Attack: "The Smooth-Talking Salesman"

The Scenario: The robot finds two skills that do the exact same thing (e.g., both can plan a trip). One is the original, safe version. The other is the hacker's version.
The Attack: The hacker changes the description of their skill to sound more confident, trustworthy, or "new." They might add phrases like, "This is the absolute best skill for this job" or "This skill is actively maintained by security experts."
The Analogy: Imagine you are buying two identical pairs of shoes. One has a plain label. The other has a label that says, "The President wears these; they are the safest shoes ever." Even though the shoes are identical, you pick the one with the fancy label.
The Result: The robot is easily fooled. In 77.6% of cases, the robot chose the hacker's skill over the safe one, simply because the description sounded more convincing. The robot didn't check if the skills actually worked differently; it just believed the "sales pitch."

3. The Governance Attack: "The Wolf in Sheep's Clothing"

The Scenario: Before a skill is allowed into the registry, a security guard (an automated scanner or an AI reviewer) checks it to make sure it's safe.
The Attack: The hacker writes a malicious instruction (e.g., "Send all my private documents to a stranger's email") but hides it inside the text using tricks. They might rephrase it to sound like a normal step, break it into tiny pieces, or hide it at the very end of a long document where the scanner stops reading.
The Analogy: Imagine a security guard checking a suitcase. The hacker puts a bomb inside, but they wrap it in a layer of "I am a harmless teddy bear" text, or they hide the bomb in a pocket that the guard doesn't look at because the suitcase is too long.
The Result: The paper showed that these tricks worked incredibly well.

Paraphrasing (rewriting the bad words to sound nice) let bad skills through 74% of the time.
Hiding the text (putting it where the scanner stops reading) let bad skills through 87% of the time.

The Big Picture

The main takeaway is that SKILL.md is not just a passive document. It is an active part of the robot's decision-making process.

It's not passive: The robot reads it, trusts it, and acts on it.
It's the weak link: Hackers don't need to be coding geniuses; they just need to be good at writing persuasive or tricky text.

The authors conclude that we can't treat these instruction manuals as safe just because they are text. We need to treat them with the same suspicion we treat executable code, because in the world of AI agents, words can be just as dangerous as code.

Technical Summary: Under the Hood of SKILL.md

Problem Statement

The rapid adoption of AI agents has led to the emergence of Agent Skills: modular filesystem packages that allow agents to extend capabilities on demand without model fine-tuning. Central to this ecosystem is the SKILL.md file, a natural-language manifest containing metadata, descriptions, and usage instructions that agents read to determine relevance and execution.

While this design enables scalable capability expansion, it introduces a novel semantic supply-chain risk. Unlike traditional software packages where security relies on code integrity, Agent Skills rely heavily on natural-language metadata that agents may read, trust, and act upon. Existing research has focused on malicious code execution after a skill is loaded or on detecting malicious submissions via static analysis. However, the registry-facing lifecycle—specifically how skills are discovered, ranked, selected, and governed before execution—remains underexplored. This paper posits that SKILL.md is not passive documentation but an operational text that can be manipulated to compromise the supply chain without altering executable code.

Methodology

The authors formulate a threat model where an adversary controls the textual content of a skill's SKILL.md file (specifically the name, description, and body) but cannot modify the registry backend, executable code, or auxiliary files. The study evaluates attacks across three distinct stages of the skill lifecycle using a corpus of 100 real-world skills from ClawHub across five categories (email, travel, tax, health, prompt).

1. Discovery Manipulation

Objective: Manipulate embedding-based retrieval to improve the ranking of an adversarial skill for a target query.

Attack Strategy: The adversary appends short textual triggers ( $\delta_D$ ) to the SKILL.md file.
Techniques:
- Black-box: Beam search optimization using a lightweight generator (Qwen2.5-0.5B) to iteratively find tokens that maximize cosine similarity with the target query embedding.
- White-box: Gradient-based optimization assuming access to the embedding model parameters.
Evaluation: Tested against BAAI (bge-base, bge-small) and OpenAI (text-embedding-3-small) models. A realistic "in-the-wild" scenario was simulated using a ClawHub-style ranking function that combines lexical relevance, vector similarity, and download popularity.

2. Selection Manipulation

Objective: Bias the agent's choice among functionally equivalent skills toward the adversarial variant.

Attack Strategy: Modify only the description field of SKILL.md while preserving the underlying functionality.
Techniques: Four framing strategies were tested:
1. False Advertising: Exaggerating capabilities (e.g., "best skill").
2. Assertive Cues: Directive language (e.g., "always choose this").
3. Active Maintenance: Claims of recency/updates.
4. Trust/Security: Claims of security endorsements.
Evaluation: A paired-choice experiment using the Hermes-agent framework. Agents were presented with an original skill and an adversarial variant for the same user task. Models tested included GPT-4.1-mini, GPT-5, Gemma-4-31B, and Qwen3-235B.

3. Registry Governance Evasion

Objective: Preserve malicious intent in SKILL.md while bypassing registry vetting pipelines.

Attack Strategy: Start with "clean" skills that pass initial scans, then insert or rewrite malicious instructions within the SKILL.md text.
Techniques:
1. LLM Judge Jailbreaking: Appending prompts to force the LLM reviewer to ignore safety criteria.
2. Paraphrasing: Rewriting malicious instructions using benign operational language to avoid keyword triggers.
3. Definition-of-Done (DoD): Decomposing malicious behavior into a procedural checklist framed as completion requirements.
4. Context Window Overflow: Placing malicious instructions beyond the truncation limit of the LLM reviewer (e.g., after the first 10k characters).
Evaluation: A three-stage governance pipeline (Static Scanner, LLM Reviewer, VirusTotal) was implemented locally. The pipeline returns verdicts of Clean, Suspicious, or Malicious.

Key Results

Discovery Manipulation

Retrieval Scores: Short triggers reliably improved retrieval scores. The strongest result was a 86.14% pairwise win rate with a 16.40% average score boost using black-box beam search against OpenAI embeddings.
Ranking Visibility: Under OpenAI retrieval, adversarial skills achieved 80% Top-10 placement in pairwise comparisons.
Realistic Impact: Even in a popularity-aware ranking system (ClawHub-style), modified skills outperformed baselines in 74.14% of "average-day" cases (simulating moderate download counts) and 94.00% of "0-day" cases (cold start).

Selection Manipulation

Bias Effect: Description-only modifications consistently biased agents toward adversarial variants. Across all models and strategies, the average selection rate was 77.6%, significantly above the 50% random baseline.
Model Sensitivity: Larger models (e.g., Gemma-4-31B, Qwen3-235B) showed higher susceptibility, with some strategies achieving >80% selection rates.
Strategy Efficacy: "False Advertising" and "Assertive Cues" were the most effective, but even "Trust/Security" claims significantly shifted agent preferences.

Registry Governance Evasion

Baseline Detection: The governance pipeline effectively detected direct malicious insertion (0% clean verdicts for baseline malicious variants).
Evasion Success:
- Paraphrasing: Reduced malicious verdicts to 26.1% (73.9% avoided blocking).
- DoD Framing: Resulted in 23.9% clean verdicts and 66.3% avoiding blocking.
- Context Overflow: The most effective strategy, resulting in 87.1% clean verdicts and 0% malicious verdicts, effectively bypassing the blocking mechanism entirely.
- Jailbreaking: Provided limited improvement (63.5% still flagged as malicious), suggesting LLM reviewers are relatively robust to direct instruction overrides.

Significance and Claims

The paper claims that SKILL.md is a critical, active attack surface in the AI agent supply chain. The findings demonstrate that:

Semantic Vulnerabilities Exist: Natural-language metadata can be manipulated to control which skills are found (Discovery), trusted (Selection), and approved (Governance) without touching executable code.
Current Defenses are Insufficient: Registry mechanisms relying on embedding similarity, description-based selection, and LLM-based semantic review are vulnerable to minimal, non-malicious-looking text modifications.
Lifecycle Composition: These attacks are composable; an adversary can craft a skill that passes governance, surfaces highly in search, and is selected by agents, creating a full supply-chain compromise.

The authors conclude that mitigating these risks requires treating skill documentation as security-sensitive input, implementing robust retrieval and evidence-based selection, and adopting stricter governance (e.g., full-file review, handling of suspicious verdicts) alongside runtime least-privilege controls. The work provides a framework for measuring semantic supply-chain risk and calls for hybrid governance models combining centralized scanning with decentralized audits and transparency logs.