💻 computer science

An Empirical Study of Downstream Adaptation for Agent Skills

This paper presents the first empirical study of downstream adaptation for LLM agent skills, analyzing 1,126 instances to reveal a reuse paradox where developers frequently rewrite skills for local contexts, and proposes a taxonomy of 46 adaptation patterns to guide improvements in skill design, standardization, and security.

Original authors: Xinjian Wu, Jingzhi Gong, Gunel Jahangirova, Zhenpeng Chen, Jie M. Zhang

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Xinjian Wu, Jingzhi Gong, Gunel Jahangirova, Zhenpeng Chen, Jie M. Zhang

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've just bought a high-end, pre-made "smart recipe" for a robot chef. This recipe (called a Skill) is supposed to be plug-and-play: you drop it into your kitchen, and the robot knows exactly how to chop, sauté, and plate a dish.

The authors of this paper wanted to see what happens when real people actually try to use these pre-made recipes in their own kitchens. They didn't just look at the recipes; they looked at the notes, scribbles, and changes people made when they tried to make these recipes work in their specific homes.

Here is the story of their findings, broken down simply:

1. The Big Surprise: "Plug-and-Play" is a Myth

The researchers expected that since these skills are designed to be reused, people would just copy them and run them.

The Reality: It's more like buying a suit that's "one size fits all" but then having to take it to a tailor to cut the sleeves, hem the pants, and change the buttons just to make it fit.
The Paradox: Even though these skills are published to be easily reused, developers spend a huge amount of time rewriting them. They have to fix how the skill is found, change the instructions to match their specific tools, and translate the language. It's not "plug-and-play"; it's "plug-and-pray-you-have-a-tailor."

2. The "Recipe Card" is the Control Center

A "Skill" isn't just one file; it's a folder with a main instruction card (called SKILL.md) and some extra tools or scripts attached.

The Finding: When people adapt these skills, they almost always (80% of the time) rewrite the main instruction card. They rarely touch the actual code scripts unless they have to.
The Metaphor: Think of the instruction card as the brain of the operation. People are constantly rewriting the brain's thoughts to fit their situation, while the tools (the hands) stay mostly the same.

3. Changes Come in Bundles (The Domino Effect)

You might think someone would just change one small thing, like "add a step to wash the vegetables."

The Finding: Changes rarely happen in isolation. If you change the steps (procedure), you almost always have to change the rules (decisions) and the constraints (policies) at the same time.
The Metaphor: It's like changing the engine of a car. You can't just swap the engine; you have to adjust the transmission, the fuel lines, and the exhaust system all together. The researchers found that these changes are tightly coupled, meaning if you miss one part of the bundle, the whole thing might break.

4. The Hidden Danger Zone: "Secrets in the Sauce"

This is the most alarming part of the study.

The Finding: Nearly 1 in 5 of the adapted skills introduced "security-sensitive" content. This means people accidentally (or intentionally) added instructions that could let the robot access private files, connect to the internet, or run dangerous commands.
The Twist: Usually, security experts scan code for viruses. But here, the dangerous instructions were hidden inside the natural language text (the recipe card).
The Metaphor: Imagine a security guard checking a suitcase for weapons (code). But the person sneaking a knife into the suitcase didn't hide it in a metal box; they wrote "I am carrying a knife" in the middle of a grocery list. The guard didn't see it because they were only looking for metal, not words. Because these risks are in the text, they bypass traditional security checks.

5. The "Commit Message" Lie

When developers save their changes, they write a note explaining what they did (a "commit message").

The Finding: These notes are terrible at explaining why the change was needed. They usually say, "I added a feature" or "I fixed a bug."
The Reality: They rarely explain the real problem, like "I had to change this because my company uses a different database" or "I had to rewrite this because the robot speaks a different dialect."
The Metaphor: It's like a traveler writing in their journal, "I changed my route," but never explaining that they did it because the bridge was out. If you only read the journal, you have no idea why the route changed.

Summary of the "Recipe"

The paper concludes that while "Agent Skills" are a great idea for reusing knowledge, the current system is messy.

Developers have to do too much manual rewriting to make skills work.
Changes are complex and interconnected; you can't just tweak one thing without checking the others.
Security is at risk because dangerous instructions are hiding in plain sight within the text, invisible to standard code scanners.
Documentation (commit messages) is often too vague to help future developers understand what actually happened.

The authors suggest we need better tools to help developers adapt these skills without breaking them, and better security checks that can read the "recipe text" to spot dangerous instructions before they cause trouble.

Technical Summary: An Empirical Study of Downstream Adaptation for Agent Skills

Problem Statement

As Large Language Model (LLM) agents increasingly rely on reusable "skills" to package workflows, decision procedures, and policies, these skills have emerged as a novel unit of software reuse. While intended for plug-and-play adoption, the literature has largely focused on skill representation, acquisition, retrieval, and evolution from the perspective of original creators. There is a significant gap in understanding downstream adaptation: how developers actually modify published skills to fit local contexts, tools, and organizational policies.

In practice, developers frequently rewrite skill components (such as the SKILL.md instruction file and bundled resources) to make them locally useful. However, the nature, frequency, and patterns of these adaptations remain unexplored. This lack of understanding hinders the development of better abstractions, standardized interfaces, and automated support for skill evolution. Furthermore, because these adaptations often involve natural language instructions rather than code, they may introduce security risks that bypass traditional code-focused review pipelines.

Methodology

The authors conducted the first empirical study of downstream skill adaptation using Mining Software Repositories (MSR) techniques on public GitHub fork histories.

Data Collection and Corpus Construction

Source: Six widely adopted, high-traffic skill repositories (e.g., anthropics/skills, obra/superpowers) were selected based on star and fork counts.
Filtering: From 67,264 public forks, the authors filtered for active forks and compared branch histories against upstream default branches.
Definition: A "skill-adaptation instance" was defined as a downstream branch snapshot where a developer introduced unique modifications to an existing skill package. Whole-skill additions/deletions and upstream synchronization were excluded.
Final Corpus: The study analyzed 1,126 skill-adaptation instances.

Taxonomy Building and Labeling

Inductive Approach: An initial taxonomy was built using a stratified sample of 300 instances, labeled by an LLM agent (claude-opus-4-8) with human oversight.
Iterative Refinement: The taxonomy was expanded over 920 additional records until saturation was reached (no new patterns in consecutive batches).
Validation: A rigorous validation process involved human auditors reviewing a random sample (293 records) plus all low-confidence labels (35 records), totaling 318 audited records. Inter-annotator agreement was high (Cohen's $\kappa$ = 0.83), and the agent's labels showed strong alignment with the human consensus.
Analysis: The final corpus was analyzed quantitatively using metrics for prevalence, skill breadth, upstream coverage, and repository-adjusted prevalence. Security-sensitive patterns were identified using a static rule-based scan adapted from prior work.

Key Contributions

First Empirical Study: The paper provides the first empirical characterization of how downstream developers adapt published LLM agent skills.
Validated Taxonomy: A taxonomy comprising 46 adaptation patterns organized into 13 families (e.g., Lifecycle, Procedure, Decision, Policy, Retarget, Resource).
Open Research Artifacts: The authors release a labeled corpus, taxonomy definitions, labeling workflows, audit materials, and analysis scripts to support future research.
RADAR Checklist: A lightweight, implication-driven review checklist (Route, Adapt, Declare, Audit, Record) to help developers assess skill adaptations.

Key Findings

1. Adaptation Patterns and Prevalence (RQ1)

The study identifies a "reuse paradox": although skills are published for reuse, developers spend significant effort rewriting them.

Most Prevalent Patterns: modify-skill-metadata (18.2%) is the most common, primarily to update the description field for better discoverability. Other top patterns include add-procedure-step, add-decision-rule, and add-hard-constraint.
Dominant Families: The Lifecycle family (40.9% prevalence) and Procedure family (40.8%) are the most widespread.
Implication: Skills are rarely adopted "as is." Developers must manually refit discovery signals and internal procedures to fit local contexts.

2. Component Distribution (RQ2)

Markdown as Control Plane: 79.8% of modifications target the SKILL.md file, the only component touched across all six repositories.
Code as Secondary: Only 23.3% of modifications target executable scripts.
Implication: The natural language instruction file has become the absolute control plane for agent behavior, relegating traditional executable code to a secondary, supporting role.

3. Family Co-occurrence (RQ3)

Adaptations are highly interdependent.

Behavioral Bundles: The most common bundles combine Procedure + Decision (18.7%) and Procedure + Policy (16.4%). This indicates that changes to workflow steps, branching logic, and constraints are tightly coupled.
Structural Bundles: Lifecycle + Retarget (7.6%) shows broad coverage, indicating that porting a skill to a new tool or language naturally requires repackaging and metadata updates.
Implication: Automated support is needed for change-impact analysis to detect inconsistent modifications across coupled families.

4. Security-Sensitive Additions (RQ4)

High Risk: Nearly 18.6% of adaptation instances introduce content matching security-sensitive patterns (e.g., access to sensitive files, external network requests, dangerous commands).
Location: 73.1% of these security-sensitive matches occur in SKILL.md or bundled documentation, not in scripts.
Implication: Security risks are embedded in natural language instructions, bypassing traditional code-focused security review pipelines.

5. Commit-Message Coverage (RQ5)

Incomplete Rationale: Commit messages describe the functional outcome (e.g., "added feature") but rarely explain the underlying customization need.
Weak Signal: The dominant rationale category is "Objective" (76.2%), which is evenly distributed across all adaptation families and fails to distinguish specific adaptation types.
Implication: Commit messages cannot serve as a reliable proxy for inferring the specific nature of skill customization; direct inspection of the diff is essential.

Significance and Implications

The paper argues that the current ecosystem of LLM agent skills suffers from a mismatch between the intended design (plug-and-play reuse) and the reality of evolution (manual, high-effort refitting).

For Researchers: The taxonomy provides variables to study skill improvement, shifting focus from "whether" a skill improved to "which types" of adaptations drove the improvement. It also cautions against using commit messages as ground truth for intent.
For Tool Engineers: There is a need for tools that support recurring customization needs (e.g., metadata editing, retargeting) as first-class operations rather than free-text edits. Tools must also provide change-impact analysis for coupled behavioral edits and surface security-sensitive changes in natural language.
For Developers: The RADAR checklist is proposed to guide developers in auditing security, declaring dependencies, and recording changes systematically.

The study concludes that improving the skill ecosystem requires better abstractions, standardized interfaces, and automated support for adaptation, particularly to address the security risks introduced in natural language instructions and the high coupling between behavioral components.