🤖 AI

SkillAlchemy: Open-World Agent Skill Creation

SkillAlchemy is a novel framework that automates the creation of open-world agent skills by identifying implicit requirements through contrastive evidence and compiling source-grounded procedures, achieving performance comparable to human-curated skills and significantly outperforming existing baselines on the SkillsBench.

Original authors: Hengjun Wang, Shuyue Wei, Boyi Liu, Jun Yang, Yongxin Tong

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hengjun Wang, Shuyue Wei, Boyi Liu, Jun Yang, Yongxin Tong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, there is a growing distinction between what a computer model knows and what it can actually do. Large language models are trained on vast libraries of text, giving them a broad, general understanding of the world. However, when asked to perform a specific, multi-step task—like analyzing a complex dataset or navigating a specialized software environment—they often stumble because they lack the precise, step-by-step instructions required for that specific job. To bridge this gap, researchers have developed the concept of "skills." Think of a skill not as a new piece of knowledge, but as a reusable tool or a pre-written manual that an agent can load to handle a specific workflow. These skills allow an AI to move beyond its general training and execute specialized procedures, much like a carpenter picking up a specific chisel for a delicate cut rather than trying to hammer a nail with a screwdriver. For these tools to be useful, they must be reliable, accurate, and applicable to a wide range of situations, not just the single example they were built on.

The challenge, however, has been how to create these skills when the task is unfamiliar. Traditionally, creating a skill required a human expert to write the manual from scratch, or for the AI to learn from a record of past successful attempts. But what happens when the task is new, and no human expert is available, and no past attempts exist to study? In such cases, the necessary information often exists scattered across the open internet in the form of documentation, code repositories, and forum discussions. The problem is that this information is messy. It is often written for a specific person, for a specific moment, or for a specific computer setup. It contains hidden assumptions and local details that do not apply to the general task at hand. If an AI simply copies these instructions, it often creates a tool that is too rigid, failing the moment the situation changes slightly.

A team of researchers has proposed a new approach to solving this problem, calling it SkillAlchemy. Instead of treating the internet as a library of ready-made answers, they treat it as a source of raw evidence that must be carefully examined and tested before it can become a usable tool. Their method starts with a simple, often vague request for a skill, such as "plan a trip" or "calibrate a model." Rather than immediately searching for a solution, the system first asks what is missing from that request. It identifies the hidden requirements that the human prompt did not state, such as the need to check for budget limits or to verify that a specific file format is compatible with the software being used. By turning these missing pieces into specific questions, the system knows exactly what to look for in the vast ocean of online information.

Once the system knows what to look for, it gathers information from various sources, but it does not accept everything it finds. The researchers designed a rigorous process to decide which pieces of information are trustworthy enough to be included in the final skill. The system looks for evidence that supports a procedure across different situations. If a method works only for one specific type of computer or one specific set of data, the system recognizes it as a local example and keeps it separate from the main instructions. It only promotes a procedure to a general rule if the evidence shows it works reliably in multiple contexts. This process acts as a filter, separating the universal principles from the temporary details. The system then compiles the approved instructions into a structured package, complete with clear steps, safety checks, and references to the evidence that supports each decision.

The researchers tested this method on 87 different tasks, ranging from software engineering and office work to financial analysis and scientific computing. They compared their system against several other methods, including those that rely on human-written skills and those that use automated tools to generate instructions directly from the internet. The results showed that the new method significantly outperformed the automated baselines. When the agents used the skills created by this system, they successfully completed tasks nearly 20 percentage points more often than when they had no skills at all. More importantly, the system performed comparably to the human-curated skills, which are considered the gold standard. In the aggregate, the automated system achieved a success rate 1.5 percentage points higher than the human-curated skills, though statistical analysis indicates these results are closely matched. Additionally, the system achieved the highest overall performance in three of the four agent-model configurations tested.

The study also revealed why previous attempts often failed. When systems simply grabbed information from the web without checking its scope, they frequently included instructions that were too specific or based on conflicting evidence. For example, a skill might include a rule that only works on a specific version of a software program, causing the agent to fail when that version was not available. The new method successfully avoided these traps by explicitly checking whether a procedure was supported by evidence across different scenarios. It also proved to be robust when faced with confusing or misleading information. When the researchers introduced conflicting or irrelevant documents into the search, the system correctly ignored them, whereas other methods often incorporated the bad advice into their final skills.

This work suggests a fundamental shift in how we might build intelligent agents. Instead of hoping that an AI can magically figure out how to do a new job, or relying on humans to write every manual, we can teach the AI to act like a careful researcher. It learns to ask the right questions, gather evidence, and distinguish between a general rule and a local exception. By grounding the creation of skills in verified evidence rather than direct copying, the system produces tools that are both flexible and reliable. The researchers found that this approach allows agents to handle unfamiliar tasks with a level of competence that was previously only possible with human intervention. While the system is not perfect and still struggles with some highly specialized media tasks, the results demonstrate that reliable skill creation is possible even when the starting information is incomplete and the sources are messy. The key lies not in finding the perfect answer immediately, but in the disciplined process of verifying what is true and what is merely a local detail.

Technical Summary: SkillAlchemy – Open-World Agent Skill Creation

1. Problem Definition

The paper addresses the challenge of creating reliable agent skills—reusable procedural artifacts that extend language agents with specialized workflows and domain behaviors—at inference time. While existing methods rely on human authorship, model priors, or execution traces, these sources are often unavailable for unfamiliar tasks.

The authors identify a gap in open-world skill creation: given an underspecified task brief and access to heterogeneous open-world sources (e.g., documentation, repositories, issue reports), how can an agent generate a reusable skill? The authors diagnose two primary failure modes in current approaches:

Underspecified Briefs: Task briefs often omit implicit operational requirements, failure modes, and boundaries necessary for a reusable skill.
Unjustified Scope: Open-world findings often mix reusable practices with local details (hardcoded parameters, specific tools). Directly adopting these without validating their scope leads to over-specific instructions that fail when generalized.

The core problem is formulated as a source-grounded procedure-admission problem: determining which candidate procedures should be admitted as general instructions, retained as scoped examples, or excluded based on evidence from open-world sources.

2. Methodology: SkillAlchemy

The authors propose SkillAlchemy, an admission-centered framework that transforms underspecified descriptions into agent-usable skills through three sequential stages. Unlike simple retrieval pipelines, SkillAlchemy explicitly separates the discovery of requirements, the validation of scope, and the compilation of the artifact.

Stage 1: Implicit Requirement Discovery ( $\Phi_1$ )

This stage lifts the task brief to an operational frame to identify omitted dimensions.

Operational Framing: The brief is converted into candidate operational factors (e.g., changing "7-day family trip" to "duration," "traveler type," and "destination").
Contrastive Evidence Acquisition: To determine if a factor is an implicit requirement, the system constructs paired acquisition targets ( $h = \langle d, x, x' \rangle$ ) where contexts differ along one factor $d$ .
Contrastive Analysis: The system queries sources to see if the procedural behavior (conditions, actions, recovery, verification) changes between $x$ and $x'$ . If evidence shows a difference, an implicit requirement is recorded. If evidence is consistent across contexts, it is treated as invariance.

Stage 2: Evidence-Grounded Procedure Admission ( $\Phi_2$ )

This stage aggregates findings and decides the status of candidate procedures.

Decision-Aligned Induction: Findings are grouped by the procedural decision they inform (e.g., route selection, failure handling).
Scope-Aware Admission: For each candidate procedure $\pi_j$ $π_{j}$ , the system constructs an admission record determining its widest justified scope ( $\sigma_j$ $σ_{j}$ ).
- General: Admitted as a reusable instruction if supported and consistent across non-equivalent contexts or explicitly stated as broadly applicable.
- Scoped: Retained as a context-bound example if supported only under narrow conditions.
- Exclude: Rejected if unsupported or conflicting.
This stage ensures that only procedures justified by evidence beyond a single local case are promoted to general instructions.

Stage 3: Skill Package Compilation ( $\Phi_3$ )

The admitted content is compiled into an installable skill package (centered on a SKILL.md file).

Grammar-Guided Rendering: A corpus-derived skill grammar ( $G_{skill}$ ) guides the organization of the package. It dictates presentation patterns for descriptions, executable sequences, applicability conditions, and safeguards.
Progressive Disclosure: Core instructions are placed in SKILL.md, while detailed evidence, scoped examples, and scripts are externalized into bundled resources (references/, scripts/, assets/) to keep the runtime context focused.

3. Key Contributions

Problem Formulation: The paper formulates open-world skill creation as a source-grounded procedure-admission problem, highlighting the dual challenges of implicit requirement discovery and procedure-scope justification.
SkillAlchemy Framework: A novel framework that operationalizes the admission process through contrastive evidence acquisition and scope-aware decision-making, rather than direct retrieval or summarization.
Empirical Evaluation: Extensive experiments on 87 tasks from SkillsBench v1.1 across four agent-model configurations.

4. Experimental Results

The evaluation compares SkillAlchemy against no-skill execution, human-curated skills, and four automated baselines (Anthropic Skill-Creator, OpenAI Skill-Creator, OpenSkill, MUSE-Autoskill).

Performance: SkillAlchemy improves the pass rate by 19.9 percentage points (pp) over no-skill execution and by 8.6 pp over the strongest automated baseline (MUSE-Autoskill).
Comparison to Human Skills: SkillAlchemy achieves an aggregate pass rate of 55.8%, which is comparable to (and in some configurations slightly higher than) human-curated skills (54.4%).
Robustness: In perturbation tests involving irrelevant, conflicting, or adversarial source documents, SkillAlchemy promoted 0/12 injected payloads, whereas baselines promoted 9/16 conflicting payloads. It maintained high downstream pass rates (17–18/20) even under conflict.
Ablation Studies: Removing any of the three core components (requirement discovery, structured findings, or procedure admission) resulted in performance drops of 5.0–15.7 pp, confirming the necessity of each stage.
Cost: SkillAlchemy requires approximately 23 minutes per task for creation, which is more efficient than OpenSkill and MUSE-Autoskill (~~35–36 mins) but slower than direct skill creators (~~6.5 mins). Execution costs are comparable to other methods.

5. Significance and Claims

The paper claims that reliable skill creation from open-world sources requires treating knowledge as evidence to be admitted under explicit scope rather than instructions to be copied directly.

Shift in Paradigm: The work moves beyond "retrieval-as-skill" to "evidence-admission-as-skill," demonstrating that open-world sources are informative but not inherently skill-ready due to implicit decisions and local context.
Bridging the Gap: SkillAlchemy demonstrates that automated systems can approach the performance of human-curated skills by systematically discovering implicit requirements and validating the scope of procedures before compilation.
Generalizability: The framework is designed to handle heterogeneous sources and underspecified briefs, making it applicable to a wide range of tasks where expert-authored skills or execution traces are unavailable.

The authors conclude that their approach provides a robust mechanism for converting raw open-world materials into reliable, reusable agent artifacts, effectively narrowing the gap between automated and human-curated skill creation.

Technical Summary: SkillAlchemy – Open-World Agent Skill Creation

1. Problem Definition

2. Methodology: SkillAlchemy

Stage 1: Implicit Requirement Discovery (Φ1\Phi_1Φ1​)

Stage 2: Evidence-Grounded Procedure Admission (Φ2\Phi_2Φ2​)

Stage 3: Skill Package Compilation (Φ3\Phi_3Φ3​)