💻 computer science

SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystems

The paper introduces SkillOps, a low-overhead, method-agnostic framework that treats LLM agent skill libraries as self-maintaining software ecosystems by representing skills as typed contracts and diagnosing library health to significantly improve task success rates without requiring additional task-time model calls.

Original authors: Hongji Pu, Xinyuan Song, Liang Zhao

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Hongji Pu, Xinyuan Song, Liang Zhao

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a digital toolbox for a robot helper (an AI agent). This toolbox contains "skills"—little programs that tell the robot how to pick up a cup, open a door, or search for an email.

At first, the toolbox is perfect. But as you keep adding new tools, fixing old ones, and reusing them in different situations, the toolbox starts to get messy. You might have:

Duplicates: Two hammers that do the exact same thing.
Broken handles: Tools that look fine but snap when you try to use them.
Wrong shapes: A square peg trying to fit into a round hole (a tool that outputs data the next tool can't understand).
Missing instructions: Tools that don't have a "safety check" to see if they worked correctly.

In the software world, this mess is called "technical debt." It's like ignoring a leaky roof; the house still stands today, but eventually, the whole thing might collapse.

The Problem: "Skill Technical Debt"

The authors of this paper noticed that while AI researchers are great at helping robots find the right tool for a job right now, they aren't very good at cleaning up the toolbox itself over time.

Existing systems try to fix a broken tool while the robot is working (like a mechanic fixing a car while it's driving). But the real problem is that the toolbox itself is getting clogged with bad tools, making it harder for the robot to find the good ones later.

The Solution: SkillOps (The "Toolbox Janitor")

The paper introduces SkillOps, a system that acts like a self-maintaining janitor for the AI's skill library. Instead of just handing the robot a messy pile of tools, SkillOps organizes, repairs, and cleans the toolbox before the robot even starts its work.

Here is how it works, using simple analogies:

1. The "ID Card" System (Skill Contracts)

Every tool in the toolbox gets a strict ID Card (called a Skill Contract). This card lists:

What you need to use it (Preconditions).
What it actually does (Operation).
What it produces (Artifacts).
A safety checklist (Validators).
Known ways it can fail (Failure Modes).

If a tool doesn't have a safety checklist, the system knows it's risky. If two tools have the same ID, the system knows they are duplicates.

2. The "Family Tree" Map (Hierarchical Skill Ecosystem Graph)

SkillOps doesn't just look at tools one by one; it draws a map (a graph) showing how tools connect.

Dependency: "Tool A must be used before Tool B."
Compatibility: "Tool A's output fits Tool B's input."
Redundancy: "Tool C is just a copy of Tool A."

This map helps the system see the big picture. If a tool at the top of the chain is broken, the system knows that every tool relying on it is also at risk.

3. The "Health Check" (Diagnosis)

SkillOps constantly runs a health check on the toolbox, looking at five specific areas:

Utility: Is this tool actually being used? If not, maybe throw it out.
Redundancy: Do we have too many copies? If so, merge them.
Compatibility: Do the tools actually fit together? If not, build a bridge (an adapter).
Failure Risk: Does this tool break often? If so, fix it or retire it.
Validation Gap: Does this tool have a safety check? If not, add one.

4. The "Cleanup Crew" (Maintenance Actions)

Based on the health check, SkillOps takes action without needing a human or a super-intelligent AI to micromanage every step. It uses simple rules to:

Merge: Combine duplicate tools into one.
Repair: Fix broken code using feedback from past failures.
Retire: Throw away tools that are rarely used or always fail.
Add Safety: Insert safety checks where they are missing.
Add Adapters: Build little bridges so tools with mismatched shapes can still work together.

Why It's Special

The paper highlights three main wins:

It Works Better: When tested on a household robot benchmark (ALFWorld), the robot using the "cleaned" toolbox succeeded 79.5% of the time. The best previous methods only succeeded about 70% of the time.
It's Cheap: The "janitor" doesn't need to call a super-intelligent AI (LLM) to do the cleaning. It uses simple, fast computer rules. This means it costs almost nothing in terms of computing power or money to keep the toolbox clean.
It's Plug-and-Play: You don't need to rebuild your robot's brain to use this. You just swap the messy toolbox for the clean one, and the robot works better immediately.

The Bottom Line

The paper argues that we should stop treating AI skill libraries as static piles of code. Instead, we should treat them like living software ecosystems that need regular maintenance. SkillOps is the framework that does this maintenance automatically, ensuring that as an AI agent grows older and more complex, its toolbox stays organized, safe, and efficient.

Technical Summary: SkillOps

Problem Statement: Skill Technical Debt

As Large Language Model (LLM) agents increasingly rely on skill libraries to solve complex, multi-step tasks, these libraries evolve from static retrieval pools into persistent software assets. However, as skills are added, patched, reused, and linked to changing dependencies, libraries accumulate skill technical debt. This failure mode refers to persistent library-level defects—such as redundancy, missing validators, interface drift, and stale implementations—that may not cause a single skill to fail locally but degrade the reliability of future retrieval, composition, and execution.

Existing skill-based agents primarily focus on task-time operations (retrieval, planning, and on-the-fly repair). They often assume the underlying library is healthy, a premise that weakens as the library grows. Consequently, task-time repairs fix individual episodes without addressing the root cause: the persistent defects within the library itself. The paper identifies a gap in library-time maintenance, where the skill library is treated as a managed software ecosystem rather than a static resource.

Methodology: The SkillOps Framework

The authors propose SkillOps, a method-agnostic, plug-in framework designed to maintain skill libraries as self-sustaining software ecosystems. SkillOps operates via two alternating loops: a Task-Time Loop for execution and a Library-Time Loop for maintenance.

1. Core Representations

Skill Contract: Each skill $s$ $s$ is modeled as a typed executable contract $(P, O, A, V, F)$ $(P, O, A, V, F)$ :
- $P$ : Preconditions.
- $O$ : Executable operation.
- $A$ : Produced artifacts.
- $V$ : Validators (checks for $A$ ).
- $F$ : Known failure modes.
- A "validation gap" exists if $V = \emptyset$ .
Hierarchical Skill Ecosystem Graph (HSEG): Skills are organized into a graph $L = (S, R)$ $L = (S, R)$ with four typed directed relations:
- Dependency ($dep$): Artifact of $s_i$ satisfies precondition of $s_j$ .
- Compatibility ($comp$): Output type of $s_i$ matches input type of $s_j$ .
- Redundancy ($red$): Equivalent interfaces ( $P_i \equiv P_j, A_i \equiv A_j$ ).
- Alternative ($alt$): Same goal but different operations.

2. The Task-Time Loop

This loop plans and executes tasks using the maintained library:

Skill Matching: Scores skills using a hybrid of lexical (BM25) and semantic relevance, filtering for satisfied preconditions.
Dependency Stitching: Constructs plans by traversing the HSEG, enforcing both dependency and compatibility constraints to avoid interface mismatches at runtime.
Validator and Adapter Insertion: Dynamically inserts validator nodes for skills with gaps ( $V=\emptyset$ ) and adapter nodes for type mismatches between dependent skills.
Local Repair: Substitutes failed skills with alternatives or re-invokes repair based on error traces.

3. The Library-Time Loop

This loop diagnoses and repairs the library after execution, using observable signals rather than additional LLM calls:

Five-Dimensional Health Diagnosis: Skills are scored on:
1. Utility: Success rate of recent calls.
2. Redundancy: Size of the redundancy cluster.
3. Compatibility: Fraction of dependency edges that are also compatible.
4. Failure-Risk: Empirical failure rate.
5. Validation-Gap: Presence of missing validators.
ContractGraph-Propagated Diagnosis (CGPD): An advanced mechanism that propagates risk scores along dependency edges. If an upstream skill has high risk, downstream skills inherit this risk, triggering preventive maintenance (e.g., adding validators) before failures occur.
Typed Maintenance Actions: Based on health scores and CGPD, the system applies specific actions:
- merge: Collapses redundant skills.
- repair: Rewrites operations using feedback.
- retire: Removes obsolete or failing skills.
- add_validator: Inserts missing validators.
- add_adapter: Inserts type-conversion shims.
- instantiate: Binds task-specific arguments.

Key Contributions

Formalization of Skill Technical Debt: The paper defines skill-library maintainability as a distinct library-time problem, introducing the HSEG and health dimensions (utility, redundancy, compatibility, risk, validation) to capture library degradation.
SkillOps Framework: A drop-in maintenance system that transforms raw libraries into maintained ones via typed actions (merge, repair, retire, etc.) without requiring changes to downstream agent code.
Method-Conditional Effects: Empirical evidence showing that maintenance benefits are contingent on the agent type; it significantly aids retrieval-heavy agents while having neutral or conflicting effects on agents with strong task-time self-repair capabilities.
Low-Overhead Implementation: The current rule-based implementation performs maintenance with nearly zero LLM calls or tokens at library time, positioning maintenance as a lightweight architectural layer rather than an inference burden.

Experimental Results

The framework was evaluated on ALFWorld, a text-based household manipulation benchmark, using a skill library ranging from 200 to 2000 skills (including synthetic technical debt).

Standalone Performance: As a standalone agent, SkillOps achieved a 79.5% task success rate (SR), outperforming the strongest baseline (LLM_Skill_Planner) by +8.8 percentage points without using additional task-time LLM calls.
Plug-in Effectiveness: When used as a maintenance layer for existing baselines, SkillOps consistently improved retrieval-heavy methods:
- Hybrid Retrieval: +2.90 pp
- BM25 Only: +1.00 pp
- Dense Only: +1.12 pp
- The gains were smaller for graph-based or LLM-planning baselines, which already possess some filtering capabilities.
Scalability and Robustness: SkillOps remained stable as library size increased to 2000 skills and degradation density rose to 90%, maintaining a lead of over 31 pp over the next-best baseline. In contrast, task-time-only baselines degraded significantly under noise.
Cost Efficiency: The maintenance loop incurred nearly zero LLM calls. Task-time token usage was generally neutral or reduced (up to -3.95% for Dense Only) because the maintained library provided a cleaner, less redundant candidate pool for retrieval.

Significance and Claims

The paper argues that skill libraries should be treated as managed software assets rather than static retrieval pools. By introducing a library-time maintenance layer, SkillOps addresses the accumulation of technical debt that undermines agent reliability over time.

The authors claim that:

Maintenance is a distinct architectural layer: It can be decoupled from task-time planning and integrated into existing systems as a "drop-in" component.
Typed contracts and graph structures are essential: They enable the detection of interface mismatches and redundancy that flat retrieval misses.
Rule-based maintenance is viable: High-quality library maintenance does not necessarily require expensive LLM inference; observable signals and rule-based actions can effectively clean and repair the ecosystem.
Context matters: The value of library-time maintenance is highest for agents relying heavily on retrieval, whereas agents with robust task-time self-repair mechanisms may see diminishing returns or conflicts.

The work suggests a shift in LLM agent design: moving from purely reactive, task-time repair to proactive, library-time ecosystem management to ensure long-term agent reliability.