💬 NLP

SkillCAT: Contrastive Assessment and Topology-Aware Skill Self-Evolution for LLM Agents

SkillCAT is a training-free framework that enhances LLM agents by employing Contrastive Causal Extraction, Assessment-Augmented Evolution, and Topology-Aware Task Execution to generate, validate, and efficiently deploy reusable skills, achieving up to a 40.40% performance improvement over baselines across various benchmarks.

Original authors: Kunfeng Chen, Qihuang Zhong, Juhua Liu, Bo Du

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kunfeng Chen, Qihuang Zhong, Juhua Liu, Bo Du

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart robot assistant how to do complex tasks, like managing a spreadsheet or answering questions from a document. The robot is already smart, but it needs a "cheat sheet" (a set of instructions or skills) to help it succeed.

The problem with older methods is that they try to write this cheat sheet by watching the robot try a task once. If it succeeds, they write down what it did. If it fails, they might guess what went wrong. Then, they just pile all these notes into one giant, messy manual. When the robot tries a new task, it has to read the entire manual, even if most of it is irrelevant, which confuses it and slows it down.

SkillCAT is a new, smarter way to build this cheat sheet. It doesn't need to retrain the robot; it just organizes the learning process into three distinct stages, like a three-step cooking competition.

1. The "Compare and Contrast" Stage (Contrastive Causal Extraction)

The Old Way: The chef (the robot) tries to bake a cake once. It comes out burnt. The cookbook writer just writes, "Don't burn the cake," without knowing why.

The SkillCAT Way: The chef tries to bake the cake five times with slightly different ingredients.

In one attempt, the cake is perfect.
In another, it's burnt.
SkillCAT looks at the perfect cake and the burnt cake side-by-side. It asks: "What was the one specific thing the chef did differently right before the disaster?"
It ignores the rest of the baking process and focuses only on that critical moment where the success and failure diverged. This ensures the lesson is about the real cause of the problem, not just random noise.

2. The "Test Drive" Stage (Assessment-Augmented Evolution)

The Old Way: The cookbook writer takes a new tip (like "add more sugar") and immediately adds it to the master cookbook, hoping it helps. Sometimes, this new tip ruins a recipe that was already working.

The SkillCAT Way: Before adding a new tip to the master cookbook, SkillCAT puts it in a test kitchen.

It takes the new tip and tries it on the exact same cake recipes the chef has already baked.
The Scorecard:
- If the tip turns a failed cake into a success, it gets a gold star (High Score).
- If the tip keeps a successful cake successful, it gets a silver star (Good Score).
- If the tip turns a successful cake into a failure, it gets a red card (Rejected).
Only the tips that pass this test drive are allowed into the final book. This prevents the robot from learning bad habits that might break things it already knows how to do.

3. The "Smart Index" Stage (Topology-Aware Task Execution)

The Old Way: When the robot needs to bake a cake, it is forced to read the entire 500-page cookbook, including chapters on how to fix a burnt pizza or how to bake a soufflé, even though it's just making a cake. This is overwhelming and distracting.

The SkillCAT Way: SkillCAT organizes the cookbook into a smart, clickable menu.

It builds a map (a topology) of all the skills.
When the robot is asked to bake a cake, the system looks at the menu and says, "Okay, we only need the 'Cake' chapter and the 'Mixing' section."
It loads only those specific pages into the robot's mind. It ignores the pizza and soufflé chapters entirely. This keeps the robot focused, fast, and less confused.

The Results

The researchers tested this system on tasks like fixing spreadsheets, answering questions from tables, and reading documents.

Better Scores: By using this three-step process, the robot improved its average score by up to 40% compared to older methods.
Cross-Model Success: Even if a different robot (a different AI model) used the cheat sheet created by the first robot, it still performed much better than before.
No Training Needed: The best part is that they didn't have to re-teach the robot from scratch. They just organized its past experiences better.

In short, SkillCAT stops the robot from guessing, tests its new ideas before they become rules, and makes sure it only reads the instructions it actually needs at that moment.

Technical Summary: SkillCAT

Problem Statement

Large Language Model (LLM) agents increasingly rely on external skill documents (e.g., structured instructions, tool rules) to execute long-horizon tasks without updating model weights. While recent "skill self-evolution" methods attempt to extract reusable skills from agent execution trajectories, existing pipelines suffer from three critical limitations:

Single-Trace Bias: Current methods often extract skills from a single trajectory per task. A successful trace may reflect accidental strategies, while a failed trace may obscure the root cause of failure, leading to weak or noisy evidence for skill extraction.
Unvalidated Merging: Candidate skill patches are typically merged into a corpus without independent verification. This allows low-quality or harmful patches to enter the skill library, potentially degrading performance on the source tasks from which they were derived.
Context Overload: As the skill corpus grows through merging, inference-time context becomes bloated with irrelevant or conflicting rules. This increases prompt length, distracts the agent, and complicates planning.

Existing approaches, such as Trace2Skill, utilize offline batch pipelines but fail to address these specific issues of evidence quality, patch validation, and context efficiency simultaneously.

Methodology: The SkillCAT Framework

SkillCAT (Contrastive Assessment and Topology-Aware Skill Self-Evolution) is a training-free framework that decomposes the skill lifecycle into three distinct stages: Contrastive Causal Extraction (CCE), Assessment-Augmented Evolution (AAE), and Topology-Aware Task Execution (TTE).

1. Contrastive Causal Extraction (CCE)

CCE addresses the single-trace bias by generating multiple trajectories for each task using multi-seed sampling.

Contrastive Pair Construction: For a given task, CCE constructs pairs of successful and failed trajectories.
Causal Watershed Identification: The system identifies the "causal watershed"—the first point where the action sequences of the success and failure traces diverge.
Record Extraction: Instead of summarizing the entire trace, CCE extracts candidate experience records focused on this divergence point, inferring the failure cause and formulating a skill-editable lesson. If no success/failure pair exists for a task, a fallback mechanism or task skipping is employed.

2. Assessment-Augmented Evolution (AAE)

AAE treats candidate patches as hypotheses to be verified rather than rules to be immediately merged.

Source-Task Replay: Each candidate patch is replayed on "source-task clones" (instances of the original task that generated the evidence).
Outcome Scoring: The system assigns a calibrated score based on the outcome transition:
- Failure $\to$ Success (Score 3.0): Strongest evidence; the patch repairs a failure.
- Success $\to$ Success (Score 2.0): Behavior preservation; the patch maintains a known success.
- Failure $\to$ Failure (Score 1.0): Weak evidence; the patch fails to resolve the issue.
- Success $\to$ Failure (Score 0.0): Rejected; the patch breaks existing functionality.
Hierarchical Merging: Only patches meeting a score threshold (e.g., $\ge 2.0$ ) are retained. These are merged hierarchically from lower to higher score tiers, ensuring that high-value patches (those that repair failures) are prioritized while preventing harmful patches from entering the global skill corpus.

3. Topology-Aware Task Execution (TTE)

TTE addresses context overload by compiling the evolved skill into a routable topology rather than a monolithic document.

Topology Construction: The offline compiler parses the merged skill into a core "always-on" component and a set of routable nodes (Level-2 sections) containing metadata (titles, keywords, dependencies).
Dynamic Routing: At inference time, an LLM router selects only the $k$ most relevant nodes based on the specific task description.
Skill Assembly: The final context is assembled by concatenating the core skill with the bodies of the selected nodes. This ensures the agent loads only the minimal, task-relevant capability nodes, reducing token costs and distraction.

Key Contributions

The paper identifies three key decisions in offline agent skill improvement—evidence extraction, patch validation/integration, and test-time deployment—and proposes SkillCAT as a solution:

Framework Design: A training-free framework that integrates contrastive causal extraction, assessment-augmented evolution, and topology-aware execution.
Problem Mitigation: Directly addresses single-trace bias, unvalidated merging, and inference-time context overload through specific architectural modules.
Empirical Validation: Demonstrates strong performance across spreadsheet, document, and cross-model settings, showing that reliable skill evolution is achievable without model training.

Experimental Results

SkillCAT was evaluated on SpreadsheetBench, WikiTableQuestions (for out-of-distribution generalization), and DocVQA (for multimodal transfer).

SpreadsheetBench: When initialized from human-written skills, SkillCAT achieved 55.50% accuracy (Vrf) with Qwen3.5-35B-A3B, outperforming the Trace2Skill baseline (29.67%) by 25.83 percentage points. In "Skill Creation" settings (starting from weaker parametric skills), SkillCAT improved accuracy by 34.33 points over the baseline.
Out-of-Distribution (WikiTableQuestions): SkillCAT demonstrated robust generalization, reaching 81.55% accuracy with Qwen3.5-35B-A3B, significantly outperforming baselines.
Cross-Model Generalization: Skills evolved by Qwen3.5 models were successfully reused by gemma-4-31B-it and gpt-5.4-mini, showing that the evolved skills are not overfit to the author model.
Multimodal (DocVQA): SkillCAT improved ANLS scores to 0.9159 (Qwen3.5-35B-A3B user), whereas unvalidated methods (Trace2Skill +Error) fell below the "No Skill" baseline, highlighting the necessity of the AAE validation step.
Ablation Studies: Removing any of the three modules (CCE, AAE, or TTE) resulted in performance degradation. Notably, removing AAE caused performance to drop below the baseline, confirming that validation is critical to prevent harmful patches.

Significance and Claims

The paper claims that SkillCAT provides a reliable, training-free pathway for LLM agents to self-evolve skills. By separating the process into evidence extraction, validation, and context-aware deployment, it overcomes the diminishing returns and noise accumulation seen in sequential update loops. The authors emphasize that the framework's ability to generalize across models and domains, while simultaneously reducing inference context through topology-aware routing, offers a practical solution for scaling agent capabilities without the computational cost of fine-tuning. The results suggest that high-quality skill evolution depends heavily on the quality of evidence (CCE) and the rigor of validation (AAE), rather than simply accumulating more trajectory data.