SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks

1. Problem Statement

Large Language Model (LLM) agents are increasingly using skills—structured documents containing instructions, workflows, and domain knowledge—to perform complex real-world tasks. While pre-built skills exist, agents facing novel tasks require the ability to continually learn and generate new skills from their own experiences (a "generate-store-reuse" cycle).

However, the field lacks a principled way to evaluate automatic skill learning. Existing benchmarks primarily measure final task success (pass/fail) or compare skills in isolation, failing to diagnose why generated skills succeed or fail. Furthermore, it is unclear which continual learning strategies (e.g., self-reflection vs. external feedback) are most effective, or if scaling model size improves skill generation.

SkillLearnBench addresses these gaps by providing the first standardized benchmark for evaluating methods that generate agent skills from scratch.

2. Methodology

A. Benchmark Construction (SkillLearnBench)

The benchmark consists of 20 verified, skill-dependent tasks across 15 sub-domains (e.g., Software Engineering, Data Analysis, Content Creation), derived from a real-world community-driven skill taxonomy.

• Skill Dependency: Tasks are designed such that agents cannot solve them reliably without skills (pass rate < 50% without skills) but can solve them with human-authored reference skills.
• Reusability Testing: Each task includes multiple instances (varying parameters, data formats, or phrasing) to test if a generated skill generalizes beyond the specific instance it was created from.
• Deterministic Verification: Tasks use automated verifiers (e.g., test suites, output checks) rather than subjective grading.

B. Three-Level Evaluation Framework

To diagnose failure points, the authors propose a layered evaluation:

1. Level 1: Skill Quality: Evaluates the generated skill document itself (without execution).
   • Coverage: Fraction of essential knowledge points from human references present in the generated skill.
   • Executability: Completeness, determinism, consistency, and usability.
   • Safety: Risk assessment across six dimensions (privacy, injection, bias, etc.).
2. Level 2: Trajectory Analysis: Evaluates agent behavior when using the skill.
   • Trajectory Alignment: Does the agent follow the correct sequence of steps?
   • Skill Usage Rate: How often does the agent actually invoke the generated skill?
3. Level 3: Task Outcome: The final result.
   • Accuracy: Binary pass/fail rate.
   • Solving Efficiency: Token consumption (lower is better).

C. Baseline Methods Evaluated

The paper compares four continual learning strategies:

1. One-Shot: Single-pass generation from task description (no feedback).
2. Self Feedback: The agent generates a skill, attempts the task, analyzes its own failure, and refines the skill (iterative self-correction).
3. Teacher Feedback: An external "teacher" (with access to human skills) provides guidance after failures, but does not reveal the ground truth.
4. Skill Creator: A structured pipeline (inspired by Anthropic) where the agent follows a multi-stage process (intent analysis, edge case checking, validation) to generate skills.

3. Key Results

A. Performance Gap

• Improvement over Baseline: All continual learning methods significantly outperform the "No Skill" baseline (e.g., ~30% accuracy vs. ~10%).
• Human Gap Remains: Despite improvements, all methods fall far short of human-authored skills. The best method covers only about 45% of the performance gap between "No Skill" and "Human."
• Scaling Laws: Using stronger LLM backbones (e.g., Opus/Pro vs. Haiku/Flash) does not reliably produce better skills. In some cases, stronger models produce more rigid, over-fitted skills that fail on varied instances.

B. Method-Specific Findings

• Self Feedback: Generally achieved the best balance of accuracy and token efficiency. It produces focused skills but struggles to improve significantly with additional iterations due to "recursive drift" (self-revision without new information leads to hallucination or drift).
• Teacher Feedback: Provided the most genuine improvement through external signals, but often resulted in lower skill usage rates (agents ignored the complex instructions) and higher token costs.
• Skill Creator: Generated highly structured skills with high usage rates, but accuracy lagged, suggesting that structure alone does not guarantee correct task logic.
• One-Shot: Surprisingly competitive, often outperforming complex iterative methods, suggesting that for some tasks, a single high-quality attempt is sufficient.

C. Task Dependency

• Workflow vs. Open-Ended: Continual learning excels on tasks with clear, reusable workflows (e.g., Software Engineering, Data Processing).
• Open-Ended Struggles: On open-ended or highly instance-specific tasks (e.g., Creative Writing, Complex Planning), generated skills often hurt performance by over-constraining the agent.

D. Reusability Analysis

• Generated skills often fail to generalize across instances. While they may pass the "seed" instance used for generation, they frequently fail on held-out instances, indicating they capture only partial task logic rather than core principles.

4. Key Contributions

1. SkillLearnBench: The first benchmark specifically for evaluating continual skill learning, featuring 20 tasks with multi-instance testing and a community-driven taxonomy.
2. Three-Level Framework: A novel evaluation protocol that decouples skill quality, execution alignment, and task outcome, allowing for precise diagnosis of failure modes.
3. Empirical Insights:
   • Identified that external feedback is crucial for genuine improvement, whereas self-feedback alone leads to drift.
   • Demonstrated that stronger LLMs do not automatically yield better skills, often due to over-fitting to specific parameters.
   • Highlighted that skill adoption (whether the agent actually uses the skill) is a major bottleneck, often more critical than the skill's content quality.

5. Significance

This work shifts the focus from simply "can the agent solve the task?" to "how well can the agent learn to teach itself?"

• For Researchers: It provides a standardized testbed to compare non-parametric continual learning methods, moving beyond parameter updates to procedural knowledge accumulation.
• For Practitioners: It reveals that current automated skill generation is not yet a drop-in replacement for human expertise. Future systems need to focus on grounding skills in core task logic (not just surface patterns) and ensuring robust agent adoption of generated instructions.
• Open Source: The authors release data and code to facilitate further research into automatic skill generation and agent continual learning.

In summary, while agents can learn to generate skills that improve performance, the current state-of-the-art methods are still far from human-level reliability, particularly in generalizing to new instances and ensuring the agent actually follows the generated instructions.