Technical Summary: SkillEvolBench

Problem Statement

Large Language Model (LLM) agents accumulate rich episodic trajectories while solving real-world tasks, yet it remains unclear whether this experience can be distilled into reusable procedural skills. While prior work demonstrates that agents can store and reuse episodic traces (e.g., Reflexion, ExpeL, Synapse) or utilize curated skills to improve performance, a critical gap exists between experience reuse and skill formation. Reusing an episode is distinct from extracting a procedure; a trajectory records a specific instance often mixed with incidental details, failed hypotheses, and mistakes, whereas a skill requires an explicit procedural form stating what to do, when, and how to verify it. Current benchmarks often measure task completion or the utility of pre-curated skills but fail to diagnose whether agents can autonomously convert noisy, one-off experiences into durable, reusable procedural knowledge that generalizes beyond the original episode.

Methodology

The authors introduce SkillEvolBench, a diagnostic benchmark designed to evaluate the evolution from episodic experience to procedural skills. The benchmark spans 180 tasks across six real-world agent environments (Code, Tool/API, Data, Documents, Research, Communication), organized into five role-conditioned task families per environment.

Experimental Protocol

The evaluation follows a lifelong episode structure for each environment:

1. Acquisition Phase: Agents attempt three learning tasks (Canonical, Enriched, Variant) within a family.
   • Evidence Generation: Each attempt produces a compacted trajectory summary and structured verifier feedback (outcome, process checks, diagnostics).
   • Skill Authoring: A host-side "Skill Author" (a separate LLM call) reviews the evidence and the current family skill state to decide whether to write a new skill, refine an existing one, or leave the library unchanged.
   • Conditions: The benchmark tests three initialization conditions:
     • Experience-based Self-Generated: Starts with an empty library; skills are induced from execution evidence.
     • Curated-Start: Starts with a "gap-exposed" curated skill (useful but incomplete) that may be refined.
     • Zero-Shot: Generates a metadata-only skill before execution.
2. Frozen Deployment Phase: The skill library is frozen. Agents face harder evaluation tasks designed to test transfer:
   • Context Shift: The skill need is embedded in a broader request.
   • Adversarial: Tasks include shortcut solutions that pass shallow checks but fail deep verification.
   • Composition: Tasks require combining the target skill with other skills.
3. Replay Phase: The original acquisition tasks are re-run using the final frozen library to measure local recovery versus transfer.

Controls and Metrics

To isolate procedural abstraction from base capability and direct trace reuse, the benchmark compares skill-based conditions against:

• No-Skill: No persistent memory.
• Raw-Trajectory: Direct retrieval of compacted episodic traces without distilling them into procedural skills.

Performance is measured via Success Rates (SR) across specific subsets:

• LSR: Learning Success Rate (acquisition tasks).
• RSR: Replay Success Rate (original tasks with frozen library).
• ESR: Frozen Deployment Success Rate (Context Shift, Adversarial, Composition).
• Tier-3 Analysis: A capacity diagnostic forcing the inclusion of resource files (scripts, references, assets) to test if larger libraries improve performance.

Key Contributions

1. SkillEvolBench Benchmark: A diagnostic framework specifically targeting the "missing step" of skill formation, distinguishing between local task recovery and the creation of transferable procedural artifacts.
2. Role-Conditioned Task Arcs: A design that separates acquisition, replay, transfer, context shift, adversarial robustness, and composition, exposing failure modes that a single success rate would obscure.
3. Comparative Controls: The inclusion of "Raw-Trajectory" and "No-Skill" baselines allows for the separation of procedural abstraction benefits from the mere availability of episodic traces or base model capabilities.
4. Capacity Diagnostic: The Tier-3 ablation tests whether increasing library size (adding scripts, references, assets) compensates for abstraction losses, revealing the limits of storage versus selective abstraction.

Results

Across ten model configurations (including Claude Opus/Sonnet, GPT-5/Codex, and Gemini) and three agent harnesses, the study yields the following findings:

• Local Adaptation vs. Robust Skill Formation: Current agents often adapt locally (improving LSR or RSR) but rarely form robust, reusable skills. Gains in acquisition or replay do not consistently transfer to frozen deployment tasks (ESR).
• The Lossy Abstraction Bottleneck: In many cases, Raw-Trajectory controls outperform distilled skills. This suggests that current abstraction procedures discard contextual and procedural cues that remain useful for future tasks. Agents often use episodic traces more effectively than the skills derived from them.
• Instability of Skill-Based Gains: While skill-based conditions can improve specific axes (e.g., Context Shift or Adversarial robustness) for specific models, these gains are unstable. A skill that improves replay success may degrade performance on adversarial or composition tasks.
• Capacity is Not the Solution: Increasing library capacity (via Tier-3 resource bundling) does not reliably improve frozen deployment success. Larger libraries often introduce "episode-specific drift" and "procedural clutter," where added resources encode details specific to the acquisition context that hinder generalization.
• Model and Environment Dependence: The efficacy of skill evolution is highly dependent on the base model's ability to interpret memory and the structure of the environment. Structured environments (e.g., Code, Documents) show more reliable skill evolution than open-ended communication tasks.

Significance and Claims

The paper positions SkillEvolBench as a testbed for measuring when one-off experience becomes durable procedural knowledge rather than task-local memory. The authors claim that the core challenge is not simply storing more experience, revising skills more often, or providing larger resource libraries. Instead, the limiting factor is selective procedural abstraction: agents must preserve details that support future invocation, verification, robustness, and composition while filtering out local repairs and episode-specific noise.

The findings suggest that current agent systems are capable of "patching" specific tasks but struggle to "learn" generalizable procedures. The benchmark provides a diagnostic tool to identify when skill abstraction loses necessary information, guiding future research toward mechanisms that can distinguish between transferable procedures and incidental episodic details.