🤖 AI

CODESKILL: Learning Self-Evolving Skills for Coding Agents

CODESKILL is a reinforcement learning-based framework that learns to extract, evolve, and maintain a compact bank of multi-granularity procedural skills from coding agent trajectories, significantly improving downstream task performance on benchmarks like SWE-Bench Verified compared to existing prompt-based or memory approaches.

Original authors: Yanzhou Li, Yiran Zhang, Xiaoyu Zhang, Xiaoxia Liu, Yang Liu

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanzhou Li, Yiran Zhang, Xiaoyu Zhang, Xiaoxia Liu, Yang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart robot assistant how to fix software bugs. Every time the robot tries to fix a problem, it goes through a long process: it looks at the code, tries a command, sees an error, tries again, and eventually (hopefully) fixes it. This entire journey is called a "trajectory."

The problem is that if you just let the robot keep a diary of every single thing it ever did, the diary would become a messy, overwhelming mountain of paper. Most of it is just specific details about one particular bug that won't help with the next one.

CODESKILL is a new system designed to turn that messy mountain of paper into a sleek, organized instruction manual that the robot can actually use.

Here is how it works, broken down into simple concepts:

1. The Problem: Too Much Noise, Not Enough Signal

Currently, when robots try to learn from their past mistakes, they often rely on rigid rules or fixed prompts (like a teacher who says, "Always do X when you see Y"). But software engineering is complex. Sometimes the robot succeeds by accident, and sometimes it fails for a weird reason. It's hard to tell what is a reusable skill (like "how to fix a broken internet connection") versus just a one-time fluke (like "how to fix the specific typo in this file").

Existing methods are like a librarian who just stacks books on the floor without organizing them. They don't know which books are actually useful for the next reader.

2. The Solution: The "Self-Evolving Librarian"

CODESKILL acts as a smart librarian that manages the robot's knowledge base (called a Skill Bank). Instead of just storing raw stories, it:

Extracts Skills: It reads the robot's past journeys and pulls out the "golden nuggets"—the general rules that apply to many situations.
Evolves Skills: If the robot tries a skill and it fails, the librarian doesn't just throw the skill away. It updates the instruction manual to say, "Oh, this rule works, unless you see this specific error message."
Maintains the Library: It constantly checks the library. If two books say the same thing, it merges them. If a book is too specific or useless, it throws it out. This keeps the library small and efficient.

3. How It Learns: The "Coach and the Athlete"

The most unique part of CODESKILL is how it learns to be a good librarian. It doesn't just guess; it gets feedback from a "frozen" athlete (the coding robot).

The Athlete: This is the coding robot that solves the actual problems. It doesn't change its brain; it just tries to solve tasks.
The Coach (CODESKILL): This is the system that manages the skills.
The Training Loop:
1. The Coach creates a new rule based on past experience.
2. The Coach gives this rule to the Athlete.
3. The Test: Does the Athlete solve the problem faster or better with this new rule?
4. The Reward: If the Athlete succeeds, the Coach gets a high score. If the Athlete fails, the Coach gets a low score.
5. The Coach uses this score to learn: "Okay, I need to write better rules next time."

The paper calls this Reinforcement Learning. It's like training a dog: you don't just tell the dog what to do; you give it a treat when it gets it right, so it learns to repeat the behavior that earns the treat.

4. Two Types of Skills

CODESKILL organizes knowledge into two types of instructions, like a cookbook with two sections:

Task-Level Skills (The Big Picture): These are high-level strategies. Example: "When you see a Java build fail, first check your internet connection, then check your dependencies."
Event-Driven Skills (The Quick Fixes): These are reactions to specific moments. Example: "If you see an error saying 'File Not Found,' immediately check if the file path has a typo."

5. The Results: A Smarter, Faster Robot

The researchers tested CODESKILL on three different "exam rooms" (benchmarks) where coding robots have to solve real-world software problems.

The Baseline: A robot with no instruction manual (just raw intelligence).
The Competition: Robots using older methods of remembering past tasks (like just reading a diary or using fixed prompts).
CODESKILL: The robot using the self-evolving skill library.

The Outcome:
CODESKILL's robot solved significantly more problems than the others.

It improved the success rate by about 9.7% compared to having no skills at all.
It beat the strongest existing "memory" methods by about 4%.
It also solved problems faster, taking fewer steps to reach the solution because it had better instructions to follow.

The Bottom Line

CODESKILL is a framework that teaches a coding robot how to learn how to learn. Instead of just memorizing every single thing that happened, it learns to distill those experiences into a compact, evolving set of rules. It acts like a coach that constantly refines the playbook based on whether the team actually wins the game, ensuring the robot gets better and more efficient over time without needing to be reprogrammed from scratch.

Technical Summary: CODESKILL

Problem Statement

Coding agents generate rich, long-horizon trajectories while solving software engineering (SWE) tasks, containing reusable experience that could theoretically guide future behavior. While recent work has explored memory mechanisms to store or summarize these trajectories, existing approaches to constructing and maintaining procedural skills (compact, reusable knowledge specifying when to act and how) face significant limitations. Current methods rely heavily on fixed prompts and heuristic update rules to determine what knowledge to extract, how to abstract it, and how to update the skill bank. This static approach fails to adapt to the specific needs of downstream agents, particularly in SWE contexts where trajectories contain dense, heterogeneous feedback. Consequently, it remains unclear how to optimally select, abstract, and maintain skills to ensure they are both reusable across tasks and actionable for concrete agent decisions.

Methodology

The authors propose CODESKILL, an LLM-based framework that reformulates skill extraction and skill-bank maintenance as a learnable management policy rather than a fixed heuristic process.

Framework Architecture

CODESKILL manages a skill bank $B$ composed of multi-granularity procedural skills:

Task-level skills: High-level strategies for task families (e.g., repository inspection, issue localization).
Event-driven skills: Local guidance for recurring execution events (e.g., handling specific error messages or command failures).

The framework operates through a loop of three core components:

Skill Extraction: Generates new task-level or event-driven skills from trajectory evidence.
Skill Evolution: Revises existing skills based on new or failed trajectory evidence to address missing cases or failure modes.
Skill-Bank Maintenance: Decides whether to add, merge, or drop candidate skills to keep the bank compact and high-quality.

Training Pipeline

CODESKILL is trained using a hybrid approach combining supervised learning and reinforcement learning (RL):

Supervised Warm-up: The model ( $M_\theta$ ) is fine-tuned on teacher-generated data (using GPT-5.4-mini) to learn the schema for skill extraction, evolution, and maintenance operations.
Reinforcement Learning (RL): The policy is optimized using Group Relative Policy Optimization (GRPO). The training loop involves:
- Hybrid Reward Design: The reward function combines:
  - Dense Rubric-Based Feedback: An LLM-as-judge evaluates skill quality (grounding, reusability, specificity, actionability) and behavior alignment.
  - Sparse Verifiable Execution Feedback: The skill's impact is measured by the performance improvement of a frozen downstream coding agent (e.g., Qwen3.5-35B-A3B) on specific tasks. The reward is the difference between the agent's success with the skill versus a no-skill baseline.
  - Alignment Factor: A mechanism to ensure the agent actually followed the skill's workflow, addressing attribution issues in long-horizon tasks.
- Three-Stage Curriculum: Training progresses from skill extraction only, to extraction + evolution, and finally to the full lifecycle including maintenance, allowing data reuse across stages.

Key Contributions

CODESKILL Framework: A novel LLM-based system that learns to extract, evolve, and maintain reusable procedural skills from coding-agent trajectories, moving beyond static prompt-based skill distillation.
Learnable Management Policy: The authors reformulate skill management as a learnable policy optimized via RL. The reward function uniquely balances dense rubric-based quality judgments with sparse, verifiable execution feedback from a frozen downstream agent.
Empirical Validation: Extensive evaluation on EnvBench, SWE-Bench Verified, and Terminal-Bench 2 demonstrates that learned skill management outperforms both no-skill baselines and strong prompt-based or memory-based baselines.

Experimental Results

The authors evaluated CODESKILL (instantiated with Qwen3.5-4B) against several baselines, including no-skill execution, subtask-level memory, and prompt-based skill management using both open-source (Qwen3.5-4B) and closed-source (GPT-5.4-mini) backbones.

Performance Gains:
- CODESKILL improved the average pass rate by 9.69% over the no-skill baseline.
- It outperformed the strongest prompt-based or memory baseline by 4.01% (a relative gain of ~11%).
- Gains were consistent across in-domain (EnvBench, SWE-Bench) and out-of-distribution (Terminal-Bench 2) tasks.
Efficiency: CODESKILL reduced the average number of reasoning steps on solved instances (from 44.12 to 35.15 under the Qwen3.5-35B-A3B policy), indicating that retrieved procedural guidance helps agents reach solutions more directly.
Generalization: The learned policy generalized effectively to a different frozen downstream agent (GPT-5.4-mini), suggesting the skill-management behavior is not overfitted to a specific solver.
Ablation Studies:
- Both task-level and event-driven skills contributed complementary knowledge.
- The full lifecycle (including maintenance) reduced the skill bank size by nearly half (from 1,252 to 676 skills) while preserving most downstream utility, demonstrating effective redundancy control.
- Supervised fine-tuning (SFT) alone did not generalize well to out-of-distribution tasks, highlighting the necessity of RL with downstream feedback.

Significance and Claims

The paper claims that learned procedural-memory management is a promising direction for coding agents. Unlike methods that rely on fixed prompts or simple retrieval of episodic memories, CODESKILL demonstrates that optimizing a skill-management policy with verifiable downstream feedback allows agents to accumulate and reuse long-horizon software engineering experience more effectively. The work highlights that static skill injection is insufficient; instead, an adaptive, learnable policy is required to ensure skills remain reusable, actionable, and appropriately sized for future task solving.

Limitations Acknowledged by the Authors:

The current system focuses on natural-language instruction skills and does not yet generate executable scripts, APIs, or tool definitions.
The action space is restricted to one operation on one candidate skill at a time, limiting the ability to express complex, multi-skill maintenance decisions (e.g., splitting a broad skill or coordinating multiple updates simultaneously).