🤖 AI

SkillForge: Self-Distilling Agents for Project-Specific Issue Resolution

SkillForge is a self-distillation framework that proactively enhances large language model agents' ability to resolve software issues in specific repositories by synthesizing and solving artificial problems derived from core functionalities, thereby distilling reusable, entity-grounded project knowledge without relying on historical repair data or costly test-time exploration.

Original authors: Silin Chen, Han Li, Xiaodong Gu, Yuling Shi, Haibing Guan

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Silin Chen, Han Li, Xiaodong Gu, Yuling Shi, Haibing Guan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, intricate landscape of modern software, code is the foundation upon which our digital world rests. When something breaks in a complex program, fixing it often requires a deep understanding of not just the broken piece, but how that piece fits into the entire machine. For decades, humans have been the primary mechanics for this work, but recently, artificial intelligence has begun to take up the tools. Large language models, which are powerful computer programs trained on massive amounts of text and code, have shown they can navigate these digital repositories, find bugs, and write fixes. However, a significant hurdle remains: these AI agents often struggle when faced with a specific, unfamiliar project. They lack the local context—the unwritten rules, the specific ways different parts of the code talk to each other, and the unique history of how that particular software was built. Without this insider knowledge, the AI is like a brilliant mechanic dropped into a garage they have never seen, forced to guess how the engine works every time they encounter a new problem.

Researchers at Shanghai Jiao Tong University have developed a new approach to solve this problem, a method they call SkillForge. Instead of waiting for the AI to stumble through real-world errors and learn from its mistakes over time, this system proactively teaches the AI about the specific project before it ever attempts a real fix. The team realized that to understand a project, the AI needs to practice on it first. So, they created a training ground where the AI generates its own fake problems. By taking the core functions of a software project—those parts that are already tested and known to work—and asking the AI to rewrite them from scratch, the system forces the AI to make mistakes. Because the AI does not have access to the original code, it must rely on its general knowledge, which inevitably leads to errors that are specific to how that project is actually built. These errors, which the researchers call synthetic issues, are then used as a learning tool. The AI attempts to fix these fake bugs, and in doing so, it reveals the hidden patterns and specific rules of that software project.

The system then takes the paths the AI took to solve these fake problems and distills them into a set of "skills." Think of these skills as a personalized guidebook for that specific software. The researchers organized this knowledge into two types. The first type is a high-level map that helps the AI understand the overall structure of the project, such as what different parts of the code are supposed to do and how they usually interact. The second type is a set of specific instructions for when the AI is actually editing the code, warning it about common pitfalls and reminding it of the project's unique quirks. When the AI is later asked to solve a real problem in that same software, it consults this guidebook. It first loads the high-level map to get oriented, and then, as it digs into the code, it receives just-in-time reminders about the specific rules it needs to follow. This happens automatically, ensuring the AI has the right context exactly when it needs it.

The researchers tested this method on a wide range of real-world software projects, using two different powerful AI models to see if it worked. They compared their system against other methods that try to help AI learn from past human fixes or by exploring the code while solving a problem. The results were clear: the SkillForge approach consistently outperformed the others. On a standard test of software engineering tasks, the system improved the success rate of the AI by nearly six percentage points compared to a baseline model that had no special project knowledge. This improvement held true even when the researchers tested the system on a different, more difficult set of challenges. The study showed that by teaching the AI about the specific project before the real work begins, the system could solve problems more accurately and efficiently than methods that tried to learn on the fly or relied on historical data that might not cover the current issue.

One of the most revealing aspects of the research was observing how the AI's behavior changed with this new knowledge. In a specific case involving a complex formatting bug in a popular web framework, an AI without this training quickly settled on a simple, incorrect solution that looked right at first glance but failed when tested against the full range of requirements. It missed the subtle, project-specific logic required to handle the data correctly. In contrast, the AI equipped with the SkillForge guidebook took a more careful, structured approach. It paused to consider the specific rules of the project, avoided a common trap that the untrained AI fell into, and ultimately found a solution that passed all the necessary tests. This demonstrated that the system was not just giving the AI more information, but was fundamentally changing how it reasoned about the code, helping it to avoid the kinds of mistakes that come from applying general rules to a specific, unique environment.

The researchers also explored whether the knowledge gained from one AI model could be transferred to another. They found that the skills distilled by one model were most effective when used by that same model. This suggests that the knowledge is deeply tied to how a specific AI thinks and writes code, rather than being a universal set of facts about the software. This finding highlights that the value of the system lies in its ability to capture the specific way a particular AI interacts with a specific project. Furthermore, the study showed that the system works best when it focuses on rewriting multiple connected parts of the code together, rather than just single functions in isolation. This approach captures the complex interactions between different parts of the software, which is where the most critical project-specific knowledge often hides.

Ultimately, this work offers a new way to bridge the gap between powerful general AI and the specific needs of real-world software engineering. By shifting from a reactive approach, where the AI learns from mistakes after they happen, to a proactive one, where it learns the rules of the game before the match begins, the researchers have shown a path to more reliable and capable software agents. The system does not require a massive history of past human fixes to work, nor does it demand that the AI waste time and resources exploring the code blindly during the actual repair process. Instead, it builds a targeted, efficient knowledge base that allows the AI to hit the ground running. The results suggest that for AI to truly become a partner in software development, it must be given the chance to learn the specific language and customs of the project it is working on, and SkillForge provides a practical, effective way to do just that.

Technical Summary: SkillForge

Problem Statement

Large Language Model (LLM) based agents have demonstrated significant proficiency in automated software issue resolution, particularly on standard benchmarks like SWE-bench. However, a critical bottleneck arises when deploying these agents on specific, real-world repositories: the cold-start problem. Agents typically lack project-specific knowledge, such as structural regularities, recurring failure patterns, specific API contracts, and implicit execution paths unique to a codebase.

Existing approaches to bridge this gap are reactive rather than proactive:

History-driven methods (e.g., EvoCoder, SWE-Exp) distill knowledge from historical issue-resolution trajectories. These are limited by the coverage and quality of past issues; they cannot learn about repository behaviors or components that have not yet been exercised by historical data.
Online methods refine agents during test-time exploration on the current issue. While effective, these incur substantial per-issue costs in terms of tokens, time, and computational resources.

The paper argues that waiting for real issues to accumulate or relying on expensive online exploration is insufficient. There is a need for a framework that proactively acquires project-specific knowledge from the repository itself before attempting to solve real issues.

Methodology: SkillForge

SkillForge is a self-distillation framework designed to proactively acquire project-specific knowledge by synthesizing issues directly from the repository's code and tests. The framework operates through four distinct stages:

1. Project-Specific Issue Synthesis

Instead of waiting for real bug reports, SkillForge generates synthetic issues by rewriting core functionalities of the repository.

Test-Driven Scope: The framework identifies test cases covering core functionalities and executes them under coverage instrumentation to generate execution traces.
Critical Segment Selection: An LLM selects high-impact code segments (functions or cohesive fragments) exercised by these tests.
Code Rewriting: The LLM is asked to re-implement these segments under a "strict-mask" constraint. Crucially, the LLM is not provided with the original implementation, receiving only surrounding context and the test's goal. This forces the model to rely on general coding knowledge, often resulting in implementation mistakes that violate project-specific conventions (e.g., API coupling, specific execution paths).
Instance Assembly: The rewritten code induces test failures. These failures are converted into synthetic issue instances (problem statement, buggy patch, reference patch) that mimic the format of standard SWE-bench tasks.

2. Skill Distillation

A SWE agent attempts to resolve these synthetic issues. The resulting action trajectories are then distilled into a dual-level skill repository:

Global Diagnostic Skills ( $M_{ext}$ ): Captures project-level knowledge for diagnosis and navigation. For each repository entity, it records:
- Purpose: The entity's role in issue resolution.
- Playbook: Reusable reasoning strategies validated during resolution.
- Related APIs: Project-specific interaction patterns between APIs.
Local Intervention Skills ( $M_{int}$ ): Captures entity-specific modification guidance. It records actionable lessons from both successful and failed trajectories, such as "avoid overriding X" or "ensure setup_query is called," to prevent recurring pitfalls.

3. Skill Adaptation (Retrieval and Injection)

During downstream real-world issue resolution, SkillForge utilizes the distilled knowledge via a two-stage mechanism:

Macro-level Initialization: Before solving an issue, relevant global diagnostic skills ( $M_{ext}$ ) are retrieved (via BM25) based on the issue description and prepended to the agent's prompt to provide high-level project context.
Micro-level Just-in-Time (JIT) Injection: As the agent interacts with the code (e.g., executing shell commands to read files), the system dynamically injects local intervention skills ( $M_{int}$ ) associated with the accessed entities. This ensures guidance is provided exactly when the agent encounters specific code regions, avoiding context noise.

Key Contributions

Proactive Self-Distillation Paradigm: The paper proposes a novel approach to the cold-start problem that acquires project-specific knowledge directly from repository tests and code, eliminating the need for rich historical issue trajectories or costly per-issue online exploration.
Dual-Level Skill Repository: The design of a structured knowledge base separating global diagnostic skills (for navigation and reasoning) and local intervention skills (for specific code modifications), enabling entity-grounded retrieval and just-in-time guidance.
Empirical Validation: Extensive experiments demonstrate that this proactive acquisition strategy significantly outperforms both history-driven and online baselines across diverse benchmarks and model backbones.

Experimental Results

The framework was evaluated on SWE-bench Verified (500 tasks) and SWE-bench Pro (731 tasks) using DeepSeek-V3.2 and GPT-5-mini.

Performance Gains:
- On SWE-bench Verified, SkillForge achieved 72.2% (DeepSeek) and 60.6% (GPT-5) Pass@1, improving over the baseline Mini-SWE-Agent by +5.8% and +5.6% respectively.
- On SWE-bench Pro, it achieved 34.1% (DeepSeek) and 51.7% (GPT-5) Pass@1, improving over the baseline by +5.8% (DeepSeek) and +4.1% (GPT-5).
Comparison: SkillForge outperformed all evaluated history-driven (e.g., MemGovern, SWE-Exp) and online (e.g., SAGE, SWE-Debate) baselines. Notably, it avoided the high per-issue costs associated with online exploration methods.
Ablation Studies:
- Removing either global or local skills resulted in significant performance drops, confirming the necessity of both knowledge types.
- Transferability: Distilled knowledge was found to be LLM-specific. Knowledge distilled by one model (e.g., DeepSeek) provided less benefit to another (e.g., GPT-5), suggesting that the "mismatches" exposed during synthesis are model-dependent.
Hyperparameters: Moderate retrieval counts ( $k_r=5$ ) and synthesis scopes ( $k_s=5$ segments) yielded optimal results; excessive retrieval or rewriting introduced redundancy or complexity that degraded performance.
Generalization: SkillForge showed consistent improvements across seven diverse repositories (e.g., Django, Scikit-learn, Sphinx) without regression, whereas history-driven baselines showed inconsistent performance and occasional regressions.

Significance and Claims

The paper claims that proactively acquiring project-specific knowledge before solving real issues is a superior strategy to reactive acquisition. By synthesizing issues that expose the gap between an agent's general coding priors and a repository's specific implementation patterns, SkillForge creates a reusable, entity-grounded knowledge base.

The authors emphasize that this approach:

Solves the cold-start problem without relying on historical data that may be sparse or biased.
Avoids the prohibitive costs of online test-time exploration for every new issue.
Provides a scalable mechanism to adapt general-purpose LLM agents to specific codebases, leading to more robust and accurate issue resolution across diverse software projects.

The work concludes that the "self-distillation" of knowledge from the repository itself, rather than from external history or real-time trial-and-error, represents an effective and scalable alternative for enhancing software engineering agents.