Repo2Skill-Evo: Repository Skills Go Stale in Silence
This paper demonstrates that while externalizing repository-specific knowledge into LLM agent skills improves performance, these skills silently degrade during software releases, and even frontier agents struggle to reliably update them without introducing significant errors in coverage or precision.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern digital landscape, software is rarely a static object; it is a living, breathing entity that changes constantly. Developers release new versions of programs daily, fixing bugs, adding features, and altering how the code works. To navigate this shifting terrain, artificial intelligence systems known as "agents" are increasingly used. These agents act like digital assistants that can read code, run scripts, and fix problems on their own. However, for an agent to work effectively on a specific software project, it needs more than just general intelligence; it needs a set of specific instructions tailored to that project's current version. Researchers call these tailored instructions "skills." Think of a skill as a concise, reusable guidebook that tells the agent exactly which buttons to press and which files to open for a particular task. Just as a human mechanic needs the latest manual for a car model that just received a new engine, an AI agent needs its guidebooks to be updated whenever the software it manages changes.
The critical question is what happens when the software changes but the guidebook does not. If a program is updated, the old instructions might become wrong, yet the AI might not realize it. It could continue to follow the outdated guide, leading to errors, without ever raising an alarm. This phenomenon, where knowledge becomes silently incorrect, is the focus of a new study called Repo2Skill-Evo. The researchers set out to see if today's most advanced AI agents could recognize when their internal guidebooks had gone stale and update them correctly on their own. They treated the maintenance of these skills not as a one-time task, but as an ongoing challenge that mirrors the constant evolution of the software itself.
To investigate this, the researchers gathered a large collection of real-world software projects, specifically looking at 57 different repositories that had undergone 105 distinct updates. For each project, they first created a perfect set of "skills" based on the old version of the software. These skills were carefully crafted to be accurate, serving as a baseline. Then, they introduced the official update patch—the exact set of changes that transformed the software from its old version to the new one. The task given to the AI agents was simple in theory but difficult in practice: look at the old skills and the new changes, identify which parts of the old guide were now wrong, remove or fix those parts, and keep everything else that was still valid. The researchers did not just ask the agents to try; they measured exactly how well the agents performed by comparing the final result against a gold standard of what should have been changed.
The results revealed a significant gap between the potential of these agents and their actual ability to handle change. Even the most sophisticated AI models available today struggled to keep these skills current. When the researchers evaluated the performance of six leading agents, the best one managed to correctly identify and update the necessary information in only about 70% of the cases on average. The others performed even worse, with some barely reaching 30% accuracy. This means that in the majority of instances, the agents either failed to remove the outdated instructions, leaving the AI with misleading advice, or they went too far, deleting information that was still correct. The study found that the agents often missed the specific files that needed attention, or they edited too broadly, removing valid content along with the bad.
A deeper look into why the agents failed showed two main bottlenecks. First, the agents frequently could not locate the specific parts of the guidebook that were affected by the software update. It is as if a librarian knows a book needs an update but cannot find the specific page where the error occurred. Second, even when the agents found the right place, they often struggled to decide exactly what to change. They tended to either leave the error untouched or rewrite large sections of the text, destroying useful information in the process. The researchers tested whether simply telling the agents exactly which files to look at would solve the problem. While this helped, it did not fix the issue entirely; the agents still made mistakes in deciding how to edit the content within those files. This suggests that the problem is not just about finding the right place, but also about understanding the subtle differences between what is old and what is new.
The study also confirmed that these skills are indeed valuable. Before testing the maintenance ability, the researchers checked if having these guidebooks actually helped the AI agents do their jobs better. They found that when agents had access to these specific skills, their performance improved dramatically, especially on tasks where they previously had little knowledge of the software. This proves that the skills are not just a theoretical concept but a practical tool that makes AI agents more effective. However, the value of these tools is entirely dependent on their accuracy. If the guidebook is outdated, it becomes a liability rather than an asset, potentially leading the agent to make mistakes that it would not have made if it were working from scratch.
The researchers concluded that the current generation of AI agents cannot be trusted to maintain their own knowledge bases as software evolves. The ability to update these skills is not a solved problem. The study highlights a "silent staleness" where outdated information persists without warning, creating a hidden risk for automated systems. As software continues to evolve at a rapid pace, the skills that guide AI agents must be treated as versioned assets that require active, human-like maintenance. The findings suggest that until agents can reliably distinguish between what is obsolete and what is still valid, their use in complex, evolving software environments will remain limited by the fragility of their own knowledge. The path forward requires new methods to ensure that these digital guides remain accurate, or that humans remain in the loop to verify them after every major change.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.