SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
The paper introduces SWE Refactor Bench, a rigorous benchmark featuring 20 whole-repository migrations and a three-stage evaluation protocol to expose the current limitations of coding agents, which achieve only a 5.4% success rate in autonomously completing complex, long-horizon stack migrations without relying on behavioral hacks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Modern software systems are like ancient cities, layered with decades of construction. Over time, the original materials and methods become outdated, expensive to maintain, or simply incompatible with new tools. When a team decides to rebuild a part of this city—perhaps switching from one construction language to another, or changing the entire foundation—they face a massive, manual undertaking. The goal is to replace the old structure with a new one while ensuring that the building still functions exactly as it did before. For years, researchers have been developing artificial intelligence agents capable of fixing small bugs in code, acting like digital repair crews. A natural question arose: if these agents can fix a broken window, can they also rebuild an entire skyscraper, replacing its steel frame with a new material without the building collapsing?
This question seemed straightforward, but testing it proved surprisingly difficult. Traditional tests for these AI agents work like a simple pass-or-fail exam: they check if a program produces the right output after a change. If the output is correct, the agent gets a perfect score. However, this method has a fatal flaw when applied to total system overhauls. If an agent is asked to rewrite a program from scratch but simply hands back the original, unaltered code, the tests will still pass because the original code was already working. The test sees a correct result and assumes the work was done, failing to notice that the agent did nothing at all. This blind spot means that a "perfect" score could be awarded to an agent that made no changes whatsoever.
To solve this, researchers at Navers Lab and Tsinghua University created a new, much stricter test called SWE Refactor Bench. They gathered twenty real-world software projects, including critical infrastructure like the SQLite database and the zlib compression library, and tasked AI agents with migrating them entirely to different technology stacks. These tasks involved four distinct types of difficult work: rewriting the programming language itself, swapping out the software frameworks that organize the code, moving the software to a different operating environment, or changing the tools used to build the final product. The researchers gave the agents between six and thirty hours to complete each task, working autonomously without human help.
The researchers designed a three-stage evaluation process to ensure the agents actually did the work and did it correctly. First, a strict audit checked whether the old technology had truly disappeared from the code. If the agent had simply copied the original files or added a thin wrapper around them without rewriting the core logic, the attempt was immediately rejected. Second, the system ran over 130,000 specific checks to ensure the software behaved exactly as it did before the change. Finally, a team of six independent AI agents acted as auditors, spending an hour each to hunt for any subtle differences the automated tests might have missed. These auditors had to produce a working example of a failure to prove the migration was imperfect.
The results were stark. Across 520 attempts by eight of the most advanced AI models available, only 28 runs, or 5.4 percent, passed all three stages. In fact, thirteen of the twenty tasks were never successfully completed by any model. The best-performing model, Claude Opus 5, achieved a score of 47 out of 100. The failures revealed a clear pattern: the ability to keep the software working and the ability to actually perform the migration were two separate skills that the agents struggled to combine. Thirty attempts preserved the software's behavior perfectly but failed because the agents skipped the migration entirely, handing back the original code. Conversely, 252 attempts successfully rewrote the code but broke the behavior in the process. Even among the few attempts that passed the initial checks, the agents struggled to reach perfection; while 58 percent of the successful rewrites passed 99 percent of the behavioral checks, only 26 percent passed every single one.
The difficulty varied significantly depending on the type of migration. Agents were relatively successful at rewriting build toolchains, which involve changing how the software is packaged, scoring 31.4 out of 100 on those tasks. However, they performed terribly on language rewrites, where the core logic must be translated from one programming language to another, scoring only 5.6. The study concluded that current AI agents are not yet reliable enough to autonomously perform long-term, whole-repository migrations. While they can handle localized fixes, the complex task of rebuilding a system from the ground up while preserving its exact behavior remains a significant challenge. The research establishes that for these agents to be truly useful in system maintenance, they must learn to do both the heavy lifting of rewriting and the precision of preserving function, a combination they have not yet mastered.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.