Evaluating Agentic Code Repair Capabilities in Distributed Systems
This paper introduces DDBench, a novel benchmark of 60 historical bugs from 13 distributed systems, to evaluate LLM-based coding agents and demonstrates that while bounded debugging context significantly improves repair success rates, distributed debugging reveals distinct reasoning challenges and performance disparities among models that single-process benchmarks fail to capture.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern digital world, software does not live in isolation. The applications that power our banks, our communications, and our infrastructure often run as vast, distributed systems. Imagine a single program not as a single worker in a room, but as a team of dozens of specialists scattered across different buildings, constantly talking to one another to get a job done. When something goes wrong in such a system, the problem is rarely a simple typo in one file. Instead, the error might be a misunderstanding between two specialists, a timing issue where one person speaks before another has finished listening, or a conflict that only appears when three different processes happen to act at the exact same moment. For years, researchers have been teaching artificial intelligence to fix code, but they have mostly tested these AI "agents" on single-file problems, like a lone worker fixing a broken tool. They have not yet figured out how to test these agents on the messy, chaotic reality of distributed systems, where the root cause of a failure is often hidden in the complex conversation between many different parts of the network.
A team of researchers has now built a new testing ground specifically designed to measure how well these AI agents can debug these complex, multi-part systems. They call their creation DDBENCH. It is a collection of sixty real-world bugs harvested from thirteen different open-source distributed systems, ranging from database engines to messaging networks. The researchers organized these bugs into three levels of difficulty. The hardest level contains problems that require the agent to understand how different computers influence one another over time, dealing with unpredictable delays and conflicting actions. To test the agents, the researchers set up a controlled experiment for every single bug. In one scenario, the agent is given only a description of the symptom—what the system is doing wrong—and the source code. It must figure out the rest on its own. In a second scenario, the agent receives the same symptom and code, but it is also handed a bundle of extra clues. These clues are like a detective's notebook: logs of what the system said, traces of how it moved, and notes on what the code was doing right before it failed. By comparing how the agents perform with and without these clues, the researchers could measure exactly how much helpful information changes the outcome.
The results of this experiment reveal that distributed debugging is a fundamentally different challenge than fixing single-file code. When the researchers tested ten of the most advanced AI models on the hardest set of bugs without any extra clues, the results were starkly different from what they see in simpler tests. On standard code-repair benchmarks, the top models perform almost identically, clustering together with very little difference in their success rates. On these distributed system bugs, however, the same models spread out dramatically. The best model solved nearly seventy percent of the hardest cases, while the weakest solved only a tiny fraction. This wide gap proves that the ability to reason about how different parts of a system interact is a distinct skill that current benchmarks fail to capture. It shows that being a "top-tier" model for simple tasks does not guarantee it will be a top-tier model for complex, multi-process problems.
The study also discovered that providing extra debugging context changes the game in surprising ways. When the agents were given the curated bundle of logs and traces, the overall success rate jumped significantly. However, the benefit was not the same for every model. The weaker models, which struggled to solve the problems on their own, saw their success rates soar when given the clues. They gained the ability to solve many more bugs because the extra information narrowed the search space they had to explore. The strongest models, which were already quite good at solving the problems, did not get much better at finding the solution. Instead, they became much faster and cheaper to run. With the clues, they needed far fewer attempts and consumed far less computing power to reach the same correct answer. This suggests that for the most capable agents, the value of extra information is not in helping them find the answer they could eventually find alone, but in saving them the time and cost of the long, expensive search.
Perhaps the most nuanced finding is that more information is not always better. The researchers found that if the extra clues are not carefully curated, they can actually mislead the agent. In some cases, a faithful log of a system failure pointed the AI toward the wrong part of the code. If the clue was too far removed from the actual root cause, the agent would get stuck investigating the wrong area, even if the clue was technically accurate. This highlights a critical lesson for the future of AI debugging tools: the quality and relevance of the information provided are just as important as the amount. A well-chosen piece of evidence can turn a failing agent into a successful one, while a poorly chosen one can waste the agent's time or send it down a dead end.
Ultimately, this work establishes a new standard for evaluating how AI handles the complexity of modern software. It moves beyond asking "can the AI fix this code?" to ask "how does the AI think when the problem spans multiple computers?" and "how much does the right information help it think?" The researchers have shown that the ability to reason across processes is a separate dimension of intelligence that separates the best models from the rest. They have also demonstrated that the tools we build to help these agents—tools that gather logs, traces, and runtime data—can be just as important as the models themselves. By providing the right context, we can make weaker models more capable and stronger models more efficient, turning a difficult, expensive debugging process into a manageable one. This opens the door for a new generation of AI tools that do not just write code, but understand the complex, living systems in which that code runs.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.