What Makes Software Issue Resolution Tasks Difficult for Agents?
This paper presents a measurement framework and large-scale empirical study demonstrating that the difficulty of software issue resolution tasks for AI agents is substantially predictable from static structural properties, particularly patch fragmentation and repository scale, thereby enabling more controlled benchmark construction.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, a new generation of software has emerged that can read code, understand problems, and write fixes on its own. These systems, often called agents, act like digital employees who can navigate complex computer projects, locate errors, and propose solutions. As these tools become more capable, researchers have begun testing them against massive collections of real-world software problems to see how well they perform. However, a simple score telling us how many problems an agent solved is not enough. Just as a test score does not explain why a student struggled with a specific math problem, a success rate does not reveal why a particular software task was too difficult for an AI to handle. Without understanding the nature of the difficulty, it is impossible to know if an agent is truly getting smarter or simply getting lucky with easier tasks. To build better tools and fairer tests, scientists need to know exactly what structural elements make a software problem hard or easy to solve.
A team of researchers set out to solve this mystery by treating software tasks like physical objects that can be measured before an agent even attempts them. They gathered a massive dataset containing over 45,000 software tasks, each consisting of a description of a problem, the code repository where the problem lives, and the correct solution that a human developer had already written. Instead of watching the AI struggle in real-time, the researchers analyzed the static properties of these tasks. They looked at the solution itself to see how many lines of code changed and how scattered those changes were across different files. They examined the repository to measure its size, how deeply its folders were nested, and how confusing the file names might be. Finally, they analyzed the text of the problem description to check for linguistic complexity, such as ambiguous pronouns or convoluted sentence structures. By feeding these measurements into computer models, they asked a simple question: can we predict whether an agent will succeed or fail just by looking at the task's structure?
The answer was a resounding yes. The researchers found that the difficulty of a software task is encoded directly in its structure, allowing them to predict an agent's success rate with high accuracy using only static features. The most powerful predictors were not the words in the problem description, but the physical layout of the code and the solution. Specifically, tasks became significantly harder when the required fix was fragmented, meaning the changes were scattered across many different files and gaps, rather than concentrated in one place. The size and complexity of the repository also played a major role; agents struggled more when they had to navigate vast codebases with deep folder hierarchies or when multiple files shared similar names, making it difficult to identify the correct file to edit. Together, these structural factors explained nearly all of the predictable variation in whether an agent would succeed.
Surprisingly, the language used to describe the problem had very little independent power to predict difficulty once the structural factors were accounted for. While the clarity of the instructions matters, the researchers found that the sheer complexity of the code change and the environment in which it must be made are the dominant forces. The linguistic features of the prompt only became a noticeable factor for tasks of medium difficulty, where the structural challenges were neither trivial nor overwhelming. In these middle-ground scenarios, ambiguity in the instructions, such as unclear references or confusing sentence connections, could tip the scales toward failure. However, for the easiest tasks, the code was simple enough that the agent succeeded regardless of the wording, and for the hardest tasks, the structural complexity was so great that even perfectly clear instructions could not help the agent succeed.
This discovery changes how we should think about testing and improving AI agents. It suggests that the difficulty of a task is not a vague quality but a measurable property that can be calculated before the AI ever sees the problem. This allows researchers to build better benchmarks that are balanced across different types of difficulty, ensuring that progress is measured fairly. It also offers a practical way for developers to know when to trust an AI tool; by understanding the structural complexity of a task, a human can better predict whether an agent is likely to succeed, rather than relying on a single, misleading average score. The study confirms that while language is important, the physical architecture of the software itself holds the true key to understanding why some problems defeat even the smartest digital workers.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.