SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

Science has always relied on tools: the telescope that revealed distant stars, the microscope that unveiled the hidden world of cells, and the particle accelerator that probed the fabric of matter. Today, a new kind of tool has become indispensable, one that lives not in a laboratory drawer but in the digital realm of computer code. This software is no longer just a helper that crunches numbers for a scientist; it has become the instrument itself. When a researcher simulates a new drug, models the climate, or analyzes the light from a distant galaxy, they are running programs that act as the very lens through which they view reality. If the code contains a flaw, the result is not merely a computer error; it is a corrupted piece of evidence that can undermine a scientific conclusion.

For years, researchers have been trying to teach artificial intelligence to fix these broken programs. They have built tests to see if a computer can read a bug report and write a patch to repair the software. These tests have shown that machines are getting better at fixing simple coding mistakes. However, a critical question remained unanswered: can these intelligent systems handle the complex, high-stakes world of scientific software? In this realm, a fix must do more than make a program run; it must preserve the delicate laws of physics, chemistry, or biology that the code is trying to represent. A repair that makes the software run faster but changes the scientific meaning is a failure, even if the computer says it passed the test.

A team of researchers has now tackled this question by creating a new testing ground called SWE-bench Science. They gathered 119 real-world tasks from 98 different software projects used by scientists in 20 distinct fields, ranging from chemistry and biology to astronomy and civil engineering. These tasks were not simple exercises; they were drawn from actual issues that scientists had encountered, such as a simulation producing the wrong energy value for a crystal or a data analysis tool misinterpreting the coordinates of a biological sample. The researchers organized these challenges into three types. Some were straightforward repairs of known bugs, others required the AI to act like an expert investigator to figure out why a scientific result looked wrong, and the rest demanded that the AI understand how different parts of a massive software system fit together to complete a whole workflow.

The results of this experiment were revealing. Even the most advanced coding agents available today struggled significantly. The best-performing system, a sophisticated model equipped with a powerful coding assistant, managed to solve fewer than half of the tasks correctly. This low success rate highlights a deep gap between what current artificial intelligence can do and what is required to maintain scientific software. The researchers found that when these systems failed, they did so in four predictable ways. Often, the AI lacked the specific scientific knowledge needed to understand the problem, leading it to propose fixes that made sense to a computer but violated the laws of nature. Sometimes, the system would fix only the surface symptom it could see, like a number that looked wrong, without tracing the error back to its root cause. In other cases, the fix worked for one part of the program but broke the connection to another part, failing to integrate the repair into the larger system. Finally, the AI frequently failed to apply a scientific principle to new situations, getting stuck on the specific example it was given and unable to generalize its solution.

To understand the role of scientific knowledge in these failures, the researchers conducted a specific experiment. They took a subset of the tasks and ran them twice: once with the AI given extra background information about the science involved, and once without it. The results showed that this extra information was not a magic bullet. For some of the stronger AI models, providing scientific context actually made them slightly less likely to solve the problem perfectly, perhaps because the extra text distracted them or led them to rely too heavily on the provided explanation rather than testing their own ideas. For weaker models, the extra information helped, but it also required them to use more computing power. This suggests that simply feeding an AI more facts does not guarantee a better repair; the system must be able to connect those facts to the code and verify them through execution.

The study concludes that while artificial intelligence is making strides in general software engineering, the unique demands of scientific computing remain a formidable challenge. Fixing scientific code requires a level of understanding that goes beyond syntax and logic; it demands a grasp of the physical and theoretical principles that the code embodies. Until these systems can reliably distinguish between a program that runs and a program that is scientifically valid, the task of maintaining the digital instruments of science will likely remain a job for human experts. The new benchmark provides a clear map of where these systems stand today, showing that the path to fully autonomous scientific software repair is still long and filled with complex obstacles.