SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction
This paper introduces SA-Bench, a diagnostic benchmark evaluating 1,491 semantic alignment units across 30 top-tier ML papers to reveal that current LLM agents suffer from significant "semantic drift," achieving low fidelity in reproducing scientific specifications despite attempting most requirements.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern landscape of artificial intelligence, a new class of software has emerged that acts as a tireless assistant for scientists. These systems, often called coding agents, are designed to read complex research papers and automatically write the computer programs needed to bring the ideas in those papers to life. The promise is transformative: if a scientist describes a new way to train a machine learning model in a journal article, an agent could theoretically generate the entire codebase, allowing other researchers to verify the results instantly without spending months deciphering dense text. This capability represents a shift from simple task completion to long-term project generation, where the computer must not just write a few lines of code, but construct a complete, working software repository that mirrors the logic of a scientific discovery. The hope is that this automation will accelerate the pace of scientific progress, turning the slow, manual process of reproducing research into a rapid, automated workflow.
However, a recent study reveals that while these agents are becoming better at writing code that runs, they are still failing at the most critical part of the job: writing code that actually means what the scientist intended. Researchers from several leading universities introduced a new diagnostic tool called SA-Bench to investigate this gap. They gathered thirty recent research papers from top-tier computer science conferences and asked twelve different combinations of AI models and software frameworks to reproduce the code described in those papers. The goal was not simply to see if the code could be executed without crashing, but to determine if the generated programs faithfully implemented the specific scientific claims, numerical details, and experimental steps outlined in the original text. The results were sobering. Even the most advanced setup, combining the strongest available AI model with a specialized coding framework, managed to correctly implement only about thirty percent of the specific requirements found in the papers. When looking at the average performance across all attempts, the success rate dropped to just twenty-two percent.
The study defines this failure as "semantic drift," a quiet divergence where the generated code looks correct on the surface but silently deviates from the paper's specifications. To measure this, the researchers broke down each research paper into hundreds of tiny, verifiable claims, which they called Semantic Alignment Units. These units ranged from specific numbers, like the learning rate of a training algorithm, to the order in which different steps of a process must occur. They then evaluated the AI-generated code against these units, looking for four specific types of errors: numerical mistakes where a number was wrong, methodological errors where a formula or algorithm step was changed, protocol errors where a dataset or baseline was missing, and ordering errors where the sequence of operations was jumbled. The analysis showed that the agents were not failing because they refused to attempt the tasks; rather, they were attempting almost every requirement but implementing them incorrectly.
The most common reasons for failure were surprisingly mundane. In many cases, the agents would write code that referenced the correct keywords but implemented a completely different logic underneath, a phenomenon the researchers called an implementation mismatch. In other instances, the agents would acknowledge a requirement but leave it as a placeholder, a stub, or a comment saying "to be done," effectively deferring the work. A significant portion of the errors also stemmed from the agents' inability to distinguish between the core contributions of a paper and the standard tools or previous work that the paper merely cited. For example, an agent might mistakenly implement a method described in a cited reference as if it were the new method proposed by the paper itself. The study found that these errors were systematic and pervasive across all the different AI models and software frameworks tested.
One of the most striking findings was that the tools designed to help agents write better code were not solving the core problem. The researchers tested three different approaches: a basic loop where the agent tries, fails, and tries again; a specialized pipeline that breaks the paper down into planning and coding stages; and a sophisticated software engineering framework that runs the code and checks for errors. While these tools helped the agents write code that could run, they did little to ensure the code was scientifically accurate. In fact, for the most capable AI models, adding these complex scaffolding tools sometimes made the performance slightly worse, as the rigid structures interfered with the model's own ability to extract and follow the paper's unique specifications. The researchers concluded that the current focus on making code executable is insufficient for scientific reproduction. To truly bridge the gap, future systems need to prioritize a different kind of verification: one that checks whether the code matches the semantic meaning of the scientific claims, rather than just whether it produces a result or passes a test. The study suggests that until agents can reliably verify that they have understood the "what" and "why" of a paper, not just the "how" of the code, the dream of fully automated scientific reproduction will remain out of reach.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.