Silent methodological failures in AI-assisted science: benchmarking large language model agents for virtual gene knockout analysis
This study benchmarks five LLM agents on virtual gene knockout analyses and reveals that while all successfully executed code, 80% produced biologically implausible results through "silent methodological failures," underscoring the critical need for biological validation and cross-model concordance beyond mere code execution in AI-assisted science.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
In the modern laboratory, a new kind of assistant has arrived, one that does not wear a lab coat or hold a pipette but instead speaks in the language of code. These large language models are being asked to perform the complex, multi-step tasks of computational biology, where researchers simulate how living cells react when specific genes are turned off. This process, known as a virtual gene knockout, allows scientists to predict the ripple effects of genetic changes without ever touching a living organism. It is a powerful tool for understanding diseases and designing treatments, but it relies entirely on the computer's ability to follow a set of instructions perfectly. The central question facing the scientific community is whether these artificial intelligence agents can be trusted to translate a researcher's intent into a correct mathematical reality, or if they might simply produce answers that look right on the surface but are fundamentally wrong underneath.
A team of researchers from the Second Affiliated Hospital of Xi'an Jiaotong University set out to test this reliability by treating five different artificial intelligence agents as independent scientists. They gave each agent the same mission: to simulate the removal of a specific gene, called albumin, from two different collections of liver cell data. Albumin is a protein made by the liver, and because its role is well understood, it serves as a reliable yardstick for checking if a simulation makes biological sense. The researchers asked the agents to use four distinct mathematical frameworks to perform this simulation, a task that requires the computer to parse data, build networks of gene interactions, and calculate the statistical significance of the changes. The goal was not to see if the computers could write code that runs, but to see if the code they wrote actually produced results that matched the known behavior of liver cells.
The results revealed a startling gap between technical success and scientific truth. Every single agent managed to complete the assigned tasks without crashing or throwing an error message. They all produced code that executed successfully, and they all returned numerical answers. However, when the researchers examined the actual numbers, the agents disagreed with each other dramatically. The time it took to finish the work varied wildly, with the fastest agent finishing in just over a minute while the slowest took more than thirty minutes. More importantly, the biological conclusions drawn by the agents were often inconsistent. In some cases, an agent would claim the gene knockout had a massive effect, while another agent, working on the exact same data, would conclude the effect was negligible. The variation in their numerical outputs was so high that the difference between the highest and lowest estimates reached more than one hundred and forty percent.
The most concerning discovery was the prevalence of what the researchers call "silent methodological failures." Four out of the five agents fell into this trap. These agents generated code that ran perfectly and produced error-free reports, yet the results they delivered were biologically impossible. For instance, one agent accidentally flipped the data matrix upside down, a mistake that changed the entire meaning of the calculation but did not trigger any warning signs. Another agent used a mathematical definition for "impact" that was different from the standard one, leading it to report a significant finding when none existed. Because the code ran without errors, these mistakes would likely go unnoticed by a researcher who only checks for technical glitches. The agents had successfully followed the letter of the instructions but failed to grasp the spirit of the scientific method, producing outputs that were computationally valid but scientifically misleading.
Not all agents performed equally poorly, and the study highlighted that success in one area does not guarantee success in another. One agent, GLM 5.1, emerged as the most balanced performer, managing to produce results that were both fast and biologically plausible in most of the tests. Another agent, Kimi, showed impressive resilience by automatically fixing a memory error that caused it to pause, though this recovery cost it a significant amount of time. In contrast, the fastest agent, Minimax, produced the quickest results but failed to generate biologically plausible outcomes in the vast majority of its attempts. The study found that the specific way an agent chose to implement a mathematical step could completely alter the final answer, even when the high-level instructions were identical. This suggests that the current generation of AI agents is not yet a reliable replacement for human oversight in complex scientific workflows.
The researchers concluded that the ability of an AI to write code that runs is not enough to prove that the science is correct. The presence of silent failures means that relying solely on automated execution could lead to false discoveries or missed insights. To ensure reliability, the study suggests that scientists should not trust a single AI agent's output. Instead, they should run the same analysis through multiple different agents and look for agreement between them. If the agents produce widely different numbers, it is a signal that something is wrong and requires human investigation. Furthermore, the instructions given to these AI systems need to be much more precise, defining exactly how mathematical operations should be performed rather than leaving room for the AI to guess the best approach. Until these safeguards are in place, the most valuable role for artificial intelligence in science remains that of a powerful assistant that requires a knowledgeable human to verify its work.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.