Verification Debt in Automated Biological Research Workflows: A Proof-of-Concept

Modern biology is entering an era where computers are no longer just tools for crunching numbers, but active partners in discovery. Scientists are beginning to use artificial intelligence agents—software programs designed to act like researchers—to plan experiments, search through vast libraries of medical literature, and connect dots between genes and diseases. These systems can access thousands of biological databases and run complex workflows that once took human teams weeks to complete. The promise is immense: faster cures, deeper understanding of life, and a scale of research that humans could never manage alone. However, this speed brings a new kind of risk. When a computer makes a decision, it is not always clear why it made that choice, what evidence it used, or if it skipped a crucial step. If a human cannot easily check the computer's work, a hidden burden of uncertainty begins to pile up. This is what researchers call "verification debt": the growing stack of unverified assumptions and hidden steps that could lead to false conclusions if left unchecked.

A team of researchers set out to measure exactly how much of this debt exists in today's biological AI. They chose to test a specific system called Biomni, an artificial intelligence agent designed to handle biomedical research. The team did not simply ask the computer if it was smart; they put it to work on ten specific, real-world biological questions. These questions ranged from simple inquiries, such as "What does this specific protein do?" to complex puzzles, like "How do RNA and protein data combine to explain a specific type of cancer?" To ensure the results were reliable, the researchers asked the AI to answer each question three separate times, treating each run as an independent experiment. They then meticulously examined every single step the AI took, from the moment it received the question to the final answer it produced. Their goal was to see if the AI's reasoning could be traced back to solid evidence, or if it was making leaps that a human scientist could not verify.

The researchers found that while the AI was capable of producing rigorous and often correct answers, it frequently left behind a trail of unverified choices. In many cases, the AI would cite a scientific study to support a claim, but it had not actually read the full text of that study. Instead, it relied on a brief summary provided by a search tool, assuming the summary contained the specific fact it needed. This created a situation where the AI appeared to be grounded in evidence, but the evidence was actually missing or incomplete. The team discovered that for well-known biological facts, the AI often skipped the search process entirely, answering from its internal memory without telling the user that it was doing so. This meant that for some questions, there was no way for a human to check where the information came from, leaving a gap in the chain of proof.

The nature of the problem changed depending on how difficult the question was. For simpler questions, the main issue was a lack of transparency; the AI would make assumptions or skip steps without explaining why. For the more complex questions, the problems shifted to interpretation. The AI would sometimes draw conclusions that were stronger than the data actually supported, or it would mix different types of evidence, such as combining results from mouse studies with human genetic data, without clarifying the difference in their reliability. The researchers noted that these errors were not consistent; they happened unpredictably. One time the AI might follow a perfect path, and the next time it might take a different route with hidden flaws. This unpredictability makes it difficult to simply trust the system, even when it gets the right answer, because the path to that answer might be flawed.

A key finding of the study was that having a record of the AI's steps, known as a "trace," was not enough to guarantee the work was correct. The AI could provide a detailed log of every tool it used and every database it queried, yet still arrive at a conclusion based on a misunderstanding of the data or an unspoken assumption. The researchers identified a specific type of risk called "dependency consequence," where a single unverified decision early in the process could ripple through the entire workflow, corrupting all the subsequent results. They found that in nearly every question they tested, most of the AI's decisions carried this kind of high-stakes risk. If a human researcher did not carefully check every link in the chain, a small error could lead to a major scientific mistake.

The study concludes that while these automated systems are powerful, they cannot be trusted to work entirely on their own. The speed and scale they offer are valuable, but they create a gap between what the computer does and what a human can verify. The researchers argue that transparency—showing the steps the AI took—is necessary but not sufficient. A system can be fully traceable and still produce unreliable results if the reasoning behind those steps is flawed. To move forward, the scientific community needs new ways to systematically check not just the final answer, but the entire journey the AI took to get there. Until these verification methods are in place, human experts must remain in the loop, ready to catch the hidden debts that the machines accumulate as they work.