BioResearchBench: 50 Long Horizon Biological Research Tasks
BioResearchBench is a challenging benchmark of 50 long-horizon biological research tasks across 10 subdomains that reveals current frontier AI models struggle with scientific reasoning and experimental judgment despite often executing individual analytical steps correctly, highlighting critical gaps in data interpretation and decision-making for real-world scientific assistance.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
In the quiet, high-stakes world of biological research, scientists spend their days trying to understand how living systems work, from the tiny machinery inside a single cell to the complex behaviors of entire organisms. Their work often involves gathering vast amounts of data—measurements of genes, proteins, and chemical reactions—and then using that information to figure out what is happening inside a living thing. The goal is to solve puzzles: why a disease develops, how a drug might stop it, or what step to take next when an experiment fails. For decades, this has been a slow, human endeavor, requiring deep expertise to interpret messy results and decide which path forward is worth the time and resources. Now, artificial intelligence has entered this arena, promising to act as a tireless assistant that can read data, run calculations, and suggest the next experiment. But before these digital tools can be trusted to guide real-world medical discoveries, we must know if they truly understand the biology they are analyzing, or if they are simply good at guessing the right words.
A new study called BioResearchBench puts this question to the test with a rigorous and realistic challenge. Researchers at Office Hours Technologies, working with over one hundred PhD biologists, created a collection of fifty complex research problems that mirror the actual work scientists face in labs every day. These are not simple trivia questions; they are multi-step tasks that require an AI to process raw data, perform specific calculations, interpret what those numbers mean in a biological context, and finally propose a logical next step. The tasks cover a wide range of fields, including genetics, cancer biology, and neuroscience, and use real-world data formats like DNA sequences, microscope images, and clinical records. The researchers then asked five of the most advanced AI models available today to solve these problems, grading their answers against strict criteria developed by human experts.
The results reveal a landscape of both surprising capability and significant limitation. While the most successful model managed to solve only about eighteen percent of the tasks perfectly, this low score does not tell the whole story. When the researchers looked closer at the answers that failed, they found that the models were often doing a great deal of the heavy lifting correctly. They could identify errors in data, perform complex statistical adjustments, and recognize patterns that human experts would spot. However, the models frequently stumbled at the final, most critical step: connecting the data to a correct biological conclusion. They would calculate a number accurately but then misinterpret what that number meant for the living system, or they would propose an experiment based on a flawed assumption.
For instance, in one task involving the recovery of cells after a treatment, several models correctly calculated that the cells' energy consumption had improved. Yet, they incorrectly concluded that the cells had fully recovered, overlooking the fact that other vital measurements, such as the number of cells and the health of their internal power plants, remained poor. In another case, models identified a protein that might be regulated by a specific genetic switch, but they went a step too far by claiming the regulation was direct and proven, even though the experiment provided only a hint that required further testing. These errors are dangerous because they could lead a researcher to pursue a dead end, wasting months of work on a theory that the data does not actually support.
The study also highlighted that different AI models have different strengths. One model was particularly good at handling uncertainty and admitting when the data was not clear enough to make a firm decision. Another excelled at designing the next experiment to test a hypothesis. Yet, no single model was consistently perfect across all areas. The researchers found that the most common failures occurred when the models had to handle data carefully, such as correcting for how samples were labeled or accounting for the time it takes for a chemical marker to appear in a new protein. They also struggled with "controls," which are the baseline comparisons that prove an experiment is working as intended, often failing to distinguish between a true cause and a side effect.
What makes these findings so important is that they show AI is not yet ready to replace human judgment in the lab, but it is becoming a powerful tool that needs careful supervision. The models can do the math and organize the information, but they often lack the deep, intuitive understanding of biology that allows a human scientist to know when a result is suspicious or when a conclusion is too strong. The study suggests that the future of AI in science lies not in letting the machines work alone, but in using them as partners. A researcher could use an AI to process thousands of data points and suggest a few possible explanations, but the human expert must then step in to verify the logic, check the assumptions, and decide which path is truly worth pursuing.
Ultimately, BioResearchBench serves as a reality check for the field. It demonstrates that while artificial intelligence has made incredible strides in understanding the language of biology, it still struggles with the nuance of scientific reasoning. The models can follow instructions and perform calculations, but they sometimes miss the forest for the trees, focusing on the numbers while losing sight of the living system they are meant to describe. As these tools continue to evolve, the goal is to train them not just to be faster calculators, but to be better thinkers—capable of understanding the limits of their own data and knowing when to pause and ask for help. Until then, the human scientist remains the essential guide, ensuring that the journey from data to discovery stays on the right path.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.