🤖 AI

BenchBench-Protocol: Evaluating Real-World Wet-Lab Protocol Reasoning and Modification

The paper introduces BenchBench-Protocol, a novel benchmark derived from real-world scientist modifications to published protocols that evaluates the ability of large language models to reason about and adapt wet-lab experimental procedures, revealing that current top models achieve only moderate success on these routine but complex tasks.

Original authors: Aditya Sivakumar, Ashu Singhal, Nicholas Larus-Stone, Nithin Parsan

Published 2026-08-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aditya Sivakumar, Ashu Singhal, Nicholas Larus-Stone, Nithin Parsan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In a laboratory, a scientist rarely follows a printed recipe exactly as written. The world of wet-lab biology is defined by adaptation. A protocol, which is simply a detailed set of instructions for an experiment, is often a starting point. When a researcher moves that experiment to a new cell type, a different machine, or a slightly different chemical environment, they must modify the steps. This is not just about changing a number; it is about understanding how one change ripples through the rest of the process. If a scientist alters the temperature in the first step, they might need to adjust the time in the third step, or swap a chemical in the final one. Doing this correctly requires a deep, practical understanding of cause and effect, where a small error in the middle can ruin the results at the end. For decades, this kind of reasoning has been a uniquely human skill, relying on experience and intuition that computers have struggled to mimic.

Researchers at Benchling, a company that builds software for scientists, have created a new way to test if artificial intelligence can master this specific type of thinking. They introduced a test called BenchBench-Protocol. Instead of asking a computer to solve a made-up problem or recite facts from a textbook, they gave it a real-world challenge: take a published scientific protocol and modify it for a new situation, just as a human scientist would. The team did not invent these challenges from scratch. They looked at thousands of actual experiments performed by real scientists on the Benchling platform. They found instances where a scientist took a standard, published protocol and changed it to fit their specific needs. By comparing the original instructions with the scientist's modified version, the researchers extracted the exact logic the human used. They then turned these real-life changes into 149 distinct test questions. Each question asks an artificial intelligence to explain how it would adapt a protocol and, crucially, to justify why those changes are necessary.

To ensure these tests were fair and accurate, the researchers did not rely on computers to grade the answers. They brought in human experts. Each of the 149 tasks was reviewed by scientists who hold advanced degrees and have spent at least three years working in a laboratory. These experts checked that the questions made sense, that the rules for a correct answer were scientifically sound, and that the task actually reflected the kind of work a real scientist does. They kept only the tasks that received the highest ratings for quality and relevance. The final set covers nine different areas of biology, from working with proteins and cells to sequencing DNA and growing microbes. The goal was to create a benchmark that measures whether a machine can reason through the downstream consequences of a change, rather than just guessing the right answer.

The researchers then put nine different artificial intelligence models to the test. These models were given the original protocol, a description of the new experimental goal, and access to search tools to find information. They were asked to write a free-form response explaining how to modify the protocol. The answers were then graded against the expert-verified rules. The results showed that while these models have made significant progress, they still have a long way to go. The best-performing model, Claude Opus 5, achieved a score of 59.2 percent. This means it correctly addressed the necessary changes and their consequences in just over half of the cases. The other models scored between 34.1 percent and 47.1 percent. Even the top performer left a large amount of credit unearned, indicating that the test is difficult and that the models are not yet perfect.

The study also looked at what happens when a model is allowed to try the same question multiple times, simulating a scientist who might run a simulation or check a few different options before deciding. When the researchers took the best answer from ten attempts, the scores improved, but the gap between the models changed. Some models that were average on a single try became much stronger when given multiple chances, while others remained consistent. This suggests that different models have different strengths; some are more reliable, while others are more likely to produce a brilliant answer if they get lucky. However, even with ten tries, the best model still did not capture all the available points, proving that the benchmark is not yet "saturated" or too easy for current technology.

The researchers found that the models performed differently depending on the type of task. They were relatively good at tasks that required looking at experimental evidence and drawing a logical conclusion. However, they struggled more with tasks that required "tacit knowledge"—the kind of gut feeling or intuition an expert has about what is the most likely or best approach in a messy, real-world situation. This was particularly true for troubleshooting, where a model needs to figure out why an experiment failed and how to fix it. The models also varied in how much they used their search tools. Some models, like Grok, consumed a massive amount of data by searching the internet extensively, while others were more conservative. The study noted that these high-reasoning settings, which allowed the models to think deeply and search widely, came with a high cost in terms of computing power and time.

Ultimately, this work provides a grounded way to measure progress in artificial intelligence for science. By building the test on real modifications made by real scientists, the researchers avoided the trap of creating artificial problems that might not reflect the complexity of actual lab work. The findings show that while artificial intelligence is becoming a helpful tool for scientists, it is not yet a replacement for human judgment in the lab. The models can recall facts and follow instructions, but they still struggle with the nuanced, step-by-step reasoning required to safely and effectively adapt a scientific protocol. As these tools become more common in research, benchmarks like BenchBench-Protocol will be essential for understanding where they succeed and where they still need to learn. The path forward involves not just making models smarter, but teaching them to understand the chain of cause and effect that defines the scientific method.

Technical Summary: BenchBench-Protocol

Problem Statement

Adapting existing laboratory protocols for new experimental contexts is a routine yet complex task for wet-lab scientists. Success in this domain requires not merely recalling facts, but reasoning over prior experimental choices and anticipating downstream consequences of modifications. While recent life-science benchmarks have shifted toward open-ended, rubric-graded tasks, they are typically constructed by eliciting tasks from experts. The authors argue that expert-suggested modifications may diverge from the distribution of actual modifications made in real-world workflows, which are constrained by specific reagents, instrument access, and prior results. Consequently, there is a need for a benchmark grounded in the actual modifications scientists perform during real experimentation to accurately evaluate Large Language Models (LLMs) on wet-lab reasoning.

Methodology

Benchmark Construction (BenchBench-Protocol)

The authors introduce BenchBench-Protocol, a benchmark comprising 149 protocol-modification tasks. Unlike previous benchmarks that generate tasks from scratch, this dataset is reconstructed from real-world data:

Data Source: The benchmark compares published protocols against modified versions created by scientists on Benchling, a life-sciences platform.
Task Derivation: Differences between the published and modified versions serve as the ground truth. These differences define the necessary modification and the downstream dependencies that must be addressed.
Domains: The tasks span nine domains of wet-lab biology, including protein biochemistry, cell culture, molecular biology, genomics, and microscopy, drawn from 96 source protocols.
Intent Classification: Tasks are labeled with primary intents such as "Plan audit," "Measurement and result interpretation," "Troubleshooting," and "Procedure specification."

Expert Review and Quality Control

To ensure scientific validity, the dataset underwent a rigorous review process:

Reviewers: Each task was evaluated by at least two independent reviewers holding PhDs in relevant fields with at least three years of bench experience.
Criteria: Tasks were rated on five dimensions: question relevance, scientific soundness, rubric reasoning depth, rubric scientific accuracy, and overall quality.
Selection: Only tasks rated "Very Strong" or "Excellent" across key dimensions were retained, resulting in the final set of 149 tasks.

Evaluation Protocol

Models Evaluated: Nine models were tested, including closed models (Claude Opus 5, GPT-5.6 variants, Gemini 3.6 Flash, Grok 4.5) and open models (Kimi K3, GLM 5.2, Inkling).
Setup: Models received a first-person scientist query, the published protocol, and internet access (via provider tools or Exa). They were required to return a free-text response with a rationale.
Scoring: Responses were graded by an LLM-as-a-judge (GPT-5.6 Terra) using weighted rubrics. Each rubric criterion was scored 0, 1, or 2, with weights reflecting importance. The final score is a normalized weighted sum.
Metrics: Performance was measured using mean normalized rubric scores (0–1) and expected best-of- $k$ scores (sampling up to 10 attempts per task) to simulate a scientist reviewing multiple candidate answers.

Key Contributions

BenchBench-Protocol Dataset: A collection of 149 expert-reviewed, open-ended protocol-modification tasks derived directly from real-world scientist adaptations, covering nine wet-lab biology domains.
Grounded Weighted Rubrics: Evaluation rubrics based on the specific modifications scientists actually made, scoring responses against downstream dependencies rather than general plausibility.
Comprehensive Model Benchmarking: An evaluation of nine frontier models under repeated sampling conditions, characterizing their performance across different task intents and protocol domains.

Results

Overall Performance: Claude Opus 5 achieved the highest mean normalized score of 59.2% (95% CI: 56.8–61.5%). Other models scored between 34.1% and 47.1%.
Benchmark Saturation: The benchmark remains unsaturated; even the best model leaves significant "headroom" (unearned weighted credit). Under best-of-10 sampling, Claude Opus 5 captured 39.5% of available headroom, while Kimi K3 captured 31.5%.
Performance by Intent: Models performed relatively well on tasks requiring reasoning from experimental evidence to conclusions. Performance dropped on tasks requiring tacit knowledge or intuition about the "most likely" expert approach, particularly in troubleshooting and adaptation scenarios.
Performance by Domain: Claude Opus and GPT-series models showed stronger performance on staining, imaging, and cytometry tasks compared to open-source models, potentially due to advancements in multimodal reasoning.
Token Usage: Models operating at maximum reasoning settings consumed high volumes of tokens, with Grok 4.5 showing the highest input token usage, likely due to extensive web search activity.

Significance and Claims

The paper positions BenchBench-Protocol as a grounded assessment of wet-lab reasoning. Its primary significance lies in shifting the evaluation paradigm from expert-elicited hypothetical tasks to reconstructed real-world modifications. By deriving tasks from the actual differences between published and adapted protocols, the benchmark offers a more direct measure of a model's ability to handle the constraints and downstream effects inherent in routine laboratory work.

The authors claim that this approach provides evidence for the utility of real-world experiments in constructing benchmark tasks. They emphasize that while current models show promise, the significant performance gap (with top models scoring below 60%) and the unsaturated nature of the benchmark indicate that substantial progress is still required before these systems can reliably assist in complex protocol adaptation. The work serves as a foundation for evaluating the utility of LLMs in life-sciences research, specifically for tasks where errors propagate into downstream experimental results.