LABBench2: An Improved Benchmark for AI Systems Performing Biology Research

This paper introduces LABBench2, an evolved benchmark comprising nearly 1,900 tasks in more realistic contexts than its predecessor, to better measure and challenge the real-world scientific capabilities of current AI systems, which, despite improvements, still face significant performance gaps on these difficult tasks.

The Gist

Imagine you've built a brilliant, hyper-intelligent robot assistant named "Science-Bot." You've trained it on every biology textbook, every research paper, and every lab manual ever written. You ask it, "What is the function of this protein?" and it answers perfectly. You feel like you've solved science!

But then, you hand Science-Bot a real-world job: "Go find a specific experiment in a 50-page PDF, figure out why the results look weird, and design a new DNA sequence to fix it." Suddenly, the robot freezes. It gets lost in the pages, confuses a chart with a graph, or writes a DNA sequence that looks right but doesn't actually work.

This is exactly what the paper LABBench2 is about. It's a new, much harder "final exam" for AI systems trying to do real biology research.

Here is the breakdown of what the authors did, using some everyday analogies:

1. The Old Test vs. The New Test

The Old Test (LAB-Bench):
Think of the first test (LAB-Bench) like a multiple-choice quiz in a high school biology class. The teacher gives you the question and the answer choices. The AI just had to pick the right letter. It was a good start, but it was a bit like playing a video game on "Easy Mode." The questions were too neat, the data was handed to the AI on a silver platter, and the AI didn't have to do any real "detective work."

The New Test (LABBench2):
The authors realized that real science isn't a multiple-choice quiz. It's more like being a detective in a chaotic library.

• No Answer Keys: Instead of picking A, B, C, or D, the AI has to write the answer from scratch.
• The Library is Messy: The AI can't just be handed the right page. It has to find the right book (a research paper), then find the right chapter, then find the right chart inside that chapter.
• New Departments: The test now includes "Patents" (legal documents for inventions) and "Clinical Trials" (medical tests on humans), which are like reading complex legal contracts and medical records, not just textbooks.

2. The Five "Gymnastics Events"

The new test has nearly 1,900 tasks divided into five main categories. Imagine these as different events in a scientific Olympics:

• Literature Retrieval (The Search): The AI has to find a specific fact hidden inside a 20-page PDF. It's like asking a librarian to find a specific sentence in a book they've never seen before, without a table of contents.
• Data Access (The Database Dive): Scientists use massive databases (like giant spreadsheets of genetic codes). The AI has to log in, search for a specific entry, and pull out the exact number. It's like finding a specific grain of sand on a beach, but the beach is made of digital data.
• Protocol Troubleshooting (The "What Went Wrong?" Game): The AI is given a recipe for a cake (a lab experiment) that has a hidden mistake (e.g., "bake at 500 degrees instead of 350"). The AI has to spot the error and explain why the cake would burn.
• Molecular Biology (The Lego Master): The AI has to manipulate DNA sequences. Imagine trying to build a specific Lego structure, but you have to cut and paste tiny bricks (genes) perfectly. If you miss one brick, the whole thing falls apart.
• Experiment Planning (The Architect): The AI has to design a whole new experiment from scratch, choosing the right tools and steps to solve a problem.

3. The Results: "Smart" but Still Clumsy

The authors tested the world's smartest AI models on this new exam. Here is what happened:

• The Score Dropped: When the AI took the new, harder test, its scores dropped by 26% to 46%. It's like a student who aced the practice quiz but failed the final exam because the questions were trickier and required more critical thinking.
• Tools Help, But Don't Fix Everything: Giving the AI a "web search" tool helped it find books faster, but it still struggled to read the charts inside those books or navigate complex databases. It's like giving a detective a car; they can get to the crime scene faster, but they still have to solve the mystery.
• The "File" Problem: When the AI had to read a file (like a PDF) instead of just having the text pasted into the chat, it got confused. It's like the difference between reading a story printed on a page versus having someone read the story to you. The AI is great at listening, but it's still learning how to read the fine print.

4. Why This Matters

The main point of the paper is this: We need to stop testing AI on how well it memorizes facts and start testing it on how well it does actual work.

If we want AI to help cure diseases or discover new materials, it can't just be a "smart encyclopedia." It needs to be a reliable research assistant. LABBench2 is the tool we use to see where the AI is still failing so we can fix it.

In a nutshell:
The old test asked, "Do you know the answer?"
The new test asks, "Can you go out there, find the answer in a messy real-world situation, and make sure it's actually correct?"

The AI is getting better, but it still has a long way to go before it can replace a human scientist in the lab.

Technical Summary

1. Problem Statement

While AI applications in scientific discovery are accelerating, current benchmarks often fail to measure the real-world capabilities required for practical research. Previous benchmarks, such as the original LAB-Bench, relied on:

• Multiple-choice formats that simplified grading but lacked open-ended reasoning requirements.
• Artificial constraints, such as providing DNA sequences inline rather than via file retrieval, or presenting figures in isolation rather than within the context of a full paper.
• Static knowledge testing rather than evaluating multi-step workflows involving information retrieval, tool use, and complex decision-making.

As frontier models have improved, many have reached "superhuman" performance on LAB-Bench subcategories, rendering the benchmark less effective at distinguishing model capabilities. There is a critical need for a more rigorous, high-fidelity benchmark that tests agentic workflows, retrieval accuracy, and application in realistic contexts (e.g., patents, clinical trials, and wet-lab protocols).

2. Methodology

LABBench2 is an evolution of LAB-Bench, comprising ~1,900 tasks distributed across five core categories. The construction methodology emphasizes realism and difficulty:

A. Task Categories & Construction

1. Literature Understanding & Retrieval:

   • LitQA3: Shifted from multiple-choice to open-response questions requiring full-text reading and reasoning.
   • FigQA2 & TableQA2: Introduced three variants to test different levels of difficulty:
     • Image: Reading a figure/table as an image (open-response).
     • Paper: Parsing a full PDF to locate the specific figure/table.
     • Retrieval: Retrieving the correct paper from scratch and then locating the data.
   • PatentQA & TrialQA: New tasks requiring retrieval and reasoning over patents and clinical trials, critical for therapeutic development.
   • SourceQualQA: A novel task assessing scientific discernment. Models must identify why experts excluded specific studies from systematic reviews, moving beyond checklist compliance to epistemic reasoning.
   • SuppQA2: Retrieval from non-standard Supplemental Information (SI) files (PDFs, Excel, CSVs).

2. Data Access (DbQA2):

   • Tasks require querying 43 specific biological databases (selected via analysis of 1,000 recent bioRxiv papers).
   • Focuses on navigating complex schemas and extracting precise data, avoiding contamination from training data memorization.

3. Molecular Biology Assistance:

   • SeqQA2 & CloningQA: Tasks involving DNA sequence manipulation (e.g., primer design, cloning).
   • Input Modalities: Tasks are delivered via inline prompts, file uploads, or retrieval, testing the model's ability to handle long sequences and external files.
   • Evaluation: Uses custom in silico verifier functions (e.g., simulating PCR) to validate designs rather than relying on text matching.

4. Protocol Troubleshooting (ProtocolQA2):

   • Models are presented with modified lab protocols containing introduced errors.
   • Tasks require identifying the specific error causing a hypothetical negative result, often within long, complex PDF documents.

5. Experiment Planning:

   • CloningQA: Requires end-to-end design of molecular cloning protocols (reagent selection, step-by-step execution) across various methods (Gibson, Golden Gate, etc.).

B. Evaluation Framework

• Models Tested: Current frontier models (e.g., GPT-4o, Claude 3.5, Gemini 1.5/2.0) evaluated in two modes: Base (no tools) and Augmented (with web search and code execution).
• Metrics: Accuracy is measured against human-expert verified ground truth. For open-ended tasks, answers are evaluated for correctness and source fidelity.

3. Key Contributions

1. Expanded Scope & Realism: Introduced nearly 1,900 tasks with a shift from multiple-choice to open-response and retrieval-based formats.
2. New Task Families: Added PatentQA, TrialQA, and SourceQualQA to cover non-paper literature and critical scientific judgment.
3. High-Fidelity Contexts: Tasks now require navigating full PDFs, supplemental files, and external databases, mimicking actual researcher workflows.
4. Rigorous Verification: Implementation of programmatic verifiers (e.g., for sequence design) and expert-reviewed ground truth to ensure objective scoring.
5. Community Release: Public release of the dataset (Hugging Face) and an evaluation harness (GitHub) to facilitate community development.

4. Results

• Significant Difficulty Increase: LABBench2 is substantially harder than LAB-Bench. Across all models and task families, accuracy dropped by 26% to 46%. This gap is attributed to open-response requirements and realistic retrieval contexts.
• Tool Augmentation Benefits:
  • Tools (web search, code execution) significantly improve performance on retrieval-heavy tasks (LitQA3, PatentQA).
  • However, tools do not fully solve the problem; DbQA2 (database access) and SuppQA2 (supplemental material) remain major bottlenecks, indicating models struggle to navigate complex, non-textual data structures.
• Visual Understanding vs. Retrieval: Models show impressive standalone visual understanding of figures/tables when provided directly. However, performance drops sharply when the model must first retrieve the correct paper and then locate the figure within it.
• Sequence Handling:
  • Input Modality: Models perform best with inline prompts and slightly worse with file uploads. Retrieval-based sequence tasks show the poorest performance.
  • Skill Variance: Models struggle with specific subsequence manipulation (e.g., primer design) but perform better on global operations (e.g., complexity scoring). Tool use acts as a "great equalizer" for calculation-heavy tasks.
• Scientific Discernment: Models struggle with SourceQualQA, often failing to identify the nuanced, epistemic reasons for study exclusion, highlighting a gap between "rubric compliance" and true scientific judgment.

5. Significance

• Benchmarking Reality: LABBench2 establishes a new standard for evaluating AI in science, moving beyond rote knowledge to applied research workflows. It reveals that while models are improving, they are not yet ready for autonomous deployment in complex scientific environments.
• Identifying Bottlenecks: The benchmark pinpoints specific failure modes:
  • Retrieval & Localization: Difficulty finding and navigating specific data within long, heterogeneous documents.
  • Database Interaction: Inability to reliably query specialized scientific databases.
  • Exact Input Fidelity: Fragility in handling exact string-level data (e.g., DNA sequences) without specialized tools.
• Future Directions: The results suggest that future AI development must prioritize:
  • Robust iterative retrieval mechanisms.
  • Better document parsing (PDFs, supplements).
  • Integration of specialized tools (sequence editors, database APIs).
  • Development of agentic systems capable of long-horizon planning and handling ambiguous outcomes.

In conclusion, LABBench2 serves as a critical "reality check" for the AI-for-Science field, demonstrating that while progress is substantial, significant gaps remain before AI can fully autonomously conduct biological research.