Agentic systems are adept at solving well-scoped, verifiable problems in computational biology

This paper introduces CompBioBench, a novel benchmark of 100 diverse computational biology tasks constructed via synthetic data augmentation and metadata scrambling to enable objective evaluation, demonstrating that leading agentic systems like Codex CLI and Claude Code can achieve high accuracy in solving complex, multi-step biological problems requiring tool use and external resource interaction.

The Gist

Imagine you have a team of incredibly smart, super-fast robots. These robots are experts at reading instructions, writing code, and using tools. In the past, we've tested them on math problems and coding challenges, and they've been amazing. But what happens when you put them in a messy, chaotic laboratory filled with biological data?

That is exactly what this paper, CompBioBench, is about. It's a report card for these "AI agents" (the robots) on how well they can solve real-world biology problems.

Here is the story of the paper, broken down into simple concepts:

1. The Problem: Biology is Messy

Think of math and coding like a Lego set. If you follow the instructions perfectly, you get the exact same castle every time. There is one right answer.

Biology, however, is more like cooking in a kitchen where the ingredients are slightly different every time. The data is "noisy." Sometimes a recipe doesn't work because the flour was damp, or the oven temperature was off. In the past, it was hard to test AI on biology because there wasn't always a single "right answer" to check against. If an AI said, "This gene is active," and a human said, "Maybe," who was right?

2. The Solution: The "CompBioBench" Test

The researchers at Genentech and Roche built a new test called CompBioBench. Think of this as a 100-question obstacle course designed specifically for biology.

To make sure the test was fair and had a clear "right answer," they used a clever trick:

• The "Fake" Data: They created synthetic data (like a fake DNA sequence) where they knew the answer beforehand.
• The "Scrambled" Data: They took real biological data and hid the labels (like taking a photo of a person and blurring their face, then asking the AI to guess who it is based on their walk).

This forced the AI to actually think and investigate rather than just guessing. The questions covered everything from finding errors in DNA sequencing to figuring out which genes are turned on in a specific cell type.

3. The Rules of the Game

The AI agents weren't given a toolbox full of pre-installed software. They were dropped into a bare-bones digital room with only the raw data files.

• The Challenge: If they needed a specific tool to analyze the data, they had to go to the internet, find it, download it, install it, and figure out how to use it—all on their own.
• The Goal: Solve the problem and give a single, exact answer.

4. The Results: The Robots are Getting Scary Good

The researchers tested the two biggest AI agents on the market: Codex CLI (from OpenAI) and Claude Code (from Anthropic).

• The Winners: The top performers were like master chefs who could not only cook the meal but also find the missing spices in the pantry.
  • Codex CLI got 83% of the questions right.
  • Claude Code got 81% right.
• The Struggle: When the questions got really hard (like trying to solve a mystery with very few clues), the scores dropped, but they were still impressive (around 60-70%).
• The "Dumb" Bots: Smaller, less powerful AI models struggled significantly, getting only about 34% right. This shows that you need a very "smart" brain to handle the complexity of biology.

5. How They Did It (The "Magic" Tricks)

The paper highlights some fascinating ways these AI agents solved problems:

• The Detective Work: In one task, the AI had to find a "double agent" (a doublet) in a crowd of cells. The AI realized that standard methods wouldn't work, so it went out, found a specialized tool called "AMULET," installed it, and used it to solve the puzzle.
• The Optimizer: In another task, the AI had to download a massive 18GB machine learning model. Instead of downloading the whole thing (which would take forever), it figured out how to download only the tiny 100MB piece it actually needed. That's like ordering a whole pizza just to get one slice, but the AI figured out how to order just the slice.
• The Persistence: Sometimes the AI would try a method, fail, realize it was wrong, and then try a completely different approach. It didn't give up easily.

6. The Catch (Limitations)

Even though the robots are great, they aren't perfect yet.

• Brittleness: Sometimes they get stuck on a "plausible" wrong path. It's like a detective who is so convinced a suspect is guilty that they ignore evidence proving the suspect is innocent.
• Time and Cost: These smart robots are expensive to run. Solving a hard problem can take 20 minutes and cost a few dollars. For a human, it might take 3 hours, but for a business, the cost adds up.
• No "New" Science: The AI is great at using existing tools and solving known types of problems. It's not yet ready to invent brand-new scientific theories or discover completely unknown biological laws.

The Bottom Line

This paper is a major milestone. It proves that AI agents are no longer just chatbots; they are becoming capable research assistants. They can navigate the messy, confusing world of biology, find the right tools, and solve complex puzzles.

While they still need a human supervisor to double-check their work (especially on the hardest problems), the day is coming soon when these AI agents will be the first line of defense in a biology lab, handling the boring, repetitive, and complex data crunching so humans can focus on the big ideas.

Technical Summary

1. Problem Statement

Computational biology is a highly interdisciplinary field involving complex workflows (e.g., genomics, transcriptomics, single-cell analysis) that require navigating heterogeneous data formats, debugging pipelines, and utilizing a "long tail" of specialized tools. While recent general-purpose agentic systems (e.g., Anthropic's Claude Code, OpenAI's Codex CLI) have shown promise in mathematics and programming, evaluating them in biology is challenging due to:

• Inherent Noise: Biological data is often noisy and open to interpretation.
• Lack of Ground Truth: Many biological analyses do not yield a single unambiguous answer.
• Ambiguity in Prior Benchmarks: Existing benchmarks (e.g., BixBench, BioAgent Bench) often focus on narrow domains, include underspecified questions, or prescribe specific workflows, reducing the need for autonomous strategy selection.

The core problem is the lack of a rigorous, objective benchmark to evaluate the end-to-end capabilities of agentic systems in computational biology, specifically their ability to reason, fetch tools, and execute multi-step workflows without human intervention.

2. Methodology: CompBioBench

The authors introduce CompBioBench, a benchmark consisting of 100 diverse tasks designed to test agentic systems under realistic constraints.

Benchmark Construction Strategy

To ensure objective evaluation while maintaining biological realism, the authors employed a specific construction philosophy:

• Single Ground Truth: Every question has exactly one correct answer, achieved through:
  • Synthetic/Augmented Data: Creating controlled scenarios (e.g., mixing RNA-seq reads from different species, introducing specific mutations).
  • Metadata Scrambling/Recovery: Taking real datasets and obscuring metadata (e.g., swapping sample labels) so the agent must infer the error.
• Non-Prescriptive: The benchmark does not specify the tools or workflows to use. Agents must autonomously select strategies, install tools, and write bespoke code.
• Diverse Scope: Tasks cover Genomics, Transcriptomics, Epigenomics, Single-cell analysis, Human Genetics, and Machine Learning.
• Difficulty Levels: Questions are annotated by experts on a 1–5 scale based on logical reasoning and step complexity.

Evaluation Environment

• Bare-Minimum Setup: Agents start in an isolated environment with only the input files. They have no pre-installed specialized tools or reference data (e.g., genome indices).
• Autonomy Requirements: Agents must:
  • Fetch data and tools from the internet.
  • Install and configure software (e.g., via conda, pip).
  • Write and debug scripts.
  • Interact with external APIs and databases.
• Evaluation Metric: Binary accuracy based on an exact string match between the agent's output and the ground truth. Numerical answers allow for limited tolerance (rounding).

Models Tested

The authors evaluated leading general-purpose agentic systems:

• Codex CLI (powered by GPT-5.4)
• Claude Code (powered by Opus 4.6, Sonnet 4.6, Haiku 4.5)
• Baselines: Non-agentic LLMs (ChatGPT 5.2, Claude Opus 4.6) running in a single-pass mode without tool access.

3. Key Contributions

1. CompBioBench Dataset: A curated set of 100 diverse, verifiable computational biology tasks that bridge the gap between theoretical AI capabilities and practical scientific workflows.
2. Novel Benchmarking Paradigm: A strategy using synthetic data augmentation and metadata recovery to create "single-answer" problems from inherently noisy biological data, avoiding the pitfalls of prescriptive checklists.
3. Comprehensive Evaluation: The first large-scale assessment of agentic systems in a "bare-metal" environment, forcing agents to handle the full lifecycle of data retrieval, tool installation, and analysis.
4. Insights into Agent Behavior: Detailed qualitative analysis of how agents fail (e.g., premature stopping, incorrect background selection in motif enrichment) and succeed (e.g., optimizing data transfer via byte-range requests).

4. Results

Overall Performance

• High Accuracy for Top Models:
  • Codex CLI (GPT-5.4): Achieved 83.3% accuracy.
  • Claude Code (Opus 4.6): Achieved 81.0% accuracy.
• Baseline Failure: Non-agentic LLMs (without tool use) performed near random guessing (3.7% – 5.3%), confirming that the tasks cannot be solved via memorized knowledge alone.
• Model Scaling: Performance dropped significantly for smaller models (Claude Code Sonnet: 70%; Haiku: 34%).

Performance by Difficulty and Domain

• Difficulty: Performance degrades as difficulty increases.
  • On Level 1 (easiest) questions, Codex CLI reached 100% accuracy.
  • On Level 4–5 (hardest) questions, Claude Code (Opus) reached 69%, outperforming Codex CLI (59%).
• Domains: Codex CLI performed exceptionally well in Genomics (97% accuracy) and maintained >70% accuracy across other domains.

Efficiency and Cost

• Time: Codex CLI was faster (avg. 11 mins/question) compared to Claude Code Opus (18 mins/question).
• Cost: Claude Code Opus was the most expensive (~$1.70/question), while Codex CLI was ~$1.00/question.
• Robustness: Agents showed high consistency; ~73% of questions were solved correctly on all three attempts for the top models.

Qualitative Observations

• Strengths: Agents demonstrated impressive ability to navigate web resources, install niche tools (e.g., Sourmash, CellTypist), and write custom code to implement algorithms (e.g., Expectation-Maximization for multi-mapping reads).
• Weaknesses (Brittleness):
  • Premature Stopping: Agents often stopped after a superficially plausible but incorrect analysis (e.g., ignoring multi-mapping reads in sequencing data).
  • Analytical Nuance: Failure to select appropriate background sequences in motif enrichment led to incorrect TF identification.
  • Tool Discovery: While generally good, agents sometimes struggled with obscure tools not in standard workflows (e.g., AMULET for doublet detection).

5. Significance and Future Outlook

• Paradigm Shift: The results suggest that general-purpose agentic systems are rapidly approaching the capability of human experts for routine, well-scoped computational biology tasks.
• Trust and Deployment: The benchmark provides a practical testbed to build trust in AI agents for scientific workflows. The authors suggest that future deployment will likely involve human-in-the-loop oversight for high-stakes or long-horizon tasks to audit intermediate outputs.
• Benchmark Design: The paper offers a blueprint for future benchmarks in other scientific domains: using synthetic/augmented data to create verifiable ground truths while preserving the complexity of real-world data.
• Limitations: The benchmark focuses on well-scoped problems and does not yet test the development of novel algorithms or the use of user-supplied tools (e.g., via Model Context Protocol). It also relies on external resources that may change over time.

In conclusion, CompBioBench demonstrates that modern agentic systems can effectively automate complex, multi-step biological analyses, provided they are given the autonomy to fetch tools and data. While still brittle on the most subtle analytical nuances, these systems represent a significant step toward autonomous scientific discovery.