Technical Summary: NatureBench

Problem Statement

As AI coding agents evolve from reproducing published code to conducting autonomous scientific research, a critical evaluation gap has emerged. Existing benchmarks for scientific agents suffer from two primary limitations:

1. Reproduction vs. Discovery: Current paper-based benchmarks (e.g., PaperBench, CORE-Bench) primarily test an agent's ability to reproduce a published method. They fail to answer whether an agent can independently discover a competitive or superior method on real scientific problems.
2. Environment Fragmentation: Engineering optimization benchmarks (e.g., MLE-bench, PostTrainBench) often rely on Kaggle-style tasks or post-training objectives that lack the domain reasoning and specialized tooling of natural sciences. Furthermore, they suffer from environment fragmentation, making independent verification and reproducible execution fragile.

There is a need for a benchmark that is both challenging and bidirectional: testing whether agents can devise methods that surpass the published state-of-the-art (SOTA) on genuine scientific problems drawn from natural sciences, rather than on engineering proxies.

Methodology: NatureGym and NatureBench

The authors introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, built upon NatureGym, an automated pipeline designed to convert papers into reproducible, containerized task packages.

1. The NatureGym Pipeline

NatureGym addresses the environment-fragmentation problem through a three-stage, review-gated pipeline that transforms a source paper into a standardized task package:

• Stage 1: Paper Filtering: A three-level cascade filter evaluates papers based on task extractability (must yield an ML task), evaluation automatability (must have a deterministic, automated metric), and data completeness (publicly accessible, <50GB). An adversarial review step targets false positives.
• Stage 2: Dataset Acquisition & Verification: The pipeline downloads data and establishes a strict information firewall. It separates the task definition (inputs) from the source paper's core algorithm (outputs/methods). Only files necessary to define the task (e.g., raw inputs, shared data preparation) are retained; files specific to the original algorithm are excluded.
• Stage 3: Task Package Construction: The final package includes a task brief, dataset, a held-out test set with hidden ground truth, and an automated evaluator. The source method is completely removed, forcing agents to discover solutions rather than reproduce them.

2. Benchmark Composition

• Source: 90 tasks derived from 10 Nature-family journals (2022–2025), including Nature Machine Intelligence, Nature Methods, and Nature Biomedical Engineering.
• Domains: Six scientific task domains: Cellular Omics (31 tasks), Protein Biology (16), Biomedical Modeling (14), Physical Modeling (13), Molecular Design (11), and Relational Reasoning (5).
• Evaluation Protocol: Agents operate in isolated Docker containers with a 4-hour wall-clock budget and restricted access (no web search). They submit iterative solutions scored against the paper's reported SOTA.
• Metric: The primary metric is the SOTA-normalized relative gap ($g$):
  $$g_i = \text{dir}_i \cdot \frac{m_i - m_{\text{sota}}}{|m_{\text{sota}}|}$$
  Where $m_i$ is the agent's score and $m_{\text{sota}}$ is the paper's reported SOTA. A value $g > 0.1$ indicates surpassing SOTA; $g \ge 0$ indicates matching SOTA.

3. Experimental Setup

Ten frontier agent configurations were evaluated under a strict web-search-disabled protocol. These included models from three harnesses:

• Claude Code: Paired with Claude Opus 4.6, 4.7, Kimi K2.6, MiniMax-M2.7, DeepSeek-V4-Pro, GLM-5.1, and Qwen 3.7 Max.
• Codex CLI: Paired with GPT-5.4 and GPT-5.5.
• Gemini CLI: Paired with Gemini 3.5 Flash.

Key Results

1. Performance vs. Published SOTA

Even the strongest agents struggle to surpass the published SOTA on real scientific problems:

• Surpass-SOTA ($g > 0.1$): The top performer, Claude Opus 4.7, surpassed SOTA on only 17.8% of tasks. The next best, Gemini 3.5 Flash and GPT-5.5, achieved 15.6% and 14.4% respectively. The weakest model (MiniMax-M2.7) achieved only 1.1%.
• Match-SOTA ($g \ge 0$): The best agent matched or exceeded SOTA on 47.8% of tasks. The remaining agents ranged between 13.3% and 37.8%.
• Validity: The Claude Opus agents demonstrated the highest integrity, with 100% completion and score rates and zero invalid (shortcut) submissions.

2. Analysis of Success and Failure Mechanisms

• Success Pathways: When agents succeed, they primarily do so through methodological translation (45.5% of successes), converting complex scientific tasks into familiar supervised prediction problems. Genuine scientific invention or domain-reasoned alternatives accounted for less than 10% of successes.
• Failure Modes: Failures are dominated by wrong method choice (45.1%) and insufficient compute budget/time (24.4%). Misunderstanding the task or producing malformed output accounts for a negligible fraction of failures.
• Domain Difficulty: A consistent difficulty gradient exists across domains. Relational Reasoning and Protein Biology are the easiest, while Biomedical Modeling and Molecular Design are the hardest. Interdisciplinary tasks (combining multiple domains) consistently show wider gaps to SOTA than single-discipline tasks.

Significance and Claims

The paper claims that NatureBench is the first benchmark to combine paper-sourced tasks, genuine scientific domains, and discovery-oriented evaluation (surpassing SOTA) into a single framework.

• Beyond Reproduction: The results indicate that current AI coding agents are capable of engineering optimization and reproducing known patterns but are not yet capable of genuine scientific discovery that consistently outperforms human-authored methods in the literature.
• The "Translation" Bottleneck: The dominant success mechanism is the translation of scientific problems into standard ML pipelines, suggesting that agents are currently "AI-for-Science" accelerators rather than "AI-native" problem solvers. They excel at applying known ML tools but struggle with the methodological choices required for novel scientific discovery.
• Benchmark Validity: The authors assert that the benchmark's design, particularly the information firewall and the SOTA-normalized metric, provides a credible, reproducible, and standardized way to evaluate autonomous scientific research, addressing the fragmentation and credibility issues of prior benchmarks.

The authors release the benchmark, the NatureGym pipeline, and a public leaderboard to facilitate future research into turning these substrates into training data for next-generation scientific discovery agents.