Technical Summary: Verifiable Benchmarking of Long-Horizon Spatial Biology

Problem Statement

Current AI agents demonstrate utility in biological data analysis but struggle to perform end-to-end scientific reasoning over raw spatial measurements. Existing benchmarks typically test broad biological knowledge, executable workflows, or localized analysis steps, rather than the ability to derive specific scientific conclusions from raw data and calibrated experimental context without prescribed methods. Furthermore, raw spatial data rarely admits a single universal ground truth; the same measurements can support multiple valid conclusions, and published claims often fail to reproduce cleanly under unbiased reanalysis. There is a lack of benchmarks that investigate whether an agent can undertake the full workflow of recovering a specific scientific conclusion from raw spatial measurements, integrating multi-step workflows, assay-specific knowledge, and experimental design.

Methodology

Benchmark Design: SpatialBench-Long

The authors introduce SpatialBench-Long, a benchmark designed to evaluate whether agents can recover biological claims from raw or near-raw spatial biology data and calibrated context.

• Scope: The benchmark comprises 24 evaluations across four distinct study systems:
  1. Primary pancreatic ductal adenocarcinoma (PDAC).
  2. Engineered glioblastoma (GBM) organoids and in vivo tumors.
  3. Cas9 lineage-traced lung adenocarcinoma.
  4. Mouse optic nerve aging and intervention systems.
• Data Modalities: The evaluations utilize diverse spatial and molecular technologies, including CosMx, Visium, Xenium, MERFISH, single-cell RNA sequencing (scRNA-seq), Slide-seq, Slide-tags, histology, and lineage-recording data.
• Task Structure: Each task provides a scientific question, raw/near-raw data, relevant assay context, and a verifiable solution. Tasks are calibrated to approximate the knowledge a scientist possesses at the start of an analysis, balancing over-specification against ambiguity. Identifying information is scrubbed to prevent literature lookup shortcuts.
• Claim Hardening: Candidate claims are not treated as automatic ground truth. They are refined through independent reproduction, randomized expert peer review, and trajectory inspection. Many candidate claims were excluded because they did not reproduce robustly from the provided data.

Grading and Diagnostics

• Deterministic Final-Answer Grading: The primary benchmark score is a binary pass/fail based on the recovery of scientific conclusions expressed through controlled biological vocabularies and ordered relationships. This grading is deterministic and does not rely on isolated numerical outputs.
• Rubric-Based Trajectory Diagnostics: Recognizing that final-answer grading provides sparse signal for long-horizon tasks, the authors employ rubric-based trajectory judging as a companion diagnostic.
  • Chokepoints: Evaluation authors define "chokepoints"—critical analysis decisions or biological constraints (e.g., choosing the correct comparison, identifying relevant spatial compartments, avoiding known traps)—that are expected to remain stable across plausible solution paths.
  • Scoring: Large Language Model (LLM) judges score model trajectories against these rubrics. These scores are treated as diagnostic evidence to identify partial progress and distinguish total failure from partial scientific success, rather than as replacement benchmark scores.

Experimental Setup

• Models and Harnesses: The benchmark was run across 15 model-harness pairs, including frontier models from OpenAI (GPT-5.5, GPT-5.4), Google (Gemini 3.5 Flash, 3.1 Pro, 2.5 Pro), Anthropic (Claude Opus 4.7/4.6, Sonnet 4.6), xAI (Grok 4.20/4.3), and Kimi (K2P6).
• Execution: Models were evaluated using harnesses such as Pi (terminal coding), OpenAI Codex, and Claude Code. Each pair was run on all 24 evaluations with three independent replicates, yielding 1,080 total trajectories (72 runs per pair).

Key Results

Low but Nonzero Success Rates

Across all 15 model-harness pairs, success rates were low but nonzero.

• Top Performers: Three pairs tied for the highest performance, each passing 8/72 runs (11.1%):
  • Gemini 3.5 Flash / Pi
  • GPT-5.5 / Pi
  • GPT-5.5 / OpenAI Codex
• Replicate Stability: Success was sparse at the evaluation level. Even among leading systems, passing at least one replicate occurred for only 4–5 of the 24 evaluations, and majority-replicate passing occurred for only 2 of 24.
• Cost and Turns: There was no clear separation in performance among leading models based on cost or number of turns, though some models (e.g., Grok 4.3) achieved low pass rates with significantly lower cost and turn counts.

Rubric Diagnostics

• Correlation with Success: Rubric scores showed a modest association with deterministic pass/fail status (Pearson r = 0.24, ROC AUC = 0.79). Trajectories that passed the endpoint grader had higher mean rubric scores (72.2%) than failing ones (45.8%).
• Signal Density: Rubric judging provided much denser signal than endpoint grading. While only 5.9% of trajectories passed the deterministic grader, 94.2% of trajectories had nonzero mean rubric scores.
• Model Bias: Rubric scores varied systematically by source model. OpenAI and Anthropic trajectories generally received higher rubric scores, while Gemini 3.5 Flash had a high verifiable pass rate but a lower mean rubric score (46.5%), suggesting potential brittleness in either the rubrics or the ground truths for that model.
• Reproducibility: Judge-to-judge reproducibility was high (mean pairwise correlation 0.93).

Failure Modes

Manual review of 75 trajectories identified seven recurring behavior groups:

1. Final-decision failures: Agents performed most of the analysis but selected the wrong final target (common in OpenAI/Anthropic trajectories).
2. Partial-credit answers: Agents returned some accepted components but lost credit due to omissions, aliases, or vocabulary mismatches.
3. Missed metadata: Agents failed to incorporate critical metadata (e.g., swapped labels in optic nerve tasks).
4. Wrong grouping variables: Agents used expression intensity or generic contrasts instead of lineage-aware comparisons (common in lineage tasks).
5. Inappropriate spatial methods: Agents clustered expression rather than neighborhoods or pooled coordinates incorrectly.
6. Prior/vocabulary shortcuts: Agents relied on prior biology knowledge rather than analyzing the specific data.
7. Runtime/data-wrangling issues: Failures in local computation or data handling.

Significance and Claims

The paper claims that SpatialBench-Long is the first benchmark to investigate whether agents can undertake the end-to-end work of recovering a specific scientific conclusion from raw spatial measurements without prescribed methods.

• Current Limitations: The results indicate that current frontier agents possess low but nonzero capability in this domain. The authors argue that the primary barrier to reliable long-horizon scientific reasoning is not a lack of general biological knowledge, but rather compounding local analysis errors.
• Procedural Competence: The paper posits that before models can reliably reason about complex disease mechanisms or drug responses, they must first become procedurally competent at concrete molecular measurements. This includes understanding the constraints of specific kits, machines, tissue types, coordinate systems, and normalization regimes.
• Progression Path: The authors suggest a progression similar to coding agents: moving from assay-specific analysis competence to emergent biological reasoning, and eventually to synthesis across modalities and translational contexts. SpatialBench-Long is designed to make this progression measurable.
• Diagnostic Utility: The study validates the use of rubric-based trajectory diagnostics as a necessary companion to sparse endpoint grading, providing higher-resolution information about partial progress and helping to identify cases where grading assumptions need revision.

The paper concludes that while current systems show promise, they are not yet capable of reliably deriving accurate scientific conclusions from complex spatial measurements, highlighting a significant gap between executing procedural analysis and deriving scientific truth.