Technical Summary: scBench-Long

Problem Statement

Single-cell biology research requires analysts to synthesize raw measurements into specific biological claims through complex, multi-step workflows. These workflows demand the integration of metadata, assay context, auxiliary evidence (such as immune repertoires or chromatin accessibility), and prior literature. While AI agents are increasingly applied to biological data analysis, existing benchmarks largely evaluate broad biological knowledge, isolated tool execution, or local analysis steps. There is a lack of verifiable benchmarks that test whether agents can perform long-horizon scientific reasoning—specifically, the ability to recover complex, claim-conditioned conclusions from raw or near-raw single-cell data without prescribed methods. Furthermore, defining verifiable ground truth in this domain is challenging, as the same dataset can support multiple valid claims, and published results do not always independently reproduce.

Methodology

Benchmark Design (scBench-Long)

The authors introduce scBench-Long, a benchmark designed to evaluate long-horizon single-cell biology tasks. The benchmark is structured around 21 evaluations derived from five distinct study systems:

1. Melanoma CD8 T-cell reactivity: Linking gene-expression states with T-cell receptor (TCR) clonotypes.
2. CD8 RNA+ATAC regulatory inference: Evaluating regulatory claims supported by both RNA expression and chromatin accessibility.
3. Human–monkey chimera development: Comparing developmental states across species and evaluating inter-population signaling.
4. KRAS-driven lung tumor aging: Separating age-associated effects from tumor-cell states and microenvironment changes.
5. Lethal COVID-19 lung pathology: Integrating single-nucleus expression with immune-receptor and orthogonal validation evidence.

Data and Inputs:
Tasks provide raw or near-raw data (e.g., 10x Chromium 5′/3′ scRNA-seq, Multiome, Smart-seq2, Parse Biosciences combinatorial sequencing) alongside calibrated experimental context and auxiliary resources (ortholog maps, ligand–receptor databases, developmental references). Crucially, the benchmark does not prescribe a solution path; agents must determine the workflow.

Evaluation Workflow:

1. Claim Generation: Candidate claims are sourced from published papers and internal reproduction notes.
2. Reproduction & Review: Tasks are retained only if the target conclusion can be independently reproduced from the staged data. This process includes randomized peer review and trajectory inspection to ensure robustness.
3. Controlled Answer Surfaces: Agents must output structured JSON objects over controlled vocabularies (e.g., cell-state identities, regulator directions, clonotype claims). Distractors are designed to penalize common shortcuts, such as relying on canonical marker priors, raw abundance rankings, or single-modality reasoning.
4. Grading: A deterministic grader evaluates the final structured answer. Pass/Fail is binary based on exact recovery of the target surface.

Diagnostic Framework:
To address the sparsity of binary pass/fail grading, the benchmark employs rubric-based trajectory diagnostics. Evaluation authors define "chokepoints" (critical analysis steps) after independent reproduction. LLM judges score trajectories against these rubrics to diagnose partial progress and failure modes, though these scores serve as diagnostics rather than the primary benchmark metric.

Key Contributions

• Long-Horizon Benchmarking: scBench-Long shifts the focus from isolated tool execution to the recovery of complex scientific conclusions from raw data, mirroring the iterative nature of real-world single-cell analysis.
• Verifiable Ground Truth: The benchmark establishes a rigorous construction workflow where claims are hardened through independent reproduction and review, ensuring that the "ground truth" is reproducible from the provided data rather than relying on the original paper as an unverified oracle.
• Multi-Modal Integration: The tasks explicitly require integrating diverse data modalities (RNA, ATAC, TCR/BCR, cross-species mapping) and distinguishing between association and mechanism.
• Diagnostic Granularity: The inclusion of rubric judges provides dense, interpretable feedback on agent performance (e.g., identifying specific failure modes like confusing abundance with importance) alongside the sparse endpoint scores.

Results

Overall Performance

Across 1,068 completed trajectories involving 17 model–harness pairs (combining models like Claude Opus, GPT-5, Gemini, Grok, Kimi, and GLM with harnesses like PI, Claude Code, and OpenAI Codex):

• Success Rate: The strongest performing pair, Claude Opus 4.8 with Claude Code, achieved a pass rate of 25.4% (16/63 runs). The second-best, Gemini 3.5 Flash with PI, achieved 22.2% (14/63).
• Reliability: Success is not yet reliable. The top pair passed at least one replicate for only 8/21 evaluations and achieved a "perfect" run (all 3 replicates passing) for only 2/21 evaluations.
• Task Difficulty: Five evaluations had zero successful trajectories. The highest-performing evaluation (a pseudotime-delay task) saw a 51.0% pass rate, while most evaluations remained in low-pass bins.

Harness and Model Effects

• Harness Sensitivity: Performance varies significantly based on the execution harness. For example, GPT-5.4 performed better with PI than with OpenAI Codex, while Claude Opus 4.8 performed better with Claude Code. This indicates that long-horizon performance depends on the model–harness system, not just the base model.
• Rubric Diagnostics: While endpoint success was sparse (11.7% of trajectories), rubric scores were nonzero for nearly all trajectories. There was a moderate correlation between rubric scores and endpoint success (AUC = 0.77), suggesting rubrics can identify partial progress. However, high rubric scores did not guarantee endpoint success, confirming they are diagnostic tools rather than replacements for deterministic grading.

Failure Modes

The paper identifies four recurring failure modes where agents failed to translate intermediate analysis into correct conclusions:

1. Reliance on Biological Priors: Agents often substituted familiar literature associations (e.g., CD39/PD-1 for exhausted T cells) for specific empirical evidence in the provided data, even when the data structure contradicted the prior.
2. Abundance vs. Importance: Agents frequently mistook large raw quantitative values (e.g., high interaction counts in ligand–receptor analysis) for biological significance, ignoring directionality or database composition.
3. Association vs. Mechanism: Agents often inferred causal mechanisms from cross-sectional data (e.g., assuming tumor programs were induced by T-cell contact simply because they co-occurred in conjugates).
4. Modality Integration Failure: Agents often analyzed single modalities (e.g., RNA only) when the task required integrating multiple data types (e.g., RNA + ATAC) to validate regulatory claims.

Significance and Claims

The authors position scBench-Long as a measuring device and a diagnostic lens for the development of AI agents in single-cell biology. The paper claims that current frontier agents show "limited but measurable capability," capable of executing plausible workflows but struggling to synthesize them into verified scientific claims.

The benchmark is presented not as a final solution but as an evolving specification for computational biology workflows. The authors emphasize that improving agent performance in this domain will require advancements in both model training and harness engineering. By providing a verifiable, claim-conditioned framework, scBench-Long aims to support test-driven development of agents that can analyze single-cell data faithfully, transparently, and reproducibly, moving beyond local analysis steps to the complex reasoning required in drug discovery and basic research.