Technical Summary: ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence

Problem Statement

Autonomous research agents have rapidly advanced in their ability to generate competitive solutions and professional-looking manuscripts for complex systems-optimization tasks. However, a critical structural tension exists between generation and verification. Current autonomous systems operate as multi-stage pipelines where errors introduced in early stages (e.g., literature review or hypothesis generation) are preserved and amplified in later stages (experimentation and writing).

The core problem identified is a verifiability gap: existing evaluation protocols assess surface presentation (e.g., writing quality, procedural completion) but fail to audit whether individual claims are supported by evidence. Consequently, autonomous agents frequently produce outputs containing:

• Hallucinated references: Citations that do not correspond to real publications.
• Unreproducible scores: Reported metrics that do not match results when re-run on canonical evaluators.
• Method–code divergence: Descriptions of algorithms in the paper that differ fundamentally from the submitted code.
• Specification violations: Solutions that exploit evaluator loopholes rather than solving the underlying task.

These failures stem from a lack of mechanisms to trace claims back to their grounding sources (e.g., execution logs, retrieved PDFs, or code artifacts).

Methodology: Chain-of-Evidence (CoE)

To address this, the authors propose Chain-of-Evidence (CoE), a verifiability framework analogous to ACID properties in database transactions. CoE defines that every research claim must be traceable, through a recorded chain of supporting claims and evidence, to a grounding source.

The framework is instantiated through three primary contributions:

1. The CoE Standard

A taxonomy of claim types (Citation, Numerical, Methodological, Conclusion) and the specific evidence chain structure required for each. For example, a numerical claim must trace from the reported value to a recorded execution log, and a citation must trace to a retrieved PDF from a scholarly database.

2. ScientistOne System

An end-to-end autonomous research system designed to satisfy CoE by construction. Its pipeline consists of three stages:

• Stage 1: Literature Grounding (Problem Investigator): Instead of relying on parametric memory, the system retrieves up to 100 full-text PDFs per topic via scholarly APIs (e.g., Semantic Scholar), builds a citation graph, and produces a structured research brief. This ensures all citations are grounded in retrieved artifacts.
• Stage 2: Discovery (Ideator & Parallel Explore-Exploit): The system generates candidate approaches and evaluates them in parallel branches. Crucially, it maintains execution logs, scores, and solver code as structured artifacts passed to the next stage.
• Stage 3: Paper Writing & Verification: The Paper Writer operates on a "provenance before prose" principle. It generates a structured markdown narrative where every factual claim carries an inline evidence tag binding it to a specific artifact (log line, score file, or citation key). A Claim Verifier then checks every claim in the draft against its declared source before the final LaTeX is produced, flagging unsupported claims, hallucinated citations, or numerical mismatches.

3. CoE Integrity Audit

A post-hoc audit protocol applied uniformly to any autonomous research system. It performs four integrity checks on submitted artifacts (paper, code, bibliography):

• I1 Score Verification: Re-runs the solution on a golden evaluator to verify reported scores.
• I2 Specification Violation: Detects if code exploits evaluator logic or violates task rules.
• I3 Reference Verification: Queries academic APIs to verify the existence and accuracy of bibliography entries.
• I4 Method–Code Alignment: Uses LLMs to judge if the paper's method description faithfully reflects the submitted code.

Experimental Setup

The authors evaluated ScientistOne and four baseline systems (Sakana AI-Scientist v2, AutoResearchClaw, DeepScientist, AI-Researcher) across 75 papers (15 per system) generated for five frontier systems-research tasks from the ADRS benchmark (Prism, Cloudcast, EPLB, LLM-SQL, TXN). All systems were adapted to use the same backbone LLM (Gemini 3.1 Pro) and identical resource budgets to ensure a fair comparison.

Key Results

1. Integrity and Verifiability

ScientistOne was the only system to achieve perfect or near-perfect scores across all four integrity checks:

• Score Verification: 12/12 (100%) of papers had scores that reproduced exactly. Baselines ranged from 42% to 92%.
• Reference Verification: 0/337 hallucinated references. Baselines exhibited hallucination rates up to 21% (DeepScientist).
• Method–Code Alignment: 14/15 (93%) papers aligned. Baselines ranged from 20% to 80%.
• Specification Violations: 0/15 violations.

In contrast, every baseline system exhibited at least one systematic failure mode. For instance, DeepScientist hallucinated references in 21% of its bibliography entries, and Sakana AI-Scientist v2 frequently cherry-picked scores from different stages of its search tree, leading to non-reproducible results.

2. Solution Discovery Performance

ScientistOne matched or exceeded human expert performance on all five ADRS tasks.

• It achieved the best overall scores on Cloudcast and EPLB.
• It demonstrated zero hallucinated references and perfect score verification, proving that maintaining strict evidence chains does not come at the cost of solver competitiveness.
• In a "Best-of-3" selection, ScientistOne achieved a 6.6 overall rating from an automated peer-review system (ScholarPeer), significantly outperforming baselines.

3. Generalizability

The system was tested on six additional tasks outside the ADRS domain, including medical imaging, fine-grained recognition, 3D perception, and parameter-constrained language modeling (Parameter Golf).

• MLE-Bench: ScientistOne achieved Gold Medals on two high-difficulty tasks (3D Object Detection, RSNA Brain Tumor) where a baseline (DeepScientist) failed entirely on 3D Object Detection.
• Parameter Golf: ScientistOne achieved State-of-the-Art (SOTA) performance (1.0600 score) under strict size constraints, while the baseline failed to produce a valid submission due to exceeding the size limit.

Significance and Claims

The paper argues that the differentiator for autonomous research is no longer just the quality of the solution code, but the trustworthiness of the research output.

• Verifiability is an architectural constraint: The authors claim that systems designed to maintain evidence chains at the time of claim production (like ScientistOne) produce significantly more verifiable outputs than those that attempt to reconstruct grounding after the fact.
• The "Verifiability Gap" is real and measurable: The audit of 75 papers demonstrates that current state-of-the-art autonomous systems are prone to systematic integrity failures that are undetectable by surface-level evaluations.
• CoE as a Standard: The paper proposes CoE not just as a feature for one system, but as a standard for the field, suggesting that future autonomous research systems must be auditable by design to be considered reliable.

The authors conclude that while challenges remain (e.g., verifying qualitative claims or citation support in open-ended domains), the Chain-of-Evidence framework provides a tractable path toward human-level, trustworthy autonomous research.