Technical Summary: BrainPilot – Automating Brain Discovery with Agentic Research

Problem Statement

Understanding the brain increasingly requires integrating evidence across scales, modalities, and disciplines. Conducting a single research question in neuroscience involves a coordinated sequence of operations: surveying prior work, designing experiments, executing analyses, and interpreting results within specific domain contexts. While AI agents promise to accelerate this process, current systems face critical limitations in brain science:

1. Lack of Domain Expertise: General-purpose agents often lack the specific, laboratory-level knowledge required for neuroscience.
2. Reliability Issues: Agents may fabricate claims, drift during multi-step reasoning, or produce results unsupported by evidence.
3. Opacity: Existing workflows offer few defined points for expert intervention, making it difficult for human researchers to inspect intermediate steps or correct methodological errors.
4. Cost and Efficiency: High-performance agent frameworks often incur significant computational and monetary costs.

The paper argues that for AI-assisted acceleration to be viable in brain science, the research process must remain trustworthy, with human expertise integrated at decision points and a clear audit trail of evidence and claims.

Methodology

The authors present BrainPilot, a fully open-source, human-in-the-loop multi-agent system designed to accelerate brain science research. The system is built on three core pillars:

1. Multi-Agent Architecture

BrainPilot employs a Principal Investigator (PI) agent that coordinates a team of specialist agents. The default team includes:

• PI Agent: Coordinates tasks, clarifies user intent, delegates subtasks, synthesizes results, and routes deliverables.
• Librarian: Surveys literature and provides knowledge grounding (Key Findings, Gaps, Hypotheses).
• Experimentalist: Designs scientific procedures, including operationalization, controls, and protocols.
• Engineer: Translates designs into reproducible code, executes it, and reports outputs.
• Writer: Drafts and polishes scientific documents based on specialist handoffs.
• Auditor: Independently verifies outputs for fabrication, checking numerical claims, file references, and citations against session evidence before delivery.

All agents share a messaging contract and a self-recording contract that logs tangible outputs to a central trace.

2. Curated Domain Knowledge

To ground agents in brain science, BrainPilot utilizes two offline-constructed assets, exposed as services rather than relying solely on parametric memory:

• Unified Knowledge Base: A retrieval corpus containing 7,233 indexed items (textbooks and papers) across 11 disciplinary groups (e.g., psychiatry, systems neuroscience, AI/ML). It is constructed via OCR, chunking, and embedding using the BAAI bge-m3 model, with a cross-encoder reranker for relevance.
• Skill Library: A collection of 72 reusable methodology units across seven research domains (e.g., cellular, cognitive, clinical). Skills are descriptive (methodological prescriptions) or executable (code routines). They are derived from LLM-authored specifications, distilled from papers, and extracted from established tools (e.g., DeepLabCut, MNE-Python, fMRIPrep). A skill router allows agents to query and load skills on demand, keeping context overhead low.

3. Graph of Trace (GoT)

The GoT is an append-only, auditable record that links subgoals, tool use, evidence, and claims. As agents complete subtasks, they report to the GoT, which constructs a directed acyclic graph (DAG) of the workflow. This allows researchers to:

• Follow the research trajectory in real-time.
• Inspect the basis of intermediate outputs.
• Identify and correct errors before they propagate.
• Verify that claims are supported by session evidence.

Key Contributions

1. System Design: A novel multi-agent framework specifically tailored for brain science that integrates curated domain knowledge, reusable skills, and an audit mechanism (Auditor agent) to mitigate hallucination and drift.
2. Knowledge Infrastructure: The creation of a unified brain science knowledge base (7,233 items) and a skill library (72 skills) that serve as a grounding layer for agentic reasoning.
3. Traceability: The implementation of the Graph of Trace, which provides a transparent, inspectable record of the entire research workflow, linking actions to evidence.
4. Benchmarking: The introduction of BrainPilotBench-v0, a preliminary benchmark specifically for brain science agents, featuring four tasks (RSC place-cell analysis, TOPS-fMRI, BCI Motor-Imagery, Sleep-EDF) with artifact-based scoring and strict isolation protocols.

Results

The paper evaluates BrainPilot across three dimensions:

1. Agents' Last Exam (ALE)

BrainPilot was tested on three brain science tasks from the ALE benchmark (skull-stripping QC, ROI resampling, C. elegans neuron tracking).

• Performance: With domain knowledge enabled, BrainPilot achieved performance comparable to state-of-the-art frameworks (Codex, Claude Code) on deterministic tasks. For instance, it achieved full credit (1.00) on T1 and T2 with specific backbone models.
• Cost & Efficiency: BrainPilot demonstrated substantial efficiency gains. Across matched backbones, it completed runs in 8–63% of the time and at 5–56% of the cost of competing frameworks. The cheapest configuration (BrainPilot + DeepSeek-V4-Pro) cost $0.08 compared to $22.53 for Codex + GPT-5.5.
• Limitation: Performance on the open-ended neuron-tracking task (T3) was lower, indicating challenges with highly unstructured problems.

2. BrainPilotBench-v0

Evaluated on four tasks spanning calcium imaging, fMRI, EEG, and sleep staging.

• Performance: BrainPilot with claude-opus-4.8 tied the highest score (1.00) on the RSC task and achieved the highest score (0.20) on the BCI task. It generally outperformed matched Claude Code configurations on fMRI tasks.
• Domain Knowledge Impact: The effect of enabling domain knowledge was mixed; it improved scores in 8 out of 26 valid cells, reduced scores in 14, and left 4 unchanged, suggesting that knowledge retrieval does not uniformly guarantee better performance in all contexts.
• Cost: Costs varied significantly by backbone. BrainPilot with deepseek-v4-pro was faster but slightly more expensive than Claude Code on the same task, while claude-opus-4.8 incurred a substantial cost premium for its higher performance.

3. Case Studies

Five end-to-end case studies demonstrated BrainPilot's ability to handle complex, multi-step workflows:

• Spatial Coding in RSC: Successfully executed a 5-step analysis (screening, visualization, decoding, correlation, dynamics) on two-photon calcium imaging data, producing coherent results and a scientific report.
• Functional Hierarchy in Mouse Visual System: Analyzed Neuropixels data to test physiological measures against anatomical hierarchy, identifying positive correlations in response latency and receptive field diameter.
• fMRI Pain Decoding: Trained a functional connectivity signature on experimental tonic pain data and validated its transfer to independent clinical cohorts, mapping weights to interpretable pain circuits.
• EEG Motor-Imagery: Designed a PyTorch model for BCI Competition IV 2a, achieving a positive trend over the EEGNet baseline (Cohen's kappa 0.493 vs 0.440).
• Sleep Staging: Implemented a 5-class sleep staging decoder on Sleep-EDF data, achieving metrics comparable to DeepSleepNet references with physiological validation.

Significance and Claims

The paper claims that BrainPilot represents a significant step toward trustworthy, efficient, and transparent agentic research in brain science. Its significance lies in:

• Bridging the Gap: Moving beyond general-purpose agents to systems grounded in specific, curated domain knowledge and reusable skills.
• Ensuring Validity: By integrating an Auditor agent and the Graph of Trace, the system addresses the critical issue of scientific validity, ensuring that claims are backed by evidence and that the workflow remains inspectable by human experts.
• Cost-Effectiveness: Demonstrating that open-source backbone models, when properly orchestrated with domain knowledge, can achieve performance comparable to expensive proprietary systems at a fraction of the cost.
• Community Framework: The release of BrainPilot, BrainPilotBench, and the underlying methodologies invites the community to contribute to a broader ecosystem of automated brain discovery, emphasizing that future progress depends on collaborative efforts to expand benchmarks and refine skills.

The authors maintain a modest tone, acknowledging that while the system accelerates research, human expertise remains essential for validating interpretations, defining methodological constraints, and steering the research process. They view BrainPilot as a foundation for future developments in multimodal integration and more complex sub-workflows.