Technical Summary: FirstResearch – Auditable Question Formation for LLM Scientific Discovery Agents

1. Problem Statement

While Large Language Model (LLM) systems are increasingly capable of assisting in scientific discovery—handling ideation, literature synthesis, experiment planning, and report generation—a critical bottleneck remains: the audibility of the initial research question.

Current autonomous research agents (e.g., AI Scientist, Agent Laboratory, AI co-scientist) often produce research questions that sound plausible but lack the structural rigor required for scientific review. Specifically, these questions frequently fail to expose:

• The underlying mechanism being tested.
• The specific assumptions made by the agent.
• A clear falsifier (an observation that would reject the hypothesis).
• A defined failure update rule (how a negative result should alter the research direction).

Without an explicit derivation record, it is difficult for human scientists or downstream agents to distinguish between a genuine discovery of a scientific tension and a superficial imitation of a research proposal. The core challenge is not merely finding a plausible topic, but forming a question that is mechanistic, falsifiable, and traceable.

2. Methodology: FirstResearch Framework

FirstResearch is a first-principles research-question formation framework designed to treat the derivation of a question as a "first-class artifact." It operates as a front-end ideation layer for scientific LLM workflows, distinct from downstream execution or data analysis.

Core Architecture

The system follows a staged derivation pipeline (Figure 1) that maps a scientific topic to an auditable research package:

1. Primitives & Assumptions: The system first defines primitive concepts and first-principle assumptions related to the topic.
2. Mechanism Model: It constructs a model of the mechanism under investigation.
3. Tension Finder: It identifies contradictions or tensions within the mechanism model.
4. Candidate Generation: It generates candidate research questions based on these tensions.
5. Certificate Construction: For each candidate, a Research Question Certificate is built.
6. Novelty Gate & Repair: A "novelty-aware gate" evaluates certificates. If a certificate is valid but generic, a CertificateRepairer is invoked to enforce "mechanism-boundary signals" (e.g., thresholds, phase transitions, failure regimes, nonlinear tradeoffs).
7. Packaging: Certificates are sorted by a tuple of metrics (average score, novelty, mechanism clarity, etc.), and the strongest is exposed to the judge.

The Research Question Certificate

The central artifact is the structured Research Question Certificate, which serves as a scientific provenance record. It explicitly requires:

• Primitive definitions and first-principle assumptions.
• A mechanism model and identified tension.
• A falsifiable hypothesis.
• A minimal decisive test.
• Expected observations.
• A failure update rule (specifying how assumptions change if the test fails).
• Quality scores for derivation, falsifiability, and clarity.

Baselines and Variants

The framework was compared against three controlled prompt-level baselines approximating existing systems:

• CoScientistBaseline: Mimics generation, debate, ranking, and evolution of hypotheses.
• AgentLabBaseline: Simulates literature review, experiment planning, critique, and synthesis.
• TreeSearchScientistBaseline: Generates a frontier of branches, selects one by novelty/feasibility, and expands it.

A combination variant, FirstResearch Debate Combo, was also tested, inserting a debate/refinement step before certificate construction to test if dialectic search complements the certificate core.

3. Experimental Setup

• Benchmark: Ten LLM-agent scientific research topics were selected, focusing on the study of LLM research agents themselves (e.g., skill discovery, process fidelity, tool routing, memory interference).
• Evaluation Protocol:
  • Primary Judge: DeepSeek (blind-judge protocol).
  • Independent Judge: Gemini-2.5-Flash (rescored the same 40 baseline packages).
  • Rubric: Five 0–5 dimensions: first-principles derivation, falsifiability, mechanism clarity, novelty, and experimentability.
  • Ablation: A "CertificateOnly" variant (keeping the core derivation but removing downstream review layers) and a "NoCertificate" variant (bypassing the certificate path entirely) were tested.

4. Key Results

• System Performance: Under the primary DeepSeek judge, FirstResearch outperformed all three controlled baselines on average rubric score and novelty.
• Judge Consistency: The system-level ranking was preserved when rescored by Gemini-2.5-Flash. FirstResearch achieved a score of 4.86/5, compared to 4.38/5 for the strongest baseline. The Pearson agreement between the two judges on average scores was 0.865.
• Ablation Findings: The certificate-centered core was identified as the strongest component:
  • CertificateOnly: Scored 4.90/5 (DeepSeek) and 4.88/5 (Gemini).
  • NoCertificate: Scores dropped below 1/5 under both judges.
  • This suggests that the explicit derivation constraints and the certificate structure are the primary drivers of quality, rather than downstream meta-layers or debate mechanisms.

5. Claims and Significance

The paper makes a narrow, preliminary scientific-discovery claim:

• Explicit derivation constraints (specifically the Research Question Certificate) are a promising mechanism for making LLM-generated scientific questions more auditable.
• The framework improves the judged quality of research packages for LLM-agent topics by forcing the exposure of primitives, mechanisms, tensions, and falsifiers.

Limitations and Scope:

• The authors explicitly state they do not claim FirstResearch is a complete autonomous scientist or that it can replace execution-heavy scientific agents.
• The results are preliminary and rely on LLM judges rather than human domain experts.
• The benchmark is a "first testbed" for scientific ideation within the domain of LLM agents, not a claim of cross-domain discovery in fields like biology or chemistry.
• The work supports the hypothesis that structured provenance records (certificates) are essential for making AI ideation inspectable before downstream execution.