Formalized scientific methodology enables rigorous AI-conducted research across domains

This paper proposes and validates a formalized, phase-gated scientific protocol for language models that decomposes research into procedural, integrity, and governance layers, demonstrating through six end-to-end projects that such constraints enable AI agents to produce rigorous, evidence-backed, and auditable scientific outputs across diverse domains while mitigating integrity risks compared to unconstrained approaches.

The Gist

Imagine you have a brilliant, incredibly fast apprentice who can read every book in the library, write a perfect essay in seconds, and solve complex math problems. This apprentice is an AI. But here's the catch: while the apprentice is smart, they don't know how to do real science. They might skip steps, hide their mistakes, or change the rules of the game halfway through just to make the result look better.

This paper introduces a solution called Amplify. Think of it not as a new brain for the AI, but as a strict, unbreakable rulebook and a team of supervisors that forces the AI to behave like a rigorous, honest scientist.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Fast & Loose" Apprentice

Without this system, an AI trying to do research is like a student who writes their history essay while they are still digging for facts in the library.

• They might write a conclusion before they've finished reading the sources.
• If they find a fact that contradicts their story, they might just ignore it (cherry-picking).
• If they make a math error, they might hope no one notices.
• Result: They produce a finished paper, but it's shaky, untrustworthy, and impossible to check.

2. The Solution: The "Three-Layer Safety Net"

The authors built a system that treats scientific research like a construction project with strict safety inspections. They broke the process down into three layers:

Layer A: The Roadmap (Procedural Workflow)

Imagine a construction site with seven specific checkpoints. You cannot pour the concrete (run the experiment) until the blueprint is signed off. You cannot paint the walls (write the paper) until the foundation is tested.

• The Rule: The AI must finish one stage completely and get a "stamp of approval" before moving to the next.
• The Safety Net: If the AI gets stuck or finds a problem, the map has red arrows pointing back. It forces the AI to go back and fix the foundation rather than trying to build a roof on a crumbling wall.

Layer B: The Integrity Police (Integrity Discipline)

This is a set of seven permanent rules that never sleep. They act like a security camera that never turns off.

• The "No Cheating" Rule: Once you decide how you will measure success (e.g., "we will use a specific test score"), you cannot change the rules later just because the score is low.
• The "Show Your Work" Rule: The AI must report everything, even the experiments that failed. No hiding bad data.
• The "Proof" Rule: Every claim the AI makes must be backed by a fresh calculation. It can't just say "I think this is true"; it has to prove it right now.

Layer C: The Project Managers (Governance)

This layer acts like a wise project manager who asks the hard questions.

• "Is this idea actually new, or are you just re-doing what others have done?"
• "If this approach keeps failing, should we stop or try a different angle?"
• It forces the AI to admit when it's going down a dead end, preventing it from wasting time on a project that will never work.

3. The "Board of Reviewers" (Multi-Agent Deliberation)

In human science, you don't just trust one person's word; you have peer review. The AI system simulates this by creating a virtual board of three different AI experts for every major decision:

1. The Domain Expert: "Does this make sense scientifically?"
2. The Skeptic: "What are the flaws? Where could this go wrong?"
3. The Editor: "Is the story clear and logical?"

These three "agents" argue with each other. They cannot agree to move forward unless all three say "Pass." If they disagree, the AI has to go back and fix the work. This catches errors that a single AI would miss, like a fake citation or a math mistake hidden in the text.

4. The Proof: The "Twin Study"

To prove this actually works, the researchers ran a controlled experiment (Project 6).

• They gave the exact same AI the exact same research task twice.
• Version A (The Wild West): The AI worked alone with no rules. It wrote a paper, but it was messy, hid some negative results, and mixed up its planning with its writing.
• Version B (The Strict Protocol): The AI used the Amplify system. It produced a paper that was modular, had all its data files neatly organized, reported its failures, and had every claim backed by fresh code.

The Result: The AI with the rulebook didn't just write a "nicer" paper; it wrote a reliable, auditable, and scientifically honest paper. It caught its own mistakes, fixed them, and produced results that could be trusted.

The Big Takeaway

The paper argues that the gap between "AI that can write about science" and "AI that can do science" isn't about making the AI smarter. It's about giving it better habits.

Just as a brilliant student needs a syllabus, a lab coat, and a strict professor to do good science, a powerful AI needs a formalized protocol to turn its raw intelligence into trustworthy discovery. The "Amplify" system is that protocol—a digital scaffold that holds the AI up until it learns to stand on its own as a rigorous researcher.

Technical Summary

1. Problem Statement

Current AI systems capable of scientific research (e.g., "AI Scientists") often lack methodological rigor. While they possess broad domain knowledge and can execute tasks, they frequently fail to adhere to the procedural norms that distinguish publishable science from exploratory exercises. Key issues include:

• Implicit Methodology: Critical steps (e.g., locking evaluation criteria before experiments, reporting negative results, excluding alternative hypotheses) are often learned through human apprenticeship rather than encoded as explicit rules.
• Auditability Gaps: Without explicit constraints, agents can silently skip steps, engage in "goalpost moving" (changing metrics post-hoc), or produce manuscripts where claims are not strictly grounded in evidence.
• Lack of Governance: Existing systems often lack mechanisms to force strategic pivots, manage failures, or enforce complete reporting, leading to integrity-relevant risks like metric drift and unverified references.

The core problem is that while AI has the capability to generate content, it lacks the methodological discipline to conduct reliable, auditable, end-to-end research.

2. Methodology: The "Amplify" Protocol

The authors propose Amplify, a formalized, executable research methodology instantiated as a phase-gated protocol with persistent constraints. This system is designed to be model-agnostic and domain-agnostic, operating as a layer atop general-purpose Large Language Models (LLMs).

The methodology is decomposed into three complementary constraint layers:

A. Procedural Workflow (The "When" and "How")

A seven-phase lifecycle with explicit entry/exit criteria and return paths for backtracking:

1. Domain Anchoring: Define scope and resources.
2. Direction Exploration: Literature review and ideation (using 6 strategies like contradiction mining and counterfactual reasoning).
3. Problem Validation: Adversarial multi-agent deliberation to test novelty and feasibility.
4. Method Design: Locking the evaluation protocol (metrics, seeds, baselines) before execution.
5. Experiment Execution: Two-stage process (exploratory probing $\rightarrow$ full execution) with mandatory iteration cycles.
6. Results Integration: Synthesizing outputs into a claim-evidence alignment table.
7. Paper Writing: Modular manuscript generation with automated reference verification.

• Validation Gates (G1–G4): Mandatory checkpoints requiring human approval (or strict automated criteria) before progression (e.g., G2 locks the protocol; G4 verifies evidence sufficiency).

B. Integrity Discipline (The "Always-On" Constraints)

Seven persistent constraints that remain active throughout the lifecycle to prevent specific failure modes:

• Metric Immutability: Evaluation criteria cannot change after locking without explicit authorization.
• Complete Reporting: All seeds, negative results, and failures must be disclosed.
• Claim-Evidence Alignment: Every assertion in the text must map to a specific code output or figure.
• Alternative-Hypothesis Exclusion: Causal claims require systematic exclusion of confounders.
• Reproducibility: Full environment logging and scriptable reruns.
• Verification: Fresh computational checks are required before finalizing claims.
• Figure Standards: Enforced visual quality and consistency.

C. Governance Layer (Strategic Oversight)

Four functions to manage the project's strategic direction:

• Novelty Assessment: Ensures the work is publishable.
• Scope Control: Manages exclusions and limitations.
• Failure Management: Forces structured reassessment (pivot/downgrade/stop) rather than unbounded iteration.
• Standards Alignment: Matches the output to target venue requirements.

D. Multi-Agent Deliberation

To simulate human peer review, the system deploys role-specialized sub-agents (Domain Expert, Skeptical Critic, Editor) at critical junctures. These agents assess artifacts independently using a shared rubric. Convergence requires unanimous "PASS"; otherwise, the artifact is modified and re-evaluated (up to 5 rounds).

3. Key Contributions

1. Formalization of Scientific Methodology: The first end-to-end decomposition of scientific practice into an executable, phase-gated protocol with persistent constraints, moving beyond ad-hoc prompting.
2. The Amplify Tool: An open-source implementation (MIT license) that acts as a plugin/controller for IDEs (e.g., Cursor), enforcing the protocol and generating auditable artifacts (planning docs, logs, claim-evidence tables).
3. Cross-Domain Validation: Successful application across five distinct scientific domains (Population Genomics, Paleogenomics, Human Evolutionary Genetics, Computational Population Genetics, Condensed-Matter Physics) without domain-specific code modifications.
4. Controlled Study Evidence: A rigorous matched comparison (Project 6) demonstrating that the protocol itself, not just the underlying model, is responsible for increased rigor and auditability.

4. Results

The authors validated the approach in six projects (5 protocol-constrained + 1 controlled study):

• Evidence-Backed Outputs: The protocol-constrained agents produced:

  • Closed-form derivations (e.g., in quantum physics).
  • Quantitative ablations resolving design choices (e.g., in HMM emission strategies).
  • Algorithmic refactors (e.g., converting binomial-likelihood EM updates to GPU-native GEMM operations, achieving a 213× speedup).
  • Biological Signal Recovery: Successfully recovered known signals (e.g., Neanderthal introgression on chromosome 21, admixture targets in the 1000 Genomes Project) as validity checks.

• Intervention Success Stories:

  • gpuADMIX: The verification constraint caught stale results from a buggy code version, forcing a re-measurement that reversed the paper's narrative from "limited performance" to "competitive performance." Multi-agent review caught a fabricated citation and formula mismatches.
  • DESI (Demography): Governance intervention at Gate 2 forced a complete project redesign when the human reviewer challenged the novelty. The agent subsequently retracted its own initial interpretation, downgraded the venue target, and performed rigorous stress tests.
  • HapGraph: Integrity constraints forced a complete rewrite of a tool when a critical flaw (insensitivity to admixture) was detected during Phase 4a. Multi-agent deliberation caught text-code mismatches in the IBD timing model.

• Controlled Study (Project 6):

  • Protocol-Free Baseline: Produced a complete manuscript but interleaved drafting with analysis, lacked externalized planning artifacts, and had higher risks of integrity issues (e.g., unstated metric changes).
  • Protocol-Constrained (Cursor+Amplify): Produced a modular manuscript with a full directory of auditable artifacts (intake anchors, locked protocols, integration blueprints). The process was transparent, and the agent explicitly documented failures and limitations.

5. Significance

• Bridging the "Capability vs. Rigor" Gap: The paper demonstrates that the gap between "AI that writes about science" and "AI that does science responsibly" is not a lack of model intelligence, but a lack of methodological transfer. By externalizing the procedural knowledge of experienced researchers, general-purpose LLMs can be steered into rigorous research practices.
• Auditability and Trust: The system transforms AI research from a "black box" generation process into a transparent workflow with auditable artifacts. This allows for turn-by-turn verification of claims, evidence, and decisions.
• Scalability: The approach is model-agnostic. As foundation models improve, the protocol layer remains fixed, allowing the system to scale in rigor without retraining. It shifts the cost from massive model training to the development of reusable, open-source protocols.
• Human-AI Collaboration: The protocol does not replace human judgment but structures it. Human researchers act as gatekeepers at critical decision points (Gates), focusing their effort on high-level scientific taste and intuition, while the AI handles the disciplined execution of the workflow.

In conclusion, this work establishes that formalizing scientific methodology as an executable protocol is a viable and effective strategy for enabling AI agents to conduct rigorous, reproducible, and auditable scientific research across diverse domains.