🔬 materials science

Toward Auditable AI Scientists: A Hypothesis Evolution Protocol for LLM Agents

This paper introduces the Hypothesis Evolution Protocol (HEP), a framework that equips large language model agents with explicit, auditable operations for hypothesis generation, testing, and belief revision, thereby enabling transparent and verifiable AI-driven scientific discovery.

Original authors: Izumi Takahara, Teruyasu Mizoguchi

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Izumi Takahara, Teruyasu Mizoguchi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a super-smart robot detective to solve a mystery about why certain materials form specific crystal shapes. In the past, these robot detectives (called LLM agents) would run around, try things, and scribble their thoughts in a messy, unorganized notebook. If you asked them, "Why did you think that?" or "How did you change your mind?", they'd just point to a jumbled page of text. It was impossible to tell if they were actually learning from their mistakes or just guessing wildly.

The authors of this paper say, "That's not good enough for real science." So, they built a new set of rules called the Hypothesis Evolution Protocol (HEP). Think of HEP not as a notebook, but as a strict, transparent courtroom where the robot detective must present its case.

The Old Way vs. The New Courtroom

In the old "planning-style" approach, the robot would just jump from "I have an idea" to "Let's test it" to "Here's the result" without ever clearly stating, "I now believe this idea is 80% true because of this specific test." It was like a detective who solves a case but refuses to show the evidence log.

With HEP, every single idea the robot has is treated like a persistent character in a story.

The Birth: When the robot comes up with a new idea (a hypothesis), it gets a unique ID card and a "belief score" (a probability from 0 to 1) showing how sure the robot is.
The Trial: The robot runs tests. If the test results are good, they get attached to the idea like evidence in a file.
The Verdict: The idea's belief score goes up or down based only on that evidence.
- If the score hits 0.8 or higher, the idea is declared "Supported" (Guilty of being true!).
- If it drops to 0.2 or lower, it's "Refuted" (Not guilty, or just plain wrong).
- If it gets stuck in the middle, it goes into "Dormant" (Sleeping until new evidence wakes it up).

The coolest part? The robot can't cheat. It can't just say, "I changed my mind because I felt like it." It has to show the specific piece of evidence that moved the score. And if the robot tries to skip a step, the system says, "Nope, try again."

What the Robot Actually Did

The researchers tested this new system on three tricky materials science puzzles. They asked the robot to figure out which crystal structure a specific computer model preferred for different families of compounds and why.

Here is what happened in these simulations:

The Robot Learned: The HEP-equipped robot didn't just guess; it actually went through the full cycle of "Idea → Test → Evidence → Belief Update." In one specific run, it generated 16 different ideas, tested them, and eventually merged them into one final "master rule" with a belief score of 0.97.
The Old Robot Failed: When they ran the exact same tasks with a standard robot (without HEP), it spent 83% of its time just running tests and 0% of its time explicitly updating its beliefs. It never really "learned" in a way you could trace.
The Flip Side: The robot was honest enough to change its mind. In one case, it started with a 50% belief in a textbook idea, but the evidence proved it wrong, dropping the score to 0.14. It then pivoted to a new, better idea. This "belief flip" happened in all three tasks.

The "Brain Power" Requirement

There's a catch, though. This new courtroom system works best if the robot has a big brain. The researchers tested HEP with three different versions of the robot's "brain" (LLMs):

The Super Brain (GPT-5.5): It generated an average of 14.7 ideas per run and went 4.7 levels deep, refining and merging ideas like a pro.
The Medium Brain (GPT-5.4-mini): It only generated 6.7 ideas on average and got stuck at a depth of 1.7.
The Small Brain (GPT-4.1): It only managed 4.0 ideas and a depth of 0.7. It often gave up, leaving ideas "open" and unresolved.

This suggests that while the rules (HEP) are great, the robot still needs to be smart enough to follow them. A less capable robot just can't handle the complex work of refining and merging ideas over time.

The Bottom Line

This paper doesn't claim to have solved all of science or created a robot that can replace human scientists. Instead, it shows that by forcing AI agents to keep a strict, auditable log of their ideas and evidence, we can finally see how they think.

In these simulations, the HEP system successfully turned a "black box" robot into a transparent one that could trace its own reasoning, change its mind based on data, and build a solid case for its final answer. It's a step toward AI scientists that we can actually trust, inspect, and learn from, rather than just taking their word for it.

Technical Summary: Toward Auditable AI Scientists via the Hypothesis Evolution Protocol

Problem Statement

While Large Language Model (LLM) agents show promise in autonomous scientific discovery, current implementations often lack rigorous scientific reasoning structures. Existing agents typically automate predefined workflows or optimize toward external goals but fail to explicitly manage the core cycle of scientific inquiry: proposing hypotheses, testing them, gathering evidence, and updating beliefs. In current systems, these processes are buried in unstructured logs or implicit internal states. Consequently, neither the agent nor human researchers can easily trace which hypotheses were considered, how evidence influenced belief updates, or how conclusions were reached. Recent analyses indicate that in many agent traces, evidence is ignored, and beliefs are rarely revised after refutation. Without an explicit, auditable record of the hypothesis–test–evidence–belief cycle, the scientific soundness of an agent's conclusions cannot be verified, nor can the agent systematically build upon or retire its own hypotheses over long research horizons.

Methodology: The Hypothesis Evolution Protocol (HEP)

To address these limitations, the authors propose the Hypothesis Evolution Protocol (HEP), an agent harness that externalizes scientific reasoning into explicit, auditable operations. HEP consists of two primary components:

HEP Registry: An append-only, event-sourced store where every hypothesis is a persistent object identified by a unique ID. Each object contains:
- A natural-language statement.
- A belief probability $P(H)$ representing the agent's confidence.
- A lifecycle state (proposed, under_test, supported, refuted, or dormant).
- A provenance record (lineage and generation mechanism).
- An append-only event log tracking all belief updates, evidence attachments, and state transitions.
HEP Tools: A set of interfaces mediating agent interaction with the Registry, categorized into:
- Propose & Evolve: Includes propose_hypothesis, refine_hypothesis, and merge_hypotheses. Hypotheses are generated via four mechanisms: de-novo, inspired-by (provenance only), refine, and merge.
- Test & Judge: Includes attach_evidence and transition_hypothesis. Belief updates are disciplined by validation gates; only evidence deemed trustworthy updates the belief. Verdicts (supported/refuted) are enforced by threshold transitions ( $P(H) \ge 0.8$ for supported, $P(H) \le 0.2$ for refuted).
- Read: Includes list_hypotheses and get_hypothesis for querying the registry.

The protocol forces the agent to operate within a structured Hypothesis–Test–Evidence–Belief cycle. Beliefs are elicited by the agent with written rationales rather than computed algorithmically, but the protocol enforces that belief changes must be anchored to specific, validated evidence.

Experimental Setup

The authors evaluated HEP on three materials-science research tasks involving open-ended questions about crystal structure preferences for specific compound families using a fixed machine-learned interatomic potential (MLIP):

Task AO2: Polymorph selection in AO2 dioxides.
Task A2BB'O6: B-site cation ordering in double perovskites.
Task MX: Prototype selection in transition-metal monochalcogenides and monopnictides.

The experiments utilized a base LLM (GPT-5.5) equipped with computational tools (structure relaxation, molecular dynamics, database queries) and the HEP harness. Comparisons were made against a standard "planning-style" agent (plan–execute–replan) lacking the HEP harness. Additionally, the scalability of HEP was tested using three LLMs of varying capability (GPT-5.5, GPT-5.4-mini, GPT-4.1).

Key Results

1. Operation of the Scientific Loop

In a single research run on Task AO2, the HEP-equipped agent generated 16 hypotheses, exercising all four generation mechanisms. Every hypothesis was eventually transitioned to a terminal state (supported, refuted, or dormant). The agent successfully executed the full cycle:

Traceability: The hypothesis tree allowed tracing of parent-child relationships and generation mechanisms.
Belief Evolution: Belief trajectories showed that changes were strictly tied to validated evidence. Hypotheses crossed support or refutation thresholds only after accumulating sufficient evidence, with "belief flips" (e.g., a hypothesis proposed with low confidence becoming supported) occurring based on data.
Comparison with Planning Agents: A planning-style agent with the same base LLM spent 83% of its steps on testing and 0% on explicit belief updates. It failed to externalize beliefs or update them systematically against evidence. In contrast, the HEP agent exhibited a clear transition flow (Hypothesis $\to$ Test $\to$ Evidence $\to$ Belief $\to$ Judgement $\to$ Update), demonstrating that HEP enforces the scientific loop that planning alone does not.

2. Generalization Across Research Questions

Applied to Tasks A2BB'O6 and MX, HEP demonstrated robustness:

The agent generated 10–20 hypotheses per task with generation depths of 3–5.
All hypotheses reached a terminal disposition; none were left unresolved.
Final governing rules were derived with high confidence (e.g., $P(H) = 0.97$ for AO2). These rules were verified by predicting structures for unseen compositions.
The protocol revealed how specific refutations (e.g., of a textbook Jahn–Teller mechanism) redirected the search toward the correct electronic exceptions, a history directly readable from the registry logs.

3. Scaling with Base LLM Capability

The efficacy of HEP was found to be dependent on the underlying LLM's reasoning capabilities:

GPT-5.5: Generated the most hypotheses (mean 14.7), utilized all generation mechanisms (including complex merges), and resolved all hypotheses.
GPT-5.4-mini: Generated fewer hypotheses (mean 6.7) and left 36% unresolved.
GPT-4.1: Generated the fewest (mean 4.0), relied heavily on de-novo proposals (62%), and failed to evolve hypotheses through refinement or merging. In one run, it abandoned all hypotheses without attaching evidence.
Conclusion: While all tested LLMs could operate individual HEP tools, only the most capable models could fully exploit the protocol's iterative evolution and lineage management.

Significance and Claims

The paper claims that HEP represents a step toward auditable AI scientists. Its primary significance lies in:

Externalization of Reasoning: It transforms the hypothesis–test–evidence–belief cycle from implicit, unstructured processes into explicit, persistent, and verifiable records.
Discipline and Verification: By enforcing belief thresholds and evidence validation, HEP ensures that an agent's conclusions are grounded in a traceable audit trail, allowing human researchers to inspect, verify, and build upon the agent's scientific reasoning.
Complementarity: The authors emphasize that HEP does not substitute for the reasoning capability of the base LLM. Instead, the harness and the capability are complementary: the harness determines whether reasoning is externalized and verifiable, while the LLM's capability determines how fully that reasoning is exploited.

The work concludes that as base LLMs advance, harnesses like HEP will be essential for converting raw computational power into trustworthy, inspectable scientific discovery. Future work suggested includes introducing independent auditor agents for evidence validation and equipping agents with first-principles tools (e.g., DFT) for more rigorous evidence validation.