🤖 machine learning

When Should an AI Scientist Stop? Verifiable Experiment Steering and Refusal for Autonomous Discovery

This paper introduces CARTOGRAPH, a verification layer for autonomous AI scientists that integrates experiment steering, ambiguity resolution, and residual-based refusal mechanisms to effectively detect structural model inadequacies and prevent false discoveries in scientific experimentation.

Original authors: Neel Tushar Shah, Manglam Kartik

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Neel Tushar Shah, Manglam Kartik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a team of autonomous AI scientists working in a high-tech lab. Their job is to run experiments to figure out how the world works. Usually, these AI systems are great at proposing new experiments and running them. But they have a blind spot: they often keep going even when they are completely wrong, or when the "rulebook" (the library of models) they are using is fundamentally broken.

This paper introduces a new "safety guard" called CARTOGRAPH. Think of CARTOGRAPH not as the scientist, but as the chief auditor sitting next to the scientist, watching every move and asking three critical questions before the scientist claims a discovery.

Here is how CARTOGRAPH works, using simple analogies:

1. The Three Questions (Select, Resolve, Refuse)

The paper frames the AI's job as making three linked decisions:

Select (The Compass): "Which experiment should we do next to clear up the confusion?"
- The Analogy: Imagine you are trying to find a hidden object in a dark room. You have a flashlight. CARTOGRAPH calculates exactly where to shine the light to reveal the most new information, rather than just shining it randomly or in a place you've already checked.
Resolve (The Finish Line): "Are we done? Do we know the answer?"
- The Analogy: It's like a puzzle. CARTOGRAPH checks if every single piece of the puzzle has been found. If the picture is complete and clear, it says, "Stop, we have the answer."
Refuse (The Stop Sign): "Wait a minute. The puzzle pieces we have don't actually fit together to make a real picture. The rulebook is wrong."
- The Analogy: This is the paper's most unique feature. Imagine you are trying to build a house using a blueprint for a boat. No matter how hard you try, the walls won't stand up. A normal AI might keep trying to hammer the nails in, insisting the house is fine. CARTOGRAPH looks at the wobbly walls, sees the blueprint is for a boat, and says, "Stop! We cannot build a house with this blueprint. We need a new blueprint."

2. How It Detects "Wrong Blueprints" (The Refusal Mechanism)

The paper shows that standard AI systems often get "overconfident." They might pick a model that fits the data okay and declare victory, even if the model is fundamentally wrong.

CARTOGRAPH uses a Residual Guard.

The Metaphor: Think of a tailor measuring a customer. If the customer is wearing a suit that is two sizes too small, the tailor measures the "gap" between the fabric and the body.
How it works: CARTOGRAPH constantly measures the "gap" (residual) between what the AI's models predict and what actually happens in the lab.
- If the gap is small, the model is good.
- If the gap is huge, CARTOGRAPH realizes the model is structurally inadequate. It doesn't just say "maybe try again"; it revokes any previous claims of success. It says, "We thought we found the answer, but the evidence shows our library of answers is broken."

3. Real-World Tests (What the Paper Actually Found)

The authors tested this system in three main ways:

The "Cascade" Test (High-Dimensional Math): They created a complex math problem with many moving parts.
- Result: When the problem got very complex (like a maze with 8 or 16 paths), CARTOGRAPH was vastly superior to other methods. It found the right path 65% of the time, while other methods only found it 2% of the time. It was like having a GPS in a massive city versus guessing directions.
The "Pharmacokinetic" Test (Drug Absorption): They tested how the body absorbs medicine.
- Result: In simple cases, CARTOGRAPH didn't do much better than standard methods. The paper is honest about this: when the problem is simple, the fancy math doesn't add much value. But, it successfully identified when the "rulebook" was wrong. In one test, it tentatively identified a drug mechanism, then revoked that identification when new data showed the model didn't fit.
The "A-Lab" Audit (Real Scientific Claims): The authors looked at 40 past claims made by a famous autonomous lab (A-Lab) that was trying to discover new materials.
- Result: CARTOGRAPH acted as a retrospective auditor. It flagged all 4 of the claims that were later proven to be inconclusive or wrong by human experts. It passed the 32 claims that were confirmed. This proves it can act as a "lie detector" for scientific claims.

4. The Bottom Line

The paper argues that for AI to be truly useful in science, it needs more than just the ability to propose experiments. It needs a verifiable "Stop" signal.

Current AI: "I think this is the answer! Let's try another experiment to be sure." (Even if the answer is wrong).
CARTOGRAPH AI: "I think this is the answer. Wait, the data doesn't fit the model. The model is broken. Stop. We need to change our library of theories before we proceed."

The authors emphasize that this system is designed for governance and safety, not just speed. It creates an "audit trail" (a log of why it stopped or revoked a claim) so humans can trust the AI's decisions, or know exactly when to step in and fix the "blueprint."

In short: CARTOGRAPH is the part of the AI that knows when to quit, when to admit the rulebook is wrong, and when to say, "We are done here," preventing the AI from confidently making mistakes.

Technical Summary: CARTOGRAPH – Verifiable Experiment Steering and Refusal for Autonomous Discovery

1. Problem Statement

Current autonomous scientific discovery loops, which integrate Large Language Model (LLM) planners, automated laboratories, and statistical interpreters, face a critical bottleneck: decision-making under structural inadequacy. While existing systems can propose experiments (Select) and interpret data, they lack a verifiable mechanism to determine when the current library of mechanistic models is fundamentally wrong.

Standard Bayesian Optimal Experimental Design (BOED) assumes the true law lies within the prior support, and classical model discrimination assumes at least one rival model is correct. In high-stakes domains like pharmacokinetics or materials synthesis, these assumptions often fail. The paper identifies a gap in the "Refuse" decision: the system currently cannot audibly signal when it should stop making claims because the hypothesis space itself is structurally insufficient.

2. Methodology: The CARTOGRAPH Framework

The authors propose CARTOGRAPH, a verification layer that couples three linked decisions: Select (experiment choice), Resolve (ambiguity closure), and Refuse (library inadequacy detection). The framework operates under a "behavioral access" model, where the agent observes numerical outputs from simulators or robots rather than symbolic equations.

2.1. Unresolved Subspace Geometry

The core mathematical object is the unresolved subspace ( $U_\tau$ ), defined as the span of right singular vectors of the accumulated disagreement matrix ( $H_{cur}$ ) corresponding to singular values $\sigma_j \le \tau$ .

Disagreement Matrix ( $H$ ): Constructed by stacking linearized pairwise predictive differences ( $D_{ij,e} = J_{i,e} - J_{j,e}$ ) across all model pairs in the library for a candidate experiment $e$ .
Select (Steering): The framework selects experiments that maximize information gain within $U_\tau$ $U_{τ}$ .
- Raw Score: The isotropic unresolved-projection score, $\|H_e U_\tau\|_F^2$ , which corresponds to the unresolved Fisher-information trace under local linear-Gaussian assumptions.
- CARTOGRAPH-A: The exact A-optimal rule, maximizing the reduction in posterior variance on the unresolved subspace: $\text{tr}(\Lambda_{cur}) - \text{tr}((\Lambda_{cur}^{-1} + G_e)^{-1})$ , where $G_e$ is the noise-aware information matrix.
Resolve: Ambiguity is declared resolved when $\dim(U_\tau) = 0$ (exact regime) or when the smallest singular value exceeds $\tau$ (approximate regime).

2.2. The Refusal Guard

Unlike standard BOED, CARTOGRAPH includes a residual-based refusal mechanism. It does not rely solely on the geometry of $U_\tau$ but monitors the physical fit of the best library member against observed data.

Residual Metric ( $\rho$ ): The normalized $\ell_2$ residual of the best-fit library member relative to physically meaningful summary features (e.g., $C_{max}$ , terminal slope).
Identification Gap ( $\mu$ ): The BIC gap between the top two library members.
Logic: A model is tentatively identified only if $\rho \le \delta$ (residual threshold) and $\mu \ge \mu_{min}$ (BIC threshold). Crucially, if $\rho$ exceeds $\delta$ in later rounds, the system revokes previous identifications, signaling that the library is structurally inadequate.

2.3. Theoretical Bridges

The paper establishes that CARTOGRAPH is not an arbitrary heuristic but sits within established theory under a local linear-Gaussian bridge:

Fisher Information: The raw projection score is exactly the unresolved Fisher-information trace.
Posterior Variance: For $k=1$ (one unresolved dimension), maximizing the projection score is equivalent to minimizing posterior variance.
EIG and Box-Hill: The framework provides first-order alignment with closed-form Expected Information Gain (EIG) and an isotropic-limit reduction of the Box-Hill criterion.
Scaling Laws: Theoretical analysis predicts that in low-dimensional settings ( $k/d \approx 1/2$ ), unresolved-projection methods will merely tie with simple disagreement heuristics, while in high-dimensional settings ( $d \to \infty$ ), they strictly dominate.

3. Key Contributions

Formal Access-Model Distinction: Separates symbolic access (coverage property) from behavioral access (rank property), defining the conditions under which the true law is recoverable.
CARTOGRAPH Framework: A unified loop integrating unresolved-subspace steering with a residual-based refusal guard, enabling "auditable stop and escalate" signals.
Local BOED Bridge: Proves exact identities between unresolved Fisher information, A-optimality, and closed-form EIG/Box-Hill in the local linear-Gaussian regime.
Scaling Theory: Derives an exact random-candidate scaling law predicting when disagreement heuristics suffice versus when projection is necessary.
Empirical Validation: Demonstrates performance across symbolic dynamical systems, pharmacokinetic (PK) benchmarks, EPA time-series data, and a retrospective audit of the A-Lab autonomous materials system.

4. Experimental Results

4.1. Select: Structured Nonlinear Cascade

In a scalable ODE system benchmark ( $d \in \{2, 4, 8, 16\}$ ):

High-Dimensional Gain: At $d=8$ , CARTOGRAPH-A selects the oracle "hidden-best" experiment in 65% of trials, compared to 2% for raw projection and disagreement baselines ( $p < 10^{-21}$ ).
Low-Dimensional Tie: At $d=2$ , all methods tie, consistent with the theoretical prediction that disagreement heuristics capture half the useful signal when $k/d = 1/2$ .
Efficiency: CARTOGRAPH-A achieves performance comparable to closed-form EIG but is computationally cheaper (one SVD per round vs. Monte Carlo integration).

4.2. Select: Pharmacokinetic Boundary Case

In a low-dimensional PK library ( $k/d \approx 1/2$ ):

The framework observes near-ties between CARTOGRAPH and disagreement heuristics, validating the scaling theory.
This result is presented as an honest negative finding: in low dimensions, complex projection offers no significant advantage over simple disagreement.

4.3. Refuse: Out-of-Library Mechanisms

The most distinctive finding is the revocation capability:

Scenario: Three out-of-library PK mechanisms (time-varying clearance, saturable elimination, enterohepatic recirculation) and one in-library control.
Behavior: CARTOGRAPH tentatively identifies a library member in early rounds (0–1) but revokes these identifications in later rounds (2–5) as normalized residuals ( $\rho$ ) cross the refusal threshold ( $\delta = 0.25$ ).
Control: The in-library control remains identified throughout.
Baseline Failure: A standard predictive-uncertainty heuristic (ensemble variance) fails to detect these failures, remaining confident even when the library is wrong.

4.4. Real-World Audit: A-Lab

A retrospective audit of 40 positive claims from the published A-Lab autonomous materials system:

Refusal Performance: The CARTOGRAPH guard flagged all 4/4 claims later marked "inconclusive" under manual re-analysis.
Pass Rate: It passed 32/36 confirmed claims.
Comparison: A residual-only baseline (Rwp) failed to flag any of the inconclusive claims.

5. Significance and Claims

The paper claims that CARTOGRAPH provides a governed approach to autonomous discovery. Its significance lies not in universal selection gains (which are modest in low dimensions) but in the auditable refusal signal.

Honesty: The framework explicitly handles the case where the library is structurally wrong, a scenario standard BOED ignores.
Revocation: It demonstrates the ability to retract tentative scientific claims as evidence accumulates, a behavior essential for high-stakes science.
Governance: The system emits a decision log ( $\rho, \delta$ , revocation events) that can be audited, treating "well-supported refusals" as scientifically credit-worthy outcomes.

The authors emphasize that the "verifiable" nature of the system comes from logged residuals and thresholds, not universal reliability certification. The framework is designed to be a verification layer for LLM planners, handling the rigorous statistical discrimination that LLMs cannot perform natively.