Technical Summary: Negative Knowledge as Failure-aware Shared Memory for AutoResearch

Problem Statement

Current AI-assisted research (AutoResearch) systems are predominantly oriented toward success-seeking. While these systems generate numerous failed attempts, such failures are typically treated as transient debugging signals or discarded entirely. They rarely become durable, shared knowledge assets. Existing agent frameworks utilize verbal reflection or self-debugging to inform immediate, in-session retries, but the failed routes themselves are not preserved as cross-agent or cross-task objects.

This absence of a structured "negative knowledge" repository mirrors historical publication bias in human science, where failed hypotheses, non-working methods, and abandoned experimental routes survive only as informal tacit knowledge. In multi-agent AutoResearch, where agents do not require human-style publication credit, there is an opportunity to assign a specific epistemic role to maintain negative knowledge as a shared asset. However, designing reward systems that value both positive and negative outcomes is challenging; rewarding only positive findings may lead to false progress, while diluting rewards for negative outcomes can obscure signal.

Methodology

The paper proposes a Negative Knowledge Memory Layer, a system-level architecture designed to convert failed attempts into reusable, structured records. The framework operates via a multi-agent workflow involving two distinct roles:

1. Curator Agent: A separate agent that processes the artifacts of a failed attempt (code, logs, traces, reasoning outputs) after the attempt concludes. Unlike the failing agent, the curator does not suffer from self-assessment bias. It converts these artifacts into a bounded, typed negative knowledge record and writes it to a shared project-level bank.
2. Research Agent: A downstream agent that, before proposing a new experiment, must explicitly inspect the shared bank. It is required to adopt or reject specific records, providing a justification tied to the specific failure mode described. This turns documented failures into active constraints on future exploration.

The Negative Knowledge Schema

The memory layer utilizes a structured schema with three key design properties:

• Attempt-level: Each record describes exactly one finished attempt.
• Bounded: Records adhere to strict length and structure limits, allowing the entire bank to be passed to an agent inline without exceeding context windows.
• Typed: Failure classification fields use closed vocabularies (e.g., layer: implementation, communication, method, hypothesis; scope: local, regime bound, general; degree: contradicted, partial, unstable). This forces the curator to commit to an explicit failure class rather than free-prose descriptions.

Records also include a prescription (recommended action and rationale) to ensure the bank acts as a constructive guide rather than merely a veto list.

Key Contributions

1. Architecture-Agnostic Memory Layer: A defined framework for maintaining negative knowledge inline with AutoResearch workflows, allowing for human auditing and revision.
2. Bounded Failure Schema: A structured format that transforms failed routes into research objects and inter-agent communication resources, distinguishing between failure classification and prescription.
3. Empirical Evaluation: A pilot study measuring the efficacy of structured negative knowledge in predicting failure repairability, reducing token costs, and supporting multi-agent sharing across same-task retries and cross-task transfers.

Results

1. Same-Task Retry (ScienceAgentBench)

Evaluated on a deterministic subset of ScienceAgentBench using Claude Sonnet 4.6, the study compared five conditions: Base, Retry (fresh agent, no memory), Self-debug (raw feedback), Negative Knowledge Retry (one structured record), and Deep Negative Knowledge Retry (one record distilled from three failed rounds).

• Performance: Plain retry did not improve performance over the Base (31.6%). Adding a single structured negative knowledge record increased the pass rate to 36.8%. Distilling three failed rounds into one deep record achieved 47.4%, outperforming raw three-round self-debug (44.7%), particularly on hard subsets where raw feedback failed.
• Efficiency: The negative knowledge approach achieved these gains with significantly fewer tokens. The deep negative knowledge record used 28.3% fewer tokens than raw self-debug, and the single-record retry used 73.3% fewer tokens.

2. Cross-Task Scientific Research (BKdV System)

The framework was tested on the coupled Burgers-swept-KdV (BKdV) system, a nonlinear PDE problem with no standard computational treatment.

• Stage 1 (Bank Construction): Agents generated a bank of 58 records (15 positive, 43 negative) from stress tests on Burgers, KdV, Gardner, and coupled BKdV settings.
• Stage 2 (Transfer): Agents attempted three new sub-tasks (soliton stability, wave-packet decomposition, bore-soliton interaction).
  • Base (No bank): Failed all three tasks (0/3).
  • Base + Positive Records: Solved only 1/3 tasks. Positive records alone were insufficient because successful methods did not always transfer between related problems.
  • Base + Negative Records: Solved 3/3 tasks. Negative records successfully warned agents where transfers would break, acting as reusable constraints.
  • Base + Positive + Negative: Also solved 3/3 tasks.
• Cross-System Transfer: When the BKdV bank was applied to a different system (Burgers-NLS), agents with the BKdV bank solved 2/4 tasks, whereas agents with no bank solved 0/4. This demonstrated that numerical failure diagnostics and method warnings can transfer across different nonlinear PDE systems.

Significance and Claims

The paper claims that structured negative knowledge functions not merely as memory augmentation or a debugging aid, but as shared epistemic infrastructure for autonomous scientific research.

• Collective Efficiency: The system allows a scientific project to "pay for a failed route only once," redirecting subsequent attempts toward unexplored regions of the research space.
• Beyond Human Limitations: While human scholars face low individual rewards for publishing negative results, AI agents can be assigned the specific role of maintaining this knowledge as a shared asset, overcoming the "file drawer problem" at a system level.
• Modest Scope: The authors position this work as a "first step" toward maintaining negative knowledge as public epistemic infrastructure. The results are based on small sample sizes and specific models (Claude Sonnet 4.5/4.6), and the framework is presented as an operational pilot rather than a fully realized solution for all scientific domains.

The core argument is that for AI-engaged science to be robust, negative knowledge must be explicitly maintained alongside positive findings, transforming failure from a transient signal into a durable, reusable constraint that guides future exploration.