Tiny Moves: Game-based Hypothesis Refinement

Technical Summary: Tiny Moves: Game-based Hypothesis Refinement

Problem: The "Black Box" of Scientific Reasoning in AI

Current machine learning approaches to scientific discovery often treat hypothesis generation as an end-to-end prediction task. While effective at producing conclusions, these models obscure the incremental, compositional nature of scientific reasoning. In empirical fields like biology, scientists rarely generate hypotheses de novo; instead, they iteratively refine, prune, and expand existing partial hypotheses based on new evidence.

Existing LLM-based agentic frameworks (e.g., "AI Co-Scientist") focus on task completion but lack a structured way to represent intermediate reasoning states. This leads to "undifferentiated rewrites," where minor errors trigger massive, non-interpretable changes to a hypothesis, making it difficult to localize errors, maintain logical consistency, or control the refinement process.

Methodology: The Hypothesis Game

The authors propose a symbolic, game-based formalism called The Hypothesis Game. Instead of broad rewrites, the framework forces LLM agents to operate on a shared hypothesis state using a fixed, interpretable grammar of reasoning moves.

1. Hypothesis Representation: Hypotheses are represented as a set of fragments (structured text or triples) that can be mapped to a graph of entities and relations.
2. Reasoning Grammar (Moves): The system uses a set of atomic and composite operations to evolve the hypothesis:
   • Prune: Remove incorrect or unsupported components.
   • Expand (with Corpus): Retrieve evidence from external biomedical literature to add information.
   • Expand (with LLM Introspection): Use the LLM’s internal knowledge to speculate on connections.
   • Debate: A multi-agent process where "ClaimSmith" agents argue different positions to reach a consensus.
3. Game Architecture: A central LLM controller, the Game Master, performs a "Diagnose" step to evaluate the current state and then selects the appropriate move(s) from the grammar.
4. Game Modes: The reasoning style is steered via natural language instructions (e.g., a "discovery mode" favors expansion, while a "validation mode" favors pruning).

Key Contributions

• Formalism: A compositional reasoning framework that treats hypothesis refinement as a structured game rather than a text-generation task.
• Implementation: A multi-agent system utilizing specialized roles (Game Master, ClaimSmiths, etc.) to execute the reasoning grammar.
• New Benchmarks: The introduction of two novel evaluation tasks for biological reasoning:
  • Corruption Recovery: Testing the ability to detect and repair controlled errors (wrong entities, wrong relationships, or irrelevant statements) in a valid pathway.
  • Reconstruction from Partial Cues: Testing the ability to rebuild a complete biochemical pathway starting from sparse, minimal information.

Results

The framework was evaluated against strong prompting baselines (Zero-Shot, Chain-of-Thought, and ReAct) using Reactome biological pathways.

• Corruption Recovery (The Primary Strength): The Hypothesis Game significantly outperformed all baselines. It achieved the highest precision, F1 score, and error removal rate. Crucially, it demonstrated "tiny moves"—making small, localized, and traceable edits—whereas the ReAct baseline tended to overwrite large portions of the hypothesis, often destroying valid information in the process.
• Reconstruction (The Competitive Setting): In the more difficult task of rebuilding pathways from scratch, the Hypothesis Game performed comparably to the strongest baseline (ReAct) and outperformed simpler prompting methods in terms of precision. While all models struggled with the inherent difficulty of the task, the game-based approach maintained tighter control over hypothesis growth.
• Robustness: The system showed remarkable stability even when the error rate in the corruption task increased from 10% to 40%.

Significance

This work shifts the paradigm of AI for science from "black-box generation" to "structured refinement." By decomposing reasoning into a formal grammar of moves, the authors provide a route toward AI systems that are:

1. Controllable: Users can steer the reasoning via modes and move budgets.
2. Interpretable: The trajectory of a hypothesis is a transparent sequence of logical edits.
3. Transferable: The symbolic grammar of moves is independent of the specific scientific domain, making it applicable to chemistry, physics, or other empirical sciences.