Technical Summary: LLM Agents for Knowledge Discovery in Atomic Layer Processing

Problem Statement

While Large Language Models (LLMs) and agentic systems show promise in materials science for synthesizing existing literature and optimizing well-defined objectives (e.g., property optimization), a significant challenge remains: distinguishing between the retrieval of latent knowledge absorbed during training and the discovery of genuinely new knowledge. Most current agentic workflows are designed to reach specific goals using pre-existing databases, making it difficult to isolate an agent's ability to interrogate an unknown system and derive rules from scratch. This paper addresses the gap in probing LLM agents' capabilities for independent knowledge discovery in data-poor conditions where the underlying rules are unknown and not part of the model's training data.

Methodology

The authors propose a framework where LLM agents are repurposed to interrogate a system using a "black box" function. The agent is given a specific system with confined input/output spaces and tasked with describing the governing rules without explicit instructions on how to discover them or what the rules might be. The study utilizes two distinct experimental setups:

1. The "Alien Market" Parlor Game: A simplified logical system where an agent must deduce a rejection rule (words containing specific letters, 'm' or 'p', are rejected) by testing inputs. This serves as a proof of concept for rule discovery.
2. Atomic Layer Processing (ALP) Reactor Simulation: A complex, simulated chemical reactor environment.
   • System: The agent controls a reactor with fictitious precursors (A, B, C, D) and surfaces. It has access to limited probes: a Quartz Crystal Microbalance (QCM) for mass change and a pressure gauge. It cannot directly observe internal reaction kinetics or chemical identities.
   • Implementation: The agent uses a LangChain ReAct architecture. It receives a "recipe" format to control valves, temperatures, and gas flows.
   • Data Handling: To manage the complexity of raw sensor data (560-dimensional vectors), a secondary LLM summarizes the experimental narrative (pressure and mass traces) into a high-level description for the main agent.
   • Constraints: The agent is given a fixed amount of "experimental time" (e.g., 3600s or 7200s) and must explore the system to generate general statements about its behavior.

The study compares different model sizes (e.g., GPT-5, Gemini 2.5 Pro/Flash) and investigates the impact of persistence (forcing a specific number of experiments) and context (providing reference values like QCM mass ranges) on discovery success.

Key Results

1. Alien Market Experiments

• Model Performance: Larger models (GPT-5) outperformed smaller ones but achieved this by conducting significantly more experiments. Smaller models often stopped investigating prematurely, providing rules that only matched their initial observations.
• Persistence: Explicitly instructing agents to perform a defined number of experiments ($n$) markedly improved rule discovery across all models, suggesting that "pushing" models to extend their investigation is a viable strategy for generalizable results.

2. ALP Reactor Experiments

• Configuration I (Favorable Kinetics): With two chemicals (A and B) and favorable reaction kinetics, agents successfully discovered self-limiting surface reactions (Atomic Layer Deposition, ALD) and exploited them. They further explored kinetic limits and decomposition growth modes within the allocated time.
• Configuration II (Difficult Kinetics): When reaction kinetics were slowed and vapor pressure lowered, agents initially got stuck in a "local minimum," describing a low-growth Chemical Vapor Deposition (CVD)-like process.
  • Time vs. Context: Merely increasing time (to 7200s) did not resolve the issue. However, providing contextual reference values (e.g., expected monolayer mass ranges for QCM) allowed the agent to break out of the local minimum in some iterations, discovering the correct self-limiting behavior or decomposition pathways.
• Configuration III (Expanded Space): With four chemicals, the agents explored a larger space including ALD, Atomic Layer Etching (ALE), and passivation.
  • Path Dependence: Results were highly path-dependent. Different iterations explored different parts of the experimental space (e.g., one iteration found ALD but missed ALE; another found ALE but missed co-dosing).
  • Visualization: Using Uniform Manifold Approximation and Projection (UMAP) on the reactor state vectors revealed that agents visited different regions of the experimental space in each iteration, suggesting that a "swarm" strategy with multiple agents could cover the space more comprehensively.

Key Contributions

• Framework for Pure Discovery: The paper demonstrates a method to isolate LLM agents' ability to discover new rules by removing prior knowledge and specific optimization objectives, relying instead on system interrogation.
• Role of Persistence and Context: The study identifies that persistence (forcing extended experimentation) and contextual hints (providing physical reference ranges) are critical for agents to escape local minima and discover complex, rare effects in high-dimensional spaces.
• Path Dependence in AI Discovery: The work highlights that AI-driven discovery is not deterministic; the trajectory of discovery depends heavily on initial conditions and random seeds. This suggests that diverse exploration strategies (e.g., varying temperature or using multiple agents) are necessary for robust scientific discovery.
• Simulation-to-Reality Bridge: The reactor simulation uses a recipe format identical to the authors' physical lab equipment, demonstrating a direct pathway for transitioning agent-controlled simulations to physical experiments.

Significance and Claims

The authors claim that this work provides a "proof of concept" that LLM agents can:

1. Interrogate unknown systems and reason moderately well on outcomes when prompted to use sufficient resources.
2. Pursue complex ideas based on initial observations.
3. Summarize findings into general system statements without predefined objectives.

The paper positions this as a step toward independent discovery in data-poor conditions, an area where AI/ML can complement traditional literature mining. The authors emphasize that while optimizing for specific objectives is a mature field, the ability to explore systems without predefined goals could help construct comprehensive scientific databases free from the "bias toward success" often found in published literature. The work concludes that while agents can be augmented with knowledge, there is value in allowing them to ignore or critically examine prior knowledge to find novel paths, provided experimental constraints are balanced between safety and flexibility.