Large language models do not replace chemists in a closed-loop catalysis experiment

Technical Summary: Large Language Models Do Not Replace Chemists in a Closed-Loop Catalysis Experiment

Problem Statement
The study addresses the capacity of Large Language Models (LLMs) to serve as autonomous reasoning agents in complex, high-dimensional, and noisy scientific experiments. While LLMs have demonstrated proficiency in knowledge retrieval and single-shot hypothesis generation, their performance in long-horizon, closed-loop laboratory settings—characterized by experimental drift, measurement noise, and the need for iterative adaptation—remains largely unexplored. The authors sought to benchmark LLM reasoning head-to-head against human domain experts in a 25-dimensional colloidal catalysis problem where the optimal solution was unknown and the system exhibited significant stochasticity.

Methodology
The researchers utilized a mobile robot (KUKA LBR iiwa 14) integrated with "Opsight," an agentic optimization software, to conduct a 528-experiment campaign aimed at maximizing hydrogen production using nanoscale covalent organic frameworks (nanoCOFs) as photocatalysts.

• Experimental Design: The problem space involved 25 dimensions, including four nanoCOFs, 12 inorganic semiconductors, three sacrificial agents, three surfactants, pH modifiers, and a platinum cocatalyst precursor. This created a combinatorial space of approximately $5.92 \times 10^{16}$ potential formulations.
• Phases of Execution:
  • Phase 0 (Pre-experiment): The LLM (GPT-5.1) was provided with 10 background resources (papers, data tables, spectra) to generate an initial search strategy.
  • Phase 1 (Open-ended Optimization): The LLM conducted 320 experiments without a fixed budget, aiming to maximize activity. It navigated experimental noise and a mid-campaign change in measurement setup (reducing photolysis time to avoid detector saturation).
  • Phase 2 (Renormalization): The system adjusted to a 15-minute photolysis time to ensure linear detector response. The LLM was instructed to re-anchor its data analysis.
  • Phase 3 (Fixed Budget Comparison): A direct comparison was conducted over 160 experiments between the LLM (run twice with different prompts) and a team of four human domain experts. Both agents had access to all prior data. The human team also utilized visual cues (phase behavior) unavailable to the LLM.

Key Results

• Performance: In the final 160-experiment phase, the human team identified a catalyst formulation that was, on average, 22% more active than the best formulation found by the LLM's weaker run and statistically comparable to the LLM's stronger run. The human team found the highest single activity (126 ± 8 µmol h⁻¹) compared to the LLM's best (117 ± 17 µmol h⁻¹).
• Speed and Cost: The LLM reasoned 35 times faster than the human team (59.2 minutes vs. 34.7 person-hours) and was estimated to be approximately 1,900 times less expensive in terms of reasoning costs.
• Exploration Strategy: The human team explored a broader chemical space (31 distinct categorical formulations) with less redundancy. The LLM was highly exploitative, focusing on a narrow subset of variables (18 distinct formulations) and performing unnecessary repeat experiments.
• Reasoning Errors: The LLM exhibited "silent errors," including:
  • A stoichiometric calculation error in Phase 0 regarding the molecular weight of the Pt precursor.
  • Context-window limitations in Phase 3 that caused it to revert to incorrect 60-minute data analysis despite instructions to use 15-minute data.
  • Logical inconsistencies, such as concluding that SDS improved dispersibility in Phase 3, contradicting its own correct Phase 0 prediction that SDS would induce aggregation (a fact confirmed by offline zeta potential and phase behavior observations).
• Discovery: Both the LLM and humans identified that the addition of the anionic surfactant SDS significantly boosted activity, though the LLM discovered this via experimentation rather than retrieving it from prior knowledge. Neither agent successfully identified productive organic-inorganic heterojunctions.

Key Contributions

1. First Direct Benchmark: This study provides a rare, direct, side-by-side comparison of LLM reasoning versus human expert reasoning in a high-dimensional, noisy, real-world closed-loop experiment.
2. Identification of Limitations: It demonstrates that while LLMs can navigate complex chemical spaces and adapt to some experimental changes, they suffer from memory limitations, silent logical errors, and a tendency toward over-exploitation without human oversight.
3. Cost-Benefit Analysis: The work quantifies the trade-off between the speed/cost efficiency of machine reasoning and the depth/reliability of human reasoning, suggesting that the optimal division of labor depends on the cost and hazard profile of the experiments.

Significance and Claims
The authors conclude that LLMs are not "out-of-the-box" replacements for expert reasoning in complex scientific problems. While the LLM (GPT-5.1) successfully directed a long-term campaign and found productive hypotheses, it failed to match the overall efficacy of human experts in the final optimization phase due to reasoning flaws and lack of sensory perception (e.g., inability to see phase changes).

The paper argues that the future of autonomous laboratories lies in hybrid systems that combine the speed and low cost of machine reasoning with the oversight, perceptual judgment, and error-correction capabilities of human experts. The authors posit that for fast, inexpensive, and low-hazard experiments, machine reasoning is highly advantageous, but as experiments become slower, costlier, or more hazardous, human intervention or close oversight becomes critical to mitigate the risks of silent machine errors. The study emphasizes that designing systems where humans and LLMs work collaboratively, rather than attempting full autonomy, is the most resource-efficient path forward.