🤖 AI

Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning

This paper introduces ChemCost, a rigorous benchmark for evaluating Large Language Models' ability to estimate chemical reaction costs by grounding identities, retrieving supplier quotes, and performing arithmetic, revealing that even advanced agents struggle with the task due to brittle parsing, ineffective evidence integration, and poor noise robustness.

Original authors: Yuyang Wu, Yue Huang, Shuaike Shen, Xujian Wang, Shuhao Zhang, Qiyao Xue, Weichen Liu, Runtian Gao, Jian Ma, Xiangliang Zhang, Olexandr Isayev

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuyang Wu, Yue Huang, Shuaike Shen, Xujian Wang, Shuhao Zhang, Qiyao Xue, Weichen Liu, Runtian Gao, Jian Ma, Xiangliang Zhang, Olexandr Isayev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to figure out exactly how much it will cost to make a specific dish for a dinner party. You have the recipe (the chemical reaction), but you don't have the prices of the ingredients, and you don't know exactly which brand or package size you need to buy to get the right amount.

This paper introduces a new "test" called CHEMCOST to see if advanced AI computers (called Large Language Models or LLMs) can act like a smart shopping assistant to solve this problem.

Here is a breakdown of what the paper found, using simple analogies:

1. The Challenge: It's Not Just "Googling"

The authors wanted to see if AI could act as a scientific procurement agent. This isn't just about writing a poem or answering a trivia question. It's a multi-step math and shopping puzzle:

The Name Game: The recipe might say "TEA." Is that Triethylamine (a chemical) or Triethanolamine (a different chemical)? The AI has to figure out exactly which molecule is meant.
The Shopping Hunt: The AI has to look up prices from a frozen database of 230,000+ supplier quotes. It can't just guess; it has to find the specific "pack" (e.g., a 1g bottle vs. a 100g bottle) that fits the recipe.
The Math: It has to convert "1.5 equivalents" into grams, calculate the cost of every single ingredient, and divide by the final amount of product to get a price-per-gram.

The Analogy: Imagine the AI is a student taking a final exam. The teacher gives them a recipe, a locked library of price tags, and a calculator. The student must find the right ingredients, buy the right sizes, do the math, and hand in a single number: the total cost.

2. The Test: CHEMCOST

The researchers built a benchmark with 1,427 different chemical recipes.

The "Answer Key": They didn't ask humans to grade the AI. Instead, they created a "frozen" database of real prices and a strict set of rules. The computer calculates the exact correct answer beforehand. This means the grading is 100% objective—no human bias.
The "Noise" Test: To see if the AI is truly smart or just memorizing patterns, they messed up the recipes. They changed the names (e.g., writing "Na2C03" instead of "Na2CO3" with a zero instead of an 'O'), removed the yield percentages, or wrote the instructions in messy, unstructured text.

3. The Results: "Tool Access" Isn't Enough

The researchers tested many different AI models, from the biggest "frontier" models to specialized chemistry models. Here is what happened:

The Ceiling: Even the smartest AI models only got about 50% of the answers right (within a 25% margin of error) on clean, easy inputs. They are far from perfect.
The "Tool" Trap: The AI models were given "tools" (like a search engine for chemical names and a price lookup). The paper found that having the tools is necessary, but not enough.
- Analogy: Giving a student a calculator and a library card doesn't guarantee they will solve the math problem correctly. They still need to know which numbers to put in the calculator and which book to open.
The "Brittle" Problem: When the input was slightly messy (like a typo in a chemical name or a weird formatting error), the AI models often gave up completely or got the answer wildly wrong.
- Analogy: If a human chef sees "Na2C03" (with a zero), they might guess it's a typo for "Na2CO3." The AI, however, often panicked, couldn't find the item, and stopped trying.

4. Where Do They Fail?

The paper broke down exactly where the AI gets stuck:

Parsing: They can't read the messy text to find the ingredients.
Evidence Integration: They find the prices but can't combine them correctly with the recipe amounts.
Pack Selection: They pick the wrong size bottle (buying a 100g bottle when they only need 1g, or vice versa).
Giving Up: When the text is messy, they often refuse to answer rather than trying to figure it out.

5. The "Human" Benchmark

The researchers also had real human chemists take the test.

The Result: Humans did better than the AI (about 67% accuracy on clean inputs), but they still made mistakes. This proves the task is genuinely hard, even for experts.
The Takeaway: The AI isn't just "dumb"; the task itself is a complex juggling act of reading, searching, and calculating that is difficult for both humans and machines.

Summary

The paper concludes that while AI is getting better at using tools, it is still not reliable enough to be trusted with real-world chemical cost estimation. It struggles when the information isn't perfectly formatted, and it often fails to connect the dots between finding a price and calculating the final total.

In short: The AI is like a very fast, very knowledgeable intern who has access to the entire internet and a calculator, but who still needs a human to double-check their work, especially when the instructions are written in a slightly messy way.

Technical Summary: Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning

Problem Statement

While Large Language Models (LLMs) have demonstrated increasing capability as tool-using agents in general domains (e.g., software engineering, web navigation), rigorous evaluation of their performance in scientific tool use remains limited. Existing benchmarks often rely on curated demonstrations, expert assessment, or "LLM-as-judge" scoring rather than exact, judge-free ground truth.

In chemistry, specifically, agents can plan syntheses and invoke domain tools, but they lack a controlled testbed for chemical procurement cost estimation. This is a practical industrial task requiring an agent to:

Ground ambiguous chemical names to unique molecular structures.
Retrieve fragmented supplier quotes from a pricing database.
Select valid purchasable packs under purity and quantity constraints.
Normalize quantities (converting equivalents, mol%, and volumes to mass).
Perform exact arithmetic to aggregate costs based on a reaction description.

Current evaluations fail to isolate specific failure modes (e.g., grounding errors vs. arithmetic errors) or provide deterministic, reproducible labels for these multi-step workflows.

Methodology: The CHEMCOST Benchmark

To address this gap, the authors introduce CHEMCOST, a benchmark designed for the deterministic evaluation of LLM agents on chemical procurement cost estimation.

Dataset Construction

Scale: The benchmark comprises 1,427 evaluable reactions drawn from five sources: the Open Reaction Database, named-reaction textbooks, Organic Syntheses, retrosynthesis route collections (PaRoutes, ChemPU), and hand-crafted examples.
Ground Truth: Labels are computed deterministically from a frozen snapshot of 230,775 pack-level supplier quotes covering 2,261 chemicals.
Procurement Rules: The ground truth assumes a fixed 1 mmol limiting-reagent scale. The cost is calculated by:
- Identifying the limiting reagent.
- Computing required mass for each non-solvent component ( $m_i = e_i \times MW_i \times 10^{-3}$ g).
- Selecting the smallest valid pack (purity $\ge$ 95%) that covers the required mass. If no single pack suffices, the minimum number of the largest packs is purchased.
- Normalizing the total purchase cost by the grams of product obtained (accounting for yield).
Noise Injection: To test robustness, a four-layer noise pipeline perturbs inputs while preserving ground truth:
1. Chemical Name Noise: Aliasing, abbreviations (e.g., TEA), and isomer ambiguity.
2. Quantity Noise: Converting between mol%, equivalents, volumes, and vague terms (e.g., "excess").
3. Missing Information: Omitting fields like molecular weights, yields, or roles.
4. Format Noise: Simulating OCR artifacts, natural language prose, and Unicode degradation.

Evaluation Protocol

Agent Setup: Agents operate as ReAct (Reasoning + Acting) loops with a maximum budget of 40 steps. They are equipped with four deterministic tools: chemical name resolution, supplier quote retrieval, molecular weight computation, and arithmetic.
Metrics: The primary metric is Cost Tolerance Accuracy (CTA@k), defined as the fraction of predictions within $k\%$ relative error of the ground truth. Additional metrics include component precision/recall, tool efficiency, and abstention rates.
Models Evaluated: A diverse set of frontier, open-weight, and chemistry-specialized models (e.g., GPT-5, Qwen3.5-Plus, DeepSeek V4 Pro, ChemDFM, LlaSMol).

Key Results

1. Tool Access is Necessary but Insufficient

Performance Ceiling: Even the strongest agents (Qwen3.5-Plus) achieve only 50.6% accuracy within 25% relative error (CTA@25) on clean inputs. Performance drops significantly under stricter metrics (CTA@10) and noise.
Specialization Gap: Chemistry-specialized models (e.g., ChemDFM, ChemLLM) show high component recognition (precision/recall) but near-zero cost accuracy. This indicates that domain knowledge alone does not translate to effective agentic reasoning or quantitative aggregation.
Tool Dependency: Zero-shot and Chain-of-Thought (CoT) approaches without tools perform poorly (CTA@25 < 8%), confirming that parametric knowledge is insufficient for retrieving specific, up-to-date supplier prices and performing exact pack selection.

2. Failure Modes and Robustness

Format Noise is Critical: Format perturbations (e.g., OCR errors, unstructured text) are the strongest trigger for agent abstention, increasing abstention rates by ~26 percentage points across frontier models. This disrupts the initial parsing stage, preventing the workflow from starting.
Tool Use Patterns:
- Necessity: There is a strong correlation ( $r=0.98$ ) between tool engagement and success; models that do not call tools fail.
- Inefficiency: Merely calling tools is not enough. Failed trajectories often involve non-convergent retrieval (repeatedly querying without selecting a valid pack) or premature termination. Failed cases often use more tool calls than successful ones.
Recognition vs. Computation: Agents can often identify chemical components correctly (high precision) but fail to assemble them into a correct procurement state (low CTA). The gap between recognition and computation widens significantly under noise.

3. Structural Difficulty

Route Depth: The number of synthesis steps is the dominant factor in difficulty. Accuracy drops from ~42% for single-step reactions to ~11-13% for multi-step routes (3-5 steps).
Component Count: Within single-step reactions, the number of components has a model-dependent effect; some models handle multi-component recipes better, possibly due to richer context for retrieval.

Significance and Claims

The paper claims that CHEMCOST provides the first rigorous, judge-free benchmark for evaluating scientific tool use in a realistic industrial workflow. Its significance lies in:

Moving Beyond "Black Box" Evaluation: By decomposing the task into grounding, retrieval, pack selection, and aggregation, the benchmark allows for stage-level diagnosis of where agents fail (e.g., brittle parsing vs. ineffective evidence integration).
Highlighting the "Grounding" Gap: The results demonstrate that while LLMs are improving at general tool orchestration, they struggle with the specific constraints of scientific domains: resolving ambiguous entities, adhering to strict procurement rules, and maintaining quantitative state tracking under noisy inputs.
Defining the Path Forward: The authors argue that progress in scientific agents requires not just better models, but reliable grounding, retrieval-aware state tracking, and calibrated abstention mechanisms. The current state of the art suggests that tool access is a prerequisite, but the integration of retrieved evidence into a coherent quantitative decision remains a significant challenge.

The authors conclude that while agents can approximate costs, reliable, exact procurement reasoning under realistic conditions remains an open problem, necessitating further research into structured tool use and robustness to input variation.