Technical Summary: PHREEQC-MCQ-200

Problem Statement

While Large Language Model (LLM) agents are increasingly integrated with scientific software, it remains unclear whether tool access genuinely improves the reliability of scientific computation or merely adds complexity. Existing evaluations often focus on whether a model can call a tool, rather than determining when tool augmentation improves a model's ability to operate a deterministic scientific workflow and when the tool loop itself becomes a source of error. Furthermore, aggregate accuracy metrics often mask item-level regressions where agents lose items they previously answered correctly without tools.

Methodology

The authors introduce PHREEQC-MCQ-200, a diagnostic benchmark designed to evaluate tool-augmented agents on deterministic aqueous-geochemistry simulations using the open-source simulator PHREEQC.

• Dataset: The benchmark consists of 200 multiple-choice questions derived from 21 validated simulation scenarios. Each item has a unique, simulator-derived target answer. The agent must construct a PHREEQC input file, execute the simulator, inspect the structured output, and commit to a final answer (A, B, C, or D).
• Agent Harness: Agents operate in an isolated workspace with a fixed maximum of 24 steps. They are provided with tools to write/read files, list files, and execute PHREEQC.
• Output-Access Protocols: The study compares two protocols for handling the simulator's often large, section-structured outputs:
  1. Agent-Raw100k: The full output is inlined but truncated to 100,000 characters (head/tail).
  2. Agent-TOC (Table of Contents): The tool returns a section-level outline (names and line numbers). The agent must explicitly request specific sections via read_file calls.
• Evaluation Metrics: Beyond aggregate accuracy, the authors employ:
  • Gain/Loss/Retention Decomposition: Tracking which items are kept, gained, or lost relative to a no-tool baseline.
  • Trajectory Analysis: Monitoring step budgets, execution failures, and commit rates.
  • Rerun Stability: Measuring variance across independent reruns.
  • Failure Taxonomy: Domain-expert review of failure modes (e.g., bad input construction vs. output misinterpretation).

Key Contributions

1. PHREEQC-MCQ-200 Benchmark: An expert-validated artifact package for evaluating LLM agents in a deterministic, unique-answer scientific software setting, including simulator inputs, outputs, trajectories, and failure-analysis queues.
2. Necessity of Grounded Execution: Evidence that scaling chain-of-thought (CoT) reasoning tokens cannot substitute for grounded simulator execution. CoT yields marginal gains (±8 pp) with massive token expansion (up to 140×), whereas simulator access provides substantial gains (15.0–41.5 pp) for capable models.
3. Non-Additive Tool Augmentation: Demonstration that tool access is not purely additive. Top- and mid-tier agents lose 10–32 items they answered correctly without tools while gaining 62–110 new correct ones, resulting in retention rates of 56.4–86.5%.
4. Output-Access Protocol Sensitivity: Identification of a capability-tier interaction effect. A metadata-guided interface (TOC) acts as a Pareto-optimal solution for top-tier models (matching or exceeding raw-output accuracy with 11–57% token reduction) but degrades performance for mid-tier models (7.5–9.5 pp accuracy loss) due to navigation overhead.
5. Diagnostic Framework: A proposal to evaluate scientific agents by reporting item-level retention, output-access sensitivity, trajectory failures, and specific break points in the computation chain, rather than relying solely on aggregate accuracy.

Key Results

• Model Performance: Top-tier models (Claude Opus 4.6, Sonnet 4.6, GPT-5.4) achieved 72.5%–84.0% accuracy with tool access, significantly outperforming their no-tool baselines (27.0%–42.0%). Mid-tier models (GPT-5.1, GPT-5.2) showed mixed results, with GPT-5.1 and GPT-5.2 losing accuracy under the TOC protocol compared to Raw100k.
• Below-Mid-Tier Failure: The Gemini 3 Flash Preview agent suffered a 17.0 pp regression under TOC, failing to commit to answers due to step-budget exhaustion caused by navigation overhead.
• Retention Analysis: GPT-family models exhibited lower retention (56.4%–59.0%) compared to Anthropic models (83.3%–86.5%), losing a higher fraction of items they previously solved correctly.
• Failure Modes:
  • Top-tier: Failures often involve step-budget exhaustion after committing to an answer or subtle input modeling flaws (e.g., omitted charge keywords).
  • Mid-tier: Failures are dominated by premature commitment to wrong answers and high-confidence incorrect PHREEQC builds.
  • Below-mid: Failures are characterized by step-budget exhaustion without ever committing to an answer.
• Efficiency: The custom harness achieved 2.7–3.3× better tokens-per-correct-answer compared to vendor-default CLI harnesses (Claude Code, Codex) on a matched subsample.

Significance

The paper argues that scientific tool use should be framed as an end-to-end diagnostic problem rather than a simple tool-calling capability. The central finding is that tool augmentation is not a monotonic improvement; it introduces a trade-off between new capabilities and the loss of existing ones, heavily dependent on the model's capability tier and the output-access protocol.

The authors claim that evaluations of scientific agents must move beyond aggregate accuracy to report:

• Item-level retention (stability of correct answers).
• Sensitivity to output-access protocols.
• Trajectory failure modes (where the computation chain breaks).
• The specific interaction between model capability and interface design.

This work provides a reusable diagnostic harness for measuring scientific tool-use agents and locating exactly where reliability is lost, offering a more nuanced view of agent performance in deterministic scientific workflows.