MolQuest: A Benchmark for Agentic Evaluation of Abductive Reasoning in Chemical Structure Elucidation

1. Problem Statement

Current Large Language Models (LLMs) show promise in scientific discovery, but existing evaluation benchmarks suffer from three critical limitations when applied to complex, real-world research:

• Static Formats: Most benchmarks (e.g., ChemBench) use single-turn, multiple-choice Question Answering (QA) formats. These fail to measure a model's ability to perform multi-step iteration, strategic planning, or handle information asymmetry.
• Synthetic Data: Many datasets rely on simulated data that lacks real-world artifacts such as spectral noise, peak overlap, and systematic deviations, leading to inflated performance metrics.
• Lack of Agency: Existing evaluations treat models as passive respondents rather than active agents capable of experimental planning and decision-making under resource constraints.

Consequently, there is a gap in assessing abductive reasoning (inferring the best explanation from incomplete observations) and strategic decision-making required for authentic scientific tasks like Molecular Structure Elucidation (deducing a molecule's structure from spectral data like NMR and MS).

2. Methodology: The MolQuest Framework

MolQuest is a novel, agent-based benchmark designed to simulate a "Senior Spectroscopist" operating in a virtual laboratory. It reframes structure elucidation as a Constraint Satisfaction Problem (CSP) within a dynamic, interactive environment.

A. Data Construction (Human-in-the-Loop)

• Source: 530 validated molecular elucidation tasks extracted from the Supporting Information of high-quality chemical literature published after 2025.
• Pipeline:
  1. Automated Extraction: Multi-agent LLMs parse raw PDFs to extract IUPAC names and spectral data.
  2. Chemical Verification: Tools like RDKit and PubChemPy verify molecular formulas and theoretical properties to prevent hallucinations.
  3. Human Review: Expert chemists review "red-flag" data to correct subtle errors (e.g., stereochemistry, peak assignment) ensuring 100% ground-truth fidelity.
• Scope: Molecules range from 150 to 500 Da, covering diverse functional groups and chiral centers.

B. Interactive Agent Environment

The framework operates as a state-machine-driven simulation where the LLM acts as an autonomous agent:

• Action Space: The agent can call 14 simulated experimental tools (e.g., Measure_MW, Get_1H_NMR, Get_IR, Calculate_DBE).
• Workflow: The agent follows a "Plan–Request–Reason" loop. It must proactively decide which data to acquire based on current hypotheses, manage resource constraints (simulated costs), and iteratively refine structural candidates.
• Termination: The agent decides when to stop and submit a final structured result (SMILES string + confidence score).

C. Evaluation Metrics

The benchmark evaluates models across multiple dimensions:

• Structural Correctness: Exact SMILES match (Accuracy).
• Chemical Plausibility: SMILES Validity Rate and Formula Conservation (ensuring the generated structure matches the mass balance of experimental data).
• Probabilistic Reliability: Root Mean Square Calibration Error (RMSCE) to measure if the model's confidence aligns with its actual accuracy.
• Efficiency: Average Interaction Rounds and Accuracy per 1M Tokens.

3. Key Contributions

1. Dynamic Benchmarking Paradigm: Shifts evaluation from static QA to a multi-turn, interactive agent framework that mimics real scientific workflows.
2. Authentic Data Integration: Unlike previous works using synthetic data, MolQuest uses rigorously validated, real-world experimental data from post-2025 literature, preserving noise and ambiguity.
3. Diagnostic Capability: Introduces a framework to distinguish between models that possess strategic reasoning (active information seeking) and those that rely on pattern matching (passive data processing).
4. Comprehensive Evaluation: Provides a multi-metric assessment (accuracy, validity, calibration, efficiency) of 12 state-of-the-art LLMs.

4. Experimental Results

The study evaluated 12 frontier models (including GPT-5.2, Gemini 3 Pro, Claude 4.5, Qwen3-Max, and DeepSeek variants).

• Overall Performance: Even the best models struggle significantly.
  • Top Performers: Gemini 3 Flash (51.51% accuracy) and Gemini 3 Pro (48.30%) lead the field.
  • Mid-Tier: Models like Claude Opus 4.5 hover around 25–28%.
  • Struggling: Many models (e.g., DeepSeek V3.1, Qwen3 Max in baseline) fall below 10–15%.
• Dynamic vs. Static (Ablation Study):
  • "Empowered" Group: Models like Qwen3 Max and GPT-5.2 showed significant accuracy gains (+4% to +10%) in the dynamic Agent mode. The interactive framework acted as a "cognitive scaffold," helping them break down complex tasks.
  • "Challenged" Group: Models like Kimi K2 Thinking and Gemini 2.5 Pro performed worse in the dynamic setting, indicating a deficit in strategic planning and information value evaluation.
• Reliability & Hallucination:
  • Formula Conservation: Gemini 3 Pro achieved 93.57% conservation, strictly adhering to mass-balance constraints. In contrast, models like DeepSeek V3.1 dropped to ~23%, indicating "connectivity hallucination" where they ignore experimental constraints.
  • Calibration: Claude Opus 4.5 demonstrated the best self-assessment (lowest calibration error), signaling when it was likely wrong.
• Efficiency: Gemini 3 models achieved a near-optimal balance between interaction rounds (~4.7) and success. Conversely, weaker models often fell into an "interaction trap," requesting redundant data without synthesizing a solution.

5. Significance and Future Directions

• Redefining AI Evaluation: MolQuest demonstrates that static benchmarks conflate pattern recognition with true problem-solving agency. The dynamic framework is essential for evaluating "AI for Science."
• Strategic Reasoning Gap: The results highlight a critical gap in current LLMs' ability to perform abductive reasoning under uncertainty. While some models can learn to plan, others fail to adapt to resource-constrained environments.
• Human-AI Collaboration: The framework provides a diagnostic tool to identify which models are suitable for autonomous scientific discovery versus those requiring human oversight.
• Future Work: The authors suggest expanding the benchmark to larger biomolecules, applying the paradigm to other scientific domains (e.g., materials science), and developing adaptive interaction protocols to train "challenged" models in strategic planning.

Conclusion: MolQuest establishes a rigorous, reproducible standard for evaluating the strategic and abductive reasoning capabilities of LLMs in chemistry, revealing that while frontier models are approaching human-level performance in specific contexts, significant gaps remain in autonomous scientific reasoning.