Technical Summary: MDGYM: Benchmarking AI Agents on Molecular Simulations

Problem Statement

The vision of autonomous scientific discovery posits that AI agents can independently formulate hypotheses, design experiments, and execute computational workflows. However, current benchmarks in software engineering (e.g., SWE-bench) primarily test syntactic correctness and behavioral verification via stack traces or test suites. These benchmarks fail to capture the unique challenges of scientific computing, where "correctness" is not defined by a known output but by physical validity.

In Molecular Dynamics (MD) simulation, a task is considered successful only if the resulting trajectory represents a physically coherent system. Unlike software engineering, MD simulations often run to completion without runtime errors even when configured with incorrect force fields, ensembles, or timesteps. The resulting output is numerically generated but physically meaningless. This "silent failure" mode, combined with the need for multi-stage reasoning (setup, minimization, equilibration, production, post-processing) and the absence of ground-truth feedback during execution, presents a uniquely demanding testbed for AI agents. The central question addressed is whether current AI agents can autonomously design and execute these physically grounded scientific workflows.

Methodology

The MDGYM Benchmark

The authors introduce MDGYM, an expert-curated benchmark comprising 169 independent molecular simulation tasks spanning two widely used MD engines: LAMMPS and GROMACS. The dataset is structured across three increasing difficulty levels:

• Level 1 (Easy): Requires minimal post-processing of native simulation outputs (e.g., temperature, pressure, total energy).
• Level 2 (Medium): Requires substantial post-processing or intermediate logic within the engine (e.g., coordination numbers, radius of gyration, hydrogen bond tracking).
• Level 3 (Hard): Involves complex, non-equilibrium setups, multi-stage runs, or specialized post-processing (e.g., curve fitting for diffusion coefficients, Young's modulus, shear viscosity).

The tasks cover diverse materials (biomolecules, metals, polymers, carbon nanomaterials) and physical states. All tasks were developed and verified by domain experts, with ground truth values obtained by running reference simulations. A rigorous quality control process involving LLMs for ambiguity detection and human visual inspection of trajectories (using VMD/OVITO) ensured physical realism.

Evaluation Framework

The benchmark employs a modular architecture with three extensible layers:

1. Agent Layer: Interfaces with CLI-accessible coding agents.
2. Orchestration Layer: Manages task prompts and execution.
3. Validator Layer: Compares agent outputs against ground truth within a 5% relative tolerance.

Agents Evaluated:

• Frameworks: OpenHands (open-source), Codex, and Claude Code.
• LLMs: Four backbones were tested, including proprietary models (OpenAI Codex gpt-5.2-codex, Anthropic Claude Code claude-4.6-sonnet) and open-source models (Qwen3-Coder, GPT-OSS).

Metrics:

• Full Success Rate: The percentage of tasks where all required metrics fall within the 5% tolerance.
• Partial-Credit Score: The fraction of individual metrics correctly computed.
• Failure Analysis: A taxonomy of failure modes derived from trajectory logs, including syntax errors, fabricated answers, premature termination, and physically incoherent configurations.

Key Results

Performance Overview

The evaluation reveals a significant performance gap across all tested agents:

• Easy Tasks: Even the strongest agents (Claude Code and Codex) achieve a full success rate of only 21%. Open-source models via OpenHands perform significantly worse (Qwen at 2%, GPT-OSS at 0%).
• Medium/Hard Tasks: Performance degrades sharply. Claude Code drops to 7% (Medium) and 4% (Hard); Codex falls to 4% across both. Open-source models converge to near-zero success rates.
• Partial Credit: While agents occasionally compute individual metrics correctly, the aggregate partial-credit scores remain low, indicating that even when parts of the workflow succeed, the end-to-end pipeline fails.

Failure Mode Analysis

Trajectory analysis identifies distinct failure patterns that differ from general software engineering benchmarks:

1. Three-Stage Drop: Agents frequently initiate simulations (Simulation Ran) but fail to produce a valid answer (Produced Answer) or a correct one (Correct Answer). For example, on easy tasks, while 84-98% of trajectories ran simulations, only 33-37% produced an answer, and only 21% were correct.
2. Physical Incoherence vs. Syntax:
   • Proprietary Agents (Claude/Codex): Failures are often due to physically incoherent configurations (e.g., wrong force fields, incompatible thermostats) rather than simple syntax errors. They successfully invoke simulation machinery but produce unstable trajectories.
   • Open-Source Agents: Show higher rates of Fabricated Answers (reporting values without running simulations) and Premature Termination (abandoning tasks upon encountering errors).
3. Pipeline Fragility: Errors compound through the simulation pipeline. Agents often succeed in initialization but fail during equilibration or production when physical parameter choices (e.g., timestep, coupling constants) determine stability. A single physical error invalidates the entire downstream result.
4. Post-Processing Errors: Even when simulations run successfully, agents often apply incorrect analysis procedures (e.g., wrong time windows for diffusion coefficients), yielding numerically plausible but physically incorrect results.

Significance and Claims

The paper claims that MDGYM serves as a critical diagnostic tool revealing a fundamental deficit in current AI agents: the inability to transfer fluent code generation to grounded physical reasoning.

• Domain-Specific Challenges: The failure modes observed in MDGYM are qualitatively distinct from those in software engineering. The lack of immediate, interpretable error signals (like stack traces) for physical invalidity means agents cannot rely on standard debugging heuristics.
• Limitations of Current Benchmarks: The results suggest that progress on general coding benchmarks (SWE-bench, HumanEval) does not translate to scientific computing capabilities. The "silent failure" nature of scientific simulations requires agents to possess domain knowledge to recognize and recover from errors without ground-truth feedback.
• Roadmap for Future Agents: The authors identify three capability gaps that must be addressed for agents to become reliable scientific partners:
  1. Physical Grounding: Reasoning about the physical coherence of a configuration before execution.
  2. Domain-Aware Error Recovery: Interpreting simulation-specific error messages to formulate physically meaningful corrections.
  3. Methodological Precision: Understanding the correct analysis procedures for specific physical quantities.

The paper concludes that while current agents show promise in syntax generation, they are not yet capable of autonomously executing end-to-end scientific workflows in molecular dynamics. MDGYM is released to track progress toward this goal and to highlight the specific capabilities required for reliable AI-driven scientific discovery.