Technical Summary: INCARBench

Problem Statement

Large language models (LLMs) are increasingly integrated into first-principles computational workflows, yet their specific ability to configure scientific calculations remains poorly understood. In the Vienna Ab initio Simulation Package (VASP), the INCAR file serves as the critical interface translating scientific intent into computational setup. This task involves selecting and coordinating distinct, often coupled parameter settings for various physical effects (e.g., magnetism, DFT+U, van der Waals interactions) and workflow types (e.g., geometry relaxation, band-structure analysis).

A key challenge identified is that an executable INCAR file does not necessarily correspond to a scientifically correct calculation. Existing evaluations of LLMs in science focus primarily on knowledge retrieval, reasoning, or property prediction, rarely assessing whether a model can translate scientific intent into a configuration that is both physically meaningful and workflow-consistent. Furthermore, practical workflows often require modifying existing configurations rather than generating them from scratch, a capability (configuration repair) that remains largely unexplored.

Methodology

To address these gaps, the authors introduce INCARBench, a benchmark designed to evaluate LLMs on VASP INCAR generation and repair tasks.

Dataset Construction

• Scope: The benchmark covers 48 materials drawn from 16 representative material families (including metals, correlated oxides, battery materials, and magnetic systems).
• Workflows: Each material is paired with four standard VASP workflow types: static self-consistent field (SCF), geometry relaxation, band-structure non-self-consistent field (NSCF), and density-of-states (DOS) NSCF.
• Cases:
  • Generation: 192 cases (48 materials × 4 workflows).
  • Repair: 576 cases derived from the generation set. Each generation case is converted into three repair variants:
    1. Correct-to-Preserve: No injected errors, testing the model's ability to avoid over-editing valid configurations.
    2. Workflow-Mistake: Errors introduced in workflow-defining settings (e.g., NSCF controls, charge-density handling).
    3. Physics-Mistake: Errors introduced in physics-defining settings (e.g., magnetism, DFT+U, van der Waals corrections, smearing).
• Challenge Types: Cases are tagged with specific physics-aware challenges (NSCF workflows, DFT+U, magnetism, vdW, smearing, symmetry) to localize failure modes.

Evaluation Framework

Configuration quality is assessed through three complementary dimensions, distinguishing parameter-level correctness from scientifically valid configurations:

1. Semantic Correctness: Measures whether the intended physical meaning (e.g., ISPIN, MAGMOM, LDAU) is correctly encoded.
2. Policy Correctness: Evaluates whether numerical and methodological choices (e.g., ENCUT, EDIFF, SIGMA) are appropriately configured, using parameter-specific tolerance rules.
3. Task-Critical Correctness (TCC): A strict case-level metric evaluating whether all task-critical parameter groups (workflow-defining and physics-defining requirements) are simultaneously satisfied. A single error in a critical group causes the entire case to fail.

The benchmark evaluates 19 frontier LLM configurations from providers including OpenAI, Anthropic, Google, DeepSeek, Alibaba, and others, using standardized prompts and parsing procedures.

Key Results

Generation Performance

• Capability Separation: There is substantial variation in performance across models, with scores spanning over 25 percentage points. GPT-5.4 achieved the highest overall score (84.7%).
• The TCC Gap: While leading models achieved high semantic and policy scores, Task-Critical Correctness (TCC) remained substantially lower across nearly all model families. This demonstrates that parameter-level accuracy does not necessarily imply a scientifically valid configuration.
• Decoupling of Skills: A "Policy–Semantic gap" was observed. Some models (e.g., Qwen3.6+ NT) exhibited stronger policy fidelity than semantic understanding (reproducing parameter patterns without full physical intent), while others (e.g., GPT-5.4 XH) showed the reverse. This suggests physical understanding and numerical-policy recovery are partially independent capabilities.

Repair Performance

• Preservation vs. Correction: Models generally found it easier to preserve valid configurations than to correct injected errors. Preservation rates were consistently higher than error-fix rates.
• TCC Deficit: Even when models successfully preserved parts of a configuration and fixed some errors, they frequently failed to restore all task-critical parameter groups simultaneously, resulting in low TCC scores.
• Behavioral Modes: Most models exhibited "conservative" repair behavior (negative Fix–Preserve gap), protecting existing settings but leaving errors unresolved. A few models showed "aggressive" behavior, improving correction rates but risking the modification of valid settings.

Failure Localization

Failure analysis revealed that errors are not uniformly distributed but concentrate in physically coupled settings:

• High-Difficulty Regimes: Transition-metal oxides, correlated oxides, and battery materials involving DFT+U constraints; magnetic materials under magnetic configurations; and layered materials under van der Waals constraints.
• Context Dependence: Difficulty arises from interactions between material-dependent physics and specific challenge constraints (e.g., DFT+U is particularly challenging when combined with correlated systems), rather than isolated parameter prediction errors.

Significance and Claims

The paper establishes scientific configuration as a measurable capability of large language models. Its primary contributions and claims are:

1. Distinction of Validity: Generating an executable INCAR file is substantially easier than producing a scientifically valid configuration. The benchmark highlights that conventional parameter-matching metrics may overestimate an LLM's ability to support real scientific workflows.
2. Multi-Dimensional Evaluation: The framework demonstrates that scientific configuration requires the simultaneous satisfaction of physical intent (semantic), numerical strategy (policy), and workflow constraints (TCC).
3. Repair Complexity: The study reveals that error correction and configuration preservation are distinct capabilities. Effective scientific repair requires not only diagnosing incorrect settings but also exercising restraint to avoid disrupting valid configurations.
4. Foundation for Future Work: INCARBench provides a foundation for developing more reliable AI systems for computational materials science. The authors note that the current benchmark focuses on input configuration and does not evaluate execution outcomes or convergence behavior, suggesting future work could extend toward execution-aware benchmarks.

The authors conclude that while current frontier models show promise, significant gaps remain in handling complex, coupled physical regimes, necessitating further development to achieve robust AI-driven scientific workflows.