Evaluating LLM-generated code for domain-specific languages: molecular dynamics with LAMMPS

This paper proposes a practical evaluation procedure, featuring normalization and syntax parsing, that enables domain experts to assess the validity of LLM-generated LAMMPS input scripts, thereby identifying common errors and establishing a path for integrating LLMs into scientific computing workflows without incurring costly computational tests.

The Gist

The Big Idea: The "Robot Chef" and the "Strict Recipe Book"

Imagine you have a brilliant Robot Chef (the Large Language Model, or LLM). This robot can read your request in plain English—like, "Make me a spicy pasta dish with garlic and basil"—and it can instantly write out a recipe.

However, the kitchen you are working in isn't a normal kitchen. It's a super-strict, high-tech molecular kitchen called LAMMPS. In this kitchen, the recipe book (the code) has very weird rules:

• If you say "salt" instead of "NaCl," the robot explodes.
• If you put the garlic in after the water boils, the whole dish turns to ash.
• If you forget to specify the temperature in Kelvin instead of Celsius, the food vanishes.

The problem is that while the Robot Chef is great at writing normal recipes, it often gets confused by the strict, weird rules of this molecular kitchen. It might write a recipe that looks like a recipe, but when you try to cook it, the kitchen shuts down, or worse, it cooks something that looks like pasta but is actually poisonous.

What the Researchers Did

The scientists at Purdue University wanted to see: Can we trust this Robot Chef to write recipes for our molecular kitchen?

To find out, they didn't just ask the robot to cook and hope for the best. They built a three-step "Safety Inspector" to check the robot's work before anyone tried to cook.

Step 1: The "Standardizer" (Normalization)

First, they took the robot's messy recipe and cleaned it up. They removed the robot's chatter ("Hey, let's make this tasty!") and turned everything into a standard, mathematical format. This is like translating a handwritten note with coffee stains into a clean, typed document so the next inspector can read it easily.

Step 2: The "Grammar Police" (The Parser)

Next, they ran the recipe through a special computer program (a parser) that acts like a Grammar Police Officer.

• This officer doesn't care if the food tastes good yet.
• They only check: "Did you use the right words? Did you put the ingredients in the right order? Did you close your parentheses?"
• If the recipe has a grammar mistake, the officer stops it immediately. This saves the researchers from wasting hours of computer time trying to cook a broken recipe.

Step 3: The "Short-Run Test" (Execution)

If the recipe passes the grammar check, they let the kitchen run the recipe, but only for 10 seconds.

• This is like asking the chef to start the stove and stir the pot for a moment to see if the fire catches.
• If the fire doesn't catch, they know there's a deeper problem (like the wrong type of fuel).
• To make sure the problem wasn't just the specific brand of fuel (the "potential" used in the simulation), they sometimes swap the fuel for a generic "zero" fuel just to see if the stove itself works.

What They Found

They asked the Robot Chef to make three different dishes, getting progressively harder:

1. Easy: A simple block of aluminum sitting still.
2. Medium: Heating up a block of nickel until it melts.
3. Hard: A high-speed crash simulation (like a bullet hitting a target).

The Results:

• The Easy Dish: The Robot Chef did pretty well! About 66% of the time, it got the recipe right on the first try.
• The Medium Dish: The success rate dropped. The robot started mixing up the "fuel types" (pair styles) and getting the heating speed wrong.
• The Hard Dish: The robot struggled badly. Only 1 out of 50 attempts was perfect.

Why did it fail?
The robot made three main types of mistakes:

1. The "Wrong Fuel" Mistake: It picked the right type of fuel (like EAM) but the wrong flavor (like eam/alloy vs. just eam). It's like ordering "soda" when the machine only accepts "Coke."
2. The "Magic Number" Mistake: When the robot didn't know a specific number (like the size of an atom), it just guessed a generic number (like "1") instead of looking it up. It's like a chef guessing the oven is 100 degrees because they forgot to check.
3. The "Hallucination" Mistake: The robot invented commands that don't exist. It wrote, "Add velocity to group," using a syntax that the kitchen doesn't understand. It made up a rule that sounded logical but wasn't real.

The Takeaway: The Robot is an Assistant, Not a Boss

The main conclusion of the paper is this: We cannot let the Robot Chef cook alone yet.

If we let the robot write the code without checking, we will waste massive amounts of time and computer power on broken simulations. However, the robot is still incredibly useful. It can write the draft of the recipe 90% of the way there.

The Solution:
We need a Human-in-the-Loop system.

1. The Robot writes the draft.
2. The Grammar Police (Parser) catches the typos and syntax errors.
3. The Human Expert checks the physics to make sure the "flavor" is right.

Why This Matters

This isn't just about molecular simulations. It's about how we use AI in science. We are moving from a world where scientists write every line of code themselves, to a world where AI helps write the code. But just like you wouldn't let a robot drive a plane without a pilot, we can't let AI write scientific code without a strict safety net.

This paper provides the blueprint for that safety net, showing us how to build tools that catch AI mistakes before they crash the simulation.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) have demonstrated strong capabilities in generating general-purpose code (e.g., Python, C++). However, their ability to generate scientifically valid code for Domain-Specific Languages (DSLs) remains largely unexplored and unverified.

• The Challenge: Scientific DSLs, such as the input language for LAMMPS (Large-scale Atomic/Molecular Massively Parallel Simulator), are syntactically complex, semi-structured, and highly sensitive to command ordering, implicit defaults, and unit consistency.
• The Gap: Unlike general programming languages, scientific DSLs often lack robust infrastructure (compilers, linters) for early error detection. Errors in LLM-generated scripts can lead to simulation crashes or, more dangerously, scientifically invalid results that appear plausible.
• The Need: There is a critical need for evaluation procedures that allow domain experts (who may not be DSL syntax experts) to assess the validity of LLM-generated inputs, identify common failure modes, and determine the feasibility of integrating LLMs into scientific workflows without costly computational trials.

2. Methodology

The authors developed a multi-stage evaluation pipeline to assess LLM-generated LAMMPS input scripts. The workflow consists of four distinct stages:

A. Experimental Setup

• Models Evaluated: Five state-of-the-art LLMs (GPT-4o, GPT-4.1, GPT-o3, GPT-5, and Claude 4 Opus) accessed via API in their base configurations (no fine-tuning).
• Tasks: Three prompts of increasing complexity adapted from prior work:
  1. Prompt 1 (Simple): Aluminum single crystal equilibration (NPT ensemble).
  2. Prompt 2 (Intermediate): Continuous heating of a Nickel single crystal (300K–2500K).
  3. Prompt 3 (Complex): Niobium spall simulation involving projectile-gap-target impact.
• Sampling: 10 independent generations per model per prompt (150 total scripts).

B. The Evaluation Pipeline

1. Normalization:
   • Scripts are processed using a custom lammps-ast Python package.
   • Operations include removing comments, merging multi-line commands, expanding loops, and resolving variables (replacing symbolic variables with numeric values).
   • Goal: Generate "canonical" files to ensure consistent parsing and eliminate trivial formatting errors.
2. Static Parsing (Syntax Validation):
   • A custom parser (built with the Lark library) converts scripts into a Typed Abstract Syntax Tree (AST).
   • This allows for programmatic inspection of command arguments, cross-referencing of regions/variables, and detection of malformed commands before execution.
3. Reduced-Step Execution:
   • Scripts are executed for only 10 steps to minimize computational cost while capturing runtime errors (e.g., command ordering issues, missing files).
   • Pair Style Zero (PSZ) Substitution: To isolate syntax/variable errors from physics errors, interatomic potential commands (pair_style) are temporarily replaced with pair_style zero. This allows the system to test script executability independent of potential file availability or parameter correctness.
4. Accuracy Assessment:
   • Scripts that pass execution are evaluated against quantitative checklists derived from the prompts.
   • Criteria include: lattice parameters, boundary conditions, ensemble settings, timestep, damping constants, and heating/cooling rates.

3. Key Contributions

• Evaluation Framework: Introduced a novel, multi-stage pipeline (Normalization $\to$ Static Parsing $\to$ Reduced Execution $\to$ Accuracy Check) specifically designed for scientific DSLs.
• Tooling: Developed lammps-ast, a parser that transforms LAMMPS scripts into ASTs, enabling static analysis and early error detection without running full simulations.
• Benchmarking: Provided the first benchmark of general-purpose LLMs on a realistic scientific DSL without domain-specific fine-tuning.
• Error Taxonomy: Identified and categorized specific failure modes unique to scientific DSLs, distinguishing between syntactic errors, semantic hallucinations, and physical reasoning failures.

4. Key Results

The study evaluated 150 scripts across 5 models and 3 prompts.

• Overall Performance:
  • Parser Pass Rate: ~74% of scripts passed the static syntax check.
  • Execution Success: Only ~32% executed without error on the first attempt.
  • One-Shot Accuracy: Only 27.3% of scripts satisfied all physical criteria specified in the prompts.
• Complexity Degradation: Performance dropped sharply as task complexity increased:
  • Prompt 1: 66% one-shot accuracy.
  • Prompt 2: 14% one-shot accuracy.
  • Prompt 3: 2% one-shot accuracy (only 1 script was fully correct).
• Model Comparison:
  • Claude 4 Opus showed the highest robustness (97% parser pass rate) but struggled with complex physics in later prompts.
  • GPT-5 achieved the highest one-shot accuracy (33%) overall and was the only model to generate a fully correct script for the most complex prompt (Prompt 3).
  • GPT-o3 exhibited the weakest performance across the pipeline.
• Common Failure Modes:
  1. Incorrect Pair Style Definitions: Models frequently confused eam vs. eam/alloy or hallucinated non-existent variants.
  2. Unit Conversion Errors: Failure to convert units (e.g., nm to Å) or assuming SI units when the DSL required specific units (e.g., units metal).
  3. Placeholder Values: Using generic defaults (e.g., lattice constant = 1 Å) instead of material-specific values.
  4. Hallucinated Syntax: Generating command arguments that look plausible but do not exist in LAMMPS documentation (e.g., incorrect velocity command syntax).
  5. Geometric/Logical Inconsistencies: In Prompt 3, models failed to coordinate boundary conditions, gap sizes, and projectile dimensions, leading to physically impossible setups.

5. Significance and Conclusion

• Limitations of Current LLMs: The study concludes that current base-configured LLMs are not yet capable of serving as autonomous scientific simulation designers. While they can capture high-level structures, they fail at the multi-step reasoning required for complex, coupled physical constraints.
• The Role of Validation: The paper argues that LLMs should be viewed as assistive components rather than autonomous agents. Their utility lies in generating starting points, which must then be rigorously validated by domain-aware tools.
• Path Forward:
  • Structured Validation: Integrating static parsers (like the AST-based tool developed here) is essential to catch errors before expensive simulations run.
  • Hybrid Workflows: Future systems should combine generation with structured validation loops, potentially using retrieval-augmented generation (RAG) to ground models in official documentation and prevent hallucinations.
  • Scientific Safety: As LLMs become more plausible, the risk of "silent failures" (scripts that run but produce wrong physics) increases. Systematic evaluation pipelines are critical for maintaining reproducibility and safety in computational science.

In summary, this work provides a critical roadmap for integrating LLMs into scientific computing by demonstrating that while the technology is promising, it requires robust, domain-specific validation infrastructure to be reliable.