SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

SimulCost introduces the first cost-aware benchmark and toolkit for evaluating LLMs in physics simulations, revealing that while multi-round agent interactions improve accuracy over single-round attempts, they remain less efficient and more expensive than traditional parameter scanning methods.

The Gist

Imagine you are trying to bake the perfect cake, but you don't have a recipe. You have a very smart, well-read assistant (the LLM) who has read millions of cookbooks. Your goal is to get a cake that tastes just right (high accuracy) without burning down your kitchen or using up all your flour and eggs (high cost).

Most previous tests for AI assistants only asked: "Did the assistant eventually bake a cake that tasted good?" They didn't care if the assistant burned 100 cakes trying to get there, or if they used a truckload of flour when a cup would have done.

SimulCost is a new, stricter test that changes the rules. It asks: "Did the assistant bake a good cake, AND did they do it efficiently without wasting resources?"

Here is a breakdown of the paper's key findings using everyday analogies:

1. The Problem: The "Free Trial" Trap

In the past, researchers tested AI on science problems by counting how many times the AI had to "try" to get the right answer. They treated every attempt as if it were free.

• The Reality: In real physics simulations (like predicting weather or designing a bridge), every "try" costs money, time, and electricity. It's like paying $1,000 for every cake you bake.
• The Flaw: If an AI guesses randomly and gets lucky on the 50th try, it passes the old test. But in the real world, you'd be bankrupt after 50 tries.

2. The Solution: SimulCost (The "Budget-Conscious Chef")

The authors created SimulCost, a benchmark with 12 different "kitchens" (simulators) ranging from fluid dynamics (how water flows) to plasma physics (how stars burn).

• They gave the AI a specific task: "Find the right settings to simulate this fluid flow."
• They measured two things:
  1. Did it work? (Success Rate)
  2. How much did it cost? (Efficiency)

3. The Results: The AI is a "Guessing Game" Master, but a "Budget" Disaster

Scenario A: The "One-Shot" Guess (Single-Round)

The AI gets one chance to guess the settings.

• The Result: The AI is okay at low-stakes tasks (getting a "good enough" cake). It succeeds about 46–64% of the time.
• The Catch: When the requirements get strict (you need a Michelin-star cake), the AI's success rate drops to 35–54%.
• The Lesson: The AI is bad at guessing the exact right settings on the first try, especially for hard problems. It often guesses settings that are "safe" but wasteful (like using a giant oven for a single cookie).

Scenario B: The "Trial and Error" (Multi-Round)

The AI gets to try again and again, learning from its mistakes (like tasting the batter and adding more sugar).

• The Result: Success rates jump to 71–80%. The AI gets the cake right!
• The Catch: It is 1.5 to 2.5 times slower (and more expensive) than just letting a computer brute-force the answer.
• The Metaphor: Imagine you are looking for a lost key in a dark room.
  • Brute Force: You turn on a light and sweep the whole floor systematically. It takes a fixed amount of time.
  • The AI: It uses its "intuition" to guess where the key might be. It finds it more often than a random guesser, but it spends so much time wandering around the room that it takes longer than just turning on the light and sweeping.

4. Key Takeaways for the Future

• Don't trust the AI's first guess for hard problems. If you need high precision, the AI's "intuition" isn't reliable enough to save you money.
• Let the AI be the Manager, not the Worker. The best strategy isn't to let the AI guess the numbers itself. Instead, let the AI call a computer program that does the brute-force scanning. The AI is good at knowing what to ask, but bad at doing the heavy lifting efficiently.
• Examples help, but they can backfire. Giving the AI examples of past successful cakes (In-Context Learning) helps it guess better the first time. However, it makes the AI too rigid. If the new problem is slightly different, the AI gets stuck trying to copy the old recipe instead of exploring new options.
• No "Magic Transfer." You can't train an AI on a cheap, simple simulation (like a toy car) and expect it to be an expert on a complex one (like a real race car). The "rules" for tuning parameters are too specific to each problem.

The Bottom Line

SimulCost tells us that while AI is getting smarter at science, it is currently too expensive to use as a standalone scientist for complex tasks. It wastes too much "fuel" trying to figure things out.

The future isn't about making the AI smarter at guessing; it's about teaching the AI to be a smart manager that knows when to stop guessing and when to let a specialized tool do the heavy lifting.

Technical Summary

1. Problem Statement

Current evaluations of Large Language Models (LLMs) in scientific domains primarily focus on task correctness (e.g., pass@k) and token costs, while largely ignoring the computational cost of tool usage. In physics simulations, tool execution (running a solver) is often orders of magnitude more expensive than LLM inference in terms of time and resources.

• The Gap: Metrics like pass@k with large $k$ implicitly treat simulation runs as "free," leading to unrealistic assessments. In real-world scientific workflows, increasing resolution or refining parameters to achieve high accuracy incurs quadratic or cubic cost scaling.
• The Challenge: LLMs lack "cost-awareness." They may generate correct solutions but do so inefficiently (e.g., choosing overly fine grids or excessive iterations), making them impractical for budget-constrained scientific tasks. There is no existing benchmark that evaluates LLM agents based on the trade-off between solution accuracy and computational cost (FLOPs/wall-clock time).

2. Methodology: The SimulCost Framework

The authors introduce SimulCost, the first benchmark designed to evaluate cost-sensitive parameter tuning in physics simulations.

A. Dataset and Scope

• Scale: The benchmark spans 12 physics simulators across three domains: Fluid Dynamics, Solid Mechanics, and Plasma Physics.
• Task Volume: It includes 2,916 single-round tasks (initial guess) and 1,900 multi-round tasks (trial-and-error adjustment), totaling 4,816 tasks.
• Cost Definition: Costs are defined analytically based on computational complexity (FLOPs) for 11 solvers, ensuring platform independence. One solver (EPOCH, a Particle-in-Cell code) uses wall-clock time on fixed hardware due to its compiled nature.
• Task Structure:
  • Single-Round: The LLM must propose a set of parameters in one shot to meet a specific accuracy threshold (Low, Medium, High) with minimal cost.
  • Multi-Round: The LLM has up to 10 trials to iteratively refine parameters based on simulation feedback (convergence status, RMSE, accumulated cost).
  • Baseline: A brute-force grid search (progressive refinement for monotonic parameters, exhaustive search for non-monotonic) serves as the reference for "optimal" cost and success.

B. Evaluation Metrics

1. Success Rate ($S$): Binary indicator (0/1) if the LLM's solution meets the accuracy threshold (RMSE or physical constraints).
2. Efficiency ($E$): A ratio comparing the brute-force reference cost ($C_{bf}$) to the LLM's cost ($C_{LLM}$):
   $$E_i = \frac{C_{bf}}{C_{LLM}} \times S_i$$
   • $E > 1.0$: The LLM found a cheaper solution than the reference (rare in single-round).
   • $E \approx 1.0$: The LLM matched the brute-force cost.
   • $E < 1.0$: The LLM was more expensive than the reference.

C. Baselines and Ablations

• Models Evaluated: GPT-5, Claude-3.7-Sonnet, GPT-OSS-120B, Llama-3-70B, and Qwen3-32B.
• Optimization Baseline: Bayesian Optimization with Gaussian Process (BO-GP) is used for multi-round comparisons.
• Ablation Studies:
  • In-Context Learning (ICL): Testing the impact of providing past experiment logs (with/without cost data).
  • Reasoning Effort: Varying the depth of Chain-of-Thought (CoT) reasoning.
  • Parameter Groups: Analyzing performance across Spatial, Temporal, Tolerance, and "Misc" (solver-specific) parameters.

3. Key Contributions

1. First Cost-Aware Benchmark: SimulCost is the first benchmark to jointly evaluate success rate and computational efficiency for LLMs in physics simulations.
2. Extensible Toolkit: The authors open-source a toolkit (simulcost-tools) with 12 solvers, standardized APIs, and platform-independent cost tracking, allowing the community to extend the benchmark.
3. Comprehensive Evaluation: Systematic comparison of frontier LLMs against traditional scanning and Bayesian optimization across diverse physics domains.
4. Practical Insights: Detailed analysis of failure modes, parameter correlations, and the efficacy of ICL and reasoning strategies.

4. Key Results and Findings

A. Performance Overview

• Single-Round Reliability: Frontier LLMs achieve 46–64% success rates. Performance drops significantly (to 35–54%) under high-accuracy requirements, indicating that initial guesses are unreliable for precise tasks.
• Multi-Round Improvement: Iterative tuning boosts success rates to 71–80%. However, LLMs are 1.5–2.5× slower (more costly) than brute-force scanning.
  • Implication: For high-accuracy tasks, LLMs should be used to invoke scanning algorithms rather than relying on their internal reasoning alone for parameter selection.

B. Efficiency Analysis

• The "Safe" Bias: In single-round mode, LLMs often choose "safe" but overly conservative parameters (e.g., excessively fine grids) that satisfy accuracy but waste compute.
• Parameter Type Differences:
  • Spatial/Temporal parameters: Easier to tune; LLMs have better priors.
  • Misc (Solver-specific) parameters: Hardest to tune; LLMs struggle with convergence tolerances and relaxation factors.
• Knowledge Transfer: There is no significant correlation between tasks within the same parameter group across different solvers. Fine-tuning on cheap simulators does not transfer well to expensive ones.

C. Ablation Insights

• In-Context Learning (ICL):
  • Pros: Improves single-round success by 15–25% by narrowing the search space.
  • Cons: Degrades multi-round performance by anchoring the model to demonstrated regimes, limiting exploration.
  • Cost Data: Including cost information in examples is critical for maintaining efficiency; omitting it leads to high-cost solutions.
• Reasoning Effort: Increasing the depth of reasoning (CoT) in GPT-5 did not significantly improve parameter selection. The bottleneck is not reasoning depth but the lack of grounded, task-specific derivation from first principles.
• Bayesian Optimization: BO-GP matches LLM success rates but suffers from higher variance and often selects extreme values early, leading to high cumulative costs. LLMs leverage physics intuition for better initial guesses in low-accuracy regimes.

D. Failure Modes

The authors identified five recurring failure patterns:

1. False Positives: Stopping prematurely when convergence is met but parameters are too loose.
2. Blind Exploration: Making random micro-adjustments without systematic strategy.
3. Instruction Misunderstanding: Continuing to refine after a valid solution is found, inflating cost.
4. Prior Bias: Memorizing "canonical" values (e.g., always setting $\beta=1.5$) regardless of context.
5. Conservative Strategy: Over-engineering solutions (e.g., choosing 100x more particles than needed) to ensure safety.

5. Significance and Future Directions

• Realism: SimulCost shifts the evaluation paradigm from "can the LLM solve it?" to "can the LLM solve it efficiently?" This is crucial for deploying agents in real scientific workflows where compute budgets are finite.
• Deployment Guidance: The paper advises practitioners to use LLMs for initial rough previews (low accuracy) or to orchestrate scanning algorithms for high-accuracy tasks, rather than relying on LLMs for direct trial-and-error tuning.
• Future Work: The authors suggest developing tool-augmented tuning (e.g., early stopping, visual feedback), cost-aware post-training, and multi-parameter joint optimization as next steps.

In summary, SimulCost demonstrates that while LLMs show promise in scientific reasoning, they currently lack the cost-awareness required for autonomous, efficient scientific discovery without human intervention or hybrid algorithmic strategies.