Agentic Scientific Simulation: Execution-Grounded Model Construction and Reconstruction

1. Problem Statement

The paper addresses a critical gap in the application of Large Language Models (LLMs) to scientific simulation: the translation of natural language descriptions into executable, physically valid simulation configurations.

• The Core Challenge: Natural language descriptions of physical models are inherently underspecified. Different resolutions of implicit choices (e.g., constitutive relations, boundary conditions, solver defaults) can produce configurations that are syntactically correct and physically valid but scientifically distinct.
• Limitations of Current Approaches: Existing "vibe coding" or generic agentic coding often prioritizes code that runs or passes basic syntax checks. However, for scientific simulation, mere execution success is insufficient. Without explicit detection and resolution of ambiguities, the correctness of the result and its reproducibility from the original description cannot be guaranteed.
• The Goal: To determine if an LLM agent can act as a genuine scientific assistant that navigates API contracts, unit conventions, and global constraints to produce a simulation that faithfully represents user intent, using the simulator itself as the authoritative arbiter of physical validity.

2. Methodology: The JutulGPT Framework

The authors propose JutulGPT, a reference implementation built on top of JutulDarcy, a fully differentiable, Julia-based reservoir simulator. The framework operationalizes an Execution-Grounded Interpret–Act–Validate Loop.

A. The Workflow Loop

The agent operates in three iterative phases:

1. Interpret: The agent parses user intent, identifies underspecified modeling choices, and determines required simulator components. Ambiguities are either resolved autonomously (with explicit assumption logging) or escalated to the user via targeted queries.
2. Act: The agent retrieves relevant documentation and examples (using Keyword Search and Semantic RAG), generates or modifies Julia code, and performs static analysis (linting, type checking).
3. Validate: The agent executes the code in a local Julia environment. Crucially, it analyzes solver diagnostics (convergence, conservation errors, timestep reductions).
   • Key Distinction: The simulator is not just a runtime; it is the truth engine. In JutulDarcy, a simulation only completes if local/global mass conservation and solver convergence criteria are met. Thus, successful completion is a certificate of physical validity, not just code stability.

B. System Architecture

• Conversational Layer: Manages dialogue, intent parsing, and ambiguity detection.
• Tool Layer: Provides structured access to documentation retrieval, code analysis, file manipulation, and execution.
• Runtime Layer: Interfaces directly with JutulDarcy, executing scripts and returning structured diagnostics (e.g., constructor mismatches, convergence failures) to the agent for revision.

C. The Simulation Substrate (Jutul)

Jutul is chosen for its residual-form architecture and automatic differentiation.

• Models are expressed as coupled balance equations and closure relations.
• The framework exposes explicit type structures and error messages, making constitutive assumptions inspectable rather than implicit.
• This allows the agent to interpret runtime failures precisely and revise the model based on physical admissibility rather than just syntax.

3. Key Contributions

1. Execution-Grounded Validation: The paper establishes a workflow where the simulator's internal enforcement of physical laws (conservation, convergence) serves as the primary correctness criterion, moving beyond simple "code that runs."
2. Structured Ambiguity Resolution: The framework explicitly detects underspecified choices. It distinguishes between explicit assumptions (logged by the agent) and tacit assumptions (inherited from simulator defaults).
3. Reconstruction Fidelity Analysis: The authors introduce a methodology to audit reproducibility by regenerating executable models from progressively abstract textual descriptions (from a detailed reproduction prompt to a journal-style summary).
4. JutulGPT Implementation: A fully open-source reference implementation demonstrating the integration of an LLM with a high-fidelity PDE solver for reservoir simulation.

4. Results

The paper evaluates JutulGPT through four dimensions:

• Documentation Navigation: The agent successfully synthesizes distributed documentation into coherent taxonomies (e.g., classifying well models), though its reliability is limited by the quality of docstrings and example scripts.
• Model Construction & Repair: In a canonical "quarter five-spot" two-phase flow test, the agent iteratively corrected API mismatches (e.g., constructor signatures) and parameter inconsistencies until the simulator ran to completion, demonstrating the power of the repair loop.
• Complex Heterogeneous Modeling: The agent successfully constructed a complex 3D reservoir model with stochastic heterogeneity, geometric deformation (anticline), and multi-well controls. It resolved underspecified inputs (e.g., porosity, viscosity) by proposing defaults with physical justification and validating them against the solver.
• Reconstruction Experiments (The Critical Finding):
  • High Fidelity: When provided with a detailed "reproduction prompt," the agent reconstructed the model with high fidelity, though minor deviations occurred due to procedural differences (e.g., random number sampling order).
  • Latent Degrees of Freedom: When provided with a "technical report" or "journal description," the agent produced physically valid but scientifically distinct results.
  • The "Tacit Assumption" Gap: A major finding was that choices inherited from simulator defaults are invisible to the agent's assumption log.
    • Example: The reference model used a compressible fluid model (a simulator default). The reconstruction agent, reasoning only from the text, explicitly chose an incompressible model because the text did not specify compressibility. This led to drastically different physical behaviors (buoyancy segregation vs. viscous fingering).
    • Conclusion: Textual descriptions alone cannot uniquely determine executable state; implicit defaults create a "structural gap" in reproducibility.

5. Significance and Implications

• Scientific Assistants vs. Code Generators: The paper argues that for scientific simulation, agents must be "execution-grounded." They must not just generate code but mediate between intent and physical validity, using the simulator as a feedback loop.
• Reproducibility Crisis: The study highlights that even with detailed textual descriptions, reproducibility is compromised by latent degrees of freedom (implicit defaults, procedural conventions). True reproducibility requires the preservation of the executable state (code + pinned versions + explicit parameterization of defaults), not just natural language descriptions.
• Design Guidelines for Future Systems:
  • Simulators must provide structured, machine-readable diagnostics to make the "Validate" phase robust.
  • Documentation must expose semantic relationships and default behaviors explicitly to reduce the "tacit assumption" gap.
  • Future agents should maintain a checklist of simulator defaults to ensure every choice (even implicit ones) is auditable.
• Future Directions: The authors suggest extending this framework to inverse problems (history matching) using Jutul's differentiability, and creating standardized tool interfaces to allow general-purpose agents to interact with domain-specific simulators.

In summary, the paper demonstrates that while LLM agents can effectively construct complex scientific simulations when grounded in execution feedback, reproducibility remains a fundamental challenge due to the invisibility of tacit assumptions inherited from software defaults. The proposed framework offers a pathway to auditing and narrowing these ambiguities.