A Preliminary Assessment of Coding Agents for CFD Workflows

This paper evaluates the effectiveness of tool-using coding agents in automating OpenFOAM CFD workflows, demonstrating that prompt-guided configurations and advanced language models significantly improve task completion rates for both tutorial-based and complex geometry simulations.

The Gist

Imagine you are trying to bake a very specific, complex cake (a Computational Fluid Dynamics or CFD simulation) using a massive, open-source cookbook called OpenFOAM.

In the old days, doing this was like trying to bake that cake by hand, blindfolded, while someone shouted instructions in a language you barely understand. You had to:

1. Find the right recipe page.
2. Manually rewrite ingredients lists (configuration files) in a dozen different notebooks.
3. Mix the batter (mesh generation).
4. Put it in the oven (run the solver).
5. If the cake collapsed, you had to read a confusing error note, guess what went wrong, and start over.

This paper is about teaching a Robot Chef (a "Coding Agent") to do this job for us, but with a specific trick to make it actually work.

The Problem: The Robot Chef Was Getting Lost

Researchers tried using advanced AI (Large Language Models) to automate this process. They built complex systems where one AI was the "Manager," another was the "Baker," and another was the "Quality Control."

But this was like hiring a whole orchestra to play a single note. It was expensive, hard to set up, and the robots often got confused. They would try to invent a new recipe from scratch instead of using a proven one, leading to burnt cakes (failed simulations).

The Solution: The "Copy-Paste-Edit" Strategy

The authors of this paper realized that human engineers don't usually invent recipes from scratch. They find a similar recipe in the cookbook, copy it, and tweak a few ingredients.

They gave their AI Robot Chef a simple, golden rule: "Don't reinvent the wheel. Find a similar tutorial first, copy it, and only change what's necessary."

They also taught the robot a second rule: "If the cake burns, read the smoke alarm (the error log) and fix just that one spot, then try again."

The Experiment: Two Types of Challenges

The researchers tested this robot on two types of tasks using a benchmark called FoamBench:

1. The "Tweak the Recipe" Tasks (Tutorial-Derivative)

• The Challenge: "Take the 'Chocolate Cake' recipe, but change the temperature to 350°F and use almond milk."
• The Result: The robot was amazing at this. By following the "Copy-Paste-Edit" rule, it successfully completed 100% of these tasks. It was faster, used less computer power, and made fewer mistakes than when it tried to write a new recipe from scratch. It was like a robot that knows exactly which page of the cookbook to flip to.

2. The "Design a New Mold" Tasks (Obstacle-Flow)

• The Challenge: "Bake a cake, but the pan has a weird diamond-shaped hole in the middle that you've never seen before. You have to design the mold yourself."
• The Result: This was much harder.
  • The "Junior Chef" (MiniMax Model): It tried to build the mold but failed. It either forgot the hole existed or made a mold that didn't fit. It couldn't visualize the 3D shape well enough.
  • The "Master Chef" (GPT-5.2): When they used a much smarter AI model, it succeeded! It could visualize the diamond shape, build a complex mold around it, and bake the cake perfectly.

The Key Takeaways (In Plain English)

1. Don't Start from Scratch: The biggest breakthrough wasn't a new AI model, but a new instruction. Telling the AI to "find a similar example first" made it 10x more reliable for standard tasks.
2. The Brain Matters: For simple tweaks, almost any decent AI works. But for complex tasks (like designing a new mold or geometry), you need a "smarter" brain (a more powerful AI model). The robot is only as good as the brain inside it.
3. Logs are Life-Savers: OpenFOAM (the software) is great at shouting "ERROR!" and telling you exactly which line is wrong. The robot learned to listen to these shouts and fix the specific line, rather than panicking and restarting.
4. Human Oversight is Still Needed: While the robot is great at fixing typos or missing files, it still struggles if the physics are weird or if the "mold" is too complex. It might bake a cake that looks fine but tastes like soap (physically incorrect results). Humans still need to taste-test the final product.

The Bottom Line

This paper shows that we don't need to build a super-complex, expensive AI system to automate engineering work. Instead, we can take a standard, off-the-shelf AI robot, give it a simple "cheat sheet" (the prompt to copy tutorials), and let it do the heavy lifting.

It's like giving a smart assistant a map and a compass instead of asking them to navigate a jungle blindfolded. For simple trips, they are perfect. For the hardest mountain climbs, we still need the best climbers (the most powerful AI models) and a human guide to make sure we don't fall off the edge.

Technical Summary

1. Problem Statement

Computational Fluid Dynamics (CFD) workflows, particularly within the open-source package OpenFOAM, are labor-intensive and prone to failure. Setting up a case requires coordinated edits across multiple interdependent dictionaries, precise command-line execution sequences, and iterative debugging.

• Current Limitations: Existing LLM-based automation systems often rely on complex, role-based multi-agent pipelines or require fine-tuning, introducing significant engineering overhead. Furthermore, single-pass automation fails to mimic the iterative nature of human engineers, who inspect intermediate artifacts (meshes, early-time solutions) and adjust settings before committing to full runs.
• The Gap: There is a need for a lightweight, reliable method to automate OpenFOAM workflows that leverages existing coding agent capabilities without requiring custom model training or complex system integration.

2. Methodology

The authors propose a minimalist, configuration-based approach using general-purpose tool-using coding agents (specifically OpenCode) to automate OpenFOAM case execution.

Core Strategy: "Tutorial-First" and "Log-Driven Repair"

Instead of asking the agent to generate a case from scratch, the system prompt guides the agent through a specific workflow:

1. Search & Reuse: The agent must first search the local OpenFOAM tutorial directory for a case similar to the user's request (using shell utilities like ls, find, grep).
2. Baseline Copy: The agent copies the selected tutorial as a baseline.
3. Minimal Edits: The agent applies only the necessary modifications (e.g., boundary conditions, geometry scaling, solver settings) while retaining the stable numerical settings of the tutorial.
4. Execution & Iteration: The agent runs the meshing and solver pipeline. If errors occur, the agent reads OpenFOAM error logs, identifies the root cause (e.g., invalid keyword, missing file), applies a minimal fix, and re-runs from the appropriate stage.

Experimental Setup

• Benchmark: The study uses FoamBench-Advanced from the CFDLLMBench suite, comprising 16 expert-authored cases.
  • 9 Tutorial-Derivative Tasks: Variants of existing tutorials requiring configuration edits.
  • 7 Planar 2D Obstacle-Flow Tasks: Complex cases requiring geometry interpretation and mesh generation (e.g., flow around cylinders, diamonds, rectangles).
• Models Evaluated:
  • MiniMax-M2.1: Used as the backbone for tutorial-derivative tasks.
  • GPT-5.2: Evaluated specifically for complex obstacle-flow tasks where mesh generation is critical.
• Protocol: Single-run execution (no human feedback loops) to test autonomous reliability, except for specific meshing ablation studies.

3. Key Contributions

1. Lightweight Automation Recipe: Demonstrates that a simple system prompt enforcing "tutorial-first retrieval" and "log-driven repair" significantly outperforms default agent prompts, eliminating the need for fine-tuning or multi-agent orchestration.
2. Agentic RAG via File System: Introduces a form of Retrieval-Augmented Generation (RAG) that relies on traditional file system commands (grep, ls) rather than vector databases, proving effective for structured, text-based CFD dictionaries.
3. Model Capability Analysis: Provides empirical evidence that while prompt engineering improves efficiency, model capability is the primary bottleneck for tasks requiring complex geometry and mesh generation.
4. Comprehensive Benchmarking: Offers a detailed evaluation of coding agents on both configuration-heavy (tutorial-derivative) and geometry-heavy (obstacle-flow) CFD tasks.

4. Results

A. Tutorial-Derivative Tasks (9 Cases)

• Success Rate: Under the default prompt, only 4/9 cases completed successfully. With the OpenFOAM-focused prompt, 100% (9/9) completed successfully.
• Efficiency: The guided prompt reduced token usage and tool calls significantly. The agent edited existing files rather than writing new ones from scratch, reducing "write" calls and increasing "read" calls.
• Accuracy: The generated cases closely matched reference structures (Mean $M_{struct} = 0.986$) and solution fields (7/9 cases had NMSE < 0.1).
• Mechanism: The agent successfully used logs to fix minor setup errors (e.g., switching perfectGas to pureMixture for compatibility with OpenFOAM v10).

B. Planar 2D Obstacle-Flow Tasks (7 Cases)

• MiniMax-M2.1 Performance: Failed to generate valid meshes for complex geometries. It often reused tutorial meshes without updating geometry offsets or failed to construct multi-block hexahedral meshes, resulting in simulations that did not capture the obstacle.
• GPT-5.2 Performance: Demonstrated a marked improvement.
  • Successfully generated accurate multi-block hexahedral meshes and hybrid meshes (using Gmsh API).
  • Completed all 4 evaluated obstacle-flow cases to the required end time.
  • Prompt Impact: While the "CFD Prompt" reduced token usage, it increased tool calls as GPT-5.2 performed more extensive tutorial searches. However, it produced a more reasonable initial mesh on the first attempt compared to the default prompt.
• Mesh Quality: GPT-5.2 could generate structured, unstructured, and hybrid meshes that captured complex obstacle boundaries, whereas weaker models could not.

5. Significance and Conclusion

• Practical Utility: Coding agents are viable for automating a significant portion of CFD workflows, particularly for 2D simulations and tasks based on existing tutorials. The "search-reuse-fix" paradigm drastically reduces the effort required for case setup.
• Limitations:
  • Geometry & Meshing: This remains the most challenging step. Success is heavily dependent on the underlying LLM's reasoning capabilities regarding geometry.
  • Physics & Stability: Agents struggle with complex physics (e.g., high-speed combustion) where errors do not trigger immediate log messages but cause divergence later.
  • Human Oversight: While agents can fix configuration errors autonomously, human supervision is still required for physics validation, mesh quality assurance, and handling non-trivial geometric complexities.
• Future Outlook: As LLM capabilities (especially in spatial reasoning and code generation) improve, coding agents are expected to become powerful tools for researchers, potentially extending to 3D simulations and industrial-grade workflows. The study suggests that model capability is currently the limiting factor, not the agent framework itself.

In summary, the paper establishes that prompt-guided coding agents can effectively automate OpenFOAM workflows by leveraging existing tutorials and error logs, but the complexity of mesh generation remains a critical dependency on the strength of the underlying language model.