Research and Prototyping Study of an LLM-Based Chatbot for Electromagnetic Simulations

This paper presents a Google Gemini 2.0 Flash-based chatbot that automates the setup, execution, and post-processing of two-dimensional finite element eddy current simulations using Gmsh and GetDP, thereby significantly reducing the time required to generate electromagnetic models with customizable geometries and analysis routines.

The Gist

Imagine you are an architect who wants to build a complex model of a city's power grid. Traditionally, to get a computer to simulate how electricity flows through this grid, you would have to spend hours writing thousands of lines of code. You'd have to tell the computer exactly where every wire goes, what shape the buildings are, and how to measure the heat generated by the electricity. It's like trying to build a house by hand-carving every single brick and nail yourself.

This paper introduces a digital assistant (a chatbot) that acts like a "magic architect." Instead of you carving the bricks, you just tell the assistant what you want in plain English, and it builds the entire simulation for you automatically.

Here is a breakdown of how this works, using simple analogies:

1. The Problem: The "Manual Labor" of Simulation

In the world of engineering (specifically electromagnetics), simulating how electricity moves through wires (called "eddy currents") usually requires two very specific, difficult tools:

• Gmsh: The "3D Printer" that draws the shapes of the wires and creates the mesh (the grid) for the computer to analyze.
• GetDP: The "Calculator" that solves the complex math equations to see how the electricity behaves.

Usually, an engineer has to write specific code to tell these tools what to do. If you want to change the shape of the wires from a circle to a square, or move them from a line to a circle, you have to rewrite the code. It's slow and tedious.

2. The Solution: The "AI Translator"

The researchers built a chatbot powered by a Large Language Model (LLM), specifically Google Gemini. Think of this chatbot as a highly skilled translator who speaks two languages:

1. Human Language: "Put 12 wires in a circle."
2. Computer Language: The specific code needed for Gmsh and GetDP.

How it works in practice:

• You speak: You type a prompt like, "Run a simulation with 100 wires arranged in a honeycomb pattern, and show me where the heat is highest."
• The AI thinks: It understands your request and instantly writes the Python code to arrange the wires and the specific math code to calculate the heat.
• The AI acts: It hands this code to the "3D Printer" (Gmsh) and the "Calculator" (GetDP), runs the simulation, and gives you the results.

3. The "Magic" Features (What the Chatbot Can Do)

The paper tested how well this chatbot could handle different levels of difficulty:

• Basic Level (The "Circle Maker"): You ask for wires in a circle. The AI writes code to place them perfectly.
• Intermediate Level (The "Custom Artist"): You ask for wires in a specific shape, like the letter "A" or a trapezoid. The AI figures out the geometry and writes the code to build it.
• Advanced Level (The "Specialist"): You ask for a very specific calculation, like "Show me the heat only on the wires at the top of the triangle." The AI has to write a specialized language (GetDP code) that it wasn't explicitly trained on, but it figures it out by looking at examples you gave it in the instructions.
• The "Reporter": After the simulation is done, the AI doesn't just give you a boring graph. It writes a summary in plain English, explaining why the heat is high in certain spots (e.g., "The wires are close together, so they are heating each other up").

4. The "Glitches" (Where it gets tricky)

Just like any new AI, it isn't perfect. The researchers found a few ways it can fail, which they call "Syntax" and "Semantics" errors:

• Syntax Error: The AI writes code that looks like English but is gibberish to the computer (like a sentence with no periods). The computer crashes.
• Semantic Error: The AI writes code that the computer can run, but the result is wrong. For example, if you asked for a square of wires, the AI might build a square with only 3 corners (because a square has 4, but the AI got confused).
• The "Hallucination": Sometimes the AI is so confident it invents things that don't exist. In one test, it tried to put wires on the "5 corners" of a square (which only has 4).

5. The Results: Speed vs. Perfection

The researchers tested this chatbot against different AI models:

• Small AI models: They were like interns who kept making spelling mistakes. They couldn't build the simulations correctly.
• Large AI models (like Gemini 2.5): They were like senior engineers. They could build complex simulations (like a "Milliken-type" conductor with 100+ wires) with high accuracy.

The Big Win:
Without this AI, a human engineer might take 2 to 8 hours to set up a complex simulation. With this chatbot, it takes seconds. Even if the AI needs to try a few times to get the code perfect, it is still much faster than doing it by hand.

The Bottom Line

This paper proves that we can use AI not just to solve physics problems, but to build the tools that solve them. It turns the process of setting up a simulation from "hand-carving a statue" into "telling a robot what to build."

While the AI still needs a human to double-check the results (to make sure it didn't build a square with 3 corners), it drastically reduces the time engineers spend on boring setup work, allowing them to focus on the actual science and design.

Technical Summary

Here is a detailed technical summary of the paper "Research and Prototyping Study of an LLM-Based Chatbot for Electromagnetic Simulations."

1. Problem Statement

The paper addresses the bottleneck in setting up numerical simulation models for electromagnetic boundary value problems (BVPs). While significant research exists in using Machine Learning (ML) to solve these problems (e.g., Physics-Informed Neural Networks), there is a gap in using AI to assist in the generation and setup of the simulation models themselves.

The specific challenge is to reduce the "time-to-experimentation" for engineers. Traditionally, setting up a finite element (FE) simulation involves manually writing code for mesh generation (geometry definition) and solver configuration (domain-specific language), which is time-consuming and prone to human error. The authors propose using a Large Language Model (LLM) to automate this workflow, translating natural language user prompts into executable simulation code.

2. Methodology

The authors developed a chatbot-driven workflow centered around the multimodal LLM Google Gemini-2.0-Flash. The system integrates open-source finite element tools:

• Gmsh: For mesh generation.
• GetDP: For solving the 2D eddy current problems.
• Python: As the orchestration layer to manage interactions between the LLM, Gmsh, and GetDP.

The Workflow Architecture:

1. Input: The user provides a natural language prompt describing the geometry (e.g., "12 conductors in a circle") and desired outputs (e.g., "plot ohmic loss").
2. Prompt Engineering: The user prompt is combined with a System Prompt containing task descriptions, rules, and specific code examples (few-shot learning).
3. Code Generation: The LLM generates:
   • Python Code: To calculate conductor coordinates based on the geometric description and call the Gmsh API.
   • Domain-Specific Language (DSL) Code: GetDP .pro files defining the physics formulation ($A-\nu$ magnetic vector potential) and post-processing routines.
4. Execution: Python executes the generated code, running the simulation via command-line interfaces (CLI).
5. Output: The system visualizes results (via Gmsh) and can generate a textual summary of the physical phenomena observed (e.g., skin and proximity effects).

Key Architectural Variations Tested:

• Python Inference: Generating coordinate lists for various geometries (circles, grids, letters).
• DSL Inference with Examples: Generating GetDP code for post-processing (e.g., ohmic loss) when the system prompt includes specific code templates.
• DSL Inference without Examples: Generating GetDP code for complex physical quantities (e.g., magnetic energy density) relying solely on the LLM's internal knowledge and user constraints.
• Textual Summarization: Converting simulation results and code logic into natural language explanations.

3. Key Contributions

• Novel Application of LLMs: Unlike most AI-for-EM research focusing on solving BVPs, this work focuses on AI-assisted model generation, prioritizing code generation and execution over numerical scheme enhancement.
• Hybrid Workflow Design: A robust pipeline that combines an LLM with deterministic open-source tools (Gmsh/GetDP) and Python automation, avoiding the need for fine-tuning the LLM.
• Failure Mode Taxonomy: The authors introduce a conceptual framework (a "stack of syntaxes and semantics") to categorize failures in AI-generated workflows:
  • Syntax Errors: Invalid code (e.g., missing brackets).
  • Semantic Errors: Code that runs but produces physically incorrect results (e.g., wrong formula factors, incorrect geometry interpretation).
  • Geometric/Semantic Mismatches: Valid code that does not match the user's intent (e.g., hallucinating a 5th vertex for a square).
• Evaluation Framework: A quantitative benchmarking study comparing different LLM models (Gemma and Gemini families) across basic, intermediate, and advanced complexity levels.

4. Results

The study evaluated five LLM models using three tiers of complexity (Basic, Intermediate, Advanced) involving conductor arrangements and post-processing tasks.

• Model Performance:
  • Gemma-3-1b-It: Failed to produce syntactically correct code for any benchmark.
  • Gemma-3-27b-It: Could handle basic syntax but failed significantly on semantic correctness (e.g., failed to generate valid geometries for intermediate prompts or correct post-processing logic).
  • Gemini-2.5-Flash: Demonstrated the highest reliability. It required the fewest attempts to achieve syntactic success and showed high rates of semantic validity (correct geometry and physics) across all complexity levels.
  • Gemini-3.1-Flash-Lite: Performed well on basic tasks but struggled with complex geometric constraints compared to Gemini-2.5-Flash.
• Code Generation Capabilities:
  • The system successfully inferred Python code for complex geometries (hexagonal packing, sinusoidal curves).
  • It successfully generated GetDP DSL code for custom post-processing (e.g., calculating ohmic loss for specific subsets of conductors), provided the system prompt included relevant examples.
  • Without examples, the LLM struggled with the specific syntax of GetDP and the precise physical constants (e.g., using a factor of 0.5 instead of 0.25 for time-averaged energy), highlighting the need for context injection.
• Textual Summaries: The LLM could generate reasonable natural language summaries of physical effects (skin and proximity effects), though the lack of a quantitative metric for summary quality remains a challenge.
• Efficiency: The AI workflow reduced the time-to-experimentation from hours (for human engineers) to seconds, with a cost of approximately €1 per full benchmark run.

5. Significance and Future Outlook

• Accelerated Engineering: The study proves that LLMs can significantly accelerate the setup phase of electromagnetic simulations, allowing engineers to explore more design scenarios rapidly.
• Declarative Workflow: The approach shifts the paradigm from imperative coding (step-by-step instructions) to declarative specification (describing the desired outcome), lowering the barrier to entry for complex simulations.
• Limitations: The probabilistic nature of LLMs means results are not deterministic. "Hallucinations" (geometric inconsistencies) and semantic errors (incorrect physics) still occur, necessitating human validation.
• Future Directions:
  • Semi-Automated Evaluation: Developing methods to automatically verify the physical correctness of generated models (e.g., using embeddings or additional LLMs for evaluation).
  • Retrieval-Augmented Generation (RAG): Integrating RAG to allow the LLM to access specific, up-to-date documentation for FE tools, reducing reliance on training data and improving DSL accuracy.
  • AI Agents: Evolving from a static workflow to dynamic AI agents capable of planning and iterating on complex control-flow graphs for multi-step simulation tasks.

In conclusion, the paper demonstrates that while current LLMs are not yet perfect for fully autonomous scientific simulation, they are highly effective as "copilots" that drastically reduce setup time, provided that the workflow is carefully engineered with system prompts and human-in-the-loop validation.