Technical Summary: VeraGrid-Agent

Problem Statement

Large Language Models (LLMs) have demonstrated significant capabilities in code generation and reasoning, leading to their exploration in scientific and engineering domains such as power systems. However, answering complex questions regarding distribution optimal power flow (D-OPF) presents a unique challenge. These questions often require solving non-convex, NP-hard numerical problems involving AC power-flow equations, distributed energy resources (DERs), and grid constraints.

When LLMs attempt to answer such questions using only parametric knowledge (memory), they are forced to guess, often producing numerically incorrect outputs. The paper argues that linguistic reasoning alone is insufficient for tasks requiring precise numerical simulation, such as determining voltage rises at PV nodes, bus capacities, DER curtailment, or feeder losses. The core problem is distinguishing between genuine computation and memorization or hallucination in scientific reasoning.

Methodology

To address this, the authors propose VeraGrid-Agent, a tool-augmented LLM framework designed to autonomously solve D-OPF problems. The methodology involves a closed-loop workflow where the LLM acts as a controller for an external solver rather than relying on internal knowledge.

1. The Agent Workflow

The agent operates within an isolated workspace and follows a ReAct (Reasoning + Acting) framework. For each query, the agent:

• Writes Input: Generates a JSON input file describing the feeder graph, buses, branches, loads, and DER configurations based on the problem description.
• Executes Solver: Calls the open-source VeraGrid simulator, which solves the AC-OPF problem numerically.
• Reads Output: The solver returns execution metadata and a table of contents rather than the full solution to keep the context window compact. The agent then performs targeted reads of specific result lines (e.g., specific line flows or bus voltages).
• Selects Answer: Interprets the retrieved numerical data to select the correct option from a multiple-choice question (MCQ).

The system is capped at a maximum number of iterations ($T_{max}$) to prevent runaway loops, though most queries are resolved in a few steps.

2. The Benchmark: VeraGrid-MCQ-150

To evaluate performance, the authors introduce VeraGrid-MCQ-150, a deterministic, template-driven benchmark consisting of 150 multiple-choice questions.

• Generation: Questions are generated by expert-defined templates where the ground truth is computed directly by the VeraGrid simulator. This ensures that every answer is a deterministic function of a specific solved network record.
• Difficulty Levels:
  • Easy (50 questions): Direct retrieval (e.g., node voltage magnitudes, element counts).
  • Medium (50 questions): Single-step derivations (e.g., per-unit conversion, line impedance magnitude, power factor).
  • Hard (50 questions): Multi-step computation and cross-table inference (e.g., aggregated branch losses, net reserve power, thermal margins).
• Evaluation Metric: Accuracy is measured as the percentage of correctly predicted labels ($\hat{\ell}$) against the ground truth ($\ell^*$), treating malformed or missing responses as incorrect.

Key Contributions

The paper makes three primary contributions:

1. Tool-Augmented Architecture: A novel framework for D-OPF analysis where the LLM writes simulator inputs, executes the VeraGrid solver, and reads relevant results, ensuring every answer is traceable to the solver output.
2. Deterministic Benchmark: The introduction of VeraGrid-MCQ-150, a reproducible testbed with verified ground truth that evaluates an agent's ability to retrieve quantitative information from a solved network, preventing reliance on static memorization.
3. Empirical Study: A comprehensive evaluation of seven different LLMs under two regimes: "no-tool" (reasoning from description alone) and "agent" (tool-augmented). The study analyzes performance gains and failure modes.

Results

The authors evaluated seven models (including GPT-5.2, Claude Sonnet 4.5, Gemini 3 Flash, Grok 4.3/4.5, Composer 2.5, and Opus 4.8) on the 8-bus radial feeder test case.

• No-Tool Regime: Without access to the solver, all models performed poorly, with overall accuracies ranging from 41.3% to 49.3%. This confirms that parametric knowledge is insufficient for these numerical tasks.
• Tool-Augmented Regime: With the VeraGrid-Agent, accuracy improved dramatically across all models, ranging from 97.3% to 100.0%.
  • Gemini 3 Flash, Grok 4.3, and Grok 4.5 achieved perfect 100% accuracy.
  • GPT-5.2 and Claude Sonnet 4.5 missed only one hard question each.
  • Opus 4.8 and Composer 2.5 missed two and four questions, respectively.
• Failure Mode Analysis: The few remaining errors were not due to solver execution failures (no timeouts or execution errors occurred). Instead, errors arose from wrong interpretations during multi-step reasoning, such as aggregating branch losses at a single terminal or confusing DER curtailment with dispatch.
• Impact of Tools: The improvement was most significant for questions requiring numerical computation (e.g., feeder losses, voltage rises). Questions that could be answered from the prompt description alone (e.g., topology, line parameters) showed little improvement, as the no-tool baseline was already relatively high for those specific categories.

Significance and Claims

The paper claims that for grounded, numerically intensive engineering tasks like grid-edge analysis, access to external computation tools is more critical than the choice of LLM backbone. Once the agent has access to a deterministic solver, the performance gap between different LLMs largely disappears.

The work demonstrates that tool-augmented agents can effectively separate genuine computation from memorization and guessing. It highlights that while current models struggle with complex mathematical analysis when relying solely on parametric knowledge, equipping them with specialized solvers allows them to achieve near-perfect accuracy on tasks that were previously unsolvable by LLMs alone. The authors note limitations, including the use of a single balanced radial feeder and the time required for solver execution, suggesting future work will focus on unbalanced three-phase systems and fine-tuning smaller models using the agent's reasoning traces.