A Benchmark on LLM-Based Power Flow Computation: Do More Structured Prompts Help?

Technical Summary: A Benchmark on LLM-Based Power Flow Computation

Problem Statement
Power flow computation is a foundational element of grid planning, operation, and stability analysis. While Large Language Models (LLMs) are increasingly explored as decision-support tools for smart grids—ranging from fault diagnosis to operator Q&A—their ability to execute precise numerical algorithms remains unverified. Specifically, the Gauss–Seidel method, a canonical iterative solver taught in power engineering curricula, requires complex arithmetic, multi-pass convergence, and careful tracking of intermediate voltage estimates. This paper addresses the gap in understanding whether LLMs can reliably execute such classical iterative algorithms and investigates how prompt engineering strategies (e.g., structured inputs, chain-of-thought, few-shot examples) influence performance on these tasks.

Methodology
The authors designed a controlled benchmark evaluating three LLMs across four distinct prompt formats using a three-bus AC power flow system.

• System and Solver: The testbed is a fully connected three-bus system (one slack bus, two PQ load buses) with randomized branch parameters ($R, X$) and load values. The reference solutions were computed numerically using Python (NumPy) with double-precision arithmetic. The Gauss–Seidel method was selected over Newton–Raphson to isolate the difficulty of iterative arithmetic from matrix factorization complexity, ensuring the benchmark tests the model's ability to follow a known algorithmic procedure.
• Models Evaluated:
  1. Gemini 2.5 Pro (Google DeepMind)
  2. Claude Sonnet 4.5 (Anthropic)
  3. GPT-3.5 Turbo (OpenAI) as a lower-capability baseline.
• Prompt Formats: Four formats were tested to vary representational complexity:
  1. Narrative-final: Natural language description of the problem (textbook style) requesting only the final answer.
  2. Narrative+Example: The same narrative plus a complete worked example from a textbook.
  3. Procedural-text: A numbered, step-by-step procedure with bullet-point data, requiring an iteration trace table and the final answer.
  4. Structured-JSON: Machine-readable JSON input with the written procedure, requiring the same iteration trace and final answer.
• Evaluation: The benchmark ran on 50 test cases, with an independent 100-case replication. Metrics included Mean Absolute Error (MAE) in MW/MVar and the percentage of cases falling within specific relative error bins (<1%, 1–5%, 5–10%, 10–20%, >20%). Failures (non-parseable outputs) were counted in the >20% error bin to reflect end-to-end reliability.

Key Contributions

1. Benchmark Design: The paper presents the first controlled study systematically varying prompt formats for a classical iterative AC power flow solver, holding the algorithm and test network constant.
2. Prompt Sensitivity Analysis: The study demonstrates that prompt engineering effects are highly model-dependent. For the most capable model (Gemini 2.5 Pro), increasing prompt complexity (moving from narrative to structured JSON with iteration traces) significantly degraded performance, increasing MAE by a factor of 3.1×. Conversely, adding a worked example degraded Gemini's performance but provided a marginal gain for Claude.
3. Capability Limits: The authors establish that no configuration tested achieved reliability sufficient for direct use as a numerical solver. GPT-3.5 Turbo failed on at least 90% of cases across all formats, while even the best-performing configuration (Gemini with a simple narrative prompt) achieved only 54% of cases within 5% relative error.

Results

• Gemini 2.5 Pro: Exhibited the widest sensitivity to prompt format. The simplest format (narrative-final) yielded the best results (MAE = 0.257 MW/MVar; 54% of cases within 5% error). The most complex format (structured-JSON) resulted in the worst performance (MAE = 0.789). Adding a worked example increased error by 73%.
• Claude Sonnet 4.5: Showed a near-flat accuracy profile across all formats, with approximately 38% of cases within 5% error regardless of the prompt structure. The procedural-text format performed slightly better for Claude than for Gemini, suggesting optimal prompting is model-specific.
• GPT-3.5 Turbo: Demonstrated near-total ineffectiveness, with 90–96% of cases exceeding 20% relative error across all formats.
• Real vs. Reactive Power: A consistent asymmetry was observed where reactive power ($Q_g$) errors exceeded real power ($P_g$) errors for most configurations, potentially due to less frequent representation of imaginary-part extraction in training data.
• Replication: The 100-case replication confirmed the qualitative ordering (Gemini $\gg$ Claude > GPT-3.5) and the lack of solver-grade accuracy. However, the optimal format for Gemini shifted from narrative to procedural-text in the larger sample, suggesting sensitivity to instance difficulty or sampling variability.

Significance and Claims
The paper concludes that while LLMs show promise for high-level grid tasks (explanation, Q&A), they currently lack the reliability required for direct numerical computation of power flow. The findings challenge the common assumption in prompt engineering that providing more explicit, structured instructions always improves arithmetic accuracy; in this domain, simpler prompts yielded superior results for the best-performing model.

The authors position this work as a diagnostic baseline for practitioners and researchers. The primary implication is that deploying LLMs for smart-grid decision support requires careful calibration of prompt formats based on the specific model, as "more detailed prompting" can be counterproductive. Furthermore, the study asserts that current LLM configurations should not be used as replacements for traditional numerical solvers in production environments, though they may serve as approximate reasoning aids or educational tools.