Technical Summary: Evaluating Quantum Solver Generation with LLMs

Problem Statement

While Large Language Models (LLMs) have demonstrated strong capabilities in code generation, their application to scientific computing—specifically for generating quantum solvers—presents unique challenges. In standard software engineering, success is often measured by syntactic correctness or the ability to pass test cases. However, in scientific computing, a script may execute without runtime errors yet produce numerically inaccurate results. This is particularly critical in quantum computing, where correctness depends on non-trivial mappings (e.g., fermion-to-qubit), hybrid quantum-classical workflows, and algorithm-specific approximations. Existing benchmarks largely focus on pedagogical use cases, symbolic derivations, or the generation of syntactically valid quantum circuits, failing to evaluate whether LLMs can generate solvers that produce numerically correct results for realistic scientific problems.

Methodology: Q-SAGE

To address this gap, the authors introduce Q-SAGE (Quantum-Solver Automated GEneration), an execution-based methodology designed to evaluate LLMs' capability to generate correct quantum solver code. Unlike static analysis, Q-SAGE employs a multi-turn iterative loop consisting of three core principles:

1. Numerical Verification: Correctness is defined by numerical agreement with a trusted reference (a "golden standard" classical solver) within a specified tolerance, rather than mere execution success.
2. Execution-Based Feedback: Failures are detected through the execution of the generated code. If the output does not match the reference, the system provides feedback to the LLM to guide refinement.
3. Closed-Loop Refinement: The process follows a generate-execute-verify-iterate cycle. The LLM generates a solver, the code is executed in a controlled environment, and the output is compared against the classical solver. If the result is incorrect, the LLM receives feedback (either just the error/output or the error/output plus the expected reference result) and attempts to revise the code.

The methodology is problem-agnostic, requiring only a machine-readable problem description, a coder prompt, a feedback prompt, and a classical solver for verification.

Experimental Design

The authors empirically evaluated Q-SAGE using:

• Five Problem Families (PFs): Spanning diverse domains and complexities:
  • PF1: Fermi-Hubbard model (Condensed Matter Physics).
  • PF2: Transverse Field Ising Hamiltonian (Many-body Statistical Physics).
  • PF3: MaxCut problem (Combinatorial Optimization).
  • PF4: Schwinger model (Lattice Gauge Theory).
  • PF5: Molecular Electronic Structure (Quantum Chemistry).
• Five LLMs: A mix of open-source and proprietary models with varying parameter sizes (e.g., DeepSeek-V3.2, GPT-OSS, Qwen3-Coder variants, and GPT-5.2).
• Algorithms: Evaluation covered various quantum paradigms, including Variational Quantum Eigensolver (VQE), Quantum Annealing, Trotterization, and Quantum Phase Estimation (QPE).
• Feedback Configurations: Two modes were tested: (1) providing only the execution output/error, and (2) providing the execution output plus the expected reference result.

Key Results

1. Effectiveness of Iterative Refinement (RQ1)

Iterative feedback substantially improves success rates, with the majority of gains occurring within the first five iterations.

• Problem Complexity: Success rates varied significantly by problem family. PF2 (Ising) and PF3 (MaxCut) achieved the highest success rates (up to ~90% at 10 turns), while PF1 (Fermi-Hubbard) remained the most challenging, with very low success rates (<3%).
• Model Performance: GPT-5.2 demonstrated the best overall performance.
• Feedback Type: Surprisingly, providing the expected numerical reference result did not yield statistically significant improvements over providing only the execution output. This suggests that iterative refinement can be effective even without a full oracle, provided the LLM receives execution feedback.

2. Failure Modes (RQ2)

The nature of failures shifts as model capability increases:

• Weaker Models: Predominantly fail due to code generation errors (syntax, undefined names) and API/dependency misuse (incorrect library versions, missing imports).
• Stronger Models: While they generate executable code more frequently, they increasingly fail due to numerical inaccuracies. The generated code runs but produces results that deviate from the ground truth.
• Problem Dependency: PF5 (Quantum Chemistry) showed a high prevalence of numerical errors across models, reflecting the strict precision requirements of the domain. PF2 and PF4 saw more API misuse, indicating challenges with specific quantum software stacks.

3. Computational Overhead (RQ3)

Achieving a correct solution via iterative refinement incurs a significant time cost.

• The time to reach the first successful solution is consistently and substantially higher than the cost of a single generation attempt (Turn 1).
• This overhead exhibits long-tail behavior, particularly for complex problem families (PF1, PF4), where iterative refinement leads to extended execution times.
• For high-performing models like GPT-5.2, the efficiency-effectiveness trade-off is more favorable, as they reach success with fewer iterations and less time overhead compared to weaker models.

Significance and Claims

The paper claims to be the first study to evaluate LLM-generated quantum solver code through multi-turn execution-based numerical verification against trusted reference implementations. Its primary contributions are:

1. A Methodology: Q-SAGE provides a framework for assessing end-to-end numerical correctness, moving beyond syntactic validation.
2. A Benchmark: It establishes a benchmark for LLMs on Hamiltonian solvers across diverse scientific domains.
3. Empirical Insights: The study highlights a critical capability limit for current LLMs: while they can effectively generate runnable code and benefit from iterative feedback to fix syntactic issues, they struggle with domain-specific numerical correctness.

The authors conclude that while LLMs can automate components of scientific workflows, the reliance on iterative refinement introduces non-negligible computational overhead. Furthermore, as models improve, the bottleneck shifts from "can the code run?" to "is the result scientifically accurate?" This suggests that for LLMs to be viable in scientific computing, future systems must integrate robust verification mechanisms and domain-specific guidance to address numerical inaccuracies that current models cannot self-correct effectively.