Gatekeepers and Hallucinations: A Layered Evaluation Framework for LLM-Driven Quantum Circuit Generation

This paper introduces a layered evaluation framework for LLM-driven quantum circuit generation that combines a physical gatekeeper rubric, fidelity analysis, and behavioral consistency metrics to identify specific failure modes and underscore the critical need for validating both model outputs and the evaluation infrastructure itself.

The Gist

Imagine you are hiring a team of brilliant, fast-talking architects (Large Language Models, or LLMs) to design the blueprints for a very specific, high-tech building: a Quantum Circuit. This isn't just any building; it's a machine meant to simulate the behavior of atoms and materials. If the blueprint has even one tiny error, the whole machine might collapse, or worse, it might look like it's working perfectly while actually doing something completely wrong.

This paper is a report card on how well these "architects" are doing, and more importantly, it introduces a new safety inspection system to catch their mistakes before they cause expensive disasters.

Here is the breakdown of their findings, using simple analogies:

1. The Problem: The "Silent Saboteur"

The authors found that these AI models are great at writing code that looks correct (like a blueprint that has the right fonts and colors), but they often fail at the physics.

• The Trap: Sometimes an AI will confidently say, "I built a circuit for a Hydrogen molecule," but if you look closely, it's actually built for a Carbon monoxide molecule.
• The Danger: In the past, we just checked if the code ran. But the authors found that some errors are "silent." The code runs, but it's solving the wrong problem. It's like a chef who follows a recipe perfectly but accidentally uses salt instead of sugar; the dish looks like a cake, but it tastes like a salty brick.

2. The Solution: The "Three-Layer Security Check"

To fix this, the team built a Layered Evaluation Framework. Think of this as a three-stage security checkpoint at an airport, but for quantum code.

• Layer 1: The Gatekeeper (The ID Check)
  Before the AI is allowed to do any heavy lifting, it must pass a quick screening. The system asks: "Do you understand the basic rules of physics? Do you know which molecule we are talking about? Do you know the correct tools to use?" If the AI fails this basic check, it's stopped immediately. This saves time and money by not letting bad ideas go further.

• Layer 2: The Fidelity Audit (The Blueprint Comparison)
  If the AI passes the gate, its blueprint is compared against a "Gold Standard" reference.

  • The Analogy: Imagine the AI claims, "I built a bridge with 3 support beams." The auditors check the math and say, "No, a bridge of this size must have exactly 3 beams based on physics laws. You said 10. You failed."
  • They found that many models guessed numbers (like the number of "knobs" or parameters in the circuit) that were physically impossible, even though the code looked perfect.

• Layer 3: The Consistency Test (The "Drunk vs. Sober" Test)
  The team asked the same AI to do the same task multiple times.

  • The Analogy: If you ask a human architect to draw a house 5 times, they might draw 5 slightly different versions. But if they are a reliable machine, they should draw the same house every time.
  • They measured "Design Entropy" (a fancy word for "how much the AI changes its mind"). They found that some models were very consistent (reliable), while others were all over the place. Interestingly, one top model (Claude Sonnet 4.5) was so consistent that it drew the exact same blueprint even when the "temperature" (randomness) of the system was changed.

3. The Big Surprise: The "Fake ID" Scandal

The most shocking part of the paper wasn't about the AI failing; it was about the testing system itself failing.

While reviewing the results, the authors noticed that two different AI models (Llama 3 and DeepSeek) seemed to have produced identical, wrong code. They thought the models were hallucinating.

• The Investigation: They dug into the "harness" (the software platform running the test) and found a bug. When the AI models failed to produce code, the testing platform silently swapped in a pre-made "fallback" template to keep the test moving.
• The Lesson: The platform accidentally lied, making it look like the AI made a mistake when the platform actually made the mistake.
• The Takeaway: You can't trust the test runner if you don't trust the test runner. The "Gatekeeper" must check the whole pipeline, including the tools used to test the AI.

4. The Five Types of "AI Hallucinations"

The paper categorizes the mistakes into five distinct types, like a medical diagnosis for AI:

1. Geometry Hallucination: "I'm building a house for a dog," but the blueprint is for a cat. (Wrong molecule).
2. Nonexistent API Usage: "I'll use the 'Super-Drill' tool." (The tool doesn't exist in the software library).
3. Runtime Integration Failure: The blueprint is perfect, but the construction crew (the software pipeline) crashes when trying to read it.
4. Constraint Violation: The instructions said "Just give me the blueprint," but the AI wrote a 10-page essay explaining its feelings instead.
5. Plausible-but-Unverifiable: The AI gives a summary ("It has 10 knobs") but no actual code, so you can't check if it's true.

Summary

The paper argues that as we start using AI to design complex quantum machines, we cannot just trust that the code "looks right." We need a strict, multi-layered inspection system that checks:

1. Does it follow the basic rules? (Gatekeeper)
2. Does the math match physical reality? (Fidelity)
3. Is the testing system itself honest? (Audit)

Without these checks, we risk building expensive quantum simulations that are beautifully written but completely useless. The authors conclude that this "Gatekeeper" approach isn't optional; it's the only way to ensure safety as AI becomes more integrated into science.

Technical Summary

Technical Summary: Gatekeepers and Hallucinations in LLM-Driven Quantum Circuit Generation

Problem Statement

As Large Language Models (LLMs) become integrated into quantum simulation workflows—serving as IDE copilots, notebook assistants, and agentic pipeline orchestrators—there is a critical gap in evaluation infrastructure. Current benchmarks often focus on syntactic correctness or executable code generation. However, for materials-informed Variational Quantum Eigensolver (VQE) tasks, the stakes are higher: models must preserve physically meaningful constraints, correctly interpret external database inputs (e.g., Materials Project), and maintain consistent design choices across runs.

The authors identify that LLM failures in this domain are not random but structured and diverse. Crucially, some failure modes are "silent": the output appears syntactically valid and plausible but is physically incorrect (e.g., wrong molecular geometry or nonexistent API calls). As model capabilities advance, the paper posits that output plausibility may increase faster than physical correctness, making robust evaluation infrastructure increasingly vital to prevent the propagation of errors through expensive quantum simulation pipelines.

Methodology

The paper proposes a layered evaluation framework designed to be reusable and model-agnostic, applied to the generation of VQE circuits for materials-informed tasks. The framework consists of three distinct stages:

1. Gatekeeper Screening: A lightweight rubric-based screening stage applied before committing to expensive materials-informed tasks. Models are tested on a baseline task (generating UCCSD code for H2/STO-3G/Jordan–Wigner) and graded on a 0–4 scale across seven criteria:

   • Physical Validity
   • Symmetry Enforcement
   • Reference State (Hartree–Fock)
   • Correlation Targeting
   • Locality
   • Framework Correctness
   • Explanation Quality

2. Structured Failure Taxonomy & Circuit Fidelity Analysis:

   • Ansatz Classification: Outputs are classified by the ansatz type actually instantiated in the code, independent of model claims.
   • Fidelity Metrics: For the H2/STO-3G/JW/UCCSD case, model outputs are compared against two reference types:
     • Analytical: Exactly 3 variational parameters (derived from first principles for a (2e, 2o) active space).
     • Reference Implementation: Specific gate counts and depth (e.g., depth 73, 24 CX gates) derived from a specific Qiskit 1.2.x decomposition.
   • Failure Taxonomy: The authors categorize failures into five distinct modes based on detectability (silent, runtime, or overt).

3. Design Entropy (Behavioral Consistency): A novel metric calculating the normalized Shannon entropy of distinct design tuples (depth, two-qubit gate count, parameter count) across repeated runs. This measures whether a model explores the design space broadly or converges to a template-driven behavior.

Experimental Setup:
The evaluation was conducted on an agentic workflow integrating the Materials Project via an MCP server. Multiple foundation models (including Claude Sonnet 4.5, Opus 4.1, Llama 3/4, DeepSeek R1, OpenAI OSS-120B, Nova Pro, and Qwen 3-32B) were tested. A forensic audit of the evaluation platform's source code was also performed to verify the origin of outputs.

Key Results

1. Failure Taxonomy and Silent Failures

The study identified five distinct failure modes:

• Geometry Hallucination: Generating valid circuits for the wrong molecule.
• Nonexistent API Usage: Calling methods or importing modules that do not exist.
• Runtime Integration Failures: Structurally correct code that fails due to pipeline crashes (e.g., null returns from database retrieval).
• Constraint Violations: Failure to follow strict output contracts (e.g., emitting chain-of-thought when code-only was requested).
• Plausible-but-Unverifiable Output: Providing metrics or summaries without runnable code.

Critical Finding: The authors discovered that two models (Llama 3 70B and DeepSeek R1) appeared to generate incorrect "wrong-molecule" code (CO instead of H2). A forensic audit of the evaluation harness revealed these were not model generations. The models failed to emit extractable code (one due to token exhaustion, the other due to no code block), triggering a silent fallback mechanism in the platform that substituted a pre-generated template with an incorrectly resolved formula. This demonstrated that evaluation infrastructure itself can be a source of silent failure, masquerading as model errors.

2. Circuit Fidelity and Parameter Counts

• Claude Sonnet 4.5 was the only model to produce confirmed, executed UCCSD output that matched all reference values (3 parameters, depth 73, 24 CX gates).
• Claude Opus 4.1 generated structurally correct UCCSD code, but the surrounding pipeline failed due to a TypeError in the response-handling layer (a runtime integration failure).
• Other Models: Most models reported parameter counts inconsistent with first principles (e.g., Nova Pro reported 10 parameters, a +233% error). OpenAI GPT produced a plausible API call but with physically inconsistent parameter counts and gate compositions.

3. Design Entropy and Stability

• Entropy: High entropy indicated broad exploration of circuit designs, while low entropy suggested template-driven behavior.
• Temperature Stability: Testing Claude Sonnet 4.5 across sampling temperatures ($T \in \{0.1, \dots, 1.0\}$) revealed that the model maintained near-identical code structure and API choices (structural similarity $\ge 0.96$ for $T \ge 0.3$). This contrasts with general code generation findings where diversity increases with temperature, suggesting a domain-specific inductive bias toward physically grounded canonical designs for this model.

Significance and Claims

The paper claims its primary contribution is not a ranking of current models, but the establishment of a shared vocabulary and methodology for characterizing failures that are structural to the task of LLM-driven quantum circuit generation.

• Gatekeeper Necessity: The authors argue that gatekeeper-style validation is a necessary safeguard, not optional, for reliable deployment. As models improve, silent failures will become harder to detect, making pre-commitment screening essential.
• Infrastructure Trust Boundary: A central claim is that the evaluation harness belongs inside the same trust boundary as the models. Pipeline-level contamination (like silent template substitution) can invalidate evaluation results, necessitating forensic audits of the infrastructure itself.
• Analytical Verification: The paper highlights that parameter count errors are the most accessible single diagnostic. Since the correct number of variational parameters for specific systems is analytically derivable, this provides a fast, definitive check that requires no circuit execution.
• Modest Scope: The authors remain modest regarding their findings. They note that the temperature stability observation is based on a single model and prompt ($n=5$) and should be viewed as preliminary. They also acknowledge limitations, such as single-rater rubric scoring and the fact that some models' true behaviors were obscured by harness failures.

In conclusion, the framework provides a foundation for transparent and reproducible assessment of agentic quantum tools, emphasizing that grounding LLM-generated code in physical constraints and external schemas is a persistent challenge that will not disappear with scale.