QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging

The paper introduces QBugLM, a multi-agent framework for automating the debugging of OpenQASM 3.0 quantum software, and demonstrates through benchmarking that iterative feedback and structured prompting significantly enhance LLMs' ability to detect and repair silent quantum bugs.

The Gist

Imagine you are building a house, but instead of bricks and wood, you are using the laws of physics to build a "quantum house." The problem is that when this house has a mistake, it doesn't crash or fall down like a normal building. Instead, it just looks perfect on the outside but gives you the wrong address when you try to live in it. These are "silent bugs," and they are incredibly hard to find.

This paper introduces a new tool called QBugLM, which is like a team of AI detectives and repairmen designed specifically to find and fix these silent mistakes in quantum software.

Here is how the system works, broken down into simple steps:

1. The Setup: Creating the "Training Ground"

Before the AI can learn to fix bugs, the researchers had to create the bugs themselves.

• QBugGen (The Bug Maker): Think of this as a mischievous robot that takes a perfect quantum program and intentionally breaks it in specific ways. It creates a "test case" where the program is broken, but the researchers know exactly what is wrong. They have a checklist of common mistakes (like using an outdated language, mixing up wires, or adding too many steps).

2. The Team: Four Specialized Agents

QBugLM isn't just one robot; it's a four-person team working together:

• The Detective (QBugFind): This AI looks at the broken code and the "crime scene." Its job is to write a report saying, "I found the mistake! It's on line 5, and it's a 'structural error'."
• The Repairman (QBugFix): This AI takes the Detective's report and the broken code. It tries to rewrite the code to fix the problem without breaking anything else.
• The Inspector (QBugCheck): This is the final judge. It runs both the original perfect program and the AI's fixed version side-by-side on a simulator. If the results match perfectly, the fix is accepted. If they differ even slightly, the fix is rejected.

3. The Experiment: Testing Two AI Stars

The researchers tested this system using two powerful AI models:

• Claude 4.6 Sonnet: A very smart, expensive, proprietary model (like a high-end consultant).
• Qwen3 Coder Next: A powerful, open-source model (like a brilliant, cost-effective engineer).

They tested them with different "instruction styles" (prompts) to see which way of talking to the AI worked best.

Key Findings (The "Aha!" Moments)

1. The "Try Again" Magic
The most surprising discovery was about patience.

• The Analogy: Imagine asking a student to solve a math problem. If you only let them try once, they might get it wrong 75% of the time. But if you say, "You got it wrong, here is the feedback, try again," their success rate jumps to over 80%.
• The Result: A single retry (one second chance) boosted the AI's success rate from below 25% to above 80%. The first attempt is often a guess; the second attempt, armed with feedback, is where the real magic happens.

2. Less Talk, More Action
Researchers expected that giving the AI a long, step-by-step thinking guide (like "Chain-of-Thought") would help.

• The Analogy: It's like telling a chef, "First, think about the heat, then the knife, then the pan..." before they cook. Sometimes, this over-thinking slows them down or confuses them.
• The Result: For these capable AI models, a simple, direct instruction ("Here is the broken code, fix it") actually worked better than complex reasoning guides. The simpler approach was faster and more accurate.

3. The Cost-Effective Winner

• The Analogy: It's like comparing a luxury car to a reliable economy car. The luxury car (Claude) is great, but the economy car (Qwen) can do the same job for a fraction of the price and much faster.
• The Result: The open-source model (Qwen) fixed most types of bugs just as well as the expensive model but cost 4 to 9 times less and was 1.5 to 4.6 times faster.
  • The Catch: For one specific type of tricky "semantic" bug (where the logic is subtly wrong), the expensive model was slightly better, but for almost everything else, the cheaper model won.

Why This Matters

Currently, fixing quantum software is like trying to fix a watch while blindfolded. This paper shows that we can build an automated system that:

1. Creates its own test cases.
2. Uses a team of AI agents to find and fix errors.
3. Verifies the fix automatically.

It proves that with the right setup (especially giving the AI a chance to retry), we can automate the debugging of quantum software, making it much easier to build reliable quantum computers in the future.

Technical Summary

Technical Summary: QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging

Problem Statement

Quantum software engineering faces unique challenges distinct from classical development. Due to the probabilistic nature of quantum computation and the lack of mature debugging toolchains, bugs in quantum programs often manifest as silent, incorrect outputs rather than explicit exceptions or crashes. This renders conventional debugging techniques ineffective. While Large Language Models (LLMs) have demonstrated proficiency in classical software engineering tasks (e.g., code generation, fault localization), their capacity to detect and repair bugs in existing quantum programs remains largely unexplored. Furthermore, existing benchmarks often focus on specific software development kits (SDKs) like Qiskit, tightly coupling evaluation to framework-specific code rather than the underlying logical quantum circuits, leaving the debugging of low-level, SDK-agnostic languages like OpenQASM under-investigated.

Methodology: The QBugLM Framework

The authors propose QBugLM, a multi-agent benchmarking framework designed to automate the quantum software debugging pipeline for OpenQASM 3.0 programs. The framework operates on an end-to-end basis, independent of specific quantum SDKs, and consists of four primary components:

1. QBugGen (Mutation Toolkit):

   • Takes a corpus of syntactically and semantically valid OpenQASM 3.0 programs (sourced from MQT Bench).
   • Systematically injects single, well-defined bugs based on a four-category taxonomy (Table I):
     • C1: Deprecated Syntax Errors (e.g., using OpenQASM 2.0 syntax in 3.0).
     • C2: Structural Errors (e.g., assigning identical indices to control and target qubits).
     • C3: Gate Overuse/Redundancy (e.g., duplicating self-inverse gates).
     • C4: Semantic Deviation (e.g., substituting gates, altering phase values, or incorrect measurement placement).
   • Outputs a controlled evaluation dataset with ground-truth annotations.

2. QBugFind (Detection Agent):

   • Invokes an LLM agent to analyze the buggy source code, program specifications, and a configurable prompt.
   • Generates a structured bug report identifying the fault location and classifying the bug according to the taxonomy.

3. QBugFix (Repair Agent):

   • Receives the buggy program and the bug report from the detection agent.
   • Delegates the repair to a second LLM agent to produce a corrected version.
   • The agent is unconstrained in repair operations, allowing substitution, insertion, removal of gates, reordering, parameter modification, and qubit index adjustments.
   • Separating detection and repair allows for independent evaluation of each capability.

4. QBugCheck (Validation):

   • Acts as a deterministic validator comparing the LLM-fixed program against the original ground-truth circuit.
   • Functional Equivalence: Measures the Total Variation Distance ($\delta$) between the probability distributions of the reference and fixed programs executed on a noiseless simulator. A fix is accepted if $\delta \leq \epsilon_\delta$.
   • Structural Check: Compares gate counts at the same transpilation optimization level.

The workflow is iterative, allowing for multiple attempts (up to $K$) where the history of previous attempts is fed back to the agents to refine the repair.

Key Contributions

1. Framework Proposal: Introduction of QBugLM, a multi-agent framework automating the debugging pipeline (injection, detection, repair, validation) for framework-agnostic OpenQASM 3.0 programs.
2. Mutation Toolkit: Development of QBugGen, which systematically injects bugs based on a defined taxonomy to create a reproducible benchmark dataset with ground-truth annotations.
3. Comprehensive Case Study: A benchmarking study of two LLMs—Claude 4.6 Sonnet (proprietary) and Qwen3 Coder Next (open-source)—across different prompting strategies, bug categories, and quantum circuits.

Experimental Results

The study evaluated the models using Pass@k metrics, token consumption, wall-clock time, and monetary cost.

• Prompting Strategies (RQ1): Contrary to the expectation that explicit reasoning scaffolds (Chain-of-Thought, ReAct) improve performance, Structured Prompting consistently outperformed both CoT and ReAct for both models. For instance, on the Bernstein-Vazirani circuit, structured prompting achieved 97% Pass@1 for Claude and 95% for Qwen3, whereas CoT dropped Claude to 90% and Qwen3 to 45%. The authors suggest that for reasoning-capable models under fixed-resource constraints, simpler structured prompts are more effective.
• Iterative Feedback (RQ2): Iterative refinement was identified as the dominant factor in repair success. A single retry increased Pass@1 from below 25% to above 80%. With two retries, both models achieved near-perfect or perfect Pass@1 (100%) on most categories. However, specific weaknesses persisted: Claude 4.6 struggled with structural errors (80% Pass@1 even after retries), while Qwen3 struggled with semantic deviations (92% Pass@1).
• Cost Efficiency (RQ3): Qwen3 Coder Next demonstrated significantly higher cost-efficiency than Claude 4.6 Sonnet across most bug categories (structural errors, deprecated syntax, gate overuse). Qwen3 achieved equal or better Pass@1 at 4 to 9 times lower cost and 1.5 to 4.6 times faster wall-clock time. The exception was semantic deviation, where Claude 4.6 achieved 100% accuracy compared to Qwen3's 92%, justifying its higher cost for this specific, complex bug type.

Significance and Claims

The paper claims to take initial steps toward benchmarking LLM capabilities specifically for debugging quantum programs. Its significance lies in:

• Bridging the Gap: Addressing the lack of systematic investigation into LLMs' ability to detect and repair bugs in existing quantum code, particularly LLM-generated code.
• Agentic Workflow: Demonstrating that a multi-agent approach with iterative feedback is critical for overcoming the limitations of single-shot debugging in quantum contexts.
• Practical Insights: Providing evidence that simpler prompting strategies may be superior to complex reasoning scaffolds for capable models in resource-constrained environments, and that open-source models can offer comparable accuracy to proprietary models at a fraction of the cost for specific bug types.
• Foundation for Future Work: Offering a framework that supports future efforts in automated quantum software repair, moving beyond framework-specific evaluations to logical circuit correctness.

The authors remain modest, noting that the study focuses on single-fault injection and specific circuits, and that future work is required to address multi-fault scenarios, larger circuits, and hybrid agent configurations.