Leveraging LLM-Based Agentic Systems to Generate Quantum Applications for Test Optimization

Technical Summary: Leveraging LLM-Based Agentic Systems to Generate Quantum Applications for Test Optimization

Problem Statement
While quantum computing (QC) shows promise for optimizing software engineering (SE) tasks—specifically Search-Based Software Engineering (SBSE) problems like Test Case Optimization (TCO)—a significant barrier remains: translating natural language (NL) task-level requirements into executable quantum applications. Current approaches require substantial expertise in both quantum mechanics and programming to handle problem formulation, encoding, solver selection, and orchestration. Existing quantum workflow engineering tools (e.g., QuantME, Q-READY) offer abstractions but do not automate the end-to-end generation of executable applications from NL requirements. Furthermore, while Large Language Models (LLMs) have demonstrated success in isolated quantum code generation or QUBO transformation, they have not yet been effectively applied to generating full, traceable, executable quantum application workflows for SE optimization tasks.

Methodology: QPipe Architecture
The authors propose QPipe, an LLM-based multi-agent architecture designed to autonomously convert NL requirements into executable quantum applications. The system operates through a structured pipeline involving eight specialized agents, supported by curated task knowledge and coding skills, and interacting with a local tooling environment.

1. Workflow Agents:

   • Parse Agent: Extracts the SE task, objectives, constraints, and project data from the NL requirement.
   • Analysis Agent: Determines quantum suitability, identifying the optimization kernel (e.g., QUBO or Ising model) and defining the classical/quantum split.
   • Blueprint Agent: Creates a hybrid workflow plan, specifying the structure, solver choice, and expected artifacts.
   • Encoding Agent: Maps the full problem instance to executable subproblems, handling qubit budget constraints by decomposing variables or instances if necessary.
   • Codegen Agent: Generates candidate Python programs (using Qiskit/Qiskit-Aer) based on the blueprint and encoding. It operates within a loop, utilizing tools for static checks, compilation, and sandbox execution to repair code iteratively.
   • Review Agent: Performs a semantic review of the generated code against the original requirement and intermediate artifacts before execution, requesting repairs for inconsistencies or objective mismatches.
   • Combination Agent/Script: Aggregates results from decomposed subproblems. It uses deterministic scripts for simple aggregations (e.g., sum) and an LLM agent for complex semantic interpretations.
   • Verification Agent: Validates the final executed application and results against the generation record and, where feasible, a classical reference solver.

2. Supporting Capabilities:

   • Task Knowledge: Provides guidance on formulation (QUBO/Ising) and grounds the problem in specific SE entities (e.g., test case attributes).
   • Coding Skill: Enforces implementation conventions, such as using specific Qiskit APIs and avoiding stubs or deprecated calls.
   • Tooling: Agents access local tools for environment inspection, code storage, static/compile checking, sandbox execution, and classical reference solving (e.g., exact eigendecomposition or brute-force enumeration for small instances).

Key Contributions

• End-to-End Automation: QPipe is the first system to autonomously generate traceable, executable quantum applications from NL SE optimization requirements, covering the full lifecycle from parsing to verification.
• Multi-Agent Decomposition: The architecture separates concerns into specialized roles (parsing, planning, coding, reviewing), enabling targeted feedback loops and preserving traceability across intermediate artifacts.
• Hybrid Execution Strategy: The system dynamically handles resource constraints (e.g., qubit limits) by decomposing large problems into subproblems and orchestrating their execution and aggregation.
• Empirical Evaluation Framework: The authors establish a rigorous evaluation protocol using 20 NL requirements across 10 real-world TCO benchmarks (Test Case Selection and Minimization), measuring generation cost, effectiveness (pass@k), and solution quality against a Genetic Algorithm (GA) baseline.

Experimental Results
The study evaluated QPipe on 20 NL requirements associated with 10 benchmarks (including elevator_o2, gsdtsr, and GNU subject programs).

• Generation Effectiveness: QPipe achieved a 100% compilation pass rate and a 96.7% success rate for both application execution and final result combination across all requirements. The average generation cost was 260.1 seconds and 1.89 million tokens per requirement.
• Solution Quality: Among successfully generated applications, the solutions outperformed a classical Genetic Algorithm (GA) baseline in most cases. The Quantum Reduction Ratio (QRR) showed positive improvements, with median QRRs of 57.6 for Test Case Selection (TCS) and 89.1 for Test Case Minimization (TCM). The worst-case scenarios still matched or slightly exceeded the GA baseline.
• Workflow Patterns: Analysis of successful runs revealed that agents autonomously converged on QUBO formulations and QAOA algorithms, despite being prompted with other algorithmic options. The system consistently decomposed tasks into 4–6 subproblems and kept circuit sizes around 12 qubits to balance feasibility and overhead.
• Ablation Studies: Removing specific components significantly degraded performance. The removal of code-generation skills caused the most drastic failure (near-zero compilation success). Removing task knowledge, review feedback, or multi-agent decomposition also reduced effectiveness and solution quality, confirming the necessity of the full architecture.
• Backbone LLM Comparison: Using Claude as the backbone yielded the highest performance (100% compilation, 93.3% execution). DeepSeek performed well in early stages and compilation but struggled in later review and verification stages. Llama failed at the code generation stage entirely, highlighting the sensitivity of the workflow to the backbone model's coding capabilities.

Significance and Claims
The paper claims that agentic coordination is a viable approach for generating executable quantum applications for real-world test optimization problems. The authors assert that QPipe demonstrates that LLM-based multi-agent systems can autonomously navigate the complexities of quantum workflow engineering—handling problem formulation, encoding, and orchestration—without requiring manual intervention for every step.

The significance lies in reducing the barrier to entry for applying quantum optimization to SE tasks. By automating the translation from NL requirements to executable code, QPipe potentially lowers the expertise threshold required for SE practitioners to leverage quantum computing. The authors modestly conclude that while the current workflow is costly in terms of tokens and time, it successfully produces high-quality solutions that often surpass classical baselines, suggesting a path toward more accessible quantum-assisted software engineering. They note that broader empirical studies are needed to generalize these findings to other SE tasks and quantum backends.