Technical Summary: Are Large Language Models Ready for Quantum Software Engineering? A Multivocal Literature Review

Problem Statement

Quantum computing is transitioning from theoretical exploration to experimental deployment, necessitating the emergence of Quantum Software Engineering (QSE) to adapt classical software engineering principles to the quantum domain. Despite this, quantum programming remains difficult due to unstable APIs, complex compilation pipelines, and a scarcity of expertise. While Large Language Models (LLMs) have demonstrated significant utility in classical software development (e.g., code generation, testing, maintenance), their application in QSE is fragmented. Existing evidence is scattered across academic and practitioner sources, lacking a consolidated view of which QSE tasks are supported, which models are effective, and what specific limitations hinder reliable adoption. The core problem addressed is the lack of a systematic synthesis of how LLMs are currently utilized in QSE and whether they are robust enough for engineering workflows.

Methodology

The authors conducted a Multivocal Literature Review (MLR), a systematic review approach that integrates peer-reviewed "white literature" with "grey literature" (e.g., technical reports, blog posts, tool documentation) to capture the rapidly evolving state of practice in tool ecosystems.

• Protocol: The study followed guidelines by Kitchenham et al. (2010) for formal literature and Garousi et al. (2019) for grey literature.
• Search Strategy:
  • Sources: IEEE Xplore, ACM Digital Library, Scopus (white literature); ArXiv and Google Search (grey literature).
  • Search String: Combined terms for "quantum," "software engineering/development," and "LLM/Generative AI" (e.g., ("quantum*" ∨"quantistic") ∧ ("software engineering" ∨"software development") ∧ ("llm" ∨"large language models" ∨... )).
  • Selection: The process included an initial search yielding 172 records, which were deduplicated to 154 unique studies. Two authors independently screened titles, abstracts, and full texts using inclusion/exclusion criteria (e.g., excluding non-English papers, short papers, or studies not focusing on LLMs in QSE).
  • Snowballing: Backward and forward snowballing were applied to the initial set, identifying additional relevant studies.
• Final Dataset: The review analyzed 24 primary sources (7 from white literature, 17 from grey literature, including ArXiv and Google results).
• Data Extraction & Synthesis: A structured extraction form was used to map studies to the QSE lifecycle (based on Weder et al., 2022), identify LLM families, and code reported limitations. Synthesis involved thematic analysis and descriptive statistics.

Key Contributions

The paper provides three primary contributions to the QSE and AI literature:

1. Lifecycle Mapping: A classification of QSE activities where LLMs are applied, identifying which phases of the software lifecycle receive attention and which remain underexplored.
2. Model Landscape Analysis: A summary of the specific LLMs and adaptation strategies used, highlighting the balance between closed-source and open-source alternatives and their implications for reproducibility.
3. Limitation Synthesis: An analysis of reported limitations (e.g., correctness, executability, evaluation gaps) to outline current barriers and future research directions.

Results

RQ1: Which QSE tasks are supported?

The evidence is heavily concentrated in the Implementation and Analysis phases of the QSE lifecycle.

• Implementation (8 studies): Dominated by quantum code and circuit generation. Studies evaluate LLMs (e.g., GPT-4) for automatically producing quantum programs, parameterized circuits, and QAOA-based optimizations. Some work involves prompt-driven refinement and interactive assistance.
• Analysis (8 studies): Features a diverse range of tasks including program repair, refactoring (e.g., migrating across Qiskit versions), code optimization, program comprehension (generating explanations), and linting for quantum-specific issues.
• Testing (2 studies): Minimal coverage, focusing on fuzz testing and detecting flaky tests due to probabilistic execution.
• Architecture/Design (1 study): Preliminary evidence of LLMs guiding ansatz architecture exploration.
• Gaps: Requirements engineering, deployment, and maintenance (beyond code evolution) are largely unaddressed.

RQ2: Which LLMs are used?

• Dominance of Proprietary Models: The GPT family (GPT-3.5, GPT-4, GPT-4o, GPT-4.1) is the most frequently adopted, appearing in the majority of studies.
• Open-Source Alternatives: Models like LLaMA (various versions), DeepSeek, StarCoder, Granite, and Qwen are increasingly represented but often used in a complementary or exploratory capacity.
• Reproducibility Concerns: The heavy reliance on proprietary models raises concerns regarding reproducibility and longitudinal comparability, as access to specific model versions may be restricted or change over time.

RQ3: What limitations are reported?

The review identified eight recurring themes of limitations:

1. Data Availability and Coverage: Training data often lacks specific quantum features, hindering generalization.
2. Performance Limitations: Models degrade significantly on complex or large-scale problems compared to simple tasks.
3. Correctness and Reliability: Frequent reports of hallucinations, semantically incorrect code, and failures in hardware simulators.
4. Comprehension Quality: Explanations are often shallow or inconsistent.
5. Prompt Dependency: Results are highly sensitive to prompt formulation.
6. Output Alignment: Generated suggestions may not match developer needs or be too specific.
7. Model Constraints: Smaller models underperform; some results are only achievable with specific variants.
8. Evaluation Scope: Many studies rely on simplified benchmarks or well-known algorithms, limiting the generalizability of results to realistic QSE contexts.

Significance and Claims

The paper concludes that LLMs are not yet ready to function as robust, lifecycle-spanning engineering agents in quantum software development. Instead, current evidence suggests they function primarily as assistive tools for bounded tasks, particularly in code generation and maintenance.

• Current State: LLM adoption in QSE is "generation-centric." While they show promise in reducing developer effort for specific artifacts, their application remains fragile, highly context-dependent, and prone to correctness issues under realistic constraints.
• Implications for Practitioners: The authors advise a "verification-first" approach. LLM outputs should be treated as provisional artifacts requiring explicit validation gates (compilation, simulation, unit testing) before integration. Teams should favor incremental edits over full solutions and treat prompts as versioned configurations.
• Implications for Researchers: The field needs to move beyond isolated generation accuracy metrics. Future work should prioritize:
  • Developing specialized or fine-tuned models for QSE.
  • Establishing rigorous evaluation protocols and benchmarks that reflect realistic constraints.
  • Adopting open-weight baselines to improve reproducibility.
  • Investigating workflow-level integration patterns rather than just code synthesis.

The review emphasizes that while LLMs offer immediate value in constrained workflows, substantial research is required to overcome barriers related to correctness, reliability, and data coverage before they can be trusted for autonomous engineering in the quantum domain.