Technical Summary: QiboAgent – A Practitioner's Guideline for Open Source Assistants in Quantum Computing

1. Problem Statement

The maintenance and evolution of scientific software ecosystems, particularly in quantum computing, face significant challenges as codebases scale. General-purpose Large Language Models (LLMs), while proficient in code generation, suffer from inherent limitations when applied to specialized domains like the Qibo middleware. Relying solely on parametric memory, standard models frequently produce hallucinations (fabricating non-existent modules or functions) and lack the reasoning depth required for complex, state-dependent engineering tasks. Furthermore, proprietary models often require massive computational resources and lack the transparency or customization needed for secure, local deployment in research environments.

2. Methodology

The authors introduce QiboAgent, a reference implementation designed to guide the development of specialized coding assistants. The system integrates two complementary paradigms to overcome the limitations of static LLMs:

A. Retrieval-Augmented Generation (RAG)

To address factual accuracy and reduce hallucinations, QiboAgent employs a RAG pipeline grounded in the Qibo codebase (source code, documentation, and notebooks). The authors evaluated two retrieval strategies:

1. Semantic Retrieval: Utilizes fixed-size chunking with a Maximum Marginal Relevance (MMR) reranking mechanism to minimize informational redundancy and ensure diverse context retrieval.
2. Hybrid Retrieval: Combines dense vector embeddings (for semantic understanding) with Okapi BM25 keyword matching (for precise technical token retrieval). This approach uses a structure-aware ingestion pipeline that respects code syntax (e.g., aggregating class definitions with their methods) and fuses scores via a normalized linear combination. This is critical for handling domain-specific nomenclature (e.g., specific Hamiltonian identifiers) that pure semantic search might miss.

B. Agentic Workflows

For complex engineering tasks requiring state management and multi-step reasoning, the system transitions from a passive generator to an autonomous agent based on the ReAct framework.

• Single-Agent Mode: Used for repository maintenance (e.g., resolving GitHub issues). The agent scrapes issue trackers, navigates the file system, and iteratively proposes code patches.
• Sequential Multi-Agent Mode: Used for large-scale refactoring (e.g., porting a Python package to Rust). The workflow is decomposed into specialized agents (Rust Architect, Bindings Generator, Compatibility Layer, Validation Agent) that operate in a Compiler-Driven Feedback Loop. This loop allows agents to execute shell commands (e.g., cargo build, pytest), parse error logs, and autonomously apply corrective patches until compilation and tests pass.

C. Model Selection

The framework prioritizes open-source, lightweight models (ranging from 20B to 120B parameters) to enable local deployment.

• Embedding: sentence-transformers/all-MiniLM-L6-v2.
• Generative: A tiered strategy is used: lightweight models (e.g., gpt-oss:20b, qwen3-coder:30b) for Q&A and documentation, and larger models (e.g., gpt-oss:120b) for complex agentic reasoning requiring robust tool-calling capabilities.
• Infrastructure: The system runs locally using Ollama for inference and LangChain for orchestration.

3. Benchmark Tasks and Evaluation

The authors established a comprehensive evaluation protocol across four distinct tasks:

1. Quantum Computing Q&A: A dataset of 50 questions covering configuration, primitives, advanced simulation, and debugging. Performance was measured by Functional Correctness (executing generated code against ground truth), Hallucination Score (tracking runtime errors like AttributeError), and Stylistic Quality (Pylint scores).
2. Automated Documentation: Generating Sphinx-compatible docstrings for new code, ensuring correct cross-referencing and usage examples.
3. New Feature Pull Request: Autonomous resolution of GitHub issues (e.g., Issue #1710 involving tensor unfolding logic) without human intervention.
4. Refactoring and Rebasing: A "greenfield" development task to create a Qibo-core module in Rust, including Python bindings via pyO3, build configurations, and unit tests.

4. Key Results

• Accuracy and Hallucination Reduction: The Hybrid Retrieval strategy consistently outperformed both the No-RAG baseline and pure Semantic Search. The system achieved a peak accuracy of 90.2% using the qwen3-coder:30b model. Crucially, the RAG approach significantly reduced hallucination rates compared to proprietary baselines (e.g., Google's Gemini-2.5-flash), which showed minimal improvement with semantic search alone, suggesting reliance on pre-training memory rather than retrieved context.
• Agentic Capabilities: The agent successfully resolved complex GitHub issues (Issue #1710) and generated a functional Proof of Concept (PoC) for the Qibo-core Rust module. While the agent's code sometimes lacked specific architectural optimizations (e.g., caching permutation indices) found in human-written patches, it produced mathematically sound, functionally complete solutions that required only minor refinement.
• Compiler-Driven Feedback: The multi-agent system successfully utilized a feedback loop to resolve compilation errors and type mismatches during the Rust porting process, demonstrating the ability to iterate and self-correct without human intervention.
• Efficiency: Lightweight open-source models (up to 30B parameters) achieved high accuracy, validating the viability of secure, cost-effective local deployment without massive proprietary architectures.

5. Significance and Claims

The paper positions QiboAgent not as a replacement for human developers, but as a practitioner's guideline and a reference architecture for building domain-aware AI assistants.

• Domain Awareness over Model Scale: The work demonstrates that for specialized scientific software, a structured knowledge base and agentic tools are more critical than raw model size or pre-training exposure. A hybrid retrieval strategy effectively bridges the gap between general LLMs and specialized requirements.
• Viability of Local Deployment: By achieving high performance with open-source models, the authors claim that research institutions can maintain data privacy and independence from proprietary platforms while automating complex maintenance tasks.
• Dynamic Adaptability: Unlike domain-specific fine-tuning, which risks overfitting and requires costly retraining for codebase changes, the RAG-based approach provides a dynamic, up-to-date context that allows the model to adapt to evolving codebases without retraining.
• Future Scope: The authors suggest that these architectures could extend beyond software maintenance to automate physical tasks, such as quantum hardware calibration via Qibocal and Qibolab, positioning open-source AI as an integral component of the full quantum computing stack.

The framework, including source code, benchmark datasets, and deployment instructions, is publicly available at https://github.com/qiboteam/qiboagent.