💬 NLP

PennySynth: RAG-Driven Data Synthesis for Automated Quantum Code Generation

PennySynth is a retrieval-augmented generation framework that leverages a curated knowledge base of PennyLane instruction-code pairs and code-aware embeddings to significantly improve the accuracy and structural validity of automated quantum code generation, outperforming state-of-the-art LLMs on QHack competition challenges.

Original authors: Minghao Shao, Nouhaila Innan, Hariharan Janardhanan, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minghao Shao, Nouhaila Innan, Hariharan Janardhanan, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but slightly scatterbrained robot how to write code for a very specific, high-tech quantum computer. The robot is smart enough to write general computer code, but when it tries to write for this specific quantum machine (called PennyLane), it starts making things up. It invents fake buttons, forgets the right settings, and builds circuits that look good on paper but fall apart the moment you try to run them. This is what the paper calls "hallucinating."

The authors of this paper, PennySynth, decided to fix this by giving the robot a personalized, verified library of cheat sheets right before it starts writing.

Here is how they did it, broken down into simple steps:

1. Building the "Cheat Sheet" Library (The Offline Stage)

First, the team needed a massive collection of correct examples. They didn't just copy-paste code from the internet because some of it was old, broken, or duplicated.

The Hunt: They scoured official documentation, community GitHub projects, and past competition archives (like the "QHack" contest).
The Filter: They used a smart robot (an AST parser) to cut out only the actual quantum functions, ignoring the boring helper code.
The Quality Check: They ran the code through a four-step safety inspection to make sure it was still valid, didn't use outdated commands, and actually worked.
The Cleanup: They used a "fingerprinting" technique to remove duplicates. If two pieces of code were 70% identical, they kept only one.
The Result: They ended up with a clean, verified library of 13,389 perfect "Instruction-to-Code" pairs. Think of this as a dictionary where every word (instruction) is perfectly matched to a working sentence (code).

2. The "Smart Search" System (The Online Stage)

When a user asks the robot to solve a new quantum problem, the system doesn't just guess. It follows a strict process:

Translation: First, it rewrites the user's question into a format the library understands better.
The Search: It uses a special "code-aware" search engine. Imagine a librarian who doesn't just look for keywords, but actually understands the structure of quantum code. This search engine is much better at finding the right cheat sheet than a standard search engine (improving search accuracy from 45% to 72%).
The Safety Net: If the search engine finds a cheat sheet that is a good match (above a certain confidence score), it hands it to the robot. If the match is weak, it tells the robot, "Don't use that; it might confuse you," and the robot relies on its own brain.
The Fix-It Loop: If the robot writes code that crashes when run, the system doesn't just give up. It shows the robot the error message and says, "Here is what went wrong, try again." It does this up to two times to fix mistakes automatically.

3. The Results: Does it Work?

The team tested this system on 74 difficult challenges from the last three years of the QHack competition.

Without the library: Even the smartest AI models (like Claude Sonnet) only got about 36% to 43% of the problems right. They kept making up fake quantum commands.
With PennySynth: The success rate jumped to 64% - 68%.
The Analogy: It's like taking a student who is guessing on a math test and giving them a reference book of verified formulas right before the exam. The student still has to do the work, but they stop making up formulas that don't exist.

4. What Makes It Special?

The paper highlights two main reasons this worked so well:

The Right Search Engine: Using a search tool trained specifically on code (not just general text) was the biggest factor. It was like using a specialized quantum dictionary instead of a general English dictionary.
The Quality of the Library: It wasn't just about having more data; it was about having verified data. The library included a mix of official examples, community projects, and competition problems, which covered different styles of coding.

5. What Didn't Work Perfectly?

The system isn't magic. It still struggles if:

The problem requires a completely new type of circuit the library has never seen.
The robot gets confused by a "cheat sheet" that is almost right but has a tiny detail that doesn't fit the current problem (like using an old version of a command).
The robot fails to follow the strict formatting rules of the test.

In summary: PennySynth is a system that stops AI from making up quantum code by giving it a curated, high-quality library of real examples and a smart way to find the right one at the right time. It turned a "guessing game" into a "reference-based task," significantly improving the success rate of automated quantum programming.

Technical Summary: PennySynth

Problem Statement

The rapid evolution of quantum programming frameworks, particularly PennyLane, has exposed critical limitations in general-purpose Large Language Models (LLMs) when tasked with generating quantum code. While LLMs excel at general coding, they struggle with specialized quantum tasks due to two primary bottlenecks:

Low-Resource Domain: PennyLane-specific code constitutes a negligible fraction of the training data for general LLMs, leading to hallucinations of gate names, incorrect device configurations, and invalid circuit structures.
Rapid API Evolution: PennyLane's interfaces, gate naming conventions, and device rules change frequently. Code samples valid in earlier versions often become obsolete, causing models to generate unsupported operations or structurally invalid circuits.

Existing solutions, such as domain-specific fine-tuning or basic Retrieval-Augmented Generation (RAG), often fail to provide the precise, up-to-date, and framework-specific context required for executable quantum code generation.

Methodology: The PennySynth Framework

The authors propose PennySynth, a RAG-driven data synthesis and retrieval framework designed to condition LLM inference on a curated, verified knowledge base. The system operates in two distinct stages:

1. Offline Stage: Automated Data Synthesis (PennySynth-13K)

The framework constructs a high-quality dataset of 13,389 instruction-code pairs through a rigorous three-stage pipeline:

Extraction: Raw Python source files are harvested from three heterogeneous sources: Official PennyLane repositories, Community GitHub repositories (searching for pennylane/qml), and QHack competition archives (2022–2024). Using AST-based function isolation, the system extracts only functions containing quantum-specific operations, filtering out unrelated helper code.
Verification: A four-layer quality check ensures syntactic validity, import correctness, preservation of quantum operations (limiting gate count changes to <20%), and semantic structure preservation. A conservative fallback policy retains original code if transformations fail.
Deduplication: MinHash LSH (Locality Sensitive Hashing) with a 70% similarity threshold removes near-duplicate entries, ensuring diversity in the final corpus.
Instruction Generation: For code samples lacking explicit task descriptions, a constrained LLM prompt generates natural-language instructions (20–40 words) focusing on operations, device/wire details, and return types. These instructions achieve 95% Top-1 retrieval accuracy against their corresponding code.

2. Online Stage: RAG Inference and Repair

Query Expansion: The system uses an LLM to rewrite challenge descriptions into focused PennyLane-oriented queries.
Code-Aware Retrieval: Unlike general-purpose embeddings, PennySynth employs st-codesearch-distilroberta-base, a model trained specifically for natural-language-to-code retrieval. This increases the average cosine similarity of retrieved pairs from 0.45 (MiniLM baseline) to 0.726.
Selective Context: The system retrieves the top-5 examples. If the maximum similarity score falls below a threshold ( $\tau=0.60$ ), it reverts to base generation to avoid context contamination.
Iterative Auto-Fix: Generated code is executed against test cases. If execution fails, the system enters a repair loop (up to 2 retries) providing the error log and original code to the LLM for correction.

Evaluation Metrics

The paper introduces a Quantum-Adapted CodeBLEU metric. It upweights qml.* tokens by 3x in the weighted n-gram component and replaces the standard dataflow matching with a Jaccard similarity over extracted quantum-specific elements (gate names, device types, measurement return types).

Key Contributions

PennySynth-13K Dataset: A verified corpus of 13,389 instruction-code pairs derived from a multi-stage extraction and verification pipeline, released to support future research.
Code-Aware RAG Pipeline: A retrieval system utilizing specialized embeddings and query expansion, demonstrating that code-aware retrieval significantly outperforms general-purpose baselines.
Quantum-Adapted CodeBLEU: A modified evaluation metric that explicitly prioritizes framework-specific tokens and quantum dataflow structures.
Systematic Benchmarking: A comprehensive evaluation across seven state-of-the-art LLMs (including Claude Sonnet 4.6, GPT-5.5, and Qwen3) on 74 QHack challenges spanning 2022, 2023, and 2024.

Results

Evaluated on QHack challenges, PennySynth (using Claude Sonnet 4.6 as the base generator) achieves significant improvements over non-retrieval baselines:

QHack 2022: 64% pass@5 (vs. 36% for base Claude; +28 pp).
QHack 2023: 68% pass@5 (vs. 43% for base Claude; +25 pp).
QHack 2024: 52% pass@5 (vs. 24% for base Claude; +28 pp).

Ablation Studies reveal that:

Code-aware embeddings are the primary driver of performance, contributing +4 to +8 percentage points over general-purpose embeddings.
Dataset expansion (adding QHack archives to the GitHub corpus) provides synergistic gains (+4 pp) only when paired with high-quality, code-aware retrieval.
Source Composition: While unofficial GitHub data provides volume, official repositories provide the highest quality per sample, significantly reducing API misuse errors.

Failure Analysis indicates that while retrieval reduces reasoning errors (from 18% to 8%), it can introduce "context-induced" errors, such as hallucinations or API misuse, when retrieved examples are partially aligned but contain outdated gate names or incompatible signatures.

Significance and Claims

The paper claims that model scale alone is insufficient for reliable quantum code generation. Instead, access to verified, up-to-date, and framework-specific code examples at inference time is critical. PennySynth demonstrates that a structured, code-aware RAG pipeline can bridge the gap between general LLM capabilities and the rigorous requirements of quantum programming.

The authors position this work as the first comprehensive baseline for competition-level PennyLane code generation, highlighting that structural code similarity (measured by their adapted CodeBLEU) and functional correctness capture distinct aspects of code quality. They conclude that while their current flat vector retrieval is effective, future work should explore Graph RAG to better capture the interconnections between gates, templates, and devices in quantum programming.