QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits

The paper introduces QASM-Eval, the first comprehensive dataset and benchmark designed to train and evaluate large language models on OpenQASM-3's advanced hardware-facing features, such as classical feedback and pulse control, demonstrating that targeted fine-tuning significantly improves model performance in these critical NISQ-era programming tasks.

The Gist

Imagine you are trying to teach a brilliant but inexperienced apprentice how to build a very delicate, high-tech machine. This machine is a quantum computer.

For a long time, the instructions we gave this apprentice were like a simple recipe: "Mix these ingredients, bake for 10 minutes." This worked for basic tasks, but the machine is now entering a noisy, difficult phase (called the NISQ era). To make it work reliably, the instructions need to get much more specific. The apprentice now needs to know exactly when to check the temperature, how to adjust the oven door mid-bake, and even how to tweak the shape of the heat waves themselves.

The language used for these ultra-precise instructions is called OpenQASM 3. It's the "hardware manual" for quantum computers.

The Problem: The Apprentice is Confused

Even though Artificial Intelligence (AI) has gotten really good at writing code, there was a major problem: No one had built a practice test specifically for this new, complex language.

Existing tests were like asking the apprentice to "bake a cake" (high-level logic) or "fix a broken toaster" (basic circuits). But they didn't test if the apprentice could:

1. Pause and think: Stop the baking process, check a sensor, and decide whether to add more sugar based on that reading (Classical Logic).
2. Time it perfectly: Wait exactly 0.0000001 seconds before opening the door, or synchronize two ovens perfectly (Timing Scheduling).
3. Tweak the waves: Manually adjust the shape of the heat waves hitting the food to prevent burning (Pulse Control).

Without a practice test for these specific skills, the AI models were guessing, and they were failing badly.

The Solution: QASM-Eval (The Ultimate Practice Exam)

The authors of this paper created QASM-Eval. Think of this as a massive, specialized training gym and a final exam for AI, designed specifically for OpenQASM 3.

• The Training Set: They generated 4,000 practice problems. These aren't just random questions; they are carefully crafted scenarios where an AI has to fill in the missing code to make the quantum machine work correctly.
• The Exam: They created a strict 100-question test.
• The Grading System: They built a special "robot teacher" (an automated verifier). This robot doesn't just check if the code looks right; it actually simulates the quantum machine to see if the code produces the correct result, follows the timing rules, and doesn't crash the system.

What They Found

The researchers put several top-tier AI models (like Llama and GPT) through this new exam. Here is what happened:

1. The "Zero-Shot" Struggle: When they asked the AI to take the exam without any help (just "here is the question, solve it"), the results were terrible. The AIs were like students who had studied general physics but had never seen the specific blueprint for this machine. They couldn't get the syntax right, let alone the timing.
2. The "Few-Shot" Boost: When the researchers gave the AI a few examples of how to solve similar problems first (like showing a sample answer key), the scores went up. It was like giving the student a cheat sheet with one example.
3. The "Fine-Tuning" Breakthrough: This was the big win. The researchers took the AI models and "trained" them specifically on their 4,000 practice problems.
   • The Result: A medium-sized AI model (Llama-8B), after this specific training, performed almost as well as the most powerful, expensive AI (GPT-5.2) that had no training at all.
   • The Champion: A larger AI model (Llama-70B), after training, became a master. It scored 85% on the exam, beating even the most powerful AI when that AI was given a few examples.

The Takeaway

The paper concludes that the bottleneck isn't that AI is "dumb" at quantum physics. The bottleneck is that AI doesn't know the specific grammar and rules of OpenQASM 3.

By creating a dedicated dataset (QASM-Eval) and training the AI on it, they proved that you can turn a general-purpose AI into a highly reliable quantum programmer. It's like taking a smart person who knows how to drive a car and giving them a specific manual and practice track for a Formula 1 car; suddenly, they can drive the race car perfectly.

This dataset is now open for everyone to use, helping to build better AI assistants that can help humans program the next generation of quantum computers.

Technical Summary

Technical Summary: QASM-Eval

Problem Statement

Quantum computing currently operates in the Noisy Intermediate-Scale Quantum (NISQ) era, where performance is heavily constrained by noise. Mitigating these errors requires hardware-facing capabilities that extend beyond standard gate-sequence circuit specifications. These capabilities include mid-circuit measurement and classical feedback for Quantum Error Correction (QEC), precise timing control for Dynamical Decoupling (DD), and pulse-level waveform access for calibration.

OpenQASM 3 was introduced to expose these hardware-level features, providing a programming interface that links algorithms to physics. However, despite the rapid advancement of Large Language Models (LLMs) in code generation, there is a critical lack of datasets designed to train and evaluate LLMs on OpenQASM 3 programs that utilize these advanced, hardware-oriented features. Existing resources either focus on high-level algorithm design (e.g., Qiskit-based datasets) or are limited to static gate-sequence circuits, failing to cover the dynamic, constraint-heavy nature of OpenQASM 3. Consequently, current LLMs struggle to generate valid code that adheres to OpenQASM 3's strict syntax and semantic constraints regarding classical logic, timing, and pulse control.

Methodology

Dataset Construction (QASM-Eval)

The authors introduce QASM-Eval, the first comprehensive dataset designed specifically for training and evaluating LLMs on OpenQASM 3. The dataset is constructed using a hybrid pipeline involving human quantum-programming experts and AI, ensuring both diversity and correctness while preventing data leakage.

1. Template Preparation: Experts design "background templates" (randomizing qubit configurations, frequencies, and initial states) and "core-task templates" (targeting specific themes like if/else flows or pulse operations).
2. Instance Generation: A coder-LLM (Qwen3-Coder-480B), augmented with OpenQASM 3 documentation, generates stylistic and structural variants of core tasks. These are combined with randomized backgrounds to create complete OpenQASM 3 programs.
3. Prompt Generation: The core task code is replaced with a natural language "TODO" comment describing the required functionality, creating a specification-solution pair.
4. Dataset Composition:
   • Test Set: 100 expert-verified tasks.
   • Training Set: 4,000 tasks.
   • Categories: The dataset covers four categories:
     • Classical Logic: Control flow, mid-circuit measurement, arithmetic, and type casting.
     • Timing Scheduling: Delay insertion, duration types, stretchable intervals, and alignment.
     • Pulse Control: Waveform manipulation, frame synchronization, and calibration.
     • Complex Tasks: Real-world workflows integrating the above, such as QEC, DD, and crosstalk calibration.

Evaluation Framework

To validate generated programs, the authors developed an extended automatic verifier that checks:

• Syntax: Parsing via the official OpenQASM 3 parser to ensure well-formed Abstract Syntax Trees (AST).
• Quantum State: Using statevector simulation (Qiskit) and pulse-level simulation (Qutip) to verify the final quantum state distribution against reference behaviors.
• Schedule: Generating a virtual timeline to verify that timing constraints (e.g., alignment, delays) are met.

The pass criterion requires successful syntax parsing and, depending on the task type, passing either the state check, the schedule check, or both.

Key Contributions

1. First OpenQASM 3 Hardware-Focused Dataset: QASM-Eval is the first resource to systematically cover the advanced features of OpenQASM 3 (classical logic, timing, and pulse control) specifically for LLM training and evaluation.
2. Automated Verification Pipeline: The authors extended existing toolchains to support automatic verification of OpenQASM 3's unique features, including dynamic timing and pulse-level semantics, achieving a 92% agreement rate with human experts.
3. Benchmarking and Fine-Tuning Results: The paper provides a rigorous benchmark showing that state-of-the-art models struggle with OpenQASM 3 tasks but can achieve significant improvements through targeted fine-tuning.

Experimental Results

The authors evaluated several models, including ChatGPT-5.2-Thinking, DeepSeek-V3, and Llama-3 variants (8B and 70B), using the QASM-Eval test set.

• Baseline Performance: State-of-the-art models exhibit limited performance. ChatGPT-5.2-Thinking achieved an overall pass@1 of 0.54, while open-source baselines (Llama-70B, Llama-8B) scored significantly lower (0.27 and 0.24, respectively). Performance was particularly poor on "Complex" tasks (0.00–0.08).
• Few-Shot Prompting: Providing three in-context examples improved performance, with ChatGPT-5.2-Thinking reaching 0.78 overall. However, this approach relies on the model's ability to infer patterns from examples rather than internalized knowledge.
• Fine-Tuning Impact: Fine-tuning on the QASM-Eval training set yielded the most substantial gains:
  • Llama-3-70B: Achieved an overall pass@1 of 0.85, outperforming the few-shot-augmented ChatGPT-5.2-Thinking (0.78).
  • Llama-3-8B: Improved from 0.24 to 0.52, approaching the zero-shot performance of ChatGPT-5.2-Thinking.
  • Fine-tuning on previous datasets (QCircuitBench, Agent-Q) provided only marginal improvements, highlighting the necessity of data specific to OpenQASM 3's hardware features.
• Error Analysis: Fine-tuning primarily reduced syntax errors (e.g., incorrect measurement assignment or invalid switch syntax), which were the dominant failure mode for base models. While syntax validity improved significantly, semantic errors (timing and distribution mismatches) remained a challenge, particularly for smaller models.

Significance and Claims

The paper positions QASM-Eval as a crucial benchmark and training foundation for the NISQ era. The authors claim that:

• Current LLMs lack the specific syntactic and semantic knowledge required for hardware-facing quantum programming.
• Targeted fine-tuning on QASM-Eval unlocks substantial capabilities, enabling mid-scale models (like Llama-3-70B) to surpass the performance of few-shot-assisted larger models.
• The dataset and verification framework provide a necessary step toward developing reliable LLM assistants that can bridge the gap between algorithmic intent and physical quantum execution, moving beyond abstract circuit design to the practical realities of noise mitigation and calibration.

The authors remain modest regarding the scope, noting that the current tasks focus on constraint-driven code translation rather than open-ended algorithmic reasoning, and that the synthetic nature of the data may not fully capture the unstructured complexity of real-world quantum codebases. However, they assert that QASM-Eval establishes a vital baseline for future research in quantum software engineering.