⚛️ quantum physics

MQSS-Selector: RL-Guided Pass Selection for an MLIR Compilation Pipeline

This paper introduces MQSS-Selector, a unified, learning-based framework that leverages reinforcement and deep learning to dynamically optimize device selection, compiler pass ordering, and job scheduling for HPC-Quantum computing workflows, aiming to simultaneously maximize fidelity while minimizing compilation time and latency in the NISQ era.

Original authors: Andre Youssefi (Leibniz Supercomputing Centre), Ercüment Kaya (Leibniz Supercomputing Centre, Technical University of Munich), Minh Chung (Leibniz Supercomputing Centre), Jorge Echavarria (Munich Quan

Published 2026-09-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andre Youssefi (Leibniz Supercomputing Centre), Ercüment Kaya (Leibniz Supercomputing Centre, Technical University of Munich), Minh Chung (Leibniz Supercomputing Centre), Jorge Echavarria (Munich Quantum Valley), Laura B. Schulz (Argonne National Laboratory), Martin Schulz (Leibniz Supercomputing Centre, Technical University of Munich)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the emerging landscape of modern computing, two powerful worlds are beginning to merge: the massive, classical supercomputers that model climate change and design new materials, and the fragile, experimental quantum machines that promise to solve problems currently beyond our reach. This convergence creates a new kind of hybrid system where a classical computer acts as the brain, sending specific tasks to a quantum processor to act as a specialized accelerator. However, these quantum machines are currently in a difficult phase of development. They are noisy, meaning their calculations are prone to errors, and they are limited in size, with only a handful of quantum bits available. Because of this, simply sending a program to a quantum computer is not enough; the program must be carefully prepared, reshaped, and matched to the specific quirks of the machine it will run on. If the preparation is wrong, the result will be garbage. The challenge lies in figuring out exactly which machine to use and how to transform the code for it, a task that involves navigating a vast and shifting maze of possibilities.

Researchers at the Leibniz Supercomputing Centre in Germany, working with partners from the Technical University of Munich and Argonne National Laboratory, have proposed a new way to solve this puzzle. They call their system MQSS-Selector, a tool designed to act as a unified guide for the entire process. Instead of treating the choice of hardware and the preparation of software as separate steps, their system learns to make both decisions at once. The team built this tool using a method called reinforcement learning, a type of artificial intelligence where a computer program learns by trial and error, much like a child learning to ride a bike. The program tries different combinations of quantum devices and code transformations, receives feedback on how well the result worked, and gradually learns to choose the best path forward.

The core difficulty the researchers tackled is that the problems of choosing a device and ordering the code steps are incredibly complex. In fact, they are so complex that finding the perfect solution for every single case is mathematically impossible to do quickly as the number of options grows. Traditional methods rely on fixed rules or guesswork, which often fail when the quantum machines change their behavior or when new types of code are introduced. The researchers demonstrated that their learning-based approach could navigate this complexity. They created a system that looks at the specific features of a quantum program, checks the current status of available machines, and then selects the most suitable device. Simultaneously, it decides which sequence of code optimizations to apply to make the program run efficiently on that specific machine.

To test their idea, the team trained their system on a large collection of quantum programs, including those designed to simulate chemical molecules. They compared their learning-based selector against standard methods and found that the guided model adopted a very conservative strategy regarding when to stop optimizing. While the model without guidance failed to call the "finish" action in almost all cases, the guided model successfully terminated the process in the vast majority of trials, though it did so with low recall, meaning it missed many potential opportunities to identify valid stopping points. In terms of program structure, the system did not produce shorter programs; on average, the depth and operation count of the resulting programs increased significantly compared to the inputs. However, the guided approach did yield better overall compilation performance by ensuring the programs were executable in 95.5% of cases, a marked improvement over the non-guided model. When the researchers introduced a special training technique to help the system explore the vast number of possibilities without getting lost, the results improved in terms of reliability, though the system still struggled to identify all valid optimization opportunities.

The study also revealed that while the system works well for choosing devices, the part responsible for ordering the code steps still needs more practice. The researchers found that without the special training guidance, the system struggled to find good solutions on its own, often wandering through useless combinations of code changes; in fact, the experimental data suggests that reinforcement learning without additional methods is not a viable approach for training a network for compilation under the resource constraints considered. However, with the guidance, the system learned to focus on the most promising paths, albeit very conservatively, leading to more reliable execution. This suggests that while the concept of a single, unified guide for quantum computing is viable, it requires careful training and significant data to be truly effective. The team's work shows that by combining the choice of hardware and the preparation of software into one intelligent decision-making process, we can move closer to making quantum computers useful tools for real-world problems. Their findings indicate that this approach is not just a theoretical idea but a practical path forward, capable of adapting to the noisy and changing nature of today's quantum machines, provided that the system is given sufficient guidance to overcome the inherent difficulties of the task.

Technical Summary: MQSS-Selector

Problem Statement and Motivation
The convergence of High Performance Computing (HPC) and Quantum Computing (QC) into unified HPCQC infrastructures necessitates software stacks capable of bridging classical and quantum workflows. However, current Noisy Intermediate-Scale Quantum (NISQ) devices are error-prone, resource-constrained, and highly sensitive to calibration and topology. Consequently, effective execution requires specialized compilation and optimization.

Existing software stacks often suffer from fragmentation, treating device selection, compiler pass optimization, and job scheduling as isolated stages. The paper identifies two critical, interdependent challenges:

Device Selection: Choosing a target quantum device from a set of available backends, considering topology, gate fidelity, queue delays, and maintenance windows.
Phase Ordering: Determining an optimal sequence of compiler passes to transform a quantum program, minimizing Just-In-Time (JIT) compilation time and execution runtime while ensuring device compliance.

The authors formally prove (Theorem 1) that both device selection and phase ordering are NP-hard problems, though they reduce to different hardness bases: device selection reduces to the Partition problem, while phase ordering reduces to the halting problem (proving undecidability). Furthermore, their interdependence creates a "bootstrapping dilemma," where optimizing one stage in isolation may suboptimalize the other. Current solutions often rely on static heuristics or separate learning tasks (e.g., supervised learning for devices, RL for passes) that fail to account for dynamic backend conditions or the joint nature of the optimization space.

Methodology
The paper proposes MQSS-Selector, a unified, learning-based framework designed to integrate device selection and pass ordering into a cohesive decision-making process. The approach leverages Reinforcement Learning (RL) and Deep Learning within the MLIR (Multi-Level Intermediate Representation) compilation framework, specifically utilizing the Quake dialect.

The methodology is structured around two conceptual schemes for a unified scheduler, which are currently implemented as separate components planned for future integration:

Approach 1 (Hybrid): A modular design where a supervised learning module handles device selection, and its output informs a separate RL-based module for phase ordering.
Approach 2 (Fully RL-based): A monolithic design where device selection and pass ordering are combined into a single, larger action space for a unified RL agent.

Key Components:

Device Selection Module: Utilizes supervised learning models (MLP, Random Forest, SVM, KNN) to predict the Hellinger fidelity of a program on various devices. The system normalizes execution time and fidelity to rank devices, prioritizing relative ranking accuracy over absolute value precision.
Pass Selection Module: Implements an Actor-Critic RL algorithm.
- State: Includes program features (qubit count, depth), backend status, and intermediate compilation results.
- Action: Selection of compilation passes from an extended MQSS pass suite (92+ passes), a mapping pass, or a "Finish" action.
- Reward Function: Designed to satisfy executability (native gate compliance), early stopping (preferring shorter paths for equivalent results), and structural optimization (reducing depth and operation count).
Dual-Annealed Exploration Priming (DAEP): A novel training strategy introduced to address the sparsity of successful compilation rollouts. DAEP introduces a guidance term in the loss function that overwrites the actor's selection with a predefined policy with a decaying probability ( $p_{DAEP}$ ) and magnitude ( $\alpha_{DAEP}$ ). This accelerates convergence by providing initial structure before the agent learns to exploit the critic's estimates independently.

Experimental Results
The authors evaluated the components using two datasets:

Device Selection: Tested on 128 MQT-Bench programs across 12 mock IBM backends and 2,880 random benchmarks on two real local devices.
- Findings: Random Forest (RF) emerged as the most suitable candidate for the mock device dataset, outperforming MLPs which struggled with infeasible program predictions. On real devices, where all programs were executable, all models performed similarly with high $R^2$ scores. The authors note that while SVM and KNN showed good fidelity prediction on limited data, they may not scale as well as RF for diverse, large-scale program sets.
Pass Selection: Evaluated on a dataset of 1,548 chemical Hamiltonian programs.
- Findings: The RL agent trained with DAEP significantly outperformed a baseline without guidance.
  - Executability: The guided model achieved a 95.5% rate of executable programs upon termination, compared to 42.1% for the unguided model.
  - Optimization: The guided model reduced program depth in 87/154 samples and operation counts in 93/154 samples, whereas the unguided model achieved these reductions in only 8 and 15 samples respectively.
  - Termination: The guided agent learned to call the "Finish" action effectively (153/154 times), whereas the unguided agent rarely terminated early.
- Critique: The authors note that the critic network requires further training, as its classification performance (True/False Positives) was conservative in the guided model, likely due to the actor learning to finish efficiently with fewer steps, thereby providing fewer training samples for the critic. Crucially, the experimental data suggests that RL without additional methods (like DAEP) is not a viable approach for training a network for compilation under the resource constraints considered.

Significance and Claims
The paper claims to demonstrate the feasibility of the foundational building blocks (supervised device selection and RL-based pass ordering) for a data-driven, MLIR-native approach to quantum compilation. Key contributions include:

Unified Framework Concept: The paper proposes a joint optimization framework to address the NP-hard, interdependent nature of device and pass selection, moving away from fragmented, stage-specific heuristics. However, the unified scheduler itself remains a conceptual design with two separate components planned for future integration, rather than a fully demonstrated, integrated system.
MLIR-Native Implementation: Unlike prior work (e.g., TuniQ) that targets Qiskit, this work extends the MQSS X Compilation Pass Suite with a large set of MLIR-based passes, enabling integration with modern compiler infrastructures.
DAEP Strategy: The introduction of Dual-Annealed Exploration Priming addresses the challenge of sparse rewards in compilation tasks, enabling the RL agent to learn effective policies where random exploration would fail.
Modest Outlook: The authors conclude that while the unified approach holds the potential to outperform independent optimization pipelines by exploiting interdependencies, the pass selection component currently requires more extensive training and guidance. They emphasize that the current work establishes the foundational building blocks for a fully integrated, adaptive scheduler capable of handling dynamic HPCQC environments, rather than claiming the unified system is fully realized.

Technical Summary: MQSS-Selector

More like this