🤖 machine learning

Quantum Reinforcement Learning for Cost and Delay Tradeoffs in Quantum Cloud Orchestration

This paper proposes QRLQ, a quantum reinforcement learning framework that integrates parameterized quantum circuits with a dueling double deep Q-network to optimize cost-delay tradeoffs in quantum cloud orchestration, demonstrating superior performance over heuristic baselines and comparable results to classical deep reinforcement learning with significantly fewer trainable parameters.

Original authors: An N. H. Phan, Dang Van Huynh, Muhammad Usman, Hoa T. Nguyen

Published 2026-09-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: An N. H. Phan, Dang Van Huynh, Muhammad Usman, Hoa T. Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where the most powerful computers on Earth are not sitting in a single room, but are scattered across the globe, accessible to anyone with an internet connection. These are quantum computers, machines that use the strange rules of physics to solve problems that would take ordinary computers thousands of years to finish. Today, scientists are building a "cloud" for these machines, a system where people can rent time on them to run their own experiments. However, just like renting a car or a hotel room, there is a price tag attached. In this digital marketplace, the cost is often tied directly to how long the computer runs. The problem is that these quantum machines are not all the same. Some are fast but expensive, while others are slower but cheaper, and some are better at handling specific types of difficult tasks. If you simply pick the cheapest machine, your task might take so long that the bill becomes huge. If you pick the fastest one, you might have to wait in a long line because everyone else wants it too. Finding the perfect balance between saving money and finishing quickly is a puzzle that has stumped researchers until now.

A team of scientists has tackled this puzzle by creating a new kind of digital manager, a system designed to decide which quantum computer should run which task. They call their creation QRLQ. Instead of using the standard, heavy software that usually runs these decision-making systems, they built their manager using a hybrid approach that mixes classical computing with a very small, specialized quantum circuit. Think of this quantum circuit as a compact, highly efficient brain that can learn from experience without needing to be massive. The researchers trained this system to look at a new task, check the status of all available quantum machines, and instantly decide where to send the work. The goal was to minimize two things at once: the time the task spends waiting in line and the time it spends actually running, which directly determines the cost.

To test if this idea worked, the researchers ran thousands of simulations on a powerful classical computer, mimicking a busy quantum cloud with five different types of machines. They fed their system a steady stream of complex tasks, similar to those used in real-world scientific research, and watched how it performed against other common strategies. Some of these rival strategies were simple rules, like always picking the machine that is free right now, or always picking the fastest machine available. Others were more advanced, using traditional artificial intelligence to learn the best moves. The results showed that the new quantum-powered manager was remarkably effective. It managed to lower the average cost of running tasks by between 5 and 11 percent compared to the simple rule-based methods. Even more impressively, it cut the average waiting time by 17 percent compared to the best-performing traditional rule, and by a staggering 82 percent compared to the weakest rule.

What makes this discovery particularly significant is not just that it works, but how efficiently it does so. Traditional artificial intelligence systems that try to solve this kind of problem often require thousands of adjustable settings, or parameters, to learn effectively. These large systems can be slow to train and difficult to run. The new QRLQ system, by contrast, achieved results that were just as good, and in some cases better, while using about 72 percent fewer adjustable settings. It is as if the researchers found a way to get the same level of intelligence from a much smaller, more streamlined engine. The system also proved to be robust; even when the researchers introduced small amounts of random noise to simulate the imperfect conditions of real quantum hardware, the system's performance remained stable, keeping waiting times low and costs down.

The study also looked at the quality of the results produced by the quantum computers. In the current era of quantum technology, machines are prone to errors, and getting a correct answer is often the most important goal. The researchers found that their new scheduling system did not sacrifice accuracy for speed. The tasks it scheduled were completed with a success rate that was nearly identical to the best possible strategy, which simply picks the machine with the lowest error rate. This suggests that the system is smart enough to know when a slightly slower machine is actually a better choice because it is more reliable, or when a faster machine is worth the wait because it is less likely to fail.

While these results are promising, the researchers are careful to note that they are based on simulations. They have not yet tested this system on a live, physical network of quantum computers. However, the findings suggest a clear path forward. By using these compact, quantum-enhanced learning models, it may soon be possible to manage the growing complexity of the quantum cloud without needing massive, energy-hungry computers to do the managing. As the quantum industry moves toward a future where these machines are more common and more powerful, having a smart, efficient way to orchestrate them will be essential. This work demonstrates that we do not need to wait for perfect, error-free machines to start optimizing how we use the ones we have today. Instead, by combining the learning power of artificial intelligence with the unique efficiency of quantum circuits, we can already begin to solve the complex logistical challenges of the quantum age.

Technical Summary: Quantum Reinforcement Learning for Cost and Delay Tradeoffs in Quantum Cloud Orchestration

Problem Statement
The paper addresses the challenge of orchestrating quantum tasks in a Quantum-as-a-Service (QaaS) environment characterized by heterogeneous hardware and uniform time-based pricing models. In such environments, the cost of a task is directly proportional to its execution time. While high-performance Quantum Processing Units (QPUs) offer faster execution, they often suffer from queue congestion, increasing waiting times. Conversely, lower-performance nodes may have shorter queues but longer execution times, potentially increasing total cost. Existing scheduling strategies face a tradeoff: heuristic methods lack adaptability to dynamic, stochastic workloads, while classical Deep Reinforcement Learning (DRL) models often require a large number of trainable parameters, which can be inefficient and difficult to train on current Noisy Intermediate-Scale Quantum (NISQ) hardware. The core problem is to jointly minimize execution time (and thus cost) and waiting time (delay) while maintaining high execution fidelity, without relying on static rules or parameter-heavy classical models.

Methodology: The QRLQ Framework
The authors propose QRLQ, a cost-delay-aware resource orchestration framework that integrates Parameterised Quantum Circuits (PQCs) with a Dueling Double Deep Q-Network (D3QN).

System Model & MDP Formulation:
- The problem is modeled as an online sequential decision-making process using a Markov Decision Process (MDP).
- State Space ( $S$ ): Encodes the incoming task features (circuit depth, gate counts, shots) and the current status of available QNodes (ready time, gate execution speeds, error rates). Features are normalized and mapped to rotation angles for quantum encoding.
- Action Space ( $A$ ): A discrete set representing the selection of one of the available QNodes for task allocation.
- Reward Function ( $R$ ): A weighted sum of normalized execution time and waiting time scores, designed to balance cost and delay.
Hybrid Quantum-Classical Architecture:
- PQC as Feature Extractor: Instead of a classical dense neural network for feature extraction, QRLQ employs a shallow PQC. The state vector is encoded into qubits via rotation gates ( $R_x, R_y, R_z$ ). The circuit includes variational layers with trainable rotation angles and entangling gates (CZ) arranged in a ring topology to capture correlations between resource features.
- Dueling D3QN Structure: The output of the PQC (measured expectation values) is fed into a classical post-processing layer split into two streams: a State-Value stream ( $V$ ) estimating the value of the current state, and an Advantage stream ( $A$ ) estimating the relative benefit of each action. These are combined to compute the final Q-value.
- Training: The framework uses Double DQN techniques to reduce overestimation bias, employing an experience replay buffer, target networks with soft updates, and gradient clipping for stability.

Key Contributions

Novel Application: This is the first work to apply Quantum Reinforcement Learning (QRL) to quantum cloud resource orchestration under a uniform time-based pricing model.
Hybrid Framework Design: The paper details the design of a hybrid quantum-classical network specifically for scheduling, including PQC configuration, state encoding, and the integration of duelling network heads.
Parameter Efficiency: By replacing dense classical layers with a shallow PQC, the framework significantly reduces the number of trainable parameters while maintaining performance.
Robustness Evaluation: The study evaluates the framework's performance under parameter perturbations, simulating the noise and gate errors inherent in current NISQ hardware.

Experimental Results
Simulations were conducted using the QSimPy tool with tasks from the MQT Bench library and fake IBM Quantum backends (ranging from 27 to 127 qubits). The QRLQ framework was compared against heuristic baselines (GEA, GFD, CRR, RUS) and classical DRL baselines (D3QN, QDQN).

Cost and Delay: QRLQ achieved a 5–11% lower mean cost compared to availability-based and rotation-based heuristics. It reduced mean delay by 17% compared to the strongest heuristic baseline and 82% compared to the weakest heuristic baseline.
Fidelity: Despite optimizing for time and cost, QRLQ maintained execution fidelity within 2% of a fidelity-greedy policy (which prioritizes the lowest error rates).
Parameter Efficiency: Compared to the classical D3QN baseline, QRLQ achieved comparable scheduling performance using 72% fewer trainable parameters (214 parameters vs. 758).
Noise Resilience: Under simulated Gaussian perturbations of the PQC parameters, QRLQ demonstrated more consistent waiting-time performance compared to a deeper QDQN baseline, suggesting that shallower circuits may be less sensitive to parameter noise.

Significance and Claims
The paper claims that QRLQ demonstrates the feasibility of using quantum-enhanced reinforcement learning for task orchestration in quantum cloud environments. The primary significance lies in proving that parameter-efficient quantum function approximators can effectively manage the cost-delay tradeoff in heterogeneous quantum clouds. The authors position this work as a step toward future fault-tolerant environments, noting that while current NISQ hardware is noisy, the proposed shallow PQC architecture offers a practical path for stable training. The study does not claim to solve all quantum scheduling challenges but highlights the potential of QRL to outperform static heuristics and reduce the computational overhead of classical DRL models in resource-constrained settings.

Technical Summary: Quantum Reinforcement Learning for Cost and Delay Tradeoffs in Quantum Cloud Orchestration

More like this