⚛️ quantum physics

Hybrid Quantum and Classical Workload Management with Graph-based Scheduling

This paper introduces Fluence, a Kubernetes scheduler plugin that leverages graph-based gang scheduling to solve the unique two-queue orchestration challenges of hybrid quantum-classical workloads, significantly reducing worker idle time, eliminating wasted node resources, and cutting costs by enabling intelligent, queue-aware device selection.

Original authors: Vanessa Sochat, Daniel Milroy

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vanessa Sochat, Daniel Milroy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-tech kitchen where chefs (classical computers) are preparing complex dishes. Usually, they just need ingredients from the pantry. But now, you've added a magical, remote oven (a quantum computer) that lives in a different city. This oven is amazing, but it has a catch: you can't just walk in and use it. You have to send a request to a separate, mysterious line of people waiting to use it.

This creates a "two-queue problem." You have your kitchen's line, and the remote oven's line. The problem? Your chefs don't know how long the oven's line is. They might start chopping vegetables and prepping pans (using up your kitchen's space and time) while waiting for the oven to be free. If the oven is busy for an hour, your chefs are just standing there, chopping air, wasting energy and space.

The authors of this paper, Vanessa Sochat and Daniel Milroy, built a new kitchen manager called Fluence. Think of Fluence as a super-smart head chef who refuses to let the prep crew start working until the remote oven is actually ready.

The Magic Trick: "All or Nothing"

In the old way of doing things (using a default scheduler), if you asked for a team of chefs to work together, the kitchen might let half the team start cooking while the other half waits for the oven. This is like sending a pizza delivery driver to the door before the pizza is even baked. It wastes time and space.

Fluence uses a strategy called "gang scheduling." It treats the whole team as a single unit. It says, "No one starts chopping, no one starts prepping, until the remote oven is 100% ready and the whole team can work at the exact same moment."

In their tests, when they tried to run groups of 4 chefs, the old manager let the team start partially, causing them to wait around for about 133 node-seconds (a measure of wasted time) while the oven line moved. Fluence kept that wasted time almost at zero. It's like the difference between a traffic jam where cars inch forward and a traffic light that stays red until the whole intersection is clear, then lets everyone zoom through at once.

The "Two-Queue" Synchronization

The biggest headache is that the remote oven's line is unpredictable. Sometimes it's empty; sometimes it's hours long.

The Old Way: The chefs start working immediately. If the oven queue is long, they sit idle for hours.
The Fluence Way: One chef (the "producer") goes to the remote oven line and checks the status. The rest of the chefs (the "consumers") are told to "wait in the wings" (a state called being gated). They don't use any kitchen space or electricity until the producer says, "The oven is free!"

In their experiments, this simple trick saved a massive amount of time. When the oven queue was short, Fluence reduced the chefs' idle time by about 5 times. But when they tested it on a real quantum device where the queue stretched out for hours, Fluence saved time by orders of magnitude. In one specific test with a group of 8 chefs, the old method made them wait 223 seconds on one simulator, while Fluence got them working in just 32 seconds. In another case with a 2.66-hour queue, the old method wasted 19,160 node-seconds of time, while Fluence's workers didn't start until the very last second, avoiding that waste entirely.

Picking the Best Oven

Fluence also helps you pick which remote oven to use. There are different ovens (simulators and real quantum devices) with different prices and wait times.

Random Choice: If you just pick an oven at random, you might pick a super expensive one or one with a long line.
Fluence's Choice: Fluence can look at the menu and say, "Let's pick the cheapest one" or "Let's pick the one with the shortest line."

In their tests, when Fluence was told to pick the cheapest option, it cut the average cost per run by roughly 70 times compared to picking randomly. It also helped finish the work faster by choosing the ovens with the shortest lines.

What Fluence is NOT

The authors are very clear about what Fluence doesn't do.

It doesn't magically make the remote oven faster. It just manages the waiting time better.
It doesn't require you to store secret passwords in the kitchen manager's main office (which is a security risk). Instead, it lets the user handle the credentials locally.
It doesn't work for every single type of quantum job. For example, if you want to send 8 different chefs to 8 different ovens at the same time, Fluence doesn't need to coordinate that because they aren't waiting on each other. Fluence is specifically for when a team of chefs needs to work together on a single task that depends on one remote oven.

The Bottom Line

The paper shows that by adding this "wait until ready" logic to the kitchen manager, you can stop wasting expensive kitchen space while waiting for a remote oven. They proved this using simulations and real tests with AWS Braket and other quantum providers.

They found that without this smart coordination, you end up with a lot of wasted time and money. With Fluence, the chefs stay fresh, the kitchen stays efficient, and the team only starts cooking when the whole meal can be served at once. It's a way to make the future of quantum computing work smoothly with the computers we already have, without needing to change the recipes the chefs are using.

Technical Summary: Hybrid Quantum and Classical Workload Management with Graph-based Scheduling

Problem Statement
As High Performance Computing (HPC) centers integrate Quantum Processing Units (QPUs) to address complex optimization problems, they face a unique orchestration challenge known as the "two-queue problem." While traditional schedulers manage local cluster resources, quantum workloads often rely on remote vendor APIs (e.g., AWS Braket, IBM Quantum Cloud) that maintain a separate, external queue. This creates a scenario where a workload manager must coordinate resources across two uncoordinated systems: the local cluster and the remote vendor queue.

Current integration efforts, such as Slurm plugins using the Quantum Resource Management Interface (QRMI), often treat the quantum device as a session opener rather than a schedulable resource. This leads to pathological behavior where classical compute nodes (workers) are allocated and begin execution while the associated quantum task waits in an external, unpredictable queue. This results in significant worker idle time, wasted node-hours, and inefficient resource utilization, particularly for hybrid algorithms like Variational Quantum Algorithms (VQAs) that require tight coupling between classical parameter updates and quantum execution. Furthermore, existing solutions often require storing vendor credentials at the system scheduler level, introducing security risks.

Methodology
The authors present Fluence, a Kubernetes scheduler plugin built upon the Fluxion graph-based scheduler from the Flux Framework. Fluence addresses the two-queue problem by modeling remote quantum backends as first-class, attributed resources within the scheduler's resource graph.

Key methodological components include:

Graph Modeling: Fluence represents the cluster as a graph where quantum devices (associated with specific vendors) sit at the same hierarchy level as racks or nodes. It performs back-to-back graph queries for both classical and quantum resources, ensuring an "all-or-nothing" (gang) allocation. A job is only scheduled if both resource types can be allocated simultaneously.
Gang Synchronization: Leveraging native gang scheduling in Kubernetes 1.36+, Fluence gates classical worker pods until the quantum task is ready.
- Producer-Consumer Model: For a gang of $N$ pods, one pod acts as a producer that submits the quantum task to the vendor API. The remaining $N-1$ pods act as consumers (workers).
- Synchronization Primitive: The producer pod includes a sidecar that polls the vendor queue. The consumer pods remain in a "gated" state (scheduling-gated) until the producer confirms the quantum task has reached the top of the external queue. Only then are the workers released to execute.
Policy-Based Backend Selection: Fluence supports attribute-based selection, allowing jobs to request specific quantum backends based on dynamic criteria such as cost or queue depth. This is facilitated by a user-space kubectl plugin that queries vendor APIs using local credentials, avoiding the need to store secrets in the scheduler namespace.
Application Patterns: The design supports various hybrid patterns, including single evaluations, variational loops (e.g., QAOA, VQE), batch processing, and fan-out designs.

Key Contributions

Characterization of the Two-Queue Problem: The paper identifies and quantifies the waste of classical resources caused by the lack of synchronization between local schedulers and remote quantum queues.
Graph-Based Hybrid Modeling: A novel approach to modeling remote quantum devices as schedulable resources within a graph-based scheduler, enabling intelligent co-scheduling.
Fluence Scheduler Plugin: A Kubernetes plugin that implements gang scheduling and custom resource handling for quantum-classical workloads without modifying user containers.
Synchronization Mechanism: A gating primitive that reduces worker idle time by preventing classical nodes from starting until the quantum task is imminent.
Dynamic Backend Selection: A mechanism to select quantum backends based on cost or queue depth, optimizing for both economic and temporal efficiency.

Experimental Results
The authors evaluated Fluence using AWS Braket simulators and real QPUs (Rigetti, IQM, IBM) across three scenarios:

Gang Scheduling Efficiency: Under node contention, Fluence's atomic gang placement eliminated the wasted node-time observed in default schedulers. While default schedulers incurred significant idle time (e.g., median of ~133 node-seconds for size $N=4$ gangs) while waiting for partial gangs to clear, Fluence maintained idle time near zero.
Idle Time Reduction: In the two-queue scenario, Fluence reduced worker idle time by approximately 5x under short device queues and by orders of magnitude when queues stretched to hours. For example, on a simulator with a 2.66-hour queue, the default scheduler accumulated ~19,160 node-seconds of idle time, whereas Fluence's gated approach minimized this waste.
Cost and Time Optimization: Using policy-based selection, Fluence reduced the mean per-run cost by roughly 70x by pinning jobs to the cheapest available backend (e.g., selecting a simulator over expensive real devices). It also reduced time-to-result from hours to under a minute by selecting backends with the shortest queues.

Significance and Claims
The paper claims that Fluence demonstrates that quantum-awareness can be added to a cloud-native scheduler without modifying user containers. By treating remote quantum devices as first-class resources in a graph-based model, the system can intelligently coordinate classical and quantum execution, mitigating the inefficiencies of the two-queue problem.

The authors emphasize that their contribution lies in the scheduling layer (co-scheduling, arbitration, and backend selection) rather than the resource control interface (QRMI). They argue that current designs often fail to coordinate the timing of classical and quantum resources, leading to pathological resource waste. Fluence offers a solution that is orthogonal to existing middleware, providing a framework for future HPC centers to support hybrid quantum-classical workflows autonomously. The work also highlights the security advantage of keeping vendor credentials in user space rather than the system scheduler.

The paper concludes that while current quantum devices are in the Noisy Intermediate-Scale Quantum (NISQ) regime, the workload management infrastructure must evolve to support the hybrid algorithms required to make these devices useful for scientific discovery. Fluence provides a foundational step toward this evolution by enabling efficient, gang-scheduled placement of hybrid workloads.