Kubernetes-Orchestrated Hybrid Quantum-Classical Workflows

1. Problem Statement

Hybrid quantum–classical workflows are essential for leveraging near-term quantum devices (Noisy Intermediate-Scale Quantum, or NISQ) alongside classical High-Performance Computing (HPC). These workflows involve iterative loops where classical processors handle optimization, error mitigation, and data post-processing, while Quantum Processing Units (QPUs) execute parameterized circuits.

However, coordinating these heterogeneous resources (CPUs, GPUs, and QPUs) at scale presents significant challenges:

• Lack of Unified Orchestration: Existing systems often address resource heterogeneity or quantum scheduling in isolation, lacking an integrated end-to-end workflow model.
• Scalability and Reproducibility: Traditional HPC schedulers (like Slurm) are designed for static, job-level execution and struggle with dynamic, multi-stage containerized workflows and complex data dependencies.
• Resource Management: QPUs have unique constraints (queueing delays, shot counts) that differ from CPUs/GPUs. Current frameworks do not effectively unify the scheduling of these disparate resources under a single operational model.
• Observability: There is a need for transparent monitoring of workflow progress, resource utilization, and performance bottlenecks across mixed hardware environments.

2. Methodology

The authors propose a cloud-native framework built on Kubernetes to orchestrate hybrid quantum–HPC pipelines. The system unifies CPUs, GPUs, and QPUs under a single orchestration layer using the following components:

• Kubernetes (Orchestration Backbone): Manages compute resources, isolation, and scalability. Workloads are encapsulated in containerized Pods. The system uses nodeSelector directives (and future Dynamic Resource Allocation) to steer tasks to specific hardware types (CPU, GPU, QPU).
• Argo Workflows (Workflow Engine): Defines hybrid pipelines as Directed Acyclic Graphs (DAGs). This allows for declarative specification of task dependencies, enabling complex multi-stage workflows where data flows between classical and quantum stages.
• Kueue (Queue-Based Scheduler): A workload-aware scheduler that manages resource allocation across heterogeneous nodes. It uses ResourceFlavors and LocalQueues to ensure fair and efficient scheduling of CPU, GPU, and QPU tasks based on availability and priority, preventing resource starvation.
• Prometheus & Grafana (Observability): Provides real-time metrics collection and visualization for task states, latency, utilization, and throughput, enabling performance tuning and bottleneck detection.

Proof-of-Concept Implementation:
The authors validated the framework using a Distributed Quantum Circuit Cutting workflow.

• Workflow Logic: A large quantum circuit is partitioned into smaller subcircuits (fragments).
• Execution: Subcircuits are distributed across CPU, GPU, and QPU nodes based on a backend-selection policy (e.g., small circuits to QPU, large simulations to GPU).
• Reconstruction: Classical nodes aggregate the results to reconstruct the global expectation value.
• Integration: The system supports both on-premise QPUs (via local worker nodes) and cloud-based QPUs (via API access with secure credential management using Kubernetes Secrets).

3. Key Contributions

• Unified Orchestration Model: The first framework to integrate CPUs, GPUs, and QPUs into a single Kubernetes-native operational model, treating QPUs as schedulable resources alongside classical accelerators.
• Dynamic Scheduling with Kueue: Demonstrates the use of Kueue to manage heterogeneous workloads, ensuring fairness and efficient resource utilization across different hardware types without manual intervention.
• End-to-End Reproducibility: By leveraging containerization, YAML-based DAG definitions, and persistent volumes, the system ensures that complex hybrid experiments are reproducible, version-controlled, and portable across different environments.
• Hybrid Backend Support: Successfully demonstrates seamless integration of local (on-premise) and remote (cloud) QPU backends within the same workflow pipeline.
• Observability Layer: Establishes a comprehensive monitoring stack specifically tailored for the unique latency and throughput characteristics of hybrid quantum-classical workflows.

4. Results

The proof-of-concept deployment yielded the following results:

• Feasibility: Successfully executed a distributed circuit-cutting workflow, demonstrating the ability to split, schedule, and recombine quantum tasks across heterogeneous nodes.
• Concurrency: Achieved concurrent execution of independent subcircuits, exploiting intra- and inter-stage parallelism to reduce wall-clock time.
• Resource Utilization: Monitoring showed sustained CPU and GPU utilization without prolonged idle periods, validating the effectiveness of the Kueue-based scheduling.
• Scalability: The architecture successfully handled the orchestration of 216 parallel parameterized tasks across different backends.
• Limitations Identified:
  • GPU Sharing: Current limitations in fine-grained GPU sharing (exclusive allocation vs. partitioned resources) constrained concurrent simulations.
  • I/O Bottlenecks: Persistent volume usage created I/O saturation during the reconstruction phase at larger scales.
  • Scheduling Intelligence: The current backend selection policy was simplistic; future work requires predictive, telemetry-driven scheduling.

5. Significance and Future Outlook

This work establishes a foundational shift from static HPC batch scheduling to dynamic, cloud-native orchestration for quantum computing.

• Beyond Slurm: While Slurm remains standard for HPC, it lacks the native support for multi-stage, containerized workflows and dynamic resource allocation that Kubernetes provides. This framework bridges the gap between traditional HPC and modern cloud-native computing.
• Future Directions:
  • Dynamic Resource Allocation (DRA): Leveraging Kubernetes 1.34+ DRA to support fractional resource allocation (e.g., sharing a single GPU or QPU among multiple workloads) and abstract resource classes.
  • Advanced Scheduling: Integrating performance-aware algorithms and real-time telemetry for automated backend selection.
  • Broader Applications: The framework is applicable to a wide range of hybrid algorithms, including Variational Quantum Eigensolvers (VQE), Quantum Approximate Optimization Algorithm (QAOA), and Quantum Machine Learning (QML).

In conclusion, the paper demonstrates that Kubernetes, augmented with Argo and Kueue, provides a scalable, reproducible, and observable foundation for the next generation of hybrid quantum-classical computing, essential for scaling quantum experiments beyond the capabilities of single devices.