Architecture-Aware Reinforcement Learning for Communication-Efficient Distributed Quantum Circuit Compilation

Technical Summary: Architecture-Aware Reinforcement Learning for Communication-Efficient Distributed Quantum Circuit Compilation

Problem Statement
Distributed quantum computing offers a pathway to scale beyond the physical limits of single quantum processing units (QPUs) by linking multiple devices. However, this architecture introduces a complex compilation challenge: quantum operations must be scheduled not only according to local hardware connectivity but also against the constraints of expensive inter-QPU communication resources. Unlike monolithic compilation, which focuses on gate synthesis and local routing, distributed compilation must determine logical qubit placement, the timing of inter-QPU communication, and the scheduling of limited communication channels.

The core difficulty lies in the trade-off between communication volume (measured by EPR-pair consumption) and communication latency (measured by communication makespan). Excessive communication increases execution time and amplifies the impact of noise and imperfect entanglement generation. Existing approaches, including heuristic mapping algorithms and partitioning frameworks, often address qubit assignment or routing in isolation. They struggle to jointly optimize communication decisions across dynamic topologies, circuit dependencies, and hardware capacity constraints, particularly when accounting for stochastic entanglement dynamics and resource contention.

Methodology
The authors propose an architecture-aware reinforcement learning (RL) framework that formulates distributed quantum compilation as a constrained Markov Decision Process (MDP). The system models the compiler as an agent that sequentially selects "compiler-level communication actions" to manage logical qubit availability across QPUs.

1. Communication Primitives: The framework utilizes two primary actions:

   • Split: Consumes an EPR pair to make a logical qubit available at an additional QPU, enabling distributed controlled-unitary operations.
   • Merge: Removes redundant distributed placement when it is no longer useful, freeing up resources.
     These actions abstract physical mechanisms like teleportation and entanglement swapping without implying the cloning of unknown quantum states.

2. System Model and Constraints: The compilation process operates under strict hardware constraints:

   • Resource Limits: Each QPU has finite qubit capacity and a finite number of communication channels.
   • Temporal Model: A round-based communication model tracks logical clocks and channel budgets. If a QPU lacks an available channel, its clock advances, introducing latency.
   • Feasibility: Actions are valid only if the QPUs are connected, channels are available, and gate dependencies are satisfied. Two-qubit gates can only execute if both operands are available at a common QPU.

3. RL Framework Design:

   • State Representation: The compiler state is encoded as a heterogeneous graph ($G_{obs}$). This graph includes nodes for QPUs, logical qubits, and gates, connected by edges representing topology, placement, dependencies, and resource availability. This structure allows the policy to reason over complex interactions between hardware and circuit logic.
   • Policy Network: The agent uses a Relational Graph Attention Network (R-GAT) to process the graph observation and select feasible split or merge actions.
   • Reward Shaping: To address the sparsity of feedback in long compilation trajectories, the reward function is decomposed into five terms:
     • Latency Reward: Penalizes increases in communication makespan.
     • Split Penalty: Directly penalizes the consumption of EPR pairs.
     • Progress Reward: Encourages the execution of gates.
     • Lookahead Reward: Rewards actions that increase the number of immediately executable gates in the future.
     • Completion Reward: A terminal bonus for successful compilation, penalized by final latency.
   • Training: The policy is optimized using Proximal Policy Optimization (PPO).

Key Contributions
The paper presents four primary contributions:

1. MDP Formulation: A novel framework modeling communication scheduling and dynamic qubit placement as a constrained MDP, explicitly handling the sequential nature of distributed compilation.
2. Dynamic Constraints: The introduction of explicit dynamic constraints on active EPR operations, enforcing physical capacity limits on concurrent communication primitives (channel contention).
3. Heterogeneous Graph State: The construction of a graph-based state representation that simultaneously encodes circuit dependencies, qubit placement, and QPU topology, enabling the RL agent to capture structural relationships.
4. Competitive Performance: Demonstration of a policy that achieves compilation performance comparable to state-of-the-art heuristics across diverse benchmark circuits.

Results and Evaluation
The framework was evaluated across benchmark circuits with varying structures. The results indicate:

• Structured Workloads: The RL policy matches the performance of state-of-the-art heuristic algorithms on structured workloads.
• Unstructured Circuits: On unstructured circuits, the inclusion of lookahead reward shaping yields modest improvements over baseline heuristics.
• Scalability: While the approach demonstrates flexibility, the authors note that scalability remains a significant bottleneck for practical, large-scale deployment.

Significance and Claims
The paper positions reinforcement learning as a flexible alternative to manual heuristics for distributed quantum compilation. It argues that by modeling the problem as a constrained MDP with a heterogeneous graph representation, the system can effectively navigate the trade-offs between communication overhead and execution time. The work highlights that while current RL methods can match existing heuristics, the field must still overcome scalability challenges to become a viable solution for practical, large-scale distributed quantum systems. The authors do not claim to have solved the scalability issue but rather demonstrate the viability of the RL approach as a foundation for future optimization.