⚡ electrical engineering

Two-Layer Reinforcement Learning-Assisted Joint Beamforming and Trajectory Optimization for Multi-UAV Downlink Communications

This paper proposes a two-layer reinforcement learning framework that synergizes a topology-aware Graph Neural Network for fast beamforming with Multi-Agent Proximal Policy Optimization for cooperative trajectory planning, effectively addressing the latency and scalability limitations of existing methods in multi-UAV downlink communications.

Original authors: Ruiqi Wang, Essra M. Ghoura, Omar Alhussein, Yuzhi Yang, Jing Ren, Shizhong Xu, Sami Muhaidat

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruiqi Wang, Essra M. Ghoura, Omar Alhussein, Yuzhi Yang, Jing Ren, Shizhong Xu, Sami Muhaidat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy city where a fleet of drone delivery pilots (UAVs) needs to drop off packages to hundreds of people on the ground. But there's a catch: these drones aren't just flying; they are also acting as Wi-Fi routers, beaming internet signals to the people below.

The problem is a chaotic juggling act:

The Drones Move: They need to fly from Point A to Point B.
The Signals Interfere: If two drones get too close, their Wi-Fi signals crash into each other, like two people shouting at the same time.
The Speed Mismatch: The wind and signal quality change in milliseconds, but the drones take seconds to turn a corner.

Trying to solve this with old-school math is like trying to calculate the perfect flight path for every drone using a calculator while they are already flying—it's too slow. Trying to use a standard AI is like giving a driver a map that doesn't show traffic; it doesn't understand how the drones affect each other.

This paper proposes a two-layer "Brain and Body" system to solve this, using a mix of Graph Neural Networks (GNNs) and Multi-Agent Reinforcement Learning (MARL).

Here is how it works, broken down into simple analogies:

1. The Fast Layer: The "Instant Reflex" (The Beamformer)

The Problem: Every millisecond, the drones need to adjust their signal beams to avoid shouting over each other. This happens way too fast for a drone to "think" about it.
The Solution: A Graph Neural Network (GNN).

The Analogy: Imagine a conductor in an orchestra. The conductor doesn't just look at one violinist; they see the entire room. If the violins are too loud, the conductor signals the cellos to quiet down.
How it works here: The GNN treats the drones and users as nodes in a living, breathing web (a graph). It instantly sees who is close to whom and who is interfering with whom. Instead of calculating complex math from scratch every time, it has "learned" the patterns of interference.
The Result: It acts like a reflex. In milliseconds, it tells the drones exactly how to angle their antennas to silence the noise and boost the signal, even if the crowd of users changes size or location. It's fast, scalable, and doesn't get confused by new arrangements.

2. The Slow Layer: The "Strategic Commander" (The Trajectory Planner)

The Problem: The drones need to fly from their hangar to their destination. They need to avoid crashing into each other, stay within the city limits, and make sure they pass over as many people as possible to deliver good service.
The Solution: Multi-Agent Proximal Policy Optimization (MAPPO).

The Analogy: Imagine a team of hikers trying to reach a summit. They can't see the whole mountain, only what's right in front of them. If they all just run toward the nearest flower, they might get lost or block each other.
How it works here:
- Centralized Training, Decentralized Execution: During practice (training), all the drones sit in a classroom with a "Super Teacher" (the Central Critic) who sees the whole map. The teacher tells them, "If you go left, you block Bob, but if you go right, you help the group."
- The "Arrival Mask": In real life, some hikers might reach the top early and just wait. The system is smart enough to tell the early birds, "Stop moving, don't mess up the data," so they don't confuse the learning process.
- The Reward: The drones get points not just for reaching the destination, but for how well they served the people along the way.
The Result: The drones learn to fly in a coordinated dance. They might take a slightly longer route to hover over a crowded park, ensuring everyone gets good internet, while automatically swerving to avoid collisions.

Why This is a Big Deal (The "Aha!" Moment)

1. Speed vs. Smarts:
Old methods were like trying to solve a Rubik's cube by turning every single face one by one (too slow). This new method is like having a robot that can solve the cube instantly because it recognizes the pattern.

2. Adapting to Chaos:
If you add 50 more people to the ground, a standard AI might get confused and crash. This system is like a chameleon; it adapts its strategy based on the size of the crowd without needing to be retrained.

3. The "Two-Speed" Trick:
The paper's genius is realizing that signals change fast but drones move slow.

The GNN handles the fast stuff (signals) like a reflex.
The MAPPO handles the slow stuff (flying) like a strategy game.
By separating these two, the system doesn't get overwhelmed.

The Bottom Line

This paper gives us a blueprint for a future where swarms of drones can act as flying 6G towers. They don't just fly blindly; they "talk" to each other, instantly adjust their signals to avoid noise, and fly in perfect harmony to ensure everyone on the ground gets the best possible internet connection, all while avoiding crashing into one another. It turns a chaotic traffic jam into a synchronized ballet.

1. Problem Statement

The paper addresses the joint optimization of beamforming and UAV trajectory planning in multi-UAV downlink communication networks (6G non-terrestrial networks). The core challenges identified are:

High Coupling: UAV mobility dictates channel topology, while beamforming capabilities and power limits restrict feasible flight paths. This creates a highly non-convex, dynamic optimization problem.
Timescale Mismatch: Channel variations (small-scale fading) occur on a millisecond scale, whereas UAV mechanical mobility operates on a second scale. Treating them synchronously creates a high-dimensional hybrid action space that is difficult to optimize.
Limitations of Existing Methods:
- Numerical Optimization: Iterative algorithms (e.g., block coordinate descent) suffer from prohibitive computational latency, making them unsuitable for real-time dynamic environments.
- Standard Deep Learning: End-to-end architectures often ignore the intrinsic graph-structured nature of user associations and interference, leading to poor generalization across varying network scales.
- Sparse Feedback: In trajectory planning, agents often receive rewards only upon mission completion, leading to sparse feedback and training instability.

2. Methodology

The authors propose a hierarchically decoupled framework that separates the joint optimization into two distinct timescales, integrating Graph Neural Networks (GNNs) and Multi-Agent Reinforcement Learning (MARL).

A. Fast Timescale: Topology-Aware GNN Beamforming (Inner Loop)

Objective: Optimize instantaneous beamforming vectors for a fixed UAV trajectory to maximize the sum rate.
Graph Modeling: The UAV-user system is modeled as a time-varying heterogeneous graph.
- Nodes: UAVs and users.
- Edges: Encode communication links and intra-cluster interference.
Architecture: A GraphNorm-enhanced GNN is developed.
- Unlike standard Batch Normalization, GraphNorm normalizes features based on the current graph instance statistics, ensuring robustness against varying cluster sizes.
- It uses a message-passing mechanism to explicitly learn interference coupling patterns without requiring fixed-dimensional inputs.
Input/Output: Takes instantaneous Channel State Information (CSI) as input and outputs beamforming vectors. It is trained via a reward-driven paradigm (maximizing sum rate) rather than supervised learning on labeled data.

B. Slow Timescale: MAPPO Trajectory Planning (Outer Loop)

Objective: Optimize UAV trajectories over the mission duration to maximize the long-term average sum rate.
Formulation: Modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP).
Algorithm: Multi-Agent Proximal Policy Optimization (MAPPO) under the Centralized Training with Decentralized Execution (CTDE) paradigm.
- Actor: Decentralized; uses only local observations (position, relative user locations, destination vector).
- Critic: Centralized; uses global state information during training to stabilize learning.
Key Innovations for Stability:
- Reachability-Aware Reward: A novel mechanism that provides dense, step-wise supervision by checking if the destination is reachable within the remaining time. It rewards feasible trajectories and penalizes deviations that make the mission impossible.
- Arrival Masking: A scheme that filters out gradients from agents that have already completed their missions, preventing them from influencing the policy update and ensuring training stability in asynchronous termination scenarios.

3. Key Contributions

Timescale-Separated Framework: Decomposes the coupled problem into an instantaneous beamforming inner-loop and a long-term trajectory outer-loop, effectively addressing the mismatch between channel dynamics and UAV mobility.
Topology-Aware GNN Beamformer: Designs a scalable beamformer that explicitly captures dynamic interference topologies using heterogeneous graphs, enabling low-latency inference suitable for real-time deployment.
Enhanced MAPPO for Multi-UAV: Solves the trajectory planning problem using MAPPO with a novel reachability-aware reward and arrival masking scheme. This overcomes challenges of sparse feedback and asynchronous task completion, significantly improving cooperative learning efficiency.
Unified Closed-Loop System: Integrates the GNN beamformer as part of the environment for the RL agent, creating a feedback loop where trajectory decisions are guided by real-time communication performance metrics.

4. Simulation Results

Extensive simulations were conducted in a 200x200 $m^2$ area with varying numbers of UAVs and users.

Beamforming Performance:
- The proposed GNN beamformer outperformed optimization-based heuristics (Genetic Algorithm) and other learning baselines (DeepSets, MS-PointNet) in achievable sum rate, particularly in high-interference and high-density regimes.
- Generalization: The GNN maintained high performance when tested on network sizes (UAV/user counts) different from the training set, whereas baselines degraded significantly.
- Latency: The GNN achieved inference times of 30–45 ms, a speedup of approximately 2,000x compared to the Genetic Algorithm (58–87 seconds), making it viable for real-time A2G channels.
Trajectory Planning Performance:
- Convergence: MAPPO demonstrated superior long-term convergence compared to Independent PPO (IPPO), Value-Decomposition Networks (VDN), and QMIX. While IPPO converged quickly to local optima, MAPPO learned sophisticated cooperative strategies (e.g., collision avoidance, load balancing).
- Sum Rate: The proposed framework achieved a stable long-term sum rate of ~65.5 bps/Hz, outperforming greedy heuristics and random strategies.
- Behavior: Visualizations showed that UAVs learned to deviate from the shortest path to service dense user clusters, dynamically trading flight distance for communication gain, while strictly adhering to safety and boundary constraints.

5. Significance

This paper presents a significant advancement in UAV-assisted communications by bridging the gap between theoretical optimization and real-time practical deployment.

Scalability: The graph-based approach allows the system to scale to varying network densities without retraining, a critical requirement for dynamic 6G networks.
Real-Time Feasibility: By reducing beamforming latency from seconds to milliseconds, the framework enables the dynamic adjustment of resources in high-mobility scenarios.
Cooperative Intelligence: The enhanced MAPPO algorithm demonstrates how multi-agent systems can learn complex, cooperative behaviors (like implicit load balancing and collision avoidance) even with partial observations and asynchronous task completion, providing a robust solution for future autonomous aerial networks.