Digital-Twin Empowered Deep Reinforcement Learning For Site-Specific Radio Resource Management in NextG Wireless Aerial Corridor

Technical Summary: Digital-Twin Empowered Deep Reinforcement Learning for Site-Specific Radio Resource Management in NextG Wireless Aerial Corridor

Problem Statement
The paper addresses the complex Radio Resource Management (RRM) challenge of jointly optimizing Base Station (BS) association and beam selection for multiple Unmanned Aerial Vehicles (UAVs) operating in a site-specific (SS) aerial corridor. This problem is characterized by:

• High-dimensional action spaces: The combinatorial nature of assigning $M$ UAVs to $L$ BSs and $N$ beams creates an NP-hard optimization problem.
• Dynamic Environments: Rapidly varying propagation channels and UAV mobility make real-time adaptation difficult.
• CSI Overhead: Conventional optimal methods require global Channel State Information (CSI), which incurs prohibitive signaling overhead and synchronization requirements in dense, mobile scenarios.
• Training Data Scarcity: While Deep Reinforcement Learning (DRL) offers a path to reduce computational complexity, training robust models typically requires extensive, site-specific datasets that are difficult to acquire from live networks without disrupting operations.

Methodology
The authors propose a two-stage, Digital Twin (DT)-enabled framework that couples physics-based modeling with Deep Reinforcement Learning.

1. Digital Twin and Channel Twin (CT) Construction:

   • A high-fidelity DT environment is constructed using the Howard University campus as a testbed, utilizing OpenStreetMap data, Blender for 3D modeling, and NVIDIA Sionna for GPU-accelerated ray tracing.
   • The Channel Twin (CT) captures site-specific propagation characteristics (multipath, diffraction, reflection) by integrating geo-spatial contexts, BS locations, and UAV positions.
   • Stage 1 (Beam Gain Optimization): A dual annealing algorithm is employed offline to precompute optimal horizontal scan angles ($\phi_{scan}$) for each BS-UAV-beam combination. This maximizes the 3GPP-compliant antenna gain ($G_{5G}$) toward the UAV's physical location, effectively creating a library of optimized beam gains.

2. Multi-Head Proximal Policy Optimization (MH-PPO):

   • Stage 2 (Association Learning): A Multi-Head Proximal Policy Optimization (MH-PPO) agent is trained on the offline dataset generated by the CT.
   • Architecture: The agent uses a shared feature extraction backbone (processing flattened channel gains and Angles of Arrival) feeding into $M$ decentralized actor heads (one per UAV) and a centralized critic network. This design allows the network to scale with UAV density without architectural changes.
   • Constraint Handling: Instead of hard constraints, the framework enforces load balancing and beam exclusivity through a penalty-based reward mechanism. If multiple UAVs select the same BS, the environment admits the top-$N$ strongest links and penalizes the excess, allowing the agent to learn load-balanced associations implicitly.
   • State Space: The input state includes channel gain, azimuth, and zenith angles of arrival derived from the CT.
   • Action Space: The agent outputs a categorical distribution over BS-beam pairs for each UAV.

Key Contributions

1. DT-Driven Training Methodology: The paper constructs a physics-consistent offline dataset using a high-fidelity CT that adheres to 3GPP TR 37.840 and ECC Report 281 specifications. This dataset underpins the training of a PPO agent, enabling robust policy learning across diverse channel conditions without requiring live network exploration.
2. Scalable MH-PPO Framework: The authors formulate the UAV-BS-beam association as a policy-learning task using a specialized multi-head actor-critic architecture. This approach maps high-dimensional CSI features directly to resource allocation decisions, eliminating the need for iterative optimization during online deployment.
3. Implicit Load Balancing: A novel reward mechanism penalizes capacity violations, enabling the agent to learn fairness and load balancing through behavior rather than explicit constraint encoding in the policy network.
4. Performance and Latency: The framework demonstrates significant throughput improvements over baselines while achieving inference latencies in the millisecond range, suitable for real-time 6G operations.

Results
Extensive simulations were conducted in a 3D virtual environment with 3GPP-standardized antenna parameters, comparing the proposed MH-PPO against a Vanilla DQN, a Two-Stage Hungarian algorithm, and several heuristics (Max-Gain, Closest-BS, Random).

• Throughput Gains: In dense scenarios ($M=30$ UAVs), the MH-PPO agent achieved a 44%–121% improvement over DQN and a 249%–807% gain over traditional heuristic schemes. For instance, at 60m altitude with 20 UAVs, MH-PPO achieved 12.59 Mbps, a 43.8% gain over DQN and a 720.6% gain over the Closest-BS heuristic.
• Robustness and Generalization: The MH-PPO agent demonstrated superior stability in high-dimensional, interference-limited environments. While DQN suffered from distribution mismatch and policy degeneration (evidenced by a long left tail in throughput distribution), MH-PPO maintained consistent performance. The 5th percentile throughput (worst-case user performance) for MH-PPO was significantly higher than DQN, indicating better fairness.
• Inference Latency: The proposed framework achieved an average inference latency of 2.43 ms, which is approximately 2.4% of the minimum channel coherence time for UAVs traveling at 20–50 km/h. This is several orders of magnitude faster than the time required for iterative optimization and well within the 40 ms latency requirement for Command-and-Control (C2) packets.
• Scalability: The inference latency scaled sub-linearly with the number of UAVs, confirming the feasibility of the approach for ultra-dense deployments.

Significance
The paper claims that this work establishes a viable pathway for real-time resource management in next-generation aerial corridor networks. By leveraging Digital Twins to generate high-fidelity training data, the authors overcome the dual challenges of computational intractability in combinatorial optimization and the impracticality of acquiring global CSI in real-time. The results demonstrate that DT-driven training pipelines can deliver high-performance, low-latency RRM policies tailored to site-specific deployments, effectively bridging the gap between theoretical optimization and practical, real-time 6G implementation. The work extends previous conference contributions by explicitly accounting for inter-cell interference in the reward design and providing a comprehensive benchmark for UAV corridor communications.