Digital Twin-Assisted Adaptive Multi-Agent DRL for Intelligent Spectrum and Resource Management in Open-RAN UAV-Enabled 6G Networks

Technical Summary: Digital Twin-Assisted Adaptive Multi-Agent DRL for Intelligent Spectrum and Resource Management in Open-RAN UAV-Enabled 6G Networks

Problem Statement
The evolution toward 6G wireless networks necessitates architectures capable of ultra-high data rates, extreme reliability, and low latency (xURLLC) under highly dynamic conditions. While Open-RAN offers a flexible, decoupled hardware-software framework, and Unmanned Aerial Vehicles (UAVs) provide critical mobility and coverage extension for ground users (GUs), their integration presents significant challenges. Specifically, managing spectrum and resources in UAV-assisted environments is complicated by non-linear system interactions, mobility-induced topology variations, and stringent constraints on latency and energy. Existing optimization and game-theoretic methods often fail to address real-time adaptability and the concurrent coordination required in 6G. Furthermore, prior Digital Twin (DT) applications have been limited to terrestrial networks or simplified models that do not fully capture the complexities of wireless communication systems.

Methodology
To address these challenges, the authors propose a hybrid framework integrating a Digital Twin (DT) with Adaptive Multi-Agent Deep Reinforcement Learning (MADRL) within an Open-RAN architecture. The system model includes High-Altitude Platforms (HAPs), Open-RAN Centralized/Distributed Units (CU/DU), multiple UAVs acting as Aerial Radio Units (ARUs), Ground Radio Units (GRUs), and distributed GUs.

The core optimization problem is formulated as a Mixed-Integer Nonlinear Programming (MINLP) task aimed at maximizing the total service provision (data rate) to GUs. The problem is decomposed into two interrelated subproblems to handle non-convexity and computational complexity:

1. UAV Trajectory Optimization: A lightweight Particle Swarm Optimization (PSO) algorithm is employed to determine optimal UAV positions. This step focuses on maximizing coverage and minimizing interference while adhering to mobility, collision avoidance, and energy constraints.
2. Resource Management: An adaptive MADRL framework, based on an enhanced Deep Deterministic Policy Gradient (DDPG) algorithm, handles dynamic resource allocation. This includes RU-GU association, bandwidth allocation, and transmission power control.
   • Architecture: The framework utilizes a hierarchical Open-RAN structure. The Non-RT RIC (hosted in the O-Cloud) runs the DT for centralized training, global state aggregation, and policy optimization. The Near-RT RIC (at the edge) executes decentralized actor policies for real-time inference and adaptation.
   • Algorithm Enhancements: The MADRL agent employs twin critic networks to counteract overestimation bias, target policy smoothing to reduce sensitivity to Q-value variations, and parameter noise adaptation for state-dependent exploration.
   • Constraints: The solution strictly enforces constraints regarding UAV battery capacity (propulsion energy model), end-to-end latency (access, backhaul, and DT synchronization delays), minimum SINR thresholds, and collision avoidance.

Key Contributions
The paper outlines four primary contributions:

• Framework Proposal: A UAV-assisted Open-RAN DT-based framework for intelligent spectrum sharing and resource management in 6G, specifically addressing latency, energy consumption, and dynamic adaptability.
• Problem Formulation: A joint data rate maximization problem that optimizes UAV positions, RU transmission power, bandwidth allocation, and RU-GU associations under strict energy, latency, and DT synchronization constraints.
• Hybrid Solution: An adaptive, iterative hybrid solution combining PSO for trajectory optimization and MADRL for decentralized resource allocation to overcome the non-convexity of the problem.
• Validation: Extensive simulations demonstrating the framework's superior performance over state-of-the-art solutions (MADDPG, MAPPO, MA Actor-Critic, and Greedy algorithms) in terms of spectral efficiency, data rate maximization, and energy utilization.

Simulation Results
The proposed DT-assisted PSO-MADRL framework was evaluated in a 1000m × 1000m area with 100 GUs and three UAV-GRU clusters. Key findings include:

• Convergence: The proposed method achieved significantly faster convergence (stabilizing after ~200 episodes) and higher average rewards compared to benchmark algorithms.
• Data Rate: The framework consistently achieved higher average data rates across varying cluster numbers, demonstrating scalability and efficient spectrum utilization in dense deployments.
• Latency: The solution achieved the lowest and most stable latency, converging rapidly to approximately 60 ms, thereby meeting xURLLC requirements more effectively than the benchmarks.
• Trajectory: PSO successfully optimized UAV paths to ensure near-complete GU coverage within clusters while maintaining energy efficiency.

Significance and Claims
The authors claim that this work represents a transformative path toward self-evolving, autonomous 6G connectivity. By integrating DT-based centralized training with decentralized execution, the framework enables intelligent, context-aware decision-making and adaptive coordination among UAVs. The study asserts that this approach effectively bridges the gap between theoretical optimization and practical, real-time implementation in dynamic UAV-assisted Open-RAN networks, offering a robust solution for scenarios such as post-disaster recovery and smart transportation where reliable, low-latency connectivity is critical. The work is supported by the Luxembourg National Research Fund (FNR).