Decision Transformer for UAV-Mounted RIS-Assisted Dynamic D2D Communications

Technical Summary: Decision Transformer for UAV-Mounted RIS-Assisted Dynamic D2D Communications

Problem Statement
This paper addresses the challenge of optimizing communication performance in dynamic indoor environments where device-to-device (D2D) links are stochastically activated and subject to blockage. The study focuses on a system assisted by a Reconfigurable Intelligent Surface (RIS) mounted on an Unmanned Aerial Vehicle (UAV). Unlike prior works that often treat RIS as static or ignore the three-dimensional (3D) rotational dynamics of the UAV, this research considers a fully coupled system where:

1. UAV Mobility and Attitude: The UAV's 3D trajectory and its rotational attitudes (yaw, pitch, and roll) are optimization variables.
2. Angle-Dependent Reflection: The RIS is modeled as a Uniform Planar Array (UPA) with practical reflection coefficients that vary based on the incident angle of electromagnetic waves.
3. Channel Dynamics: The system operates under time-varying Rician fading channels where the Line-of-Sight (LoS) components depend on the UAV's position and orientation.

The core objective is to jointly optimize the UAV's flight trajectory, 3D attitude angles, and RIS phase shifts to maximize the average sum rate of all D2D pairs, subject to constraints on mobility, energy, and hardware limitations. The problem is non-convex and highly coupled, making traditional optimization methods difficult to apply.

Methodology
The author proposes a framework based on Deep Reinforcement Learning (DRL) enhanced by a Decision Transformer (DT) to achieve robust control and generalization across different scenarios.

1. System Modeling:

   • Channel Model: Utilizes Rician fading with LoS and Non-Line-of-Sight (NLoS) components. The array response is calculated based on the specific azimuth and elevation angles relative to the RIS.
   • Reflection Model: Adopts a practical model where the reflection coefficient of RIS elements is a function of the incident angle, rather than assuming ideal unitary reflection.
   • Motion Model: Incorporates a 3D rotation matrix to update the RIS normal vector and incident angles based on UAV yaw, pitch, and roll.

2. Optimization Framework:

   • MDP Formulation: The problem is formulated as a Markov Decision Process (MDP) where the state includes UAV position/velocity/attitude and channel matrices, and the action includes trajectory updates, attitude adjustments, and RIS phase shifts.
   • Expert Data Generation: Several DRL algorithms (DDPG, PPO, SAC, TD3) are trained in multiple scenarios. Comparative analysis identifies DDPG as the superior "expert" algorithm for this continuous, high-dimensional control problem.
   • Decision Transformer (DT) Training:
     • Offline Pre-training: The DT is pre-trained on a large dataset of expert trajectories generated by the DDPG algorithm across multiple initial UAV positions. The DT learns to predict actions conditioned on historical states, actions, and "returns-to-go" (target future rewards).
     • Online Fine-tuning: For unseen scenarios, the pre-trained DT is deployed in a "zero-shot" manner. To adapt to the new environment, the system collects interaction data, selects the top-performing trajectories (based on cumulative returns), and uses them to fine-tune the DT policy. This avoids the performance degradation often seen in direct policy transfer.

Key Contributions

• Integrated 3D Modeling: The paper presents a comprehensive model that jointly optimizes UAV trajectory and 3D attitude (yaw, pitch, roll) alongside RIS phases, explicitly accounting for the coupling between UAV orientation and the angle-dependent reflection characteristics of the RIS.
• Generalizable Control via Decision Transformer: Instead of relying solely on standard DRL which may struggle with unseen environments, the author utilizes a Decision Transformer. This approach frames reinforcement learning as conditional sequence modeling, enabling the policy to learn from offline expert data and generalize to new scenarios without extensive retraining.
• Hybrid Training Strategy: The proposed framework combines offline pre-training on high-quality expert trajectories with an online fine-tuning mechanism that selectively uses high-return trajectories from the new environment. This balances sample efficiency with adaptability.

Results
Numerical simulations were conducted in a 3D industrial manufacturing scenario with four D2D pairs and a 4x4 RIS array.

• DRL Algorithm Selection: Among the tested algorithms (DDPG, PPO, SAC, TD3), DDPG demonstrated the fastest convergence and highest average sum rate (approx. 29.5 bps/Hz), making it the chosen expert for generating training data.
• Generalization Performance: In unseen scenarios (different initial UAV positions), the Zero-Shot DT significantly outperformed policies obtained by directly transferring a DDPG model trained on a single or neighboring scenario.
• Fine-tuning Efficacy: The Fine-tuned DT achieved performance nearly identical to a scenario-specific DDPG expert but required significantly fewer online interactions to adapt.
• Stability: The DT-based approaches showed more stable performance with fewer fluctuations compared to direct transfer methods, which were sensitive to distribution shifts in channel states and user pairings.

Significance and Claims
The paper claims that the proposed Decision Transformer framework offers a promising solution for the control of UAV-mounted RIS systems in dynamic environments. By leveraging offline expert data and conditional sequence modeling, the method overcomes the robustness limitations of conventional DRL policies. The author asserts that their approach enables effective cross-scenario generalization and efficient online adaptation with fewer interactions, addressing the critical need for flexible and reliable communication in complex, time-varying wireless networks. The work highlights the potential of expert-data-driven sequence modeling to solve complex joint optimization problems in UAV-RIS communications without requiring real-time, computationally expensive online training from scratch.