DRL-based Power Allocation in LiDAL-Assisted RLNC-NOMA OWC Systems

Technical Summary: DRL-based Power Allocation in LiDAL-Assisted RLNC-NOMA OWC Systems

Problem Statement
Optical Wireless Communication (OWC) utilizing Non-Orthogonal Multiple Access (NOMA) offers a promising solution for high-speed indoor connectivity, enabling multiple users to share the optical spectrum via the power domain. However, the performance of NOMA-based OWC systems is significantly degraded by imperfect Channel State Information (CSI) and residual decoding errors, particularly in dense-user indoor scenarios involving user mobility. Traditional power allocation (PA) strategies often rely on static channel assumptions or standard probabilistic distributions (e.g., Rayleigh, Gaussian), which fail to capture the dynamic reality of indoor environments characterized by shadowing, quantization errors, and user movement. Furthermore, optimizing continuous PA dynamically to maximize system throughput is computationally prohibitive, as the problem is often NP-hard. Existing heuristic methods (e.g., Genetic Algorithms) may converge to local optima, while standard Reinforcement Learning (RL) algorithms struggle with large, continuous state-action spaces due to the "curse of dimensionality" and slow convergence.

Methodology
To address these challenges, the authors propose a LiDAL-assisted RLNC-NOMA OWC system integrated with a Deep Reinforcement Learning (DRL) framework. The methodology comprises three core components:

1. System Model (LiDAL and RLNC):

   • LiDAL Integration: A Light Detection and Localization (LiDAL) system is employed to exploit spatio-temporal information. By passively capturing user presence and location, LiDAL provides auxiliary data to refine CSI estimation. The system models the imperfect CSI as a function of the Cramer–Rao Lower Bound (CRLB) of location errors and the Line-of-Sight (LOS) distance, acknowledging that optical source nonlinearity and eye-safety constraints reshape localization accuracy differently than in RF systems.
   • RLNC Integration: Random Linear Network Coding (RLNC) is integrated into the NOMA framework. The optical access point encodes data packets as random linear combinations over superimposed NOMA signals. This allows receivers to reconstruct original information reliably once a sufficient number of independent combinations are received, thereby mitigating error propagation caused by imperfect Successive Interference Cancellation (SIC).

2. Optimization Formulation:
   The authors formulate a complex optimization problem aimed at maximizing the average sum rate of the system while ensuring user fairness and meeting minimum data rate constraints ($\gamma_{min}$). The problem involves determining optimal power allocation coefficients ($\alpha$) for users grouped under specific optical Access Points (APs), subject to constraints on total power, power ordering (for SIC), and minimum rate requirements.

3. DRL Solution (NAF Algorithm):
   To solve the non-convex, dynamic optimization problem efficiently, the authors propose a Deep Reinforcement Learning framework based on the Normalized Advantage Function (NAF) algorithm.

   • Agent Architecture: The NAF agent utilizes a deep neural network architecture with three heads: a state-value function $V(s)$, a mean-action function $\mu(s)$ (policy), and a quadratic advantage function $A(s, a)$. This structure allows for direct optimization of actions in continuous spaces.
   • Learning Process: The agent interacts with the environment (modeled as a Markov Decision Process) where the state includes estimated channel gains, location noise coefficients, and current power factors. The reward is defined as the average sum rate, penalized if constraints are violated. The NAF algorithm uses a target network and an evaluation network with soft updates to stabilize learning and accelerate convergence compared to traditional actor-critic methods like Deep Deterministic Policy Gradient (DDPG).

Key Contributions

• System Modeling: The paper models a novel LiDAL-assisted RLNC-NOMA OWC system that jointly leverages location-based information for enhanced CSI estimation and RLNC for robust data transmission in the presence of residual decoding errors.
• Algorithmic Innovation: A NAF-based DRL model is introduced to solve the continuous power allocation problem in dynamic, high-dimensional environments. This approach overcomes the limitations of traditional RL algorithms (like Q-learning) and actor-critic methods (like DDPG) regarding convergence speed and stability in continuous action spaces.
• Performance Analysis: The study provides a comprehensive evaluation of user location estimation accuracy and its impact on network performance. It benchmarks the proposed NAF algorithm against DDPG, Gain Ratio Power Allocation (GRPA), and exhaustive search.

Results
Simulation results, conducted in an indoor study space with 8 optical APs and up to 32 users, demonstrate the following:

• Convergence and Efficiency: The NAF algorithm converges significantly faster than DDPG (approximately 56% faster convergence time) and requires less training time per iteration (0.27s vs. 0.44s for DDPG). While it has slightly higher memory demands due to its quadratic advantage structure, it offers a clear implementation advantage over exhaustive search, which is computationally infeasible for dense scenarios.
• Sum Rate Performance: The NAF-based power allocation achieves an average sum rate of 1.882 Gbps. This represents a 4.6% improvement over the conventional GRPA scheme and a 1.64% improvement over the DDPG agent.
• Optimality: The performance of the NAF algorithm closely matches that of the exhaustive search method, which is considered the global optimum within the discrete grid, validating the near-optimality of the learned policy.
• Fairness and Robustness: The proposed method ensures high user fairness (Jain's fairness index), comparable to exhaustive search, and demonstrates robustness against location estimation errors. The algorithm effectively allocates higher power to users with higher location estimation errors to maintain system reliability.

Significance
The paper claims that the proposed framework offers a practical and efficient solution for optimizing power allocation in realistic, dense-user indoor OWC systems. By integrating LiDAL and RLNC, the system addresses the dual challenges of imperfect CSI and residual decoding errors. The adoption of the NAF-based DRL approach provides a low-complexity, high-performance alternative to computationally prohibitive exhaustive search and slower-converging DRL methods. The results suggest that this approach is robust to the uncertainties inherent in dynamic indoor environments, making it a viable candidate for next-generation high-speed optical wireless networks.