Autonomous Collision Avoidance Decision-Making Method for UAV-USV Based on PD3QN Algorithm

Technical Summary: Autonomous Collision Avoidance Decision-Making for UAV-USV Based on PD3QN

Problem Statement
Unmanned Surface Vehicles (USVs) face significant challenges in autonomous navigation within complex, dynamic marine environments. Traditional environmental perception methods relying on onboard sensors (radar, LiDAR, sonar) suffer from limitations such as short detection ranges, blind spots caused by occlusions, and sensitivity to environmental conditions. Furthermore, existing collision avoidance algorithms often struggle to balance real-time computational efficiency with strict adherence to the International Regulations for Preventing Collisions at Sea (COLREGs). While Deep Reinforcement Learning (DRL) offers promise, traditional Q-learning algorithms face the "curse of dimensionality," and standard DRL approaches often lack sample efficiency or fail to adequately optimize path quality (smoothness and efficiency) alongside safety. Additionally, current UAV-USV collaborative research often treats UAVs merely as external auxiliary tools without effectively fusing their high-altitude visual data into the USV's decision-making core.

Methodology
This paper proposes an autonomous collision avoidance decision-making framework that integrates Unmanned Aerial Vehicle (UAV) visual perception with a Prioritized Dueling Double Deep Q-Network (PD3QN) algorithm.

1. Perception and Input: The system utilizes a UAV as an aerial perception platform to capture high-definition environmental images. These images, representing the surrounding navigation environment, are preprocessed and resized to 120×120×3 pixels to serve as the primary input for the neural network. The state space also includes the USV's position, target coordinates, speed, and heading angle.
2. Algorithm Architecture (PD3QN): The core decision-making engine is an improved D3QN algorithm.
   • D3QN Foundation: It combines the Double DQN (to mitigate Q-value overestimation) and Dueling DQN (to separate state value and advantage functions) architectures.
   • Prioritized Experience Replay (PER): A PER mechanism is integrated to enhance sample utilization. Samples with higher Temporal Difference (TD) errors are assigned higher priorities for training, accelerating convergence. Importance Sampling Weights (ISW) are applied to correct bias introduced by non-uniform sampling.
   • Network Structure: The neural network consists of three convolutional layers (with ReLU activation) followed by fully connected layers that split into value and advantage branches.
3. Reward Function Design: A comprehensive, multi-component reward function is designed to guide the agent:
   • Primary Rewards: Include guidance rewards (based on Euclidean distance to the target), collision/arrival rewards (large positive for success, large negative for collision), and COLREGs compliance rewards (positive for adherence, negative for violation).
   • Secondary Rewards: Focus on path quality, including trajectory smoothing (penalizing frequent rudder changes) and direct navigation rewards (encouraging straight sailing).
4. Action Space: The action space is defined as discrete steering angles ranging from -20° to +20° with a 5° resolution, prioritizing heading changes over speed adjustments as per COLREGs guidelines for early and substantial maneuvers.

Key Contributions
The paper identifies three main contributions:

1. UAV-Enhanced Perception: The integration of UAV visual data as direct input for the DRL network, effectively expanding the USV's perception horizon and mitigating onboard sensor blind spots.
2. Algorithm Optimization: The development of the PD3QN algorithm, which utilizes Prioritized Experience Replay to significantly accelerate convergence speed and improve training efficiency compared to standard DQN and D3QN.
3. Comprehensive Reward Mechanism: The design of a novel reward function that simultaneously optimizes for collision avoidance safety, COLREGs compliance, and path quality (smoothness and efficiency), addressing the common issue of sparse rewards and poor trajectory quality in existing methods.

Experimental Results
The proposed method was validated through simulation experiments using the OpenAI Gym framework.

• Training Efficiency: In comparative training against DQN and D3QN, the PD3QN algorithm demonstrated superior performance. It converged approximately 16.7% faster than D3QN (3,500 episodes vs. 4,200) and achieved a 28.6% higher average reward value (approx. 3,600 vs. 2,800). In terms of time, PD3QN converged in 35.5 hours, compared to 43.2 hours for D3QN and 61.6 hours for DQN (which did not fully converge).
• COLREGs Compliance: Experiments in Head-on, Overtaking, Crossing (give-way), and Crossing (stand-on) scenarios confirmed that the trained USV successfully adhered to COLREGs, executing appropriate maneuvers (e.g., passing on the port side in head-on situations, maintaining course as a stand-on vessel).
• Multi-Ship Scenarios: In complex scenarios involving multiple target ships (TS) and static obstacles, PD3QN outperformed both the Dynamic Window Approach (DWA) and standard DQN.
  • In Scenario I, PD3QN reduced path length by 4.0% compared to DWA and 2.1% compared to DQN, while reducing total heading variation by 23.0% and 5.8%, respectively.
  • In Scenario II, PD3QN reduced path length by 6.4% (vs. DWA) and 1.9% (vs. DQN), and reduced heading variation by 20.2% and 6.9%.
  • The results indicate that PD3QN generates smoother, more efficient paths with fewer unnecessary maneuvers.

Significance and Claims
The authors claim that the proposed method enables USVs to make safer and more efficient decision-making behaviors with excellent environmental adaptability. By leveraging UAV visual perception and the PD3QN algorithm, the system effectively addresses the limitations of onboard sensors and the computational inefficiencies of traditional DRL methods. The study asserts that the method has significant potential for engineering applications, particularly in complex marine environments where compliance with COLREGs and path efficiency are critical. The authors note that the trained model exhibits low hardware resource occupancy during inference, laying a foundation for future deployment on actual hardware platforms.

Limitations and Future Work
The paper acknowledges several limitations:

• The current action space is limited to heading adjustments and does not include speed control (deceleration).
• The framework does not yet account for environmental interferences such as communication delays, perception noise, or UAV energy constraints.
• Future work aims to expand the action space to include speed control and introduce more comprehensive environmental disturbance factors (wind, waves, currents) to simulate realistic marine conditions.