Cross domain Persistent Monitoring for Hybrid Aerial Underwater Vehicles

1. Problem Statement

The paper addresses the challenge of persistent monitoring using Hybrid Unmanned Aerial Underwater Vehicles (HUAUVs). These vehicles are capable of operating in both aerial and underwater environments, making them ideal for complex tasks like inspecting oil platforms, submerged industrial facilities, and search-and-rescue operations.

However, autonomous operation across these two distinct domains presents significant hurdles:

• Domain Shift: The dynamics of flight (air) and hydrodynamics (water) differ drastically (e.g., drag, acceleration, control loops).
• Sensing Limitations: Traditional visual sensors (cameras) and GPS often fail underwater due to turbidity and signal loss.
• Task Complexity: Persistent monitoring requires continuous observation of moving targets to minimize "uncertainty" (the time elapsed since the last observation).
• Current Gaps: Existing Deep Reinforcement Learning (DRL) approaches are often domain-specific (trained only for air or water) or rely on visual/GPS data, limiting their scalability and robustness in cross-domain scenarios.

2. Methodology

The authors propose a framework that combines Deep Reinforcement Learning (DRL) with Transfer Learning to create a unified policy capable of operating in both air and water without retraining.

A. Problem Formulation

• Environment: Bounded 2D scenarios ($E \subset \mathbb{R}^2$) containing $N$ mobile targets moving along predefined Lissajous curves.
• State Space: The agent observes its own 2D position ($s_t$) and noisy estimates of target positions ($\hat{g}_{i,t}$).
• Action Space: Linear velocity ($v_t$) and angular velocity ($\omega_t$) setpoints.
• Uncertainty Model: Each target $i$ has an uncertainty $\sigma_{i,t}$ that grows linearly over time. If the agent enters the sensing radius ($r_{sense}$) of a target, the uncertainty resets to zero, and the agent receives a positive reward.
• Sensors: The system relies exclusively on ranging sensors:
  • Air: LiDAR (simulated Hokuyo UST-10LX).
  • Water: Forward-Looking Sonar (FLS).
  • Note: No visual or GPS data is used, ensuring the methodology is robust to environmental opacity.

B. Algorithm: Distributional Soft Actor-Critic (DSAC)

The core learning algorithm is DSAC, an extension of Soft Actor-Critic (SAC) that incorporates distributional reinforcement learning.

• Mechanism: Instead of learning a scalar expected Q-value, DSAC models the full distribution of returns using a categorical distribution (quantiles).
• Benefits: This approach captures uncertainty in value estimates more effectively, reduces overestimation bias, and provides more stable policy gradients, which is crucial for continuous control tasks in dynamic environments.
• Training Setup:
  • Parallel Training: A distributed setup with $K=5$ parallel simulation workers accelerates learning.
  • Zero-Shot Transfer: The policy is trained only in the aerial domain (using LiDAR). It is then deployed directly into the underwater domain (using Sonar) without any fine-tuning or retraining.

C. Simulation Environment

• Platform: Gazebo simulator integrated with ROS.
• Physics:
  • Air: RotorS framework with wind simulation (Ornstein-Uhlenbeck noise).
  • Water: UUV Simulator with hydrostatic/hydrodynamic forces and thruster models.
• Scenarios:
  • Env 1: Obstacle-free 10m $\times$ 10m tank.
  • Env 2: Same layout but with four cylindrical obstacles (simulating drilling risers) to test obstacle avoidance.

3. Key Contributions

1. Cross-Domain Unified Policy: Demonstrated that a single DRL policy trained on LiDAR data in the air can be successfully transferred to operate underwater using Sonar data, handling the domain shift in vehicle dynamics (air vs. water) without retraining.
2. Distributed Parallel DRL: Proposed a scalable training architecture that accelerates policy convergence compared to non-parallel methods.
3. Sensor-Agnostic Approach: Validated a methodology based solely on range data (LiDAR/Sonar), avoiding the limitations of visual/GPS-based approaches in turbid or GPS-denied environments.
4. Benchmarking: Systematically compared the DRL approach against a behavior-based baseline (Bug2 algorithm) in both domains.

4. Experimental Results

The study evaluated performance over 100 trials in both environments, comparing the DSAC policy against a Bug2 baseline.

• Performance Metrics: Average time to first visit ($t$) and average target uncertainty ($\bar{\sigma}$).
• Key Findings:
  • Superiority over Baseline: The DSAC policy significantly outperformed the Bug2 algorithm in both domains. In the obstacle-free aerial domain, DSAC reduced average uncertainty by ~3-4 points compared to Bug2. In the underwater domain, the gap widened, with Bug2 showing much higher uncertainty (e.g., 42.12 vs. 23.84 for one target in Env 1).
  • Obstacle Handling: In Env 2 (with obstacles), the DSAC policy maintained robust performance, effectively navigating detours while keeping target uncertainty low. The Bug2 baseline struggled significantly, leading to much higher uncertainty and longer visit times.
  • Zero-Shot Transfer Success: The policy trained in air successfully managed underwater dynamics. While the absolute time to visit increased underwater (due to slower hydrodynamic response), the uncertainty reduction capability remained consistent between domains.
    • Example (Env 1 Underwater): DSAC achieved average uncertainties of ~27.75, 27.45, and 23.84, comparable to its aerial performance.
  • Uncertainty Peaks: Visual analysis of uncertainty evolution (Fig. 2) showed that DSAC resets target uncertainties more frequently and keeps peak values lower than the Bug2 baseline.

5. Significance and Future Work

Significance:
This work provides a foundational proof-of-concept that Deep Reinforcement Learning can bridge the gap between aerial and underwater autonomy. By relying on range-based sensing and transfer learning, it offers a scalable solution for HUAUVs to perform complex, continuous monitoring tasks in mixed environments without needing separate control systems for each domain. This is particularly valuable for search and rescue and industrial inspection where environmental conditions are unpredictable.

Limitations & Future Directions:

• Current Scope: The study uses a 2D abstraction and a simplified linear uncertainty model with noiseless target estimates.
• Future Work:
  • Extending the framework to 3D motion (adding altitude/depth and vertical thrust commands).
  • Incorporating sensor noise and occlusions for more realistic simulation.
  • Developing multi-agent coordination strategies for large-scale monitoring.
  • Testing in real-world HUAUV deployments to validate the simulation-to-reality transfer.