⚡ electrical engineering

Multi-AUV Trajectory Learning for Sustainable Underwater IoT with Acoustic Energy Transfer

This paper proposes a centralized Proximal Policy Optimization (PPO) framework that jointly optimizes multi-AUV trajectory control and acoustic energy transfer to sustain long-term Internet of Underwater Things (IoUT) operations, achieving significant improvements in data freshness, fairness, and collection efficiency compared to heuristic baselines.

Original authors: Mohamed Afouene Melki, Mohammad Shehab, Mohamed-Slim Alouini

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohamed Afouene Melki, Mohammad Shehab, Mohamed-Slim Alouini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the ocean floor is a vast, dark city where tiny, battery-powered sensors are scattered everywhere. These sensors are like streetlights that need to stay lit to monitor the temperature, pollution, and health of the ocean. But there's a catch: they can't be plugged into the wall, and changing their batteries is like trying to swap a lightbulb on a skyscraper while standing on a tiny raft in a storm—it's dangerous, expensive, and often impossible.

To solve this, scientists have deployed Autonomous Underwater Vehicles (AUVs). Think of these AUVs as underwater delivery drones. Their job is two-fold:

Deliver Data: They fly by to pick up the information the sensors have collected.
Recharge the Sensors: They use sound waves (acoustic energy) to wirelessly "zap" the sensors with a little bit of power, keeping them alive without needing a battery swap.

The Problem: The Rush Hour Dilemma

The challenge is that the ocean is huge, the sensors are many, and the drones have limited battery life of their own. If a drone just flies randomly or picks the nearest sensor every time (a "greedy" approach), some sensors might get visited constantly while others are left in the dark for weeks. This makes the data "stale" (like reading a newspaper from last year).

Furthermore, if you have multiple drones, they need to coordinate so they don't crash into each other or waste energy flying in circles.

The Solution: The "Smart Coach" (AI)

This paper proposes a new way to train these drones using Deep Reinforcement Learning (DRL).

Think of the AI not as a computer program, but as a super-smart coach watching a team of drones.

The Training: The coach doesn't just tell the drones where to go. Instead, it lets them play a game over and over again. Every time a drone does something good (like picking up fresh data or recharging a sensor), it gets a "point." If it crashes, wastes energy, or leaves a sensor waiting too long, it loses points.
The Learning: Over time, the drones learn the perfect dance. They figure out that sometimes it's better to fly a little further to recharge a dying sensor, or to split up with a teammate so they can cover more ground simultaneously.

Key Features of This New System

1. The "Age of Information" (AoI) Score
Imagine a scoreboard that tracks how "fresh" the data is. If a sensor hasn't been visited in a long time, its score goes up (bad). The AI's main goal is to keep this score as low as possible for everyone, ensuring no sensor is forgotten.

2. The Wireless Power Zap
The drones don't just pick up data; they act as wireless power banks. Using sound waves (which travel well underwater, unlike Wi-Fi), they beam energy to the sensors. The AI learns exactly when to stop and "zap" a sensor to keep it alive, balancing the energy it spends flying against the energy it gives away.

3. Teamwork Makes the Dream Work
The paper tests this with two drones working together.

The Old Way (Greedy): One drone might chase the nearest sensor, while the other chases a different one, often ignoring the sensors in the middle. It's like two people in a grocery store both grabbing the same item while ignoring the rest of the aisle.
The New Way (AI): The two drones learn to divide the ocean into territories. One takes the left side, the other takes the right. They communicate (implicitly through the AI) to ensure every sensor gets visited fairly and frequently.

The Results: Why It Matters

When the researchers tested this system, the AI-driven drones performed much better than the old "follow the nearest neighbor" strategy:

Fresher Data: The information collected was much newer and more reliable.
Fairness: No sensor was left behind; the system treated all sensors equally.
Efficiency: Even though the drones flew more complex paths, they collected more total data because they didn't waste time revisiting sensors that were already full or ignoring those that were dying.

In a Nutshell

This paper is about teaching underwater robots to be smart, cooperative, and energy-conscious. Instead of just flying blindly, they use a "coach" (AI) to learn the best routes to keep the underwater internet alive, fresh, and fair for everyone, ensuring we can keep monitoring our oceans for years to come without needing to send divers down to change batteries.

1. Problem Statement

The paper addresses the critical challenge of sustaining long-term operations in the Internet of Underwater Things (IoUT). Underwater sensor networks face two primary constraints:

Energy Scarcity: Replacing batteries in deep-sea sensors is costly and often infeasible.
Information Freshness: Data collected by sensors must be timely. The Age of Information (AoI) measures the time elapsed since the last successful update; high AoI renders data obsolete.

The specific problem involves coordinating multiple Autonomous Underwater Vehicles (AUVs) to simultaneously:

Collect data from static sensor nodes via acoustic uplinks.
Wireless power transfer to sensor nodes using Acoustic Energy Transfer (AET) to recharge their batteries.
Navigate efficiently while adhering to propulsion energy limits, safety constraints (collision avoidance), and mission goals (docking).

The objective is to minimize the average AoI across all nodes while maximizing data collection and ensuring fairness in service, all within a finite mission horizon.

2. Methodology

The authors propose a Centralized Deep Reinforcement Learning (DRL) framework based on Proximal Policy Optimization (PPO).

A. System Modeling

Kinematics & Energy: The AUVs are modeled with continuous 2D kinematics (heading and speed). Energy consumption is explicitly coupled to motion dynamics, modeled as a cubic function of velocity (propulsion drag) plus a constant "hotel" load.
Acoustic Channel: The system utilizes a realistic underwater acoustic channel model (Thorp's formula) to calculate signal attenuation, source levels, and required transmission power for both data uplink and energy transfer.
State Space: The global state includes the positions, headings, and speeds of all AUVs; the AoI and remaining energy of all sensor nodes; and relative distances between agents.
Action Space: The action is a joint vector for all AUVs, comprising:
- Discrete control indices for heading and speed adjustments.
- Selection of a specific sensor node for Data Collection.
- Selection of a specific sensor node for Wireless Energy Transfer (WET).

B. Optimization Formulation

The problem is formulated as a finite-horizon Markov Decision Process (MDP). The goal is to minimize a composite cost function:
$\text{Cost} = \text{Average AoI} + \lambda_f (1 - \text{Jain's Fairness Index})$
This is subject to constraints including:

Bounded operational area and docking requirements.
Maximum speed and acceleration limits.
Energy causality (nodes must have enough energy to transmit).
Collision avoidance between AUVs.

C. Reward Function Design

The PPO reward function is carefully engineered to guide the agents:

Positive Rewards: Progress toward the docking zone, reduction in AoI, and successful energy-feasible transmissions.
Negative Penalties: Boundary violations, stalling (zero movement), collisions (Gaussian distance-based penalty), and unfair service distribution.
Shaping: An energy-margin term encourages selecting nodes that have sufficient stored energy for transmission.

3. Key Contributions

Multi-Agent Coordination: Extends previous single-AUV work to a coordinated multi-AUV setting, demonstrating how spatial parallelism improves network performance.
Realistic Energy-Motion Coupling: Integrates a physics-based propulsion energy model (cubic velocity dependence) directly into the learning framework, ensuring trajectories are energy-feasible.
Continuous Control Formulation: Uses continuous 2D motion control (heading/speed evolution) rather than discrete grid movements, allowing for more physically consistent and smooth trajectory generation.
Joint Optimization: Simultaneously optimizes trajectory control, acoustic energy transfer, and data collection, balancing the trade-off between mobility costs and communication gains.
Fairness Regulation: Explicitly incorporates Jain's Fairness Index into the reward function to prevent the "starvation" of distant or hard-to-reach nodes.

4. Results

The proposed PPO-based approach was evaluated against a Greedy Baseline (which steers AUVs to the nearest node and uses round-robin data collection) across varying network sizes (1 to 2 AUVs, up to 7+ nodes).

AoI Reduction: The PPO approach consistently achieved the lowest average AoI. In multi-AUV scenarios, it significantly outperformed the greedy baseline, especially as network size increased.
Fairness: The RL agents maintained a higher Jain's Fairness Index, ensuring that all nodes received updates more uniformly compared to the greedy strategy, which tended to favor nearby nodes.
Data Collection: Despite the energy costs of maneuvering, the RL agents collected more total data within the mission horizon due to better scheduling and reduced revisit delays.
Scalability: The performance gap between RL and the greedy baseline widened as the number of sensor nodes increased, highlighting the RL agent's ability to learn complex, long-term scheduling patterns that simple heuristics cannot capture.
Trajectory Behavior: Visualizations showed that RL agents learned to adjust speed and heading dynamically to service multiple nodes before docking, whereas the greedy baseline followed rigid, almost fixed headings.

5. Significance

This work represents a significant step toward sustainable and autonomous underwater monitoring. By proving that DRL can effectively manage the complex trade-offs between mobility, energy harvesting, and data freshness in a multi-agent underwater environment, the paper provides a blueprint for:

Maintenance-free IoUT: Enabling sensor networks to operate indefinitely via acoustic recharging.
Real-time Ocean Sensing: Ensuring data is fresh enough for critical applications like disaster management and infrastructure inspection.
Scalable Control: Demonstrating that learning-based coordination scales well with network size, offering a solution for large-scale ocean deployments where classical optimization methods (NP-hard) fail.

The study validates that cooperative multi-AUV control driven by deep reinforcement learning is superior to heuristic approaches for next-generation underwater IoT systems.