⚡ electrical engineering

Multi-Objective-Optimization Assisted Data Collection Framework for IoUT Based on Offline Reinforcement

This paper proposes a multi-AUV assisted data collection framework for Information Updating Networks that leverages a multi-agent offline reinforcement learning approach with a semi-communication decentralized training paradigm to simultaneously maximize data rate and information value while minimizing energy consumption and ensuring collision avoidance in challenging underwater environments.

Original authors: Yimian Ding, Xinqi Wang, Jingzehua Xu, Guanwen Xie, Weiyi Liu, Yi Li

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yimian Ding, Xinqi Wang, Jingzehua Xu, Guanwen Xie, Weiyi Liu, Yi Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the ocean as a giant, chaotic library where books (data) are scattered across thousands of shelves (underwater sensors). The problem is that the library is underwater, the shelves are shaking violently due to strong currents (turbulence), and the librarians (Autonomous Underwater Vehicles, or AUVs) can't talk to each other very well. If they try to learn how to organize the library while they are actually working, they might crash into each other, waste a lot of energy, or get lost in the noise.

This paper proposes a new way to train these robotic librarians so they can collect data efficiently without crashing or wasting power. Here is the breakdown in simple terms:

The Problem: Learning While Swimming is Dangerous

Traditionally, these underwater robots learn by trial and error. They swim around, try to grab data, make mistakes, and slowly get better. But underwater, "making a mistake" is expensive. It burns battery, risks damaging the robot, and the ocean is too noisy and unpredictable for them to learn quickly. It's like trying to learn to ride a bike on a slippery, stormy slope while blindfolded.

The Solution: The "Study Hall" Approach (Offline Reinforcement Learning)

Instead of learning by swimming around in the real ocean, the authors suggest a "Study Hall" method called Offline Reinforcement Learning.

The Dataset: First, they use a very smart, expert robot to swim around and collect a massive amount of data in a simulation. This is like a master librarian writing down every single move they made to organize the library perfectly.
The Training: The new robots then sit in a "classroom" (a computer) and study this pre-recorded data. They don't need to touch the real ocean to learn. They learn from the "homework" of the expert robot. This saves time, energy, and prevents them from crashing while learning.

The Teamwork Strategy: "SC-DTDE"

Since there are multiple robots (AUVs) working together, they need to coordinate without getting in each other's way.

The Old Way: Either they all talk to a central boss (which is slow and causes delays), or they act completely alone (which leads to chaos).
The New Way (SC-DTDE): The authors created a "Semi-Communication" system. Imagine a group of friends playing a game where they can whisper a little bit of information to each other about where they are, but they don't need to share their entire life story. This allows them to coordinate their movements to avoid collisions without waiting for slow messages.

The "Smart Brain" Algorithm (MAICQL)

To make sure the robots don't get overconfident and try things they haven't seen before (which leads to crashes), they use a special algorithm called MAICQL.

The Analogy: Think of this as a strict teacher who says, "You can only try moves that look very similar to the expert's moves we studied." If a robot tries a crazy new path that isn't in the study notes, the algorithm says, "No, that's too risky." This keeps the robots safe and efficient.

What They Optimized (The Three Goals)

The robots aren't just trying to grab data; they are trying to do three things at once, like a tightrope walker balancing three poles:

Get the most data: Collect as many "books" as possible.
Get the right data: Some data is more urgent than others (like a book about a storm vs. a book about a calm day). The system prioritizes urgent data so it doesn't get "stale."
Save energy: Don't swim in circles or fight the currents unnecessarily.

They also have to dodge "turbulent whirlpools" (ocean currents) that could push them off course.

The Results: Did It Work?

The authors ran simulations to test their idea:

Noise Resistance: Even when they added "static" (noise) to the data to mimic a messy ocean, the robots still learned well. They were robust.
Team Size: They tested with 1, 2, and 3 robots. They found that 2 robots were the sweet spot. One was too slow, but three caused too many near-misses and collisions.
Comparison: When compared to other methods (like robots that just copy what they see or robots that learn the hard way), this new method collected more data, earned higher "scores" (rewards), and was more stable.

The Bottom Line

The paper claims that by having underwater robots study a "textbook" of expert moves instead of learning by crashing in the real ocean, and by using a smart team-coordination system, they can collect data much faster, safer, and with less energy, even in a stormy ocean.

Note: The paper focuses entirely on the simulation and algorithm design for underwater data collection. It does not claim to have tested this on real physical robots in the ocean yet, nor does it discuss medical or clinical applications.

Technical Summary: Multi-Objective-Optimization Assisted Data Collection Framework for IoUT Based on Offline Reinforcement Learning

Problem Statement
Information Updating Networks (IUNs) hold significant potential for ocean exploration but face critical challenges due to dynamic underwater environments, severe system attenuation, and the limitations of current data collection methods. Traditional approaches, such as opportunistic routing and clustering, often suffer from high communication overhead and failure risks. While Autonomous Underwater Vehicles (AUVs) offer mobility and intelligence, existing online Reinforcement Learning (RL) methods for AUV trajectory planning incur high computational costs, low data utilization, and pose risks of physical damage due to extensive environment interaction. Furthermore, multi-AUV systems struggle with cooperative control and communication constraints in turbulent ocean conditions.

Methodology
To address these issues, the authors propose a multi-AUV assisted data collection framework for IUNs based on Multi-Agent Offline Reinforcement Learning (MARL). The core components of the methodology include:

System Modeling:
- Environment: The framework models a scenario with $N$ AUVs and $\lambda$ IUNs nodes, distinguishing between ordinary nodes and data-collecting nodes with backlogs.
- Physics & Constraints: The model incorporates underwater acoustic channel characteristics (using the Thorp formula for absorption), noise models (flow-field, platform, wind, thermal), AUV energy consumption (thrust vs. power relationships), and a turbulent ocean environment modeled via superimposed viscous vortex functions.
- Value of Information (VoI): A metric is introduced to quantify the timeliness and importance of data, decaying over time to prioritize urgent updates.
- Optimization Formulation: The task is formulated as a multi-objective optimization problem aiming to maximize data rate, total VoI, and channel capacity while minimizing energy consumption and ensuring collision avoidance.
Algorithm Design:
- SC-DTDE Paradigm: The authors introduce a Semi-communication Decentralized Training with Decentralized Execution (SC-DTDE) model. Unlike fully centralized training (which causes delays) or fully decentralized execution (which may yield suboptimal results), SC-DTDE allows AUVs to possess partial knowledge of other agents' states without requiring complete environmental information sharing during execution.
- MAICQL Algorithm: The framework utilizes a Multi-Agent Independent Conservative Q-Learning (MAICQL) algorithm. This extends Conservative Q-Learning (CQL) to a multi-agent setting to mitigate extrapolation errors common in offline RL.
- Training Process: Instead of direct environment interaction, the algorithm trains on a pre-collected offline dataset generated by a Soft Actor-Critic (SAC) agent operating in a turbulent environment. The MAICQL algorithm optimizes policies by penalizing actions that deviate from the expert dataset while minimizing overestimation of value functions.

Key Contributions

Framework Proposal: A novel multi-AUV data collection framework specifically designed for IUNs in turbulent ocean environments using offline RL to reduce interaction risks and computational costs.
Algorithm Innovation: The development of the MAICQL algorithm, which integrates the SC-DTDE paradigm to handle multi-agent cooperation and competition effectively without direct environmental interaction.
Multi-Objective Optimization: The integration of data rate, VoI, energy consumption, and collision avoidance into a unified reward structure, addressing the complex trade-offs inherent in underwater data collection.
Robustness Mechanism: The use of offline datasets with added Gaussian noise to simulate real-world uncertainties, enhancing the algorithm's robustness against environmental variations.

Results
Extensive simulations were conducted to evaluate the proposed framework:

Performance Metrics: The system demonstrated improvements in cumulative reward, sum data rate, and sum VoI across training epochs. The inclusion of noise in the dataset was found to improve sampling efficiency and robustness.
Comparative Analysis: MAICQL outperformed Behavioral Cloning (BC), Generative Adversarial Imitation Learning (GAIL), and the Multi-Agent Soft Actor-Critic (MAISAC) algorithm in terms of cumulative reward and stability.
Scalability and Trade-offs: Experiments varying the number of AUVs (1, 2, and 3) indicated that while increasing AUVs from 2 to 3 improved the sum data rate, it also significantly increased the collision rate. The study identified 2 AUVs as the optimal configuration balancing efficiency and safety.
Trajectory Visualization: In both turbulence-free and turbulent environments, the trained AUVs successfully prioritized urgent, nearby nodes and maintained effective hovering for data collection, demonstrating the framework's capability in complex scenarios.

Significance and Claims
The paper claims that the proposed framework significantly enhances data collection efficiency, robustness, and scalability for IUNs tasks. By leveraging offline RL, the approach avoids the high costs and risks associated with online exploration in dangerous underwater environments. The authors assert that the SC-DTDE model and MAICQL algorithm effectively tackle the challenges of multi-agent cooperation and environmental turbulence. The work concludes by noting that future efforts will focus on "sim2real" experiments to validate the framework in physical environments.