💻 computer science

VDSAC-TD3-based collaborative coverage optimization in satellite-UAV heterogeneous networks with energy and safety constraints

This paper proposes VDSAC-TD3, a novel multi-agent deep reinforcement learning framework that integrates value decomposition, maximum entropy regularization, and twin delayed mechanisms to optimize collaborative coverage in satellite-UAV heterogeneous networks while effectively addressing energy constraints, safety requirements, and dynamic environmental challenges.

Original authors: Rou Zhang, Yongsheng Ma, Wei Zhao, Wenzhe Li, Qingtao Zeng

Published 2026-10-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rou Zhang, Yongsheng Ma, Wei Zhao, Wenzhe Li, Qingtao Zeng

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where the sky is no longer empty, but crowded with a new kind of infrastructure. High above, low-orbit satellites circle the Earth, acting as massive, distant eyes. Closer to the ground, fleets of unmanned drones hover, ready to zoom into specific neighborhoods. On the surface, mobile stations wait to connect people to this vast network. Together, these layers form a space-air-ground system designed to provide seamless internet and communication services to every corner of the planet, even in the most rugged mountains or bustling cities. The challenge, however, is coordination. These different machines move at different speeds, have different battery lives, and must avoid crashing into each other or into obstacles like tall buildings. If they do not work together perfectly, the network develops blind spots, leaving people without service, or the drones run out of power before finishing their jobs.

For years, engineers have tried to solve this using rigid mathematical rules or simple trial-and-error methods. But real-world environments are too messy for simple rules. The terrain is uneven, the weather changes, and the demand for data shifts by the minute. To handle this complexity, a team of researchers from the Beijing Institute of Graphic Communication and other institutions has developed a new way to teach these machines how to cooperate. They created a smart system that allows satellites, drones, and ground stations to learn from their own experiences, much like a student learning to navigate a city by walking its streets, rather than just reading a map. Their goal was to find a way for these diverse machines to cover the most ground possible, use their energy wisely, and stay safe, all while working together without a central commander micromanaging every move.

The researchers built a computer simulation to test their idea, creating a virtual world that mimicked the difficult conditions of the real Earth. In this digital landscape, they placed two satellites, four drones, and three ground stations. They also scattered five specific points of interest that needed to be covered, along with three obstacles like mountains or buildings that the drones had to avoid. The drones had limited battery power, and if they flew too far without recharging, they would stop working. The satellites moved in fixed paths, while the drones could fly freely but had to stay within communication range of the others. The team then introduced their new learning algorithm, which they named VDSAC-TD3. This system acts as a coach that watches all the agents at once during training but lets them make their own decisions during the actual mission. It teaches them to balance three competing needs: covering as much area as possible, saving battery life, and avoiding collisions.

To see if this new approach worked, the researchers compared it against four other established methods used in similar fields. They ran thousands of training sessions, letting the agents fail and try again until they learned the best strategies. The results showed that their new system was significantly better than the others. In the simulations, the team's algorithm achieved an average coverage rate of 79 percent, meaning it successfully monitored or served nearly four-fifths of the target area. It also used resources more efficiently, improving load balancing by nearly 18 percent compared to the next best method. Perhaps most importantly, the new system was much safer. It reduced the number of times the drones bumped into each other or obstacles, keeping the network running smoothly without crashes. The system also proved to be very stable; even when the researchers changed the number of target points or shuffled the starting conditions, the algorithm continued to perform well, whereas the older methods often became erratic or failed to find good solutions.

The secret to this success lies in how the algorithm handles the different types of machines. Because a satellite cannot move like a drone, and a ground station cannot fly, they have different capabilities and limitations. The new system uses a technique that breaks down the overall goal into smaller, manageable pieces for each machine, then combines them to see the big picture. It also includes a safety mechanism that prevents the system from overestimating how well it is doing, a common problem that causes other learning systems to become unstable. By encouraging the drones to explore new paths while still respecting their energy limits, the system found a way to keep the network alive and active for longer periods. The researchers found that the system worked best when its internal "brain" had a specific size; making it too large actually made it worse, as it got confused by too much information.

This study does not claim to have solved every problem in space and air communication, nor has it been tested on real hardware in the sky yet. The findings are based entirely on computer simulations, which are powerful tools but not a perfect substitute for the real world. However, the results suggest that this new approach offers a robust and reliable way to manage complex networks of different machines. It demonstrates that by teaching these systems to share information and learn from their mistakes, we can create communication networks that are not only wider and faster but also more efficient and safer. As the world moves toward a future where our phones, cars, and cities are connected to a web of satellites and drones, methods like this will be essential for ensuring that the connection never breaks, no matter where you are or what the terrain looks like.

Technical Summary: VDSAC-TD3-based Collaborative Coverage Optimization in Satellite-UAV Heterogeneous Networks

1. Problem Statement

The paper addresses the challenge of efficient area coverage within Space-Air-Ground Integrated Networks (SAGIN), specifically focusing on heterogeneous networks comprising Low-Earth Orbit (LEO) satellites, Unmanned Aerial Vehicles (UAVs), and ground stations. While SAGIN is critical for 6G global coverage and multidimensional sensing, deploying these networks in real-world environments faces significant hurdles:

Complex Environments: Non-convex terrains, dynamic obstacles, and heterogeneous node constraints (e.g., fixed satellite orbits vs. limited UAV battery capacity).
Multi-Objective Trade-offs: The system must simultaneously maximize coverage rates, minimize energy consumption, maintain communication connectivity, and avoid collisions.
Algorithmic Limitations: Traditional geometric optimization methods lack real-time adaptability in high-dimensional continuous spaces. Existing Multi-Agent Deep Reinforcement Learning (MADRL) approaches often struggle with training instability, value overestimation, and effective credit assignment among heterogeneous agents in large-scale dynamic environments.

The core objective is to design a control policy that maximizes long-term cumulative rewards while adhering to physical constraints (orbital dynamics, energy limits, obstacle avoidance) and maintaining network topology.

2. Methodology: VDSAC-TD3 Framework

The authors propose VDSAC-TD3 (Value-Decomposed Soft Actor-Critic with Twin Delayed mechanisms), a robust Multi-Agent Deep Reinforcement Learning (MADRL) framework operating under the Centralized Training with Decentralized Execution (CTDE) paradigm. The architecture integrates three key mechanisms:

Value Decomposition via MixNet: To handle credit assignment among heterogeneous agents (satellites, UAVs, ground stations), the algorithm employs a non-linear mixing network (MixNet). This network aggregates local Q-values ( $Q_i(o_i, a_i)$ ) derived from individual agents' observations and actions into a global Q-value ( $Q_{tot}$ ), conditioned on the global state $S$ .
Entropy Regularization (SAC): Drawing from Soft Actor-Critic (SAC), the framework incorporates maximum entropy regularization. This encourages exploration and robustness by maximizing both the expected return and the policy entropy, preventing premature convergence to suboptimal policies.
Twin Delayed Mechanisms (TD3): To mitigate value overestimation and ensure training stability, the algorithm utilizes:
- Clipped Double Q-Learning: Two target mixing networks ( $Q'_{tot,1}, Q'_{tot,2}$ ) are maintained, and the minimum of their estimates is used for target value calculation.
- Delayed Policy Updates: The actor network is updated less frequently than the critic networks.
- Target Policy Smoothing: Added noise to target actions to reduce overfitting to specific Q-value peaks.

System Modeling & Reward Design:
The environment is modeled as a Markov Decision Process (MDP) with a continuous action space. The total reward function ( $R_{total}$ ) is a weighted sum of:

Coverage Reward ( $R_{cov}$ ): Based on the weighted coverage of Points of Interest (POIs).
Completion Reward ( $R_{comp}$ ): Incentives for covering high-priority targets.
Energy Penalty ( $P_{eng}$ ): Proportional to energy deficits, encouraging sustainable operation.
Collision Penalty ( $P_{coll}$ ): A logarithmic function of collision counts to prevent penalty explosion while maintaining safety constraints.
Time Factor: A linear gain to encourage sustained task execution.

Energy dynamics for UAVs include baseline static consumption, dynamic motion consumption, and a recharging model when within the radius of ground charging stations.

3. Key Contributions

Novel Algorithm Integration: The paper introduces VDSAC-TD3, which uniquely combines value decomposition (MixNet), entropy regularization (SAC), and twin delayed critics (TD3) specifically tailored for heterogeneous SAGIN coverage.
Comprehensive System Modeling: The study constructs a detailed simulation environment incorporating satellite orbital dynamics, UAV energy consumption/recharging models, static obstacles, and communication topology constraints.
Robustness to Heterogeneity: The framework effectively addresses the challenges of coordinating agents with vastly different motion characteristics (fixed orbits vs. controllable flight) and resource constraints.

4. Experimental Results

The proposed algorithm was evaluated in a Python-based simulation (600×600 area) involving 2 satellites, 4 UAVs, and 3 ground stations. It was compared against four baselines: MAQMIX, MAVEN, MADDPG, and MATD3.

Performance Metrics: VDSAC-TD3 outperformed all baselines in cumulative reward, average coverage rate, energy efficiency, load balancing, and collision frequency.
Coverage & Efficiency: The proposed method achieved an average coverage rate of 79%, improving resource utilization by 5.55% and load balancing by 17.75% compared to the second-best baseline (MAQMIX).
Safety: It demonstrated a significantly lower collision frequency than all comparison algorithms.
Robustness:
- POI Sensitivity: Performance advantages became more pronounced as the number of Points of Interest increased (5 to 15).
- Hyperparameter Sensitivity: The algorithm performed optimally with a hidden layer dimension of 256; performance degraded at 512 due to overfitting and increased sample complexity.
- Stability: Across different random seeds, VDSAC-TD3 exhibited a more concentrated reward distribution with higher median values compared to the dispersed distributions of baselines like MADDPG.

5. Significance and Claims

The paper claims that VDSAC-TD3 provides a stable and robust solution for coverage control in heterogeneous space-air-ground integrated networks. By integrating value decomposition, entropy regularization, and twin delayed critic mechanisms, the approach effectively resolves the trade-offs between global optimization and local cooperation in complex, non-convex environments.

The authors assert that the method offers an effective solution for adaptive resource allocation and sustainable operation in integrated communication infrastructures. The study concludes that the algorithm successfully optimizes network resource allocation, enhances overall coverage performance, and reduces collision occurrences, exhibiting strong adaptability across varying network configurations and task demands. Future work is suggested to explore scalability in larger-scale environments and real-world deployment constraints.