💻 computer science

Multi-UAV Path Planning in 3D based on APF-MADDPG

This paper proposes a hierarchical deep reinforcement learning framework that integrates an improved artificial potential field-based dense reward function with the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to achieve efficient and optimal 3D path planning for multiple UAVs in unknown environments with complex obstacles.

Original authors: Ming Bai, Yan Chen, Yi Liu

Published 2026-08-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ming Bai, Yan Chen, Yi Liu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a sky filled with small, autonomous aircraft, each tasked with flying from different starting points to a single meeting spot. Their mission is not just to get there, but to arrive at the same time, weaving through a three-dimensional world that might be filled with invisible walls, dangerous zones, and other moving aircraft. This is the challenge of multi-drone path planning. For decades, engineers have tried to solve this using rules based on geometry or simple trial-and-error, but these methods often get stuck or fail when the environment becomes too complex or changes unexpectedly. In recent years, a different approach has emerged: teaching these machines to learn by doing. By letting them interact with a simulated world and rewarding them for good decisions, researchers can help them figure out how to navigate on their own. However, this learning process is often slow and difficult, especially when the machines have to coordinate with each other without crashing.

A team of researchers at Yan'an University has developed a new method to make this learning process faster and more reliable for groups of drones flying in three dimensions. They combined two existing ideas: a way for multiple machines to learn together while watching each other, and a classic navigation concept that uses invisible forces to guide movement. In their approach, they created a system where the drones receive constant feedback, rather than just a single "good job" or "bad job" signal at the very end of a flight. This constant feedback acts like a gentle, continuous nudge, telling the drones exactly how close they are to their goal and how far they are from obstacles at every single moment. This helps the drones learn much quicker than before.

The researchers tested their method in a computer simulation involving three drones. The drones started at different locations within a space measuring 10 kilometers by 10 kilometers by 10 kilometers. Their goal was to fly to a specific meeting point located at coordinates 7, 4, and 1 kilometers, all while avoiding two large spherical obstacles and one tall cylindrical obstacle. The team compared their new method against two other common approaches. One of the comparison methods had the drones learning entirely on their own without sharing information, while the other allowed them to learn together but without the special continuous feedback system. The results showed that the drones using the new method learned the best strategy the fastest. They reached the target successfully and avoided all obstacles, whereas the drones learning alone failed to avoid collisions in several attempts.

When looking at the efficiency of the flights, the new method also produced shorter total travel distances. The group of drones using the new approach flew a combined total of 24.7056 kilometers to complete their mission. In contrast, the group using the standard cooperative learning method flew a longer distance of 25.9426 kilometers. The drones that learned without cooperation failed to complete the mission safely at all. The researchers found that by giving the drones a steady stream of information about their progress and safety, the system could find a better path more quickly. They also structured the training in two layers: one layer focused on planning the overall route, and another focused on executing the specific movements, which helped the drones learn the complex task more effectively.

This work suggests that by refining how machines receive feedback during training, we can help them solve complex navigation problems much faster. The study did not test these drones in the real physical world, but the simulations provided a clear picture of how the method performs under controlled conditions. The findings indicate that for groups of autonomous vehicles to work together safely in a crowded, three-dimensional space, they need a way to understand their progress continuously, rather than waiting until the end of a task to know if they succeeded. The new method offers a way to provide that understanding, leading to safer and more efficient flights.

Technical Summary: Multi-UAV Path Planning in 3D based on APF-MADDPG

1. Problem Statement

The paper addresses the challenge of multi-UAV path planning in 3D unknown environments. This problem is characterized by a high-dimensional action space, strict collision avoidance constraints (both between UAVs and with obstacles), and an NP-hard computational nature.

Specific challenges identified include:

Collaboration vs. Individuality: Unlike single-UAV planning which focuses on individual optimality, multi-UAV planning requires balancing collaboration, collision avoidance, and simultaneous mission completion (aggregation).
Limitations of Traditional Methods: Geometric methods (e.g., Voronoi) struggle with dynamic changes; Artificial Potential Field (APF) methods suffer from local minima; and heuristic algorithms (e.g., Genetic Algorithms, PSO) often face slow convergence or local optimization issues in high-dimensional spaces.
Limitations of Existing RL Approaches: While Deep Deterministic Policy Gradient (DDPG) works for single agents, it ignores inter-agent strategic differences, leading to collisions. Multi-Agent DDPG (MADDPG) addresses collaboration but often suffers from sparse rewards, making reward function design difficult and convergence slow in complex environments.

2. Methodology

The authors propose a novel algorithm, APF-MADDPG, which integrates the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework with an improved reward mechanism based on the Artificial Potential Field (APF).

2.1 Problem Modeling

Mission Model: Defined as an aggregation mission where $N$ UAVs start from dispersed locations and must reach a common target area simultaneously while avoiding threat zones, no-fly zones, and obstacles.
Motion Model: A 3D kinematic model is used where UAVs control linear acceleration, pitch angular acceleration, and yaw angular acceleration. Constraints are applied to velocity, acceleration, and angles.
Obstacle Model: Obstacles are modeled as convex bodies (spheres or cylinders) defined by geometric parameters.
Game Theoretic Formulation: The problem is formulated as a Partially Observable Markov Game (POMG).
- State Space ( $S_i$ ): Includes position, velocity, pitch, and yaw angles.
- Action Space ( $A_i$ ): Linear acceleration, pitch angular acceleration, and yaw angular acceleration.
- Observation Space ( $o_i$ ): Local observations including the UAV's own state, relative states of other UAVs, and relative positions of obstacles.

2.2 Core Algorithm: APF-MADDPG

The algorithm utilizes a distributed execution and centralized training framework.

Hierarchical Training Mechanism: A two-layer structure is introduced:
- Path Planning Layer: Each UAV has an independent actor-critic network. The actor takes local observations to generate actions with exploration noise.
- Action Execution Layer: UAVs execute actions in the environment, generating new observations and rewards.
Dense Reward Function (Key Innovation): To overcome sparse rewards, the authors design a composite reward function ( $R_{sum}$ $R_{s u m}$ ) based on APF principles:
- Attraction ( $R_{ut}$ ): Encourages movement toward the target using an exponential decay function based on Euclidean distance.
- Repulsion ( $R_{uu}$ ): Penalizes proximity between UAVs to prevent collisions.
- Obstacle Repulsion ( $R_{uo}$ ): Penalizes proximity to obstacles using an exponential function.
- Departure Reward ( $R_{depart}$ ): Rewards progress toward the target by calculating the difference between the previous distance and the current distance to the target ( $d(p_{last}, p_{target}) - d(p_{now}, p_{target})$ ). This mechanism is designed to solve the problem of unreachable targets in APF and prevent the agent from wandering near the target to accumulate maximum rewards without reaching it.
- Total Reward: $R_{sum} = w_1R_{ut} + w_2R_{uu} + w_3R_{uo} + w_4R_{depart}$ .

2.3 Training Process

The algorithm employs the standard MADDPG update rules with target networks for stability.

Critic Update: Minimizes the loss between the predicted Q-value and the target value ( $y_i$ ), which incorporates the immediate reward and the discounted future Q-value from target networks.
Actor Update: Maximizes the expected return using the policy gradient.
Soft Updates: Target network parameters are updated via soft updates ( $\tau \ll 1$ ) to ensure training stability.

3. Key Contributions

The paper explicitly claims two primary contributions:

Dense Reward Function: The integration of an APF-based dense reward function into the MADDPG framework. This addresses the issues of sparse rewards and slow convergence found in traditional multi-agent reinforcement learning for path planning.
Hierarchical Training Mechanism: The establishment of a hierarchical deep reinforcement learning structure comprising a path planning layer and an action execution layer to efficiently train optimal strategies.

4. Experimental Results

The proposed method was evaluated in a 3D simulation environment (10 km $\times$ 10 km $\times$ 10 km) with three UAVs and three obstacles (two spheres, one cylinder). It was compared against MADDPG (using traditional sparse rewards) and ILDDPG (Independent Learner DDPG).

Convergence Speed: The episode reward curves (Fig. 4) demonstrate that APF-MADDPG converges significantly faster than both MADDPG and ILDDPG.
Policy Quality:
- ILDDPG: Failed to avoid obstacles in the simulation (UAVs 1 and 2 collided), rendering it unsuitable for this multi-agent scenario.
- MADDPG: Successfully avoided obstacles but resulted in a longer total path length.
- APF-MADDPG: Successfully avoided all obstacles and achieved the shortest total path length.
Path Length Comparison:
- MADDPG Total Length: 25.9426 km
- APF-MADDPG Total Length: 24.7056 km
- The APF-MADDPG algorithm reduced the total path length compared to the standard MADDPG.

5. Significance and Claims

The paper concludes that the APF-MADDPG algorithm outperforms both ILDDPG and standard MADDPG in terms of:

Policy Learning Efficiency: Faster convergence to an optimal strategy.
Path Planning Optimality: Generating shorter, more efficient paths.
Safety: Effectively avoiding collisions with obstacles and other UAVs in a 3D environment.

The authors assert that the combination of the APF-based dense reward and the hierarchical training mechanism effectively solves the challenges of sparse rewards and local minima in multi-UAV 3D path planning, providing a robust solution for collaborative missions in unknown environments.