⚡ electrical engineering

DQN Based Joint UAV Trajectory and Association Planning in NTN Assisted Networks

This paper proposes a Deep Q-Network (DQN) based algorithm for joint UAV trajectory and association planning within an integrated GEO-terrestrial network architecture to minimize energy consumption, handover frequency, and disconnection for Advanced Air Mobility applications.

Original authors: Afsoon Alidadi Shamsabadi, Cosmas Mwaba, Thomas Nugent, Jie Gao, Pablo Madoery, Halim Yanikomeroglu, Subhadeep Pal

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Afsoon Alidadi Shamsabadi, Cosmas Mwaba, Thomas Nugent, Jie Gao, Pablo Madoery, Halim Yanikomeroglu, Subhadeep Pal

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are flying a high-tech drone (a UAV) to deliver a package across a city. Your drone has a limited battery, and its most important job is to stay connected to a "control tower" so you can steer it. If the connection drops, the drone might crash or get lost.

Here is the problem: The ground is full of cell towers (like the ones for your phone), but they are designed to look down at people on the street. If your drone flies high, the signal is weak. Plus, if you fly over a park or a rural area, there might be no towers at all, leaving your drone in a "dead zone."

This paper proposes a clever solution: Don't just rely on the ground; look up to the sky.

The Big Idea: A Two-Story Internet

Think of the internet connection for your drone as a two-story building:

The Ground Floor (Terrestrial Network): This is made of local cell towers. They are fast and have low delay (like a local courier), but they have "blind spots" where the signal doesn't reach.
The Penthouse (GEO Satellite): This is a giant satellite hovering high above the Earth. It covers the entire city and even the countryside. It's like a lighthouse that never goes out. However, talking to it takes a tiny bit longer (higher latency) and uses more energy.

The challenge is that your drone has to decide: "Should I talk to the fast ground tower, or the reliable satellite?" And at the same time, it needs to decide: "Which path should I fly to save my battery?"

If the drone switches back and forth too much between towers, it gets confused (this is called a "handover"), which wastes energy and risks dropping the connection.

The "Smart Pilot": The DQN Brain

The authors created a "Smart Pilot" for the drone using a type of Artificial Intelligence called Deep Q-Network (DQN).

Imagine training a video game character. At first, the character flies randomly, crashes into dead zones, and runs out of battery. But every time it makes a mistake, it gets a "negative score." Every time it flies efficiently and stays connected, it gets a "positive score."

Over thousands of practice flights (simulations), the AI learns the perfect strategy:

When to fly straight: To save battery.
When to switch to the satellite: When it's about to enter a dead zone or when the ground signal is too weak.
When to stick to the ground: When the signal is strong and switching would waste energy.

The AI doesn't just memorize one route; it learns a policy (a set of rules) that helps it adapt to any situation.

What the Experiments Showed

The researchers tested this "Smart Pilot" in a simulated 3km x 3km city. Here is what they found:

The Hybrid Approach Wins: When the drone could use both the ground towers and the satellite, it flew a shorter, more direct path. It was like having a GPS that could switch between a local road map and a satellite view depending on traffic.
Fewer Dead Zones: In a system with only ground towers, the drone hit "dead zones" (where it lost signal) five times. With the satellite help, it only hit one dead zone.
The Balancing Act: The AI is smart enough to listen to your instructions.
- If you tell it, "Save battery!", it will fly the shortest path and use the satellite to avoid weak spots, even if it means switching connections a bit more.
- If you tell it, "Don't switch connections!", it will stick to the ground towers even if it has to take a longer, winding route to avoid switching to the satellite.

The Takeaway

This paper shows that by combining ground cell towers with satellites and using AI to make the decisions, we can make drone delivery and air travel much safer and more efficient.

In simple terms: Instead of forcing a drone to choose between a fast but spotty ground connection and a slow but reliable sky connection, this AI acts as a wise conductor, orchestrating a seamless dance between the two. This ensures the drone never loses its way, never runs out of juice, and gets to its destination safely.

1. Problem Statement

The paper addresses the challenges of maintaining reliable, energy-efficient, and continuous communication for Uncrewed Aerial Vehicles (UAVs) in Advanced Air Mobility (AAM) scenarios.

The Challenge: Terrestrial networks (Macro Base Stations or MBSs) often fail to provide ubiquitous coverage for UAVs due to:
- Antenna down-tilting (optimized for ground users), leading to weak signal power for aerial users.
- Coverage gaps in rural or remote areas where MBS deployment is impractical.
- Frequent handovers and "ping-pong" effects when switching between cells.
The Objective: To design a Joint UAV Trajectory and Association Planning (JUTAP) algorithm. The goal is to optimize the UAV's flight path and its network association (switching between terrestrial MBSs and a Geostationary Earth Orbit (GEO) satellite) to simultaneously minimize:
1. Energy Consumption: Due to limited on-board battery.
2. Handover Frequency: To prevent link degradation and signaling overhead.
3. Disconnectivity: To avoid coverage holes where signal quality falls below a threshold.

2. System Model

The authors propose an integrated network architecture consisting of:

Terrestrial Tier: $N_B$ tri-sector MBSs distributed over a 3km $\times$ 3km area. These operate in a specific frequency band and cause co-channel interference.
Non-Terrestrial Tier: A single GEO high-throughput satellite providing wide-area coverage via fixed beams. It operates in a distinct frequency band, eliminating inter-tier interference.
UAV Dynamics: The UAV flies in a 3D Cartesian space at a fixed altitude ( $h_{UAV} < 300$ m). The environment is discretized into a grid ( $N_G \times N_G$ ). The UAV moves between grid centers, choosing one of 8 movement directions.
Channel Models:
- MBS-UAV: Modeled with probabilistic Line-of-Sight (LoS), down-tilted antennas, and path loss based on 3GPP standards.
- GEO-UAV: Modeled with near-guaranteed LoS, including free-space path loss, atmospheric loss, and scintillation.
Optimization Formulation: The problem is formulated as a mixed-integer non-convex optimization problem. The objective function minimizes a weighted sum of energy cost, handover cost, and disconnectivity penalties, subject to the UAV's total energy constraint.

3. Methodology: DQN-Based Algorithm

Due to the "curse of dimensionality" in large state spaces (many grids $\times$ many MBSs), traditional Q-learning is inefficient. The authors propose a Deep Q-Network (DQN) approach.

State Space ( $s_t$ ): Encoded as $[x_t, y_t, \hat{c}_t]$ , representing the UAV's normalized coordinates and a binary association indicator ( $\hat{c}_t = 0$ for Terrestrial, $1$ for GEO).
Action Space ( $a_t$ ): Defined as $(m_t, \hat{c}_{t+1})$ $(m_{t}, \overset{c}{^}_{t + 1})$ , where $m_t$ $m_{t}$ is the movement direction (8 options) and $\hat{c}_{t+1}$ $\overset{c}{^}_{t + 1}$ is the target association mode for the next step. Total action space size = $8 \times 2 = 16$ $8 \times 2 = 16$ .
- Simplification: Instead of selecting a specific MBS index (which would explode the action space), if $\hat{c}_t=0$ , the UAV automatically associates with the MBS offering the highest SINR at that location.
Reward Function ( $r_t$ ): Designed to penalize the three objectives:
- Energy: Penalized based on distance traveled and direction (diagonal moves cost more).
- Handover: High penalty for MBS-to-GEO switches (due to latency/cost), moderate penalty for MBS-to-MBS, and rewards for staying connected.
- Disconnectivity: Penalty if the next grid is a coverage hole.
Learning Mechanism: The agent uses an $\epsilon$ -greedy strategy, experience replay, and a target network with soft updates to learn the optimal policy $\pi(s, a)$ that maximizes cumulative reward.

4. Key Contributions

Integrated Architecture: Proposes a seamless integration of GEO satellites and terrestrial MBSs specifically tailored for UAV connectivity, addressing the limitations of terrestrial-only networks.
Joint Optimization: Formulates the JUTAP problem to jointly optimize trajectory and network association, rather than treating them as separate sub-problems.
DQN Adaptation: Develops a specialized DQN algorithm that reduces the action space complexity by using a binary association mode, making it computationally feasible for large-scale grid environments.
Comprehensive Reward Design: Creates a reward function that explicitly balances energy efficiency, handover minimization, and connectivity reliability through tunable weighting factors.

5. Simulation Results

The algorithm was tested in a 3km $\times$ 3km environment with 10 MBSs and one GEO satellite.

Convergence: The DQN algorithm converged stably after approximately 400 training episodes, demonstrating the ability to learn the underlying network dynamics.
Integrated vs. Standalone:
- Integrated (GEO + Terrestrial): The UAV took a shorter path (4,072.94m) with only 1 coverage hole. The satellite acted as a "bridge" over coverage gaps and allowed for a more direct trajectory.
- Standalone Terrestrial: The UAV took a longer, winding path (4,354.1m) to avoid holes, resulting in 5 coverage holes and higher energy consumption.
Weight Sensitivity:
- Prioritizing Handover ( $w_2=0.6$ ): The UAV avoided the satellite (due to high handover cost penalties) and stayed on terrestrial links, resulting in a longer path but fewer handovers.
- Prioritizing Energy ( $w_1=0.6$ ): The UAV took the shortest path, utilized the satellite to bypass holes, and accepted a slightly higher handover count to minimize total distance and energy.

6. Significance

This work demonstrates that AI-driven decision-making is critical for the future of AAM. By leveraging Deep Reinforcement Learning, UAVs can dynamically adapt to complex, hybrid network environments. The study proves that integrating Non-Terrestrial Networks (NTN) with terrestrial infrastructure significantly enhances:

Reliability: Eliminating coverage gaps.
Efficiency: Reducing energy consumption through optimized trajectories.
Scalability: Providing a framework that can handle the high-dimensional state spaces required for real-world UAV operations.

The proposed framework offers a viable solution for ensuring continuous, high-quality connectivity for UAVs in both urban and remote environments, a prerequisite for the widespread adoption of next-generation air transportation systems.