💻 computer science

WDQAR:A Weighted Double Q-Learning Based Adaptive Routing Protocol for Flying Ad Hoc Networks

This paper proposes WDQAR, an adaptive routing protocol for Flying Ad Hoc Networks that leverages a weighted double Q-learning framework combined with dynamic neighbor discovery and sector-based filtering to mitigate Q-value estimation bias and optimize routing performance in highly mobile UAV environments.

Original authors: Qingzhong Ni, Yijie Bai, Daojie Yu, Qi Zhang, Fan Meng

Published 2026-09-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qingzhong Ni, Yijie Bai, Daojie Yu, Qi Zhang, Fan Meng

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the sky above disaster zones, battlefields, or remote wilderness, small flying robots known as unmanned aerial vehicles, or UAVs, are increasingly working together in swarms. These machines do not rely on cell towers or fixed internet cables; instead, they talk directly to one another, forming a temporary, self-organizing network that floats in the air. This type of network is called a Flying Ad Hoc Network. The challenge for these swarms is that the air is a chaotic place. The drones move quickly, their positions change every second, and the invisible radio links connecting them can snap apart just as easily as they form. To keep a message moving from one drone to the next until it reaches a ground station, the network needs a routing system that can make split-second decisions about which neighbor to trust with the next hop. Traditional methods often struggle here, either reacting too slowly to changes or wasting precious battery power by constantly shouting updates to every neighbor.

A team of researchers at the People's Liberation Army Information Engineering University has proposed a new way to solve this problem, called WDQAR. Their approach treats the pathfinding problem not as a static map to be drawn, but as a learning process. Imagine a drone that has never flown this specific route before; it must learn which neighbors are reliable, which links are likely to break, and which direction leads most efficiently to the goal. The researchers used a technique from the field of artificial intelligence known as reinforcement learning, where an agent learns by trial and error, receiving rewards for good choices and penalties for bad ones. However, standard learning methods can sometimes be overly optimistic, guessing that a path is better than it actually is. To fix this, the team introduced a "weighted double" learning method. This system maintains two separate internal records of what it has learned. By comparing these two records and blending their predictions, the system avoids the trap of overconfidence, leading to more stable and accurate decisions in the fast-moving sky.

The protocol operates in two main phases. First, the drones must know who is around them. In a fast-moving network, sending a "hello" message to announce one's presence too often wastes energy, but sending them too rarely means the drone might not realize a neighbor has drifted out of range. The WDQAR system solves this by making the timing of these messages adaptive. If a drone is flying in a stable pattern with neighbors moving in the same direction, it slows down the frequency of its hello messages. If the wind or maneuvering causes the links to become shaky, it speeds up the messages to stay updated. This dynamic adjustment ensures the network stays aware of its changing shape without drowning the airwaves with unnecessary chatter.

Once a drone knows its neighbors, it must decide where to send the data packet next. Instead of asking every single neighbor, the system first filters the list based on geography. It creates a cone-shaped zone pointing toward the destination and only considers neighbors inside that cone. This narrows the field of choices, allowing the learning algorithm to focus on the most promising paths. The decision of which neighbor to pick is then guided by a reward system that weighs four specific factors: how much battery power the neighbor has left, how long the connection is expected to last, how much closer the neighbor is to the final destination, and how many other reliable neighbors that neighbor has. By balancing these factors, the system avoids sending data to a drone that is about to run out of power or one that is heading into a dead end.

To make this learning happen faster, the researchers gave the drones a head start. Instead of beginning with no knowledge at all, the system assigns an initial value to each neighbor based on its position relative to the destination. A neighbor that is physically closer to the goal gets a better starting score. This prevents the drones from wasting time exploring useless paths early in the process. Furthermore, the system adjusts its own learning speed based on how stable the network is. If a link is slow or unreliable, the system learns quickly from that failure. If the connection is steady, it learns more slowly, trusting its past experience. This flexibility allows the network to adapt instantly to sudden changes in speed or direction.

The researchers tested their idea using a computer simulation that modeled a disaster monitoring scenario with up to one hundred drones flying in a three-dimensional space. They compared their new protocol against other existing methods that also use learning algorithms. The results showed that the WDQAR system was significantly more effective. It delivered a higher percentage of data packets to the ground station, reaching an average success rate of over ninety-two percent, even as the number of drones increased. It also managed to get the data there faster, with an average delay that was nearly forty percent lower than the next best competitor. Perhaps most importantly for the longevity of the swarm, the new protocol consumed less energy and generated less control traffic. By sending fewer hello messages and choosing more efficient paths, the drones preserved their battery life and kept the network running smoothly.

The study confirms that by combining a smart way to filter neighbors with a more careful method of learning from experience, it is possible to create a routing system that is robust enough for the unpredictable nature of flight. While the findings come from simulations rather than physical flight tests, the data suggests that this approach could make future drone swarms more reliable in real-world situations where communication infrastructure is absent or damaged. The work highlights that in a world of moving parts, the best strategy is not just to react, but to learn, weigh options carefully, and adapt continuously to the environment.

Technical Summary: WDQAR - A Weighted Double Q-Learning Based Adaptive Routing Protocol for FANETs

1. Problem Statement

Flying Ad Hoc Networks (FANETs), composed of Unmanned Aerial Vehicles (UAVs), face significant challenges in maintaining reliable routing due to the rapid mobility of nodes, frequent topology changes, and unstable communication links. While existing routing protocols for Mobile Ad Hoc Networks (MANETs) and Vehicular Ad Hoc Networks (VANETs) exist, they often fail to meet FANET requirements due to high control overhead, sensitivity to location inaccuracies, and an inability to learn from dynamic conditions.

Specifically, Q-learning-based routing protocols, though adaptive, suffer from three primary limitations in this context:

Biased Q-value Estimation: Conventional Q-learning tends to overestimate future values because it uses the same Q-table for both action selection and value evaluation.
Slow Convergence: Random or zero initialization of Q-values leads to inefficient early exploration and slow adaptation to the current network environment.
High Control Overhead: Static HELLO message intervals often result in either outdated topology information (if too long) or excessive control traffic (if too short), failing to adapt to varying link durations and node speeds.

2. Methodology: WDQAR

The paper proposes WDQAR (Weighted Double Q-Learning based Adaptive Routing), a two-stage protocol designed to address the aforementioned issues. The protocol operates within a 3D disaster monitoring scenario where UAVs communicate with a Ground Control Station (GCS).

2.1 Adaptive Neighbor Discovery

To manage topology changes efficiently, WDQAR introduces a dynamic HELLO message broadcast interval adjustment mechanism.

Mechanism: Instead of a fixed interval, the broadcast period ( $HP_i^t$ ) is calculated based on the predicted Link Duration ( $LD_{ij}$ ) and the Cosine Similarity ( $CS_i^t$ ) of velocity vectors between a node and its neighbors.
Logic: If nodes are moving apart rapidly (low link duration) or have divergent velocities, the interval shortens to update neighbor tables quickly. If the topology is stable, the interval lengthens to reduce control overhead.
Bounds: The interval is constrained within $[HP_{min}, HP_{max}]$ to prevent extreme values.

2.2 Routing Decision Module

The routing phase employs a reinforcement learning framework with several specific optimizations:

A. Candidate Neighbor Filtering

To reduce the action space and accelerate convergence, the protocol filters potential next hops using a sector-based mechanism.

A spherical sector is defined with the current node as the center, the GCS as the central axis, and a half-cone angle ( $\phi \approx 60^\circ$ ).
Only neighbors within this sector are considered primary candidates. If no neighbors exist in the sector, the protocol falls back to a greedy candidate set (nodes closer to the GCS).

B. Q-Value Initialization

To avoid inefficient early exploration, WDQAR uses a position-based initialization strategy rather than random or zero initialization.

The initial Q-value ( $Q_{init}$ ) is assigned based on the ratio of the candidate neighbor's distance to the destination versus the current node's distance.
Neighbors closer to the destination receive higher initial Q-values, providing the agent with prior knowledge of forwarding capabilities.

C. Reward Function Design

The reward function ( $r_i^t$ ) is a weighted sum of four normalized metrics, with weights determined dynamically using Information Entropy:

Residual Energy ( $E_j$ ): Prevents energy depletion on specific nodes.
Link Duration ( $LD_{ij}$ ): Prioritizes stable links.
Geographic Progress ( $F_{ij}$ ): Encourages forward movement toward the destination while minimizing deviation angles.
Valid Neighbor Ratio ( $VNR_{ij}$ ): Reflects the density of reliable neighbors available for subsequent hops, reducing the risk of routing holes.

D. Weighted Double Q-Learning Update

To mitigate the overestimation bias of standard Q-learning and the potential underestimation of Double Q-learning, WDQAR utilizes a Weighted Double Q-Learning approach.

Dual Tables: Each UAV maintains two independent Q-tables ( $Q_1$ and $Q_2$ ).
Update Process: One table is randomly selected to update, while the other is used to evaluate the action value.
Adaptive Weighting: An adaptive weight ( $\beta$ ) is calculated based on the spread of values between the two estimators for the best and worst actions. This weight balances the single and double estimators, yielding a more accurate future value estimate ( $Q_W$ ) and enhancing stability.

E. Dynamic Learning Parameters

The learning rate ( $\alpha$ ) and discount factor ( $\gamma$ ) are adaptively adjusted:

Learning Rate: Increases when the next-hop delay is higher than the average (indicating instability) to update values faster; decreases for stable links to rely on historical experience.
Discount Factor: Adjusted based on the stability of the neighbor sets between consecutive time steps, reducing reliance on future rewards when topology variation is drastic.

3. Key Contributions

The paper outlines four main contributions:

Adaptive HELLO Mechanism: A dynamic adjustment of broadcast intervals based on link duration and velocity similarity to balance timely topology updates with control overhead reduction.
Weighted Double Q-Learning Framework: A distributed routing protocol that maintains two Q-tables with an adaptive weighting mechanism to reduce estimation bias and improve stability in highly dynamic environments.
Comprehensive Reward Function: A multi-metric reward function incorporating residual energy, link duration, geographic progress, and valid neighbor ratio, with weights dynamically determined via information entropy.
Optimization Strategies: The integration of position-based Q-value initialization and adaptive learning parameters to accelerate convergence and improve adaptability.

4. Simulation Results

The protocol was evaluated using NS-3 in a 3D disaster monitoring scenario with 40–100 UAVs moving at 10–30 m/s. WDQAR was compared against QFANET and QGEO.

Q-Value Stability: Weighted double Q-learning reduced Q-value volatility by approximately 45.62% and total variation by 33.03% compared to single Q-learning.
Packet Delivery Ratio (PDR): WDQAR achieved the highest PDR, averaging 92.62% (approx. 3.42% higher than QFANET and 6.50% higher than QGEO). It maintained superior performance across varying node densities and UAV speeds.
End-to-End Delay: WDQAR demonstrated the lowest average delay (16.80 ms), representing a reduction of 40.93% compared to QFANET and 44.30% compared to QGEO.
Energy Consumption: The protocol reduced average energy consumption by 12.34% (vs. QFANET) and 6.86% (vs. QGEO) by balancing load and reducing unnecessary retransmissions.
Routing Overhead: WDQAR achieved the lowest overhead (9.94%), reducing it by 29.43% and 34.80% compared to the comparative protocols, primarily due to the adaptive HELLO interval.

5. Significance and Claims

The authors claim that WDQAR significantly enhances routing reliability and learning stability in highly dynamic FANET environments. By addressing the specific limitations of existing Q-learning approaches—namely estimation bias, slow convergence, and static control overhead—the protocol offers a robust solution for UAV swarms operating in resource-constrained, high-mobility scenarios.

The paper concludes that the combination of adaptive neighbor discovery, sector-based filtering, and weighted double Q-learning allows the network to identify optimal routes more precisely and adapt to topology changes more effectively than current state-of-the-art protocols. Future work is noted to include extending the protocol to larger, more complex scenarios and validating the approach through hardware-in-the-loop and real-world UAV experiments.