🤖 machine learning

Hierarchical Multi-Agent DRL Based Dynamic Cluster Reconfiguration for UAV Mobility Management

This paper proposes a hierarchical multi-agent deep reinforcement learning framework for UAV mobility management that dynamically reconfigures access point clusters and allocates power to balance reliability, energy efficiency, and reconfiguration frequency, achieving near-centralized performance with superior scalability through a novel action-observation transition-driven learning algorithm.

Original authors: Irshad A. Meer, Karl-Ludwig Besser, Mustafa Ozger, Dominic Schupke, H. Vincent Poor, Cicek Cavdar

Published 2026-01-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Irshad A. Meer, Karl-Ludwig Besser, Mustafa Ozger, Dominic Schupke, H. Vincent Poor, Cicek Cavdar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy city where thousands of drones (UAVs) are flying around, delivering packages or taking photos. In the old days, each drone would talk to just one ground station (an Access Point or AP). But if the drone moved too fast or the signal got blocked, the connection would drop, like a phone call cutting out when you walk behind a big building.

To fix this, the paper proposes a new way of thinking: instead of one station, a drone should talk to a team of nearby stations at the same time. This is called "multi-connectivity." It's like having a group of friends all shouting instructions to you at once; even if one friend is muffled by noise, you can still hear the others.

However, managing this team is tricky. You have three main problems:

Who is on the team? (Clustering) You need to pick the best group of ground stations for each drone as it flies.
How loud should they shout? (Power Allocation) You don't want them screaming (using too much battery) if a whisper will do, but you need them loud enough to be heard clearly.
Don't switch teams too often. If you keep changing the group of friends helping the drone every second, the drone gets confused and wastes energy switching connections.

The Problem with Old Methods

The paper says that trying to solve all these problems at once with a single "super-brain" (a central computer) is too slow and heavy. It's like trying to manage a whole football league with one referee; as the league gets bigger, the referee gets overwhelmed. On the other hand, letting every ground station decide for itself is chaotic; they might all shout at once, causing a mess of interference.

The Solution: A Two-Level "Coach and Players" System

The authors propose a Hierarchical Multi-Agent Deep Reinforcement Learning (H-MADRL) system. Think of this as a sports team with a Head Coach and Players.

The Head Coach (High-Level Agent): This agent lives in the "Edge Cloud" (a powerful computer nearby). Its job is simple: look at where all the drones are and decide which team of ground stations should serve each drone. It doesn't worry about the volume of the shout; it just picks the lineup.
The Players (Low-Level Agents): These are the individual ground stations. Once the Coach tells them, "You are on Team A for Drone X," they figure out how much power to use to talk to that drone. They listen to their local environment (is it windy? is there a building blocking the signal?) and adjust their volume accordingly.

The Secret Sauce: "Action-Observation Transition"

Here is the clever part. Usually, the Players (ground stations) don't know what the Coach is thinking. But in this new system, the Coach's decision (the team lineup) is passed down to the Players as part of their "observation."

The Analogy: Imagine a player on a soccer field. Usually, they just look at the ball and the opponent. In this new system, the Coach whispers, "We are playing a defensive formation," and the player hears that before they decide where to run. This helps the player make a smarter move because they know the big picture strategy. This allows the ground stations to coordinate their power usage much better, even though they are making their own decisions.

What Did They Find?

The authors ran computer simulations to see how well this "Coach and Players" system worked compared to other methods.

It's Fast and Scalable: When they doubled the number of ground stations, the old "Super-Brain" method got 90% slower. The new "Coach and Players" method only got 10% slower. It handles growth much better.
It Saves Energy: The system learned to use just enough power to keep the connection reliable, avoiding the waste of shouting unnecessarily loud.
It's Reliable: The drones stayed connected with very few errors, even when flying fast or in tricky conditions.
It's Stable: The system didn't keep changing the teams (clusters) every second. It found a good group of stations and stuck with them longer, reducing confusion.

Summary

In short, this paper introduces a smart, two-layer system for managing drone internet connections. Instead of one giant brain trying to do everything, or everyone doing their own thing, they use a Coach to pick the teams and Players to manage the volume. This approach is faster, saves battery, and keeps the drones connected reliably, even as the network grows larger.

Technical Summary: Hierarchical Multi-Agent DRL-Based Dynamic Cluster Reconfiguration for UAV Mobility Management

1. Problem Statement

The paper addresses the challenge of mobility management for Unmanned Aerial Vehicles (UAVs) in multi-connectivity wireless interference networks. Unlike traditional cell-centric networks where users connect to a single Access Point (AP), multi-connectivity architectures (e.g., cell-free networks, CoMP) serve users via dynamic clusters of distributed APs. This paradigm offers robustness for high-mobility 3D UAV trajectories but introduces three critical challenges:

Reliability under Finite Block-Length: UAV command-and-control applications require Ultra-Reliable Low-Latency Communication (URLLC). In finite block-length regimes, decoding error probabilities are non-zero, necessitating strict Signal-to-Interference-plus-Noise Ratio (SINR) constraints to meet reliability targets.
Energy Efficiency: Concurrent transmissions from multiple APs increase total power consumption. The system must minimize joint power consumption while maintaining communication quality.
Cluster Stability: Frequent reconfiguration of serving clusters as users move generates significant control overhead and latency. The system must minimize reconfiguration frequency without compromising Quality of Service (QoS).

Conventional optimization, matching, and game-theoretic approaches struggle with these problems due to their reliance on perfect, real-time channel state information (CSI) and their inability to scale as network size increases. While Deep Reinforcement Learning (DRL) offers adaptability, centralized DRL lacks scalability, and standard Multi-Agent DRL (MADRL) often suffers from performance limitations due to partial observability and the complexity of heterogeneous, tightly coupled objectives (clustering vs. power control).

2. Methodology

The authors propose a Hierarchical Multi-Agent Deep Reinforcement Learning (H-MADRL) framework, specifically implemented as H-MAPPO (Hierarchical Multi-Agent Proximal Policy Optimization). The approach decomposes the joint optimization problem into two interdependent subproblems managed by agents at different network levels.

System Architecture

High-Level Agent (Edge Cloud): Responsible for dynamic cluster reconfiguration. It observes global network states (user locations, AP loads, channel conditions) and determines the optimal serving cluster $\Gamma$ for each UAV.
Low-Level Agents (APs): Responsible for energy-efficient power allocation. Each AP acts as an independent agent that allocates transmit power to its assigned users based on local observations and the clustering decision received from the high-level agent.

Novel Mechanism: Action-Observation Transition

To address the partial observability issue in MADRL and improve coordination, the paper introduces an action-observation transition mechanism. The action taken by the high-level agent (the clustering decision) is injected directly into the observation space of the low-level agents. This allows low-level agents to incorporate global clustering context into their local power allocation decisions, creating a cohesive system where lower-level policies are guided by higher-level strategies.

Mathematical Formulation

The problem is modeled as a Partially Observable Markov Decision Process (POMDP).

Objective: Maximize a weighted sum of cluster stability (minimizing reconfigurations) and energy efficiency (maximizing bits transmitted per unit power), subject to:
- Maximum decoding error probability ( $\varepsilon_{max}$ ) constraints derived from finite block-length information theory.
- Maximum transmit power constraints ( $P_{max}$ ).
- Minimum SINR thresholds ( $\gamma_{th}$ ).
Reward Functions:
- High-Level: Rewards stable clusters and penalizes SINR outages.
- Low-Level: Rewards power minimization and penalizes error probability violations.

Training and Deployment

The framework is trained using Proximal Policy Optimization (PPO) for the high-level agent and Multi-Agent PPO (MAPPO) for the low-level agents. Training occurs in a digital twin environment (OpenAI Gym with Ray RLlib) using on-policy rollouts. The resulting policies are deployed as "frozen" networks, where the edge cloud issues clustering commands over optical backhaul, and APs perform local inference for power control without online exploration.

3. Key Contributions

Optimization Formulation: The authors formulate a multi-objective optimization problem for joint dynamic clustering and power allocation in a wireless interference network, explicitly accounting for finite block-length reliability constraints and error probabilities.
Hierarchical Framework: They propose a scalable H-MADRL framework that decouples clustering and power control. A high-level agent in the edge cloud optimizes clustering, while distributed low-level AP agents optimize power, balancing scalability with performance.
Action-Observation Transition: A novel mechanism is introduced where the high-level agent's action (clustering) becomes part of the low-level agents' observation space. This enhances coordination and allows low-level agents to allocate power more efficiently based on the global clustering context.
Validation: The feasibility and performance of the proposed H-MAPPO algorithm are validated through extensive numerical simulations against centralized and distributed baselines.

4. Numerical Results

The paper compares H-MAPPO against several baselines: MSAC (centralized), standard MAPPO (distributed), Opportunistic clustering, JCPC (optimization-based), and a "Closest" AP scheme.

Learning Efficiency: H-MAPPO converges approximately 15% faster and achieves higher cumulative rewards than the distributed MAPPO baseline. The hierarchical decomposition simplifies the action space, aiding faster exploration and stabilization.
Reliability (Outage Probability): H-MAPPO achieves an SINR outage probability near $10^{-8}$ , comparable to the optimization-based JCPC scheme ( $10^{-9}$ ) and significantly better than centralized MSAC ( $10^{-5}$ ) and distributed MAPPO. It effectively meets strict Decoding Error Probability (DEP) thresholds.
Power Efficiency: H-MAPPO achieves the required reliability with the lowest total transmit power among learning-based methods. It operates below 60% of maximum available power in 90% of instances, whereas non-learning opportunistic schemes utilize >90% of power.
Cluster Stability: H-MAPPO demonstrates superior stability with a cluster reconfiguration rate of only 9%, compared to 83% for the JCPC optimization approach. This indicates the learning agent successfully captures long-term mobility patterns to avoid unnecessary reconfigurations.
Scalability: When the number of APs doubles (from 16 to 32), the decision time for H-MAPPO increases by only 10%. In contrast, the centralized MSAC approach sees a 90% increase in decision time, and standard MAPPO sees a 70% increase. This highlights the superior scalability of the hierarchical approach.

5. Significance and Claims

The paper claims that the proposed H-MAPPO framework offers a practical solution to the trade-off between scalability and performance in multi-connectivity UAV networks. By decomposing the problem and utilizing the action-observation transition mechanism, the system achieves:

Comparable performance to centralized optimization (specifically in reliability and power efficiency) but with significantly better scalability.
Reduced control overhead through minimized cluster reconfigurations.
Energy efficiency by dynamically adapting power allocation to meet strict reliability constraints without over-provisioning.

The authors conclude that this hierarchical approach effectively bridges the gap between the adaptability of distributed learning and the coordination required for stringent QoS in dynamic 3D mobility scenarios. They acknowledge limitations regarding coordination complexity between levels and the reliance on a central high-level controller, suggesting future work could explore fully decentralized hierarchical formulations.