Communication-Aware Multi-Agent Reinforcement Learning for Decentralized Cooperative UAV Deployment

1. Problem Statement

The paper addresses the challenge of deploying autonomous Unmanned Aerial Vehicle (UAV) swarms for tasks such as aerial relays and sensing in dynamic environments. Key constraints include:

• Partial Observability: Individual UAVs can only sense entities within a limited radius ($r_s$), lacking a global view of the environment.
• Restricted Communication: UAVs can only exchange messages with neighbors within a specific communication radius ($r_c$), leading to intermittent and distance-limited peer-to-peer links.
• Scalability: Traditional centralized controllers are fragile and do not scale well with the number of agents or dynamic environments.
• Optimization Complexity: The underlying problem (Maximum Coverage Location Problem) is NP-hard, making real-time, optimal solutions via Mixed-Integer Linear Programming (MILP) computationally intractable for online control.

The goal is to develop a scalable, decentralized control system where UAVs can cooperatively position themselves to maximize coverage (or win in adversarial settings) using only local observations and limited communication.

2. Methodology

The authors propose a Graph-based Multi-Agent Reinforcement Learning (MARL) framework utilizing a Centralized Training with Decentralized Execution (CTDE) paradigm.

A. Environment Modeling: Agent-Entity Graph

The environment is modeled as a graph $G = (V, E)$ where:

• Nodes: Represent UAV agents and observable environment entities (e.g., ground nodes).
• Edges: Encode sensing relationships (UAV to entity) and communication relationships (UAV to UAV).
• Sets: Each UAV $i$ maintains a sensed-entity set $N_s(i)$ and a communication-neighbor set $N_c(i)$.

B. Dual-Attention Graph Encoder

The core innovation is a dual-attention mechanism that processes information in two stages:

1. Agent-Entity Attention (Local Embedding):
   • Each UAV encodes its local state and the entities it senses.
   • Uses scaled dot-product attention where the UAV embedding acts as the query and sensed entity embeddings act as keys/values.
   • This aggregates variable-sized local observations into a fixed-size latent embedding, invariant to the number of sensed entities.
2. Neighbor Self-Attention (Message Aggregation):
   • UAVs exchange messages over the communication graph.
   • Uses self-attention to aggregate messages from neighbors $N_c(i)$.
   • This allows the agent to weigh the importance of information from different neighbors dynamically, rather than using simple averaging.

C. CTDE Training & Execution

• Training: A centralized critic has access to global state information (all UAV positions and all ground nodes) to stabilize learning and compute value functions. The actor networks are trained to maximize the team reward.
• Execution: Each UAV executes a shared policy using only its local observations and the aggregated messages from its communication neighbors. No global state or central coordinator is required during deployment.

3. Key Contributions

1. Scalable MARL Framework: A novel CTDE framework specifically designed for multi-UAV deployment under partial observability and distance-limited communication.
2. Dual-Attention Architecture: Introduction of a graph encoder combining agent-entity attention (for local perception) and neighbor self-attention (for communication), enabling robust handling of variable agent counts and dynamic topologies.
3. Generalization: Demonstration that the framework transfers effectively from fully cooperative tasks to mixed cooperative-competitive settings without architectural changes.
4. Zero-Shot Generalization: The policy trained on a specific number of UAVs generalizes to unseen team sizes without fine-tuning.

4. Experimental Results

The framework was evaluated on two primary scenarios:

A. DroneConnect (Cooperative Relay Deployment)

• Task: UAVs reposition to maximize coverage of mobile ground nodes.
• Performance:
  • Achieved 74% coverage with 5 UAVs and 10 nodes under restricted communication and partial observability (RC+PO).
  • Performance remained competitive with a static MILP upper bound (which is offline and computationally expensive).
  • Ablation Studies: Removing inter-UAV communication or entity attention significantly degraded performance (dropping to ~63-65% coverage), proving the necessity of both components.
  • Zero-Shot Transfer: A policy trained on $M=5$ UAVs successfully generalized to teams with $M=3, 4, 6, 7$ UAVs with stable coverage ratios.
  • Coordination: The learned policy reduced coverage overlap (12% vs. 19% in non-communicating baselines), indicating effective task division.

B. DroneCombat (Adversarial Engagement)

• Task: A 5-vs-5 team-based combat scenario where drones eliminate opponents.
• Performance:
  • The same framework achieved a 62% win rate, significantly outperforming non-communicating baselines (49%) and independent agents (42%).
  • This validates the framework's applicability beyond purely cooperative settings.

5. Significance

This work bridges the gap between theoretical optimization and practical, real-time UAV swarm control.

• Practicality: By operating under partial observability and restricted communication, the solution is viable for real-world disaster response or military scenarios where infrastructure is damaged or denied.
• Scalability: The graph-based approach allows the system to handle dynamic changes in swarm size and topology, a critical requirement for autonomous swarms.
• Efficiency: It provides a decentralized alternative to centralized optimization (MILP), offering near-optimal coverage with significantly lower computational overhead during execution.
• Robustness: The ability to generalize to unseen team sizes without retraining makes the system highly adaptable to mission changes.

In summary, the paper presents a robust, communication-aware MARL architecture that enables UAV swarms to achieve high-performance cooperative and competitive behaviors in complex, constrained environments.