Scaling up Energy-Aware Multi-Agent Reinforcement Learning for Mission-Oriented Drone Networks with Individual Reward

This paper proposes an energy-aware Multi-Agent Reinforcement Learning model utilizing Deep Q-Networks with individual reward functions to enhance the robustness, energy efficiency, and success rate of mission-oriented drone networks, particularly when scaling up environment size and agent numbers compared to traditional shared reward approaches.

The Gist

Imagine a fleet of delivery drones as a team of hired movers trying to pack up a house (the mission) and get everything back to the garage (the base station) before their batteries die.

This paper tackles a tricky problem: How do you teach a whole team of drones to work together efficiently when they are running on limited battery power?

Here is the breakdown of the paper's ideas, using simple analogies:

1. The Problem: The "Group Project" Dilemma

In the past, researchers tried to teach these drone teams using a method called Shared Reward.

• The Analogy: Imagine a group project in school where the teacher gives the entire group an "A" if the project is finished, regardless of who actually did the work.
• The Issue: If one drone gets lost or wastes energy, the whole team gets punished. If one drone does all the work, the lazy drones still get the same reward. This makes it hard for the drones to figure out exactly what they personally should do to help. It's like trying to learn a dance routine where everyone gets the same applause, so no one knows if they stepped on the wrong foot.

2. The Solution: The "Individual Report Card"

The authors propose a new method called Individual Reward.

• The Analogy: Instead of a group grade, every drone gets its own report card based on its specific actions.
• How it works:
  • If a drone moves closer to a task, it gets a small "point."
  • If a drone finishes a chunk of a task, it gets more points.
  • If a drone is running low on battery, it gets a "penalty" (a negative score) to encourage it to save power.
  • Crucially: The drones still want the whole mission to succeed (because that's the ultimate goal), but they learn faster because they know exactly which of their own moves earned them points.

3. The "Brain" of the Drones

The paper uses a type of AI called Deep Q-Networks (DQN).

• The Analogy: Think of this as a very smart GPS for each drone. It doesn't just know where the task is; it learns by trial and error.
  • Trial: "If I fly here, I use too much battery." -> Error: "Ouch, negative points."
  • Error: "If I hover here and scan this turbine, I get points." -> Success: "Good job!"
  • Over time, the GPS learns the perfect path to finish the job without running out of juice.

4. The Real-World Challenge: Wind Turbines

The paper uses inspecting wind turbines as a real-world example.

• Unlike a simple delivery where you drop a package at a fixed spot, inspecting a turbine is messy.
• Some turbines are damaged and need 10 minutes of inspection; others need only 2.
• Sometimes one drone can't do it alone; two might need to work on the same turbine at the same time.
• The environment is chaotic: tasks appear in random spots, and they take random amounts of time.

5. What the Experiments Showed

The authors ran thousands of computer simulations to test their "Individual Reward" idea against the old "Shared Reward" idea.

• The "Small Room" Test: In small, simple environments, both methods worked okay.
• The "Big Room" Test (Scalability): This is where the magic happened. When they made the environment bigger (more tasks, more drones, larger map):
  • The Shared Reward team got confused. As the map got bigger, their success rate crashed. They couldn't figure out who was doing what.
  • The Individual Reward team stayed strong. Even in huge, complex environments, they maintained a nearly 100% success rate.
• Why? Because in a big room, the "Group Grade" system is too blurry. The "Individual Report Card" system kept every drone focused on its own clear goals, making the whole team more efficient and energy-saving.

6. The Bottom Line

The paper claims that by giving each drone a clear, personal score based on its own actions and battery life, the whole team becomes much better at:

1. Planning paths (not wasting energy flying in circles).
2. Sharing tasks (knowing when to help others).
3. Scaling up (working well even when the job gets huge and complicated).

In short: The paper argues that to make a team of battery-powered robots work perfectly in a chaotic world, you shouldn't just praise the team; you need to grade each robot individually so they know exactly how to help.

Technical Summary

Technical Summary: Scaling up Energy-Aware Multi-Agent Reinforcement Learning for Mission-Oriented Drone Networks with Individual Reward

Problem Statement
The paper addresses the challenge of executing collaborative missions in drone networks where tasks have dynamic locations, non-binary lengths (requiring multiple time steps to complete), and strict energy constraints due to limited battery capacity. While Multi-Agent Reinforcement Learning (MARL) has been applied to drone trajectory planning, existing approaches often struggle with the "credit assignment" problem in cooperative settings. In traditional shared-reward MARL, agents receive identical rewards based on global outcomes, which can blur individual contributions and lead to inefficiency. Furthermore, standard algorithms often fail to scale effectively when the environment size or the number of agents increases, and they frequently overlook the specific energy constraints required for safe return to base stations.

Methodology
The authors propose an energy-aware MARL model utilizing Deep Q-Networks (DQN) with an individual reward function designed specifically for mission-oriented drone networks.

• System Model: The simulation environment consists of a grid of trajectory points with a central base station. Tasks are distributed randomly across these points, each requiring a specific number of time steps ($T_i \geq 1$) to complete. Drones consume energy in three modes: forward flight, hovering, and task execution (which includes powering onboard facilities like sensors or neural networks). A mission is successful only if all tasks are completed and all drones possess sufficient energy to return to the base station.
• Algorithm: Each drone operates with its own DQN, featuring a policy network and a target network. The state space is five-dimensional, encompassing task locations, drone locations, actions taken, remaining task lengths, and battery levels. The action space includes moving to adjacent grid points, hovering, and executing tasks.
• Reward Formulation: The core innovation is the individual reward mode. Unlike shared reward models where all agents receive the same feedback, this model calculates rewards based on each drone's specific action and its impact on the global state. The reward function ($R_{t+1,k}$) is driven by:
  1. Task Execution Progress: The reduction in remaining task time steps ($E(t, k)$).
  2. Battery Status: A coefficient ($\mu$) representing the percentage of remaining energy.
  3. Constraints: Penalties are applied if the mission fails due to insufficient energy for return (Formula 5) or if tasks remain incomplete (Formula 4).
• Training: The model employs experience replay and an Adam optimizer. Training involves an exploration-exploitation trade-off using an $\epsilon$-greedy strategy, with hyperparameters (batch size, target network update frequency, and exploration cutoff) tuned via extensive simulation.

Key Contributions

1. Practical Scenario Modeling: The study introduces a simulation framework where tasks have non-binary lengths and require multiple time steps, differing from standard Vehicle Routing Problems (VRP) that assume fixed positions and binary task completion.
2. Energy-Aware Reward Design: The paper formulates a DQN reward function explicitly driven by task progress and battery levels, marking one of the first attempts to integrate battery capacity constraints directly into the MARL reward structure for drone networks.
3. Credit Assignment Analysis: The work systematically compares individual reward versus shared reward modes. It demonstrates that individual rewards provide clearer goal signals for agents, mitigating the credit assignment problem and improving scalability.
4. Scalability and Robustness: The proposed model is evaluated against varying task densities, task lengths, locations, and grid sizes, showing superior robustness compared to shared-reward baselines.

Results
Extensive simulations on a 5x5 grid (and scaled up to 8x8) yielded the following findings:

• Success Rate: The proposed individual reward model achieved a success rate of at least 80% across various dynamic environments. When task density was high (approaching 40% of grid points), the success rate reached nearly 100%.
• Efficiency: The model required fewer execution steps to complete missions compared to the shared reward baseline, indicating higher energy efficiency.
• Scalability: As the grid size increased (from 5x5 to 8x8), the shared reward model's performance degraded significantly, with success rates dropping and execution steps approaching the failure threshold. In contrast, the individual reward model maintained a stable success rate near 100% and showed only a slight increase in required steps, proving its robustness to environmental expansion.
• Task Density: Higher task densities generally improved learning efficiency for both models, but the individual reward mode consistently outperformed the shared reward mode, particularly in scenarios with 6 and 8 tasks.

Significance and Claims
The paper claims that its primary significance lies in bridging the gap between theoretical MARL and practical, energy-constrained drone operations. By shifting from a shared reward to an individual reward paradigm, the authors demonstrate that agents can learn more effective collaborative strategies without the ambiguity of global reward signals. The study asserts that this approach is particularly vital for scaling drone networks, as it prevents the performance collapse often seen in shared-reward systems when the number of agents or the environment size grows.

The authors modestly acknowledge limitations, noting that performance can fluctuate in specific scenarios (e.g., random locations with fixed lengths) and that the current simulation is limited to a 2D environment. They suggest future work should explore 3D environments, real-world deployment, and the integration of more advanced collision avoidance and communication techniques, while maintaining the simplicity of the current framework for embedded system compatibility.