🤖 AI

Deep Reinforcement Learning based Autonomous Decision-Making for Cooperative UAVs: A Search and Rescue Real World Application

This paper presents a first-of-its-kind end-to-end framework for autonomous cooperative UAVs in GNSS-denied indoor search-and-rescue missions, integrating APF-enhanced DRL guidance, GAT-based task allocation, and a lightweight altitude stabilization filter to achieve centimeter-level stability and secure first place in the 2024 NATO-Sapience competition.

Original authors: Thomas Hickling, Maxwell Hogan, Abdulla Tammam, Nabil Aouf

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Hickling, Maxwell Hogan, Abdulla Tammam, Nabil Aouf

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a lost child in a massive, pitch-black, multi-story maze filled with furniture, narrow hallways, and dead ends. You can't use a GPS because the walls block the satellite signals. You have two small, flying robots (drones) to help you, but they are blind without your help.

This paper is the story of how a team of researchers taught these two drones to work together as a super-smart rescue team, winning a major international competition to prove it works in the real world.

Here is the breakdown of their "magic trick," explained simply:

1. The Problem: The "Blind" Maze

Usually, drones fly using GPS (like a car's navigation). But inside a building, GPS doesn't work. The drones have to figure out where they are just by looking at the walls with cameras and lasers.

The Challenge: It's like trying to walk through a dark room while wearing a blindfold, but you have to remember where you've been and avoid bumping into chairs.
The Specific Issue: The lasers (LIDAR) were great at seeing left and right, but they kept getting confused about "up" and "down." The drones would think they were flying at the same height, but they were actually slowly drifting up or down like a balloon losing air.

2. The Solution: Three Superpowers

The team gave their drones three specific "superpowers" to solve these problems.

Superpower A: The "Ghost Guide" (Artificial Potential Fields)

Imagine you are walking toward a goal, but there are walls around you.

Old Way: You just try to walk straight to the goal. If you get too close to a wall, you panic and stop. This makes for jerky, unsafe movements.
The New Way (APF): The researchers gave the drone a "Ghost Guide." Imagine invisible forces:
- A magnet pulling the drone toward the goal.
- Repulsive balloons pushing the drone away from walls.
How it helped: The drone didn't just learn by trial and error (which takes forever). Instead, the "Ghost Guide" showed the drone the ideal smooth path. The drone's brain (AI) learned to copy this smooth path. This meant the drone flew gracefully, not like a jittery insect.

Superpower B: The "Smart Manager" (Graph Attention Network)

You have two drones and a building full of rooms to search. Who goes where?

Old Way: You might tell Drone A to go left and Drone B to go right, or let them guess. This often leads to them both going to the same room (wasting time) or missing a room entirely.
The New Way (GAT): The team built a "Smart Manager" sitting on a laptop. It looks at the whole building like a map of connected dots (a graph).
- It sees: "Drone A is close to Room 1, but Drone B is closer to Room 2."
- It uses Attention (like a spotlight) to focus on the most important tasks.
- It instantly assigns the best job to the best drone. It's like a traffic controller at an airport, ensuring no two planes crash and everyone gets to their gate efficiently.

Superpower C: The "Altitude Anchor" (Fixing the Drift)

Remember the problem where the drones drifted up or down?

The Fix: The team realized the drone's laser sensor was bad at measuring height in long, empty hallways. So, they added a second camera (a depth camera) looking straight down.
The Analogy: Think of the drone's internal gyroscope (IMU) as a person trying to count their steps to know how high they've climbed. They get tired and lose count. The depth camera is like a friend shouting, "Hey, you are actually at step 5!"
They combined the "step counter" with the "friend's shout" using a simple math filter. This kept the drone at the exact right height, centimeter by centimeter, even without GPS.

3. The Real-World Test: The Competition

The team didn't just test this in a video game. They built a real, messy, multi-story disaster zone (a "mock-up" of a collapsed building) and entered the NATO Sapience Competition.

The Mission:
1. Map the building: Fly around and create a 3D picture of the maze.
2. Deliver supplies: Find specific spots and drop off "medical aid" (simulated packages).
3. Work together: Sometimes, two drones had to land at the same time to deliver a heavy load.
The Result: Their system won First Place.
- They mapped the building 53% faster than a single drone could.
- They navigated tight corners and different floor levels without crashing.
- They proved that AI trained in a computer game could actually fly a real, heavy drone in a dangerous, real-world environment.

The Big Takeaway

This paper is important because it bridges the gap between "cool computer simulations" and "real life."

Before this, most AI drone research stayed in the computer. This team showed that if you teach the AI with the right "Ghost Guide" (APF), give it a "Smart Manager" (GAT) to coordinate, and fix its sense of height, you can have a team of robots that can save lives in a real disaster zone, even when the GPS is broken.

In short: They taught two drones to be a coordinated, height-aware, obstacle-avoiding rescue team that actually works in the real world, not just in a video game.

1. Problem Statement

The paper addresses the critical challenge of deploying autonomous Uncrewed Aerial Vehicles (UAVs) for Search and Rescue (SAR) operations in GNSS-denied indoor environments. These environments present unique difficulties:

Navigation: Lack of GPS requires robust odometry, but standard LIDAR-SLAM suffers from "Z-drift" (vertical altitude errors) in featureless corridors.
Control: Traditional discrete action spaces or simple distance-based rewards often fail to produce smooth, safe trajectories in cluttered, multi-level indoor spaces.
Cooperation: Efficiently allocating tasks between multiple agents in real-time without centralized human intervention is complex, especially when agents must avoid collisions and cover large areas.
Sim-to-Real Gap: Most Deep Reinforcement Learning (DRL) solutions for UAVs remain confined to simulation, failing to translate to real-world physics, sensor noise, and momentum constraints.

2. Methodology

The authors developed an end-to-end framework comprising three core modules deployed on two 1-meter class quad-rotors equipped with LIDAR, RGB-D cameras, and NVIDIA Jetson Orin Nano processors.

A. Autonomous Guidance (DRL with APF Shaping)

Algorithm: The system uses Twin Delayed Deep Deterministic Policy Gradient (TD3) for continuous control (velocity-level commands: $v_x, v_z, \omega$ ).
Reward Shaping (Key Innovation): Instead of a sparse reward based solely on distance to the goal, the authors introduced an Artificial Potential Field (APF)-referenced reward.
- An APF calculates an "optimal" reference action ( $a^*_t$ ) based on attractive forces (towards the goal) and repulsive forces (away from obstacles).
- The DRL agent is rewarded for aligning its executed action ( $a_t$ ) with this reference ( $r \propto 1 - ||a_t - a^*_t||^2$ ).
- This provides dense, physically meaningful feedback, accelerating convergence and ensuring smooth, collision-free trajectories.
Architecture: A multi-branch neural network fuses depth images (8 conv layers), 1D LIDAR scans (5 conv layers), and positional state vectors to output continuous velocity commands.

B. Centralized Task Allocation (Graph Attention Networks)

Approach: Task allocation is modeled as a centralized decision process using a Graph Attention Network (GAT).
Mechanism: The system constructs a graph where nodes represent UAVs and tasks (waypoints). Edge attributes encode collision-free shortest-path distances.
Function: The GAT-actor-critic learns to assign tasks by reasoning over the graph structure, dynamically prioritizing tasks based on agent proximity and state, outputting an assignment matrix in a single forward pass.

C. Robust Localization (Altitude Drift Suppression)

Problem: Standard LIDAR-SLAM accumulates vertical drift in narrow corridors due to a lack of vertical features.
Solution: A lightweight complementary filter fuses:
1. IMU vertical velocity: Provides short-term stability.
2. Depth Camera Altimetry: A median depth calculation from five regions of an Intel RealSense D455f image provides absolute height references relative to the floor.
Result: This fusion corrects Z-drift to achieve centimeter-level altitude stability without external beacons.

3. Key Contributions

End-to-End Real-World Deployment: Unlike prior work limited to simulation or small-scale labs, this is the first demonstration of a cooperative DRL policy for multi-UAV SAR in a complex, multi-level indoor disaster mock-up.
APF-Referenced Action Alignment: A novel reward shaping technique that blends APF guidance with continuous DRL control. It outperformed standard distance-to-goal baselines, enabling stable training and deployment where others failed.
GAT-Driven Cooperative Allocation: A centralized task allocator that uses Graph Attention Networks to reason over drone-task graphs, producing near-optimal allocations with negligible onboard compute.
Z-Drift Mitigation: A practical sensor fusion method (Depth Camera + IMU) that solves the notorious vertical drift issue in indoor LIDAR-SLAM, enabling reliable multi-level navigation.

4. Results and Performance

The system was tested in the NATO SAPIENCE Autonomous Cooperative Drone Competition (2024), where the team secured 1st place.

Simulation Validation: The APF-rewarded model achieved 304/304 goals in validation, whereas standard distance-based rewards failed to produce a deployable controller under the same constraints.
Real-World Mapping:
- Two UAVs completed a 3D mapping task of a complex building in 134 seconds.
- A single UAV took 287 seconds for the same task, demonstrating a 53.3% reduction in time due to cooperation.
- The system successfully generated coherent 3D point clouds despite GNSS denial.
Delivery Task: The UAVs successfully completed 8 deliveries, including synchronized landings on different elevation levels (0.2m floor and 1.2m platform), demonstrating precise altitude control and coordination.
Alternative Layout: On a secondary layout, the GAT-DRL allocator reduced total travel distance by 19.4% and makespan by 19.8% compared to classical greedy and auction-based methods.
Localization: The depth-IMU fusion successfully maintained accurate altitude estimates, correcting the raw LIDAR-SLAM drift which otherwise failed to track height changes.

5. Significance and Impact

Bridging the Sim-to-Real Gap: The paper proves that DRL policies, when properly shaped with domain knowledge (APF) and validated against real-world physics (momentum, sensor noise), can scale from simulation to full-scale, safety-critical operations.
Operational Viability: The system demonstrates that autonomous cooperative UAVs can perform complex SAR tasks (mapping, detection, delivery) in GNSS-denied environments without human intervention, significantly reducing risk to human responders.
Scalability: The framework provides a blueprint for future multi-agent systems, showing that centralized graph-based allocation and decentralized continuous control can coexist effectively in cluttered, dynamic environments.
Future Roadmap: The work lays the foundation for future outdoor/mixed-environment operations, decentralized task allocation (peer-to-peer), and collaborative perception to further reduce drift and improve safety.

In conclusion, this work represents a significant leap forward in autonomous robotics, moving DRL from theoretical simulation to a proven, winning solution for real-world disaster response scenarios.