Technical Summary: DSSE – A Drone Swarm Search Environment

Problem Statement

The paper addresses the critical challenge of locating missing individuals in vast oceanic environments, a task complicated by the complexity of marine conditions and the sheer scale of search areas. While drone swarms have emerged as a promising tool for search and rescue (SAR), the development of effective multi-agent reinforcement learning (RL) algorithms for this domain is hindered by a lack of accessible, standardized simulation environments. Previous research has often relied on proprietary, non-public environments, creating barriers to reproducibility, transparency, and collaborative advancement in the field. The authors identify this gap as a significant limitation, preventing researchers from validating and building upon existing work.

Methodology

To address these limitations, the authors developed DSSE (Drone Swarm Search Environment), a simulation environment built upon the PETTINGZOO library, designed to interface with multi-agent and single-agent reinforcement learning algorithms. The environment simulates a scenario where drone agents must locate a shipwrecked person without prior knowledge of the target's exact position or direct distance-based rewards.

Core Mechanics and Simplifications

The environment employs several strategic simplifications to manage complexity while maintaining relevance:

• Grid-Based Simulation: The search area is discretized into a grid. Drones move in cardinal directions (up, down, left, right) or perform a "search" action. Diagonal movement is excluded to simplify the model.
• Target Dynamics: The scenario assumes a single shipwrecked person. The target's movement is not modeled via complex fluid dynamics but rather through a directional vector representing ocean currents. The target's position is determined stochastically based on a probability matrix, where the target tends to move toward adjacent cells with higher probabilities, simulating survival-driven decision-making.
• Probability Matrix: This is the primary input for agents. It represents the likelihood of the target being in a specific cell.
  • Initialization: The matrix begins with 100% probability at the target's last known location.
  • Evolution: Over time, the probability disperses according to a directional vector (current) and a growing uncertainty radius.
  • Calculation: Probabilities are distributed using a 2D Gaussian function. The values are normalized so the sum equals one, creating a dynamic probability map that agents must interpret.

Reward Structure

The environment utilizes a sparse and penalized reward system to guide agent behavior:

• Exploration: Agents receive a small positive reward (+1) for standard movement to incentivize grid exploration.
• Penalties: Severe negative rewards are applied for leaving the grid (−100,000), colliding with other agents (−100,000), or exceeding the timestep limit (−100,000). A moderate penalty (−1,000) is applied for failing to find the target within the episode.
• Search Actions:
  • Searching a cell with <1% probability yields a penalty (−100).
  • Searching a cell with ≥1% probability yields a reward proportional to the cell's probability value.
  • Successfully locating the target yields a reward calculated by the equation $r = 10000 + 10000 \times (1 - \frac{timestep}{timestep\_limit})$, incentivizing rapid discovery.

Implementation

The environment is implemented as a Python package adhering to PETTINGZOO standards, ensuring compatibility with existing RL algorithms. The source code is publicly available on GitHub, providing documentation, installation instructions, and usage examples.

Key Contributions

1. Open-Source Environment: The primary contribution is the release of a public, standardized simulation environment for drone swarm SAR, addressing the reproducibility crisis caused by proprietary tools in previous literature.
2. Dynamic Probability Input: The system introduces a mechanism where agents receive dynamic probability maps rather than direct distance rewards, forcing the development of algorithms capable of handling probabilistic inputs and uncertainty.
3. Standardization: By aligning with PETTINGZOO, the tool allows researchers to test and compare different RL algorithms within a consistent framework without needing to rebuild the simulation logic.

Results and Claims

The paper does not present experimental results comparing specific RL algorithms or claiming superior performance over existing methods. Instead, the "results" are the successful design, implementation, and release of the tool itself. The authors note that a peer-reviewed paper describing version 2 of this software, which includes expanded functionality, has been published separately [6].

The current preprint serves as a description of the initial design and release. The authors explicitly state that the simulation is a simplified representation of reality, noting limitations such as:

• The target does not exit the grid boundaries but bounces off corners.
• Drone movement is discrete rather than continuous.
• Ocean and target movement models are simplified and may not fully capture real-world complexities.

Significance

The paper positions DSSE as a foundational tool to foster collaboration and transparency in reinforcement learning research for maritime search and rescue. By providing a public, standardized environment, the authors aim to enable the research community to validate existing solutions and develop more sophisticated algorithms for finding shipwrecked individuals. The significance lies not in a specific algorithmic breakthrough presented in this text, but in the removal of barriers to entry for researchers wishing to tackle this complex, real-world problem. The authors conclude that this open approach encourages external modification and improvement, potentially driving future algorithmic advancements in the field.