Technical Summary: MultiUAV-Plat

Problem Statement

While Large Language Models (LLMs) show promise for high-level robotic task planning, their application to Multi-UAV collaborative task planning lacks systematic evaluation frameworks. Existing UAV simulators (e.g., AirSim, Flightmare) focus on high-fidelity physics, perception, and low-level control, which are often inaccessible to LLM agents. Conversely, existing LLM-agent benchmarks (e.g., ALFWorld, AgentBench) rarely capture critical aerial-robotics constraints such as partial observability, spatial coverage, UAV assignment, and multi-vehicle coordination.

Current research often assumes agents have access to full simulator states or structured optimization objectives. However, a realistic LLM-controlled UAV agent must interpret natural-language objectives, select appropriate UAVs, actively collect missing local observations, issue executable API calls, and verify progress under hidden mission conditions without privileged global state access. There is a need for a reproducible benchmark that evaluates agents through restricted APIs and hidden validators rather than offline question answering or full-information prompting.

Methodology

The authors propose a three-component methodology: a simulation platform, a benchmark suite, and a specialized agent framework.

1. MultiUAV-Plat (Simulation Platform)

MultiUAV-Plat is a lightweight, open-source, LLM-agent-oriented simulation platform designed to bridge the gap between natural language and executable UAV control.

• Architecture: The platform consists of a Model Layer (encapsulating Drone, Target, Obstacle, Environment, Session, and Task entities), a Control Layer (handling lifecycle management, collision detection, and logical arbitration), and a Service Layer (providing RESTful APIs).
• Agent Interface: Agents interact via concise, tool-like RESTful APIs for actions (takeoff, landing, waypoint navigation, sensing, photography) and receive agent-facing observations. Crucially, the platform enforces partial observability: agents only see information within a UAV's local perception radius. Global targets, obstacles, and validation logic are hidden.
• Security & Visualization: A role-based permission system prevents agents from accessing privileged global data. Optional 2D (pygame-based) and 3D visualization views allow for human-in-the-loop inspection without compromising the agent's restricted information boundary.

2. MultiUAV-Plat Benchmark

Built on the platform, this benchmark suite evaluates LLM agents under realistic constraints.

• Scale: It contains 75 mission sessions, 1,500 natural-language tasks, and 9,396 hidden validation checks.
• Scenario Families: The benchmark covers three primary domains:
  1. Area Search: Spatial exploration and coverage of circular/polygonal regions.
  2. Area Assignment and Patrol: Mixed missions involving discrete point targets and area coverage/patrol.
  3. Target Assignment: Allocation of multiple targets across UAVs for reaching, inspecting, or photographing.
• Difficulty: Sessions are stratified into five difficulty levels (Easy to Extreme), varying fleet size (3–30 UAVs), target density (3–100), and obstacle complexity.
• Evaluation Protocol: Tasks are scored via hidden platform-side validators. Metrics include Task Pass Rate, Average Check Pass Rate (per-task), Global Check Pass Rate (aggregate), and Total Failed Task Rate.

3. Agent4Drone (Framework)

To address the specific challenges of multi-UAV collaboration, the authors propose Agent4Drone, a task-specific LLM agent framework. Unlike generic workflows (e.g., ReAct), Agent4Drone explicitly structures behavior into six modules:

1. Observation: Active gathering of local perception and UAV states.
2. Task Understanding: Structuring natural-language instructions into intents and constraints.
3. Memory: A shared blackboard for tracking state, assignments, and history without accessing hidden validation rules.
4. Planning: Decomposing tasks into short-horizon subtasks, selecting UAVs based on battery/position, and avoiding coordination conflicts.
5. Execution: Grounding plans into bounded tool chains (action sequences).
6. Verification: Comparing expected vs. observed states and triggering replanning or retry logic upon failure.

Key Results

Experiments compared Agent4Drone against a standard ReAct baseline across multiple backend LLMs (doubao-2-pro, qwen3.5, deepseek-v4-pro/flash) on the full benchmark.

• Performance Gains: Using the doubao-2-pro backend, Agent4Drone achieved a 57.93% Task Pass Rate, significantly outperforming ReAct at 30.60% (a gain of 27.33 percentage points).
• Check-Level Metrics: Agent4Drone improved the Average Check Pass Rate from 47.91% to 74.58% and the Global Check Pass Rate from 43.15% to 71.96%.
• Failure Reduction: The Total Failed Task Rate (tasks with zero passed checks) was reduced from 32.40% (ReAct) to 12.93% (Agent4Drone).
• Robustness: With stronger backends (deepseek-v4-pro), Agent4Drone maintained a lead (70.27% vs. 63.60% for ReAct), demonstrating consistent improvements across difficulty levels and scenario families.
• Failure Analysis: While Agent4Drone excelled in movement and route-following checks, persistent challenges remained in target grounding, final-condition verification, and coordinated execution under partial observability, particularly for complex target-assignment tasks.

Significance and Claims

The paper claims that MultiUAV-Plat and its associated benchmark provide the first reproducible foundation for studying LLM-driven multi-UAV autonomy under realistic information and execution constraints.

• Bridging the Gap: It addresses the disconnect between high-level LLM reasoning and the low-level, restricted interfaces of real-world UAV operations.
• Executable Evaluation: By enforcing hidden validators and partial observability, the benchmark moves beyond "conversational compliance" to evaluate closed-loop, executable mission planning.
• Framework Utility: Agent4Drone demonstrates that structuring LLM agents around specific mission constraints (memory, partial observation, bounded replanning) yields substantial performance gains over generic prompting strategies.
• Future Direction: The work establishes a baseline for future research in agent workflow design, planning under partial observability, and collaborative mission execution, while acknowledging that challenges in target localization and final verification remain open problems.