Technical Summary: Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly utilized as reasoning modules for embodied agents, mapping natural language instructions and visual observations to actions. However, it remains unclear how well frozen, general-purpose MLLMs can execute complex, long-horizon embodied tasks from a single high-level instruction without task-specific adaptation. Existing benchmarks, such as Vision-Language Navigation (VLN) or object navigation, typically evaluate path-level instructions or decomposed subgoals. These settings fail to assess whether MLLMs can coordinate perception, planning, control, and reporting in a single closed-loop episode. Aerial environments present a particularly demanding testbed due to viewpoint sensitivity, open-endedness, and the need for continuous pose control, yet current aerial benchmarks largely focus on route following or isolated object search rather than full mission execution.

Methodology: MissionBench

To address these gaps, the authors introduce MissionBench, a benchmark designed for mission-level evaluation of MLLMs in aerial 3D environments.

Benchmark Construction

• Scope: The benchmark comprises 120 missions distributed across five high-fidelity simulated environments (Neighborhood, City, Forest, Savannah, and AirSimNH) built on Unreal Engine 5 and Cosys-AirSim.
• Task Families: Missions are categorized into four families grounded in real-world UAV operations:
  1. Reporting: Extracting information from observations (e.g., reading a license plate).
  2. Inspection: Reaching an observation-ready configuration to view an object (e.g., inspecting a fire).
  3. Manipulation: Performing physical actions like payload delivery or landing.
  4. Patrol: Following routes along environmental structures.
• Input/Output: Each mission begins with a single natural-language instruction. The agent receives egocentric RGB images (current and up to two previous frames) and action history. It must output a structured response including a bounding box, reasoning trace, an action primitive with continuous magnitude (direction and distance/angle), and a completion flag.
• Control: Unlike prior works using discrete primitives (e.g., fixed 3m steps), MissionBench requires agents to predict continuous action magnitudes, allowing for finer viewpoint selection and control.
• Evaluation: A closed-loop framework evaluates agents using multiple metrics:
  • Success Rate (SR): Task-specific completion (e.g., correct text extraction, proximity thresholds, or trajectory overlap for patrol).
  • Oracle Success Rate (OSR): Whether the agent ever passed within a threshold distance of the target, distinguishing navigation failure from perception/reporting failure.
  • Mission Progress (MP): Continuous measure of progress toward the objective.
  • Step Efficiency (Eff) & Collision Rate (CR): Measures of resource usage and spatial awareness.

Experimental Setup

The authors evaluated 22 open- and closed-source MLLMs in a zero-shot setting (no aerial-specific fine-tuning). Models included various sizes of Gemini, GPT, Claude, Qwen, and InternVL families. Human baselines were established using both the same text-action interface and continuous keyboard control.

Key Results

• Performance Gap: Even the strongest model, Gemini 3.1 Pro, achieved a Success Rate (SR) of only 34.8%, significantly lower than the 84.4% achieved by humans using continuous keyboard control (and 70.0% for humans restricted to the MLLM interface). This highlights the difficulty of multi-step embodied tasks for current general-purpose models.
• Scaling Trends: Performance generally improves with model scale within families (e.g., Gemini 3.1 Pro outperforms Flash variants; Qwen 27B outperforms 2B). This suggests that larger general-purpose models possess stronger zero-shot embodied capabilities.
• Perception vs. Execution: A strong correlation between single-frame spatial perception (bounding box localization and distance estimation) and end-to-end mission success was not found. High-performing models in perception proxies did not necessarily succeed in missions, and vice versa. This indicates that mission success requires capabilities beyond static perception, including multi-step planning and adaptive reasoning.
• Failure Modes: Analysis revealed distinct failure patterns:
  • Premature Termination: Agents declaring success while far from the target.
  • Drift and Oscillation: Agents exhausting step budgets by circling landmarks without converging.
  • Unmet Objectives: Agents reaching the target vicinity but failing to extract the correct information (high OSR, low SR).
• Ablation Studies: Experiments on design choices (temperature, resolution, history length, output fields) showed that mission success is highly sensitive to the structured output format (specifically the reasoning field) and moderate stochasticity (temperature), while higher resolution primarily aids final task completion rather than coarse navigation.

Significance and Claims

The paper claims that MissionBench provides the first joint evaluation of frozen, off-the-shelf MLLMs across multiple mission families with continuous-magnitude pose control and explicit reporting in a closed-loop setting.

The authors argue that:

1. Mission-level competence is multifaceted: It requires coordinating spatial perception with multi-step planning and adaptive reasoning, capabilities that current MLLMs struggle to integrate without specific training.
2. Scaling offers promise and risk: While scaling improves zero-shot embodied capabilities, the gap between simulation performance and real-world reliability remains. The emergence of these capabilities in general-purpose models raises concerns about safety and uncontrolled capability emergence in real-world systems.
3. Evaluation necessity: Standard proxy tasks (like static perception) are insufficient predictors of embodied mission success. Closed-loop, mission-level evaluation is necessary to diagnose specific failure modes such as control instability or premature termination.

The work concludes that while general-purpose MLLMs show potential for embodied AI, they are not yet ready for reliable deployment in safety-critical aerial missions without further development in reasoning, planning, and safety constraints.