Drones that Think on their Feet: Sudden Landing Decisions with Embodied AI

1. Problem Statement

Autonomous drones operating in dynamic urban environments face critical challenges when sudden events occur (e.g., GPS spoofing, sensor degradation, or adverse weather). Traditional recovery strategies rely on pre-defined "safe sets" or hand-coded rules, which are brittle and cannot anticipate novel or evolving contingencies.

• The Core Challenge: When an emergency arises, a drone must immediately identify a safe landing spot that is not pre-specified. This requires common-sense reasoning to interpret complex scenes (e.g., distinguishing a flat but dangerous highway from a flat but safe rooftop) and to adapt to dynamic changes (e.g., moving traffic) in real-time.
• Limitations of Existing Work: Previous methods either assume the target is known in advance, rely on specific failure models, or use deep learning models trained on static datasets that lack semantic understanding of context and safety constraints.

2. Methodology

The authors propose a hybrid, modular pipeline that integrates Large Visual-Language Models (LVLMs) for high-level semantic reasoning with traditional perception and control modules for geometric precision and safety.

A. System Architecture

The pipeline consists of three interconnected modules (Figure 1.2):

1. Surface ID Module (Perception):
   • Uses a downward-facing monocular RGB camera and a LiDAR sensor.
   • Employs a lightweight depth estimation model (Depth Anything V2) to generate dense depth maps.
   • Identifies flat, planar regions by thresholding gradient magnitudes and applying morphological filtering.
   • Outputs cropped image patches of candidate surfaces to reduce the LVLM's search space and prevent hallucinations.
2. LVLM Ranking Module (Reasoning):
   • Role: Acts as the "human operator," applying common-sense reasoning to evaluate candidates.
   • Two-Stage Process:
     • Ranking: Evaluates up to 5 cropped candidate images to select the best landing spot based on flatness, obstacle presence, and context (e.g., avoiding highways).
     • Confirmation: Once the drone moves to the target, the LVLM re-evaluates a close-up view to ensure the site remains safe (checking for dynamic obstacles like moving cars).
   • Prompt Engineering: Uses Chain-of-Thought prompting with strict safety constraints (e.g., "MUST be clear of people") to minimize hallucinations.
3. Movement Planner Module (Control):
   • Converts the LVLM's 2D image selection into 3D world coordinates using Inverse Perspective Mapping (IPM) and LiDAR data.
   • Translates coordinates to the North-East-Down (NED) frame for the drone's controller (PID/SimpleFlight).
   • Executes the maneuver and manages the descent, only proceeding if the LVLM confirms safety.

B. Deployment Strategy

The paper explores three deployment tiers to balance latency and reasoning capability:

• Onboard: Lightweight models (e.g., GPT-5 Nano, InternVL) for low latency but potentially lower reasoning depth.
• Edge: Mid-sized models on ground stations or nearby UAVs.
• Cloud: Large, powerful models (e.g., GPT-5, Gemini 2.5) for maximum reasoning, accepting higher latency.

C. Experimental Setup

• Simulator: Unreal Engine 5 (City Sample) integrated with Cosys-AirSim for realistic physics, sensors, and dynamic urban environments (moving cars, pedestrians, weather).
• Models Tested: GPT-5, GPT-5 Mini, GPT-5 Nano, Gemini 2.5 Flash, Mistral 3 Large, and InternVL 2.5.
• Scenarios: Curated scenarios with clear safe/unsafe options, adverse weather (fog/rain), and open-world city-wide tests using Halton sequence sampling for diverse launch points.

3. Key Contributions

1. LVLM-Driven Pipeline: A novel architecture combining perception, LVLM-based semantic reasoning, and traditional control for adaptive emergency landing.
2. Reproducible Benchmark: An open-source benchmark (Airsim-closeloop) featuring realistic, dynamic urban scenarios with configurable obstacles and weather conditions.
3. Empirical Validation of Embodied AI: Demonstration that embodied AI enables recovery behaviors (e.g., rejecting a "flat" highway due to traffic risk) that are infeasible to design with hand-coded rules.
4. Deployment Trade-off Analysis: A comprehensive study on the latency vs. accuracy trade-offs across different model sizes and deployment tiers (Cloud vs. Edge vs. Onboard).

4. Results

• Module Performance:
  • The Surface ID Module achieved high accuracy (Jaccard Index ~87% in simple scenarios, ~60% in complex ones).
  • The Movement Planner guided the drone within ~1 meter of the target.
• LVLM Performance:
  • Large Models (GPT-5, Gemini 2.5): Achieved near-perfect success rates (>95%) in selecting safe rooftops and avoiding hazards (roads, pedestrians) across all scenarios, including adverse weather.
  • Small Models (GPT-5 Nano, Mistral): Struggled significantly (20–30% success in some cases), often misclassifying textures (e.g., labeling shingles as "uneven" or missing traffic on highways). They were highly sensitive to context; providing full images improved Nano's performance but it still lagged behind larger models.
• Full Pipeline in Open Environments:
  • In 20 randomized city-wide trials, the system achieved a 75% safe landing rate.
  • Failure Modes: Included landing on obstructed rooftops (HVAC units), highways with hidden traffic, and timeouts caused by the LVLM hallucinating non-existent obstacles (e.g., "birds" on a roof).
  • Efficiency: The pipeline is lightweight (avg. 715 MB RAM, 0.31 GB GPU), allowing it to run on standard laptops, though cloud-based LVLMs introduced latency (up to 25s for GPT-5).
• Latency: Cloud models (Gemini 2.5) were fastest (3.7s), while GPT-5 was slowest (25s). Edge models (InternVL) offered a middle ground (~8.5s).

5. Significance and Future Work

• Significance: This work proves that Embodied AI can bridge the gap between rigid control systems and the unstructured nature of real-world emergencies. It moves beyond "pre-defined safe sets" to dynamic, context-aware decision-making, significantly enhancing the resilience of autonomous aerial systems.
• Key Insight: Larger models are currently required for reliable safety reasoning, but their latency and compute requirements pose challenges for onboard deployment.
• Future Directions:
  • Hierarchical Inference: Combining lightweight onboard geometric checks with edge/cloud LVLMs.
  • Uncertainty Quantification: Explicitly modeling LVLM confidence to trigger fallback heuristics when the model is unsure.
  • Adversarial Robustness: Hardening the system against visual perturbations and deceptive inputs.

In conclusion, the paper demonstrates that while LVLMs introduce latency and resource challenges, their ability to reason about complex, dynamic safety constraints makes them a critical component for the next generation of trustworthy, autonomous drones.