Technical Summary: VLN-Pilot

Problem Statement

Vision-and-Language Navigation (VLN) has advanced significantly for ground-based robots, yet its application to autonomous indoor drones remains a challenging frontier. Indoor drone navigation faces unique constraints, including confined spaces, dynamic obstacles, and the absence of GPS signals. Currently, human-in-the-loop piloting is the standard for many industrial applications. The paper addresses the need for a system that can interpret free-form natural language instructions and ground them in visual observations to plan and execute drone trajectories in GPS-denied indoor environments, aiming to reduce operator workload while maintaining safety and mission flexibility.

Methodology

The authors propose VLN-Pilot, a framework where a Large Vision-Language Model (VLLM) acts as a high-level supervisory controller for an indoor drone. The system operates within a custom photorealistic Unity-based simulation replicating a DJI Tello drone. The architecture consists of three primary modules:

1. Unity Simulator: Provides the virtual environment (a furnished cabin with a living room/kitchen, bedroom, and bathroom). It generates observations including frontal and rear RGB images, drone position (X, Y, Z), yaw orientation, and collision status via the ML-Agents framework.
2. Python Controller (FSM): Manages low-level execution using a Finite State Machine (FSM). The FSM defines semantic states such as "Recognize room," "Search open door," "Orient towards door," "Go through door," "Search object," and "Reach object." The controller handles communication between the simulator and the VLLM, parses the model's output, and executes predefined motion commands (e.g., forward 10/25/50cm, rotate 15/45/90 degrees).
3. VLLM Module: Acts as the "pilot." It receives a structured prompt containing the user's natural language instruction, the current visual observation, the topological map (room connectivity graph), the current FSM state, and the previous action. Based on this input, the VLLM (tested with GPT-4.1 and Gemini-2.5-flash) determines:
   • The next movement command.
   • The next discrete state in the FSM.
   • The current room identification.
   • A brief visual description and reasoning for the decision.

The system relies on a topological map rather than precise metric coordinates, defining nodes as rooms and edges as navigable connections.

Key Contributions

• Novel Framework: Introduction of VLN-Pilot, which integrates VLLMs with a rule-based FSM to enable autonomous indoor drone navigation driven by natural language.
• Hybrid Architecture: A design that leverages the high-level reasoning and semantic understanding of VLLMs for decision-making while offloading low-level stability and collision avoidance to a deterministic state machine.
• Benchmarking: Evaluation of the framework on a custom photorealistic indoor simulation benchmark involving complex, long-horizon instruction-following tasks (e.g., navigating to specific rooms and locating objects like refrigerators, mirrors, or sinks).
• Empirical Analysis: A comparative study of GPT and Gemini models in the context of drone control, highlighting specific behavioral differences in spatial reasoning and state transitions.

Experimental Results

The system was evaluated over five repetitions for various queries starting from different spawn points. Key findings include:

• Performance Comparison: The GPT model generally outperformed the Gemini model. For instance, in the task "Go to the bedroom" starting from the living room, GPT achieved a 4/5 success rate compared to Gemini's 0/5. Similarly, for "Find the refrigerator," GPT succeeded 4/5 times while Gemini failed all 5 attempts.
• Failure Modes:
  • Gemini: Exhibited a tendency toward "oscillatory re-centering." It interpreted "being near the door" as requiring a large distance and strict centering within the field of view. This led to repeated alignment adjustments that prevented the drone from crossing. When prompts were modified to encourage closer approach, the drone frequently collided with door frames or propellers due to a lack of volumetric awareness.
  • GPT: Demonstrated more confident crossing behaviors with tighter proximity thresholds. However, it also occasionally failed to account for the drone's physical dimensions relative to door clearances, leading to minor collisions (e.g., propeller clips) despite perceived correct alignment.
• Success Rates: Both models achieved high success rates (5/5) in simpler tasks, such as returning to the starting room or navigating to the bathroom from the bathroom, but struggled with complex multi-step navigation and object localization in the presence of spatial reasoning limitations.

Significance and Claims

The paper claims that VLN-Pilot demonstrates the potential of replacing remote human pilots with language-guided autonomous agents for indoor UAVs. The results suggest that VLLMs can significantly reduce operator workload while maintaining effective navigation performance in constrained environments.

The authors modestly conclude that while VLLMs show promise, they currently lack true volumetric awareness and precise spatial reasoning regarding physical dimensions (e.g., drone size vs. door width). They emphasize that relying solely on textual prompt adjustments to fix navigation issues (like oscillation) can inadvertently introduce new failure modes (like collisions). The work opens avenues for scalable, human-friendly control in tasks such as inspection, search-and-rescue, and facility monitoring, but highlights the necessity for integrating structured spatial constraints and volumetric awareness into future VLLM-driven navigation frameworks.