QuadAgent: A Responsive Agent System for Vision-Language Guided Quadrotor Agile Flight

1. Problem Statement

The paper addresses the critical challenges in deploying autonomous quadrotors that can interpret complex natural language instructions and perform agile flight in real-world environments. Existing approaches face three main limitations:

• End-to-End Limitations: Models that map language directly to control actions require massive amounts of expert flight data for training and lack explicit memory or structured scene priors, hindering long-horizon reasoning.
• Serial "Stop-and-Infer" Bottlenecks: Traditional agent architectures (e.g., ReAct) operate sequentially. The drone must halt its physical motion while the LLM/VLM reasons and invokes tools. This blocking logic creates significant latency, making the system unresponsive to dynamic inputs and inefficient for agile flight.
• Mismatched Characteristics: UAVs have limited flight time and require fast reaction times, whereas foundation models are computationally heavy and slow. Furthermore, 3D navigation requires different reasoning than 2D visual tasks.

The core problem is how to achieve non-blocking, responsive interaction where high-level semantic reasoning occurs concurrently with low-level physical flight control, allowing the drone to navigate cluttered spaces at high speeds (up to 5 m/s) while processing complex conditional instructions.

2. Methodology

The authors propose QuadAgent, a training-free, asynchronous multi-agent system that decouples high-level reasoning from low-level control.

A. Asynchronous Multi-Agent Architecture

The system separates the cognitive layer (ground station) from the physical layer (onboard UAV) using an event-driven, non-blocking protocol:

• Foreground Workflow Agents: Handle active tasks and user commands. They include an Orchestrator (routes control flow), a Planner (decomposes instructions into task trees), and an Executor (handles atomic sub-tasks).
  • Mechanism: Instead of blocking, the Executor sends a path command to the physical layer and immediately yields control, entering an idle state. This allows the system to monitor user inputs ($\epsilon_{usr}$) and physical feedback ($\epsilon_{phy}$) in parallel with flight execution.
• Background Agents: Perform look-ahead reasoning. While the Foreground agents handle the current task, Background agents asynchronously pre-execute upcoming sub-tasks (e.g., searching the map for the next target) using mnemonic skills.
  • Benefit: Retrieved context is cached in a shared buffer and injected into the Executor's prompt when needed, effectively masking inference latency and reducing "hovering" time.

B. The Impression Graph (Scene Memory)

To replace dense 3D reconstruction (which is computationally expensive) and static scene graphs (which lack flexibility), the authors introduce the Impression Graph:

• Structure: A lightweight topological map ($G_{imp}$) built from sparse keyframes. Nodes contain multimodal data (visual streams, pose, CLIP embeddings, semantic captions).
• Connectivity: Edges are established not just by data collection paths, but via a pyramidal frustum intersection criterion. By projecting depth map patches into geometric pyramids, the system calculates volumetric overlap to determine traversability. This allows the discovery of "aerial shortcuts" (e.g., flying over obstacles) that were not traversed during initial mapping.
• Semantic Enrichment: Nodes are annotated using a multi-depth masked prompting pipeline (Near, Far, Full views) to generate accurate spatial captions, reducing hallucinations.

C. Skill Library & Physical Layer

• Skills: Agents invoke skills on demand, including Mnemonic (retrieving from the Impression Graph), Perceptual (reading state or analyzing live images), and Navigation (computing topological paths to waypoints).
• Physical Control: The onboard layer runs a finite state machine (Hover, Align, Avoid, Align-End). It utilizes a differentiable, vision-based obstacle avoidance policy ($\pi_\theta$) trained via differentiable physics. This acts as a safety net, ensuring collision avoidance at 30Hz even if the high-level agent is delayed or fails.

3. Key Contributions

1. QuadAgent System: A training-free framework integrating foundation models (LLM/VLM) with low-level vision-action networks for agile flight, capable of handling complex, conditional natural language instructions.
2. Asynchronous Multi-Agent Architecture: A novel design that decouples reasoning from execution. By using Background Agents for pre-execution and a suspend-resume protocol, it eliminates the "stop-and-infer" bottleneck, enabling continuous flight during reasoning.
3. Impression Graph: A lightweight topological representation that enables agile 3D navigation based on 2D visual reasoning. It supports dynamic shortcut discovery and avoids the computational overhead of dense 3D reconstruction.

4. Results

The system was evaluated in simulation (AirSim) and real-world experiments.

• Simulation Performance:
  • Success Rate (SR): QuadAgent achieved 80.0%, outperforming baselines like Serial Plan-ReAct (78.3%) and Scene Graph Agent (15.0%).
  • Efficiency: It reduced Hovering Time (HT) to 118.0s (vs. 249.9s for Serial Plan-ReAct) and achieved 58.3% Interaction Availability (IA), proving its non-blocking nature.
  • Path Planning: By discovering shortcuts via frustum intersection, QuadAgent achieved a higher Success weighted by Path Length (SPL) of 58.6 compared to 49.8 for trajectory-based baselines.
• Real-World Experiments:
  • Agility: The drone successfully navigated cluttered indoor spaces at speeds up to 5.1 m/s.
  • Robustness: It handled unmapped dynamic obstacles using the vision-based avoidance policy.
  • Responsiveness: In experiments, the system successfully processed conflicting user commands (re-routing mid-flight) and informational queries (answering "Where is the drone?" while flying) without interrupting the trajectory.

5. Significance

QuadAgent represents a significant leap forward in autonomous aerial robotics by solving the latency-agility conflict.

• Paradigm Shift: It moves away from the "stop-and-infer" paradigm common in current LLM-robotics research, demonstrating that complex reasoning can occur while the robot is in motion.
• Scalability: Being training-free and relying on lightweight topological maps rather than dense 3D reconstruction, the system is more scalable and adaptable to new environments without retraining.
• Safety & Speed: The integration of a reactive safety layer allows for high-speed flight (5 m/s) in unstructured environments, a speed rarely achieved by semantic navigation agents which typically operate below 2 m/s.

This work bridges the gap between high-level cognitive capabilities of foundation models and the low-level, real-time demands of agile physical flight.