OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency

OnFly is a fully onboard, real-time framework for zero-shot aerial vision-language navigation that employs a shared-perception dual-agent architecture, hybrid memory, and semantic-geometric verification to overcome existing limitations in decision stability and safety-efficiency trade-offs, achieving a task success rate of 67.8% in simulation.

The Gist

Imagine you are teaching a drone to fly through a complex, unfamiliar city based only on a spoken sentence like, "Fly to the red mailbox, then circle the fountain, and stop near the blue bench."

This is the challenge of Aerial Vision-Language Navigation (AVLN). The drone needs to "see" the world, "understand" your words, and "fly" safely without crashing.

The paper introduces OnFly, a new system that acts like a super-smart, self-contained pilot for drones. Here is how it works, broken down into simple concepts and analogies.

The Problem: The "Overworked Driver"

Previous attempts to make drones do this had three main flaws, like a driver who is trying to do too many things at once:

1. The "Multitasking Mess": Old systems tried to do two very different jobs at the same time: steering the wheel (high-speed decisions) and checking the GPS map (slow, big-picture planning). Because these jobs happen at different speeds, the system got confused, stuttered, and made bad decisions.
2. The "Short Memory": To know if it's making progress, the drone needed to remember the whole trip. But old systems used a "sliding window" memory—like a tape recorder that constantly erases the beginning of the song to make room for new notes. Eventually, the drone forgot where it started, got lost, and couldn't tell when to stop.
3. The "Safety vs. Speed" Trap: To be safe, drones would hover and wait (very slow). To be fast, they would fly blindly toward a target (very dangerous). They couldn't be both safe and efficient.

The Solution: OnFly

OnFly solves these problems with three clever tricks, turning the drone into a highly organized team rather than a confused individual.

1. The "Dual-Agent" Team (The Driver and the Navigator)

Instead of one brain trying to do everything, OnFly splits the work into two specialized agents that share the same eyes but have different jobs:

• The Driver (High-Frequency Agent): This agent is the reflex. It looks at the camera and instantly decides, "Turn left now," or "Go forward." It doesn't worry about the big picture; it just keeps the drone moving smoothly.
• The Navigator (Low-Frequency Agent): This agent is the strategist. It checks the map every few seconds to ask, "Are we close to the mailbox yet? Did we get lost?"
• The Magic: Because they don't fight for the same computer resources, the drone never stutters. The Driver keeps the plane steady, while the Navigator ensures they are going the right way.

2. The "Hybrid Memory" (The Photo Album vs. The Scrapbook)

To remember the whole journey without getting confused, OnFly uses a special memory system:

• The Old Way: A "Sliding Window" is like a scrapbook where you keep tearing out old pages to add new ones. You lose the context of the start of the trip.

• The OnFly Way: It uses a Hybrid Memory. Imagine a photo album that always keeps:

  1. The First Photo (where you started).
  2. Key Snapshots (important landmarks you passed).
  3. The Current View (what you see right now).

  This way, the drone never forgets where it began, but it also doesn't get bogged down by remembering every single second of the flight. It keeps the "prefix" of the memory stable, which makes the computer think much faster.

3. The "Safety Double-Check" (The Spotter and the Pilot)

When the AI suggests a target (e.g., "Fly to that tree"), it might be wrong. Maybe the tree is behind a wall, or the depth is wrong. OnFly adds a safety layer:

• The Semantic-Geometric Verifier: Before the drone flies, this module acts like a spotter. It checks: "Does that tree actually look like the one in the instructions? Is there a wall in the way?" If the AI is hallucinating, this step corrects the target.
• The Receding-Horizon Planner: Once the target is safe, a planner calculates the smoothest, crash-free path to get there, avoiding obstacles like a skilled pilot weaving through traffic.

The Results: From "Clumsy" to "Pro"

The researchers tested OnFly in a virtual world and then on a real drone flying outdoors.

• Success Rate: Old methods succeeded only about 26% of the time. OnFly succeeded 68% of the time.
• Safety: It crashed significantly less often.
• Speed: It flew more efficiently, not stopping and starting unnecessarily.

The Bottom Line

OnFly is like giving a drone a dedicated driver, a dedicated navigator, a perfect photo album memory, and a safety spotter, all running on a small computer attached to the drone itself. It allows drones to follow complex human instructions in the real world safely and efficiently, without needing a giant computer in the cloud to tell them what to do.

Technical Summary

Here is a detailed technical summary of the paper "OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency."

1. Problem Statement

Aerial Vision-Language Navigation (AVLN) involves guiding Unmanned Aerial Vehicles (UAVs) through complex 3D environments using natural language instructions. While recent Zero-Shot AVLN methods leverage Vision-Language Models (VLMs) to generalize to unseen environments, they face three critical bottlenecks preventing real-world deployment:

1. Decision Instability: Existing methods often couple high-frequency flight control (real-time target generation) with low-frequency task progress monitoring into a single inference stream. This causes mutual interference, slow target output, and unstable decision-making due to conflicting update rates.
2. Unreliable Long-Horizon Monitoring: Traditional sliding-window memory designs lose global trajectory context over time and disrupt KV-cache prefix stability (crucial for efficient LLM inference), leading to high latency and unreliable termination signals (e.g., failing to stop when the goal is reached).
3. Safety vs. Efficiency Trade-off: Conservative approaches prioritize safety via short-horizon, iterative actions, causing "stop-and-go" behavior and low efficiency. Aggressive long-range prediction improves efficiency but lacks systematic safety mechanisms, leading to collisions.

2. Methodology: The OnFly Framework

OnFly is a fully onboard, real-time framework designed to address these challenges through three core innovations:

A. Shared-Perception Dual-Agent Architecture

To decouple conflicting demands, OnFly separates the system into two agents that share a visual backbone but maintain independent inference streams:

• Decision Agent (High-Frequency): Runs at a high rate to generate real-time flight targets (2D waypoints in image coordinates). It uses a structured prompt with a lightweight "history point" (reprojected previous goal) to maintain temporal continuity without heavy computation.
• Monitoring Agent (Low-Frequency): Runs asynchronously to track task progress and determine termination (Stop), continuation (Continue), or recovery (Lost).
• Shared Perception: Both agents share the Vision Transformer (ViT) backbone and cached visual features to eliminate redundant encoding, while maintaining separate KV-caches to match their respective update rates.

B. Hybrid Keyframe-Recent-Frame Memory

To enable reliable long-horizon monitoring while preserving KV-cache efficiency, the Monitoring Agent utilizes a custom memory structure ($M_t$) composed of:

1. Initial Frame ($o_1$): The starting observation.
2. Keyframes ($Q_t$): A set of representative frames selected based on geometric distance and visual feature similarity. These are deduplicated and serialized to ensure the input prefix remains stable across steps, maximizing KV-cache reuse.
3. Latest Frame ($o_t$): The current real-time observation.
   This design preserves global trajectory context (preventing "amnesia") while ensuring the input prefix remains identical for efficient inference, enabling accurate termination and recovery signals.

C. Semantic-Geometric Verification & Receding-Horizon Planning

To bridge the gap between VLM semantic intent and geometric safety:

• Semantic-Geometric Verifier: Refines the VLM-predicted 2D target. It uses VLM feature matching to ensure the target remains in a semantically consistent region and applies depth constraints to avoid obstacles. It also implements bearing-aware gating, reducing forward range if the target is near the image boundary (preventing aggressive turns).
• Receding-Horizon Local Planner: Takes the verified 3D goal and generates a collision-free trajectory using an ESDF (Euclidean Signed Distance Field) map. It enforces UAV dynamic constraints and regularizes the yaw angle to align with the goal bearing, balancing safety and efficiency.

D. Deployment Optimization

The system runs fully onboard a Jetson Orin NX. Optimizations include:

• TensorRT acceleration for VLM inference.
• AWQ quantization for the LLM component.
• KV-cache reuse and CUDA Graphs to minimize decoding overhead.

3. Key Contributions

1. Dual-Agent Architecture: A novel design that decouples high-frequency control from low-frequency monitoring, stabilizing decision-making and eliminating inference interference.
2. Hybrid Memory System: A memory scheme that balances global context preservation with KV-cache prefix stability, enabling reliable long-horizon progress tracking on edge devices.
3. Safety-Aware Planning Pipeline: An integrated semantic-geometric verifier and local planner that refines VLM outputs into executable, collision-free trajectories.
4. Real-World Validation: Successful deployment on a physical quadrotor platform, demonstrating feasibility in unknown 3D environments.

4. Experimental Results

The system was evaluated in high-fidelity simulations (10 Unreal Engine scenes, 150 tasks) and real-world flights.

Simulation Results (vs. State-of-the-Art Baselines: TypeFly, PIVOT, SPF):

• Success Rate (SR): Improved from 26.4% (SPF, the strongest baseline) to 67.8%.
• Oracle Success Rate (OSR): Improved from 61.5% to 78.1%.
• Collision Rate (CR): Drastically reduced from 42.7% to 2.7%.
• Flight Time (FT): Reduced from 39.2s to 27.1s, indicating higher efficiency.

Ablation Studies:

• Removing the Dual-Agent architecture reduced SR to 48.6% and increased flight time significantly.
• Removing the Hybrid Memory (using sliding windows) increased collision rates and reduced success due to poor context retention.
• Removing the Planner increased the collision rate to 37.5%, highlighting the necessity of geometric safety constraints.

Real-World Experiments:

• Successfully executed four complex tasks: pedestrian tracking, long-range corridor navigation, precise outdoor navigation over stepped terrain, and multi-level obstacle avoidance.
• Latency: On the Jetson Orin NX, the optimized deployment achieved a 4.73× speedup in decision latency and 6.50× in monitoring latency compared to baseline implementations, confirming real-time feasibility.

5. Significance

OnFly represents a significant step forward in deployable autonomous robotics. It demonstrates that zero-shot AVLN can move beyond cloud-dependent or simulation-only prototypes to fully onboard, real-time operation. By solving the fundamental conflicts between decision frequency, memory efficiency, and safety constraints, OnFly provides a robust framework for UAVs to perform complex, instruction-guided tasks in open-world environments without prior training on specific scenes. The release of the code and the focus on edge deployment make it a critical resource for future research in safe and efficient autonomous navigation.