🤖 AI

Fly0: Persistent Metric Anchoring for Zero-Shot Aerial Vision-Language Navigation

Fly0 is a novel framework that decouples semantic reasoning from geometric planning by leveraging Multimodal Large Language Models for 2D instruction grounding and depth-based 3D localization, thereby achieving robust, low-latency zero-shot aerial navigation that significantly outperforms state-of-the-art baselines in both success rate and navigation error.

Original authors: Zhenxing Xu, Yihong Lu, Weidong Bao, Zhengqiu Zhu, Jingxuan Zhou, Zhichuang Wang, Ji Wang, Lihua Liu, Wei He

Published 2026-07-22

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenxing Xu, Yihong Lu, Weidong Bao, Zhengqiu Zhu, Jingxuan Zhou, Zhichuang Wang, Ji Wang, Lihua Liu, Wei He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your drone isn't just a remote-controlled toy, but a smart explorer that can understand your voice commands like, "Fly to the red barn behind the big oak tree." This is the dream of Vision-Language Navigation. For a long time, scientists have been trying to teach robots to listen to words and look at pictures to figure out where to go. But there's a catch: thinking and moving happen at very different speeds. A smart computer brain (like a large language model) might take a second to figure out what "red barn" means, but a drone flying through the air needs to make safety decisions hundreds of times every second to avoid crashing into a tree branch. If the drone waits for the brain to think every single time, it will crash. The big question researchers are asking is: How do we let a slow, smart brain give directions without making the fast, flying body panic or crash?

This paper introduces a clever solution called Fly0, which acts like a translator between a slow, thoughtful brain and a fast, reactive pilot. Instead of asking the smart brain to steer the drone every second, Fly0 lets the brain do its homework only occasionally—about once every two seconds—to set a stable, invisible "target anchor" in the real world. Think of it like a game of Marco Polo. The smart brain (the caller) shouts out a location, "You are near the big tree!" and then the drone (the swimmer) starts swimming toward that spot. Even if the tree gets hidden behind a bush or the camera loses sight of it for a moment, the drone doesn't panic. It keeps swimming toward the invisible anchor the brain set earlier, using its own fast sensors to dodge branches and rocks along the way. The brain only needs to speak up again if the drone gets too far off course or if the target changes.

The researchers tested this idea in both high-tech computer simulations and real-world flights with actual drones. They found that this "set it and forget it" approach works much better than letting the smart brain try to steer the drone directly. In their tests, Fly0 helped drones reach their goals successfully about 23.7% more often in simulations and 21.4% more often in real life compared to other smart methods. It also crashed less often and made fewer mistakes when the target was hidden or when there were confusing objects nearby. The paper suggests that the secret isn't just having a smarter brain, but building a better bridge between that brain's slow thoughts and the drone's fast movements. By keeping the "thinking" and the "flying" separate but connected, drones can finally understand complex instructions without losing their minds—or their wings.

The Problem: A Brain Too Slow for a Fast Body

To understand why Fly0 is special, you have to imagine the drone's two main jobs: thinking and flying.

The "thinking" part is done by a massive AI model (called an MLLM). This model is like a genius librarian who can read a map, understand a sentence like "fly to the front of the tree," and point to the right spot on a picture. But this librarian is slow. It takes time to read, think, and write down an answer.

The "flying" part is done by the drone's flight controller. This is like a reflexive athlete who needs to adjust the wings and motors 50 to 100 times every second to stay in the air and avoid hitting things.

In older methods, scientists tried to make the librarian steer the athlete. They asked the slow brain to give a new steering command every time the drone moved. This caused three big problems:

The Speed Mismatch: The drone would wait for the brain to think, and by the time the answer came, the drone had already drifted or was about to hit a wall.
The Memory Gap: If the drone turned its head and the target (like a tree) disappeared from the camera view, the slow brain would forget where it was. The drone would stop or fly in circles, confused.
The Precision Problem: The brain might say "fly to the tree," but the drone needs to know exactly where in 3D space to stop so it doesn't crash into the trunk. A simple text answer isn't precise enough for high-speed flying.

The Solution: The Persistent Anchor

Fly0 solves this by changing the rules of the game. Instead of asking the brain to steer, it asks the brain to set a destination and then step back.

Here is how Fly0 works, step-by-step:

The "Contract" (Structured Output): When the drone sends a picture and a command (e.g., "Fly to the front of the utility pole on the right") to the brain, the brain doesn't just guess. It follows a strict rulebook. It must return a specific package of information: a point on the image, a box around the object, a relationship word (like "front" or "above"), and a confidence score (how sure it is). This prevents the brain from giving vague or messy answers.
The "3D Lift" (Uncertainty Check): The brain's answer is just a 2D dot on a picture. Fly0 takes that dot and uses depth sensors (like a 3D camera) to lift it into the real world. But here's the smart part: it checks for errors. If the depth data is shaky or the object is hidden, Fly0 says, "I don't trust this," and ignores the bad data. It only accepts the target if it's solid.
The "Free-Space" Adjustment: If the command is "fly to the front of the tree," the brain might point at the tree trunk. But the drone can't fly into the tree. Fly0 has a rule that says, "Okay, the tree is here, but the 'front' is actually 2 meters in front of it, in the empty air." It shifts the target to a safe, open spot where the drone can actually hover.
The "Persistent Anchor" (The Magic Trick): Once the target is set, Fly0 saves it as a persistent 3D anchor. This is like a GPS coordinate that stays in the drone's memory even if the camera can't see the tree anymore. The drone's fast flight controller now takes over. It ignores the slow brain and just flies toward that saved anchor, using its own fast sensors to dodge obstacles. The brain only needs to wake up again if the drone gets close to the target or if the target changes.

What the Results Show

The researchers tested this system in two ways: in a super-realistic computer simulation and in real life with actual drones flying in parks and on campuses.

Success Rates: In the computer tests, Fly0 succeeded 23.7% more often than the next best method. In real life, it succeeded 21.4% more often. This is a huge jump.
Safety: The drones crashed less often. The collision rate dropped significantly, showing that the "anchor" method keeps the drone safer than letting the slow brain try to steer directly.
Handling Confusion: When there were multiple similar trees (a "repeated object" problem) or when the target was hidden behind leaves, Fly0 handled it much better. Because it kept the anchor in memory, it didn't get confused when the view was blocked.
Speed: The system works even if the connection to the "brain" (which was running on a powerful server) is slow or has a delay. The drone kept flying smoothly because it didn't need to ask for new directions every second.

What Fly0 Is NOT

It is important to note what this paper says Fly0 is not.

It is not a new flight controller. The drone still uses standard, proven flight software (Ego-Planner) to actually move.
It is not a new AI brain. They used existing large language models (like Qwen2.5VL) and didn't invent a new one.
It is not a magic fix for everything. If the drone is in a very dense forest with no open space, or if the instructions are incredibly complex and nested (like "the tree behind the sign which is to the left of the lamp"), the system can still get confused. The paper admits that understanding complex spatial relationships is still a hard problem.

The Takeaway

The main lesson from Fly0 is that to make drones smart, we don't just need smarter brains. We need better interfaces—better ways to connect the slow, thoughtful part of the system with the fast, reactive part. By treating the AI as a "goal setter" rather than a "steerer," and by giving the drone a stable memory anchor to hold onto, we can make autonomous flight safer, more reliable, and capable of understanding the messy, real world. The paper suggests that this separation of duties is the key to unlocking the next generation of flying robots.

Technical Summary: Fly0 – Persistent Metric Anchoring for Zero-Shot Aerial Vision-Language Navigation

1. Problem Formulation

The paper addresses the challenge of Zero-Shot Aerial Vision-Language Navigation (VLN) for Unmanned Aerial Vehicles (UAVs). The core difficulty lies in the fundamental mismatch between language reasoning and flight control:

Representation Mismatch: Language provides sparse, open-vocabulary semantic goals, while flight control requires dense, stable metric signals in continuous 3D space.
Time-Scale Mismatch: Multimodal Large Language Models (MLLMs) operate at low frequencies (due to inference latency), whereas UAV controllers require high-frequency updates (50–100 Hz) for safety and stability.
Memory Mismatch: Existing approaches often lose the goal when the target is occluded or leaves the field of view because they lack a persistent metric anchor in the world frame.

Current methods either rely on end-to-end policies (lacking interpretability and sim-to-real transfer) or embed MLLMs directly into the control loop, causing the model to struggle with both semantic disambiguation and high-frequency geometric control simultaneously.

2. Methodology: The Fly0 Framework

Fly0 reformulates aerial VLN as a semantic-to-geometric interface problem. Instead of using the MLLM as a recurrent controller, Fly0 treats it strictly as a low-frequency semantic localizer that generates a persistent metric anchor. The system decouples semantic reasoning (0.5–1 Hz) from geometric planning and control (50–100 Hz).

The architecture consists of five key stages:

A. Structured Semantic Grounding

The MLLM is constrained to output a structured JSON object rather than free-form text or actions. This output includes:

Point ( $p_{ts}$ ) and Region ( $b_{ts}$ ): Grounded 2D coordinates and bounding boxes.
Relation Token ( $r_{ts}$ ): Spatial prepositions (e.g., "front," "above," "left").
Confidence ( $c_{ts}$ ): A model-reported confidence score.
A deterministic clause parser decomposes multi-step instructions into sub-goals, grounding only the active clause to prevent conflating sequential targets.

B. Uncertainty-Aware 3D Goal Lifting

To avoid fragile single-pixel back-projection, Fly0 aggregates depth over the grounded region ( $b_{ts}$ ).

It computes the median depth and variance of valid pixels within a support patch.
Uncertainty Checks: If the valid depth ratio is low (occlusion) or variance is high (depth discontinuity), the update is rejected.
The 2D region is lifted to a 3D object-centric target in the camera frame and transformed to the world frame using the UAV's pose.

C. Relation-Aware Navigable Goal Generation

The system converts object-centric estimates into reachable free-space anchors.

Instead of targeting the object surface (e.g., the tree trunk), it applies a relation-aware offset (e.g., $+2.0$ meters in the "front" direction relative to the UAV's heading).
A projection module ( $\Pi_{F_t}$ ) maps this raw target to the nearest collision-free point in the local LiDAR-based free-space set, ensuring the goal is physically navigable.

D. Persistent Goal Memory and Re-grounding

The validated goal is stored as a persistent 3D anchor ( $\bar{g}_t$ ) that persists even when the target is occluded.

Fusion: New valid updates are fused with the existing anchor using a confidence-weighted average.
Consistency Checks: To prevent "false locks" (e.g., locking onto the wrong repeated object), the system performs jump checks (residual distance) and multi-view confirmation. If a new anchor is inconsistent or ambiguous, it is marked tentative and requires confirmation before replacing the current anchor.
Fallback: If the semantic service is unavailable, the UAV continues using the last valid anchor for a hold period before triggering hover or return-to-home routines.

E. LiDAR-Based Safe Trajectory Optimization

The geometric controller (Ego-Planner) operates independently at high frequency (50 Hz). It receives the persistent anchor as a goal and generates smooth, jerk-minimized B-spline trajectories that avoid local obstacles detected by onboard LiDAR. The MLLM is not queried during this loop.

3. Key Contributions

Problem Formulation: The authors define zero-shot UAV-VLN as a persistent metric anchoring problem, decoupling low-frequency semantic reasoning from high-frequency geometric control.
Structured Interface: Introduction of a structured grounding contract (point, region, relation, confidence) and a pipeline for uncertainty-aware 3D lifting and relation-aware goal synthesis.
Systematic Evaluation: A comprehensive evaluation protocol including:
- Fairness-controlled baselines: Comparing Fly0 against other methods using identical MLLM backbones, planners, and sensor stacks to isolate the contribution of the interface.
- Real-world benchmark (Fly0-Real): A new dataset with 18 scenes and 800 flight episodes involving occlusion, repeated objects, and dynamic obstacles.
- Granular metrics: Analysis of grounding accuracy, latency, network robustness, and failure taxonomy.

4. Experimental Results

Fly0 was evaluated on AerialVLN, OpenFly (simulation), and Fly0-Real (physical flights).

Performance Gains:
- Simulation: Fly0 improved the Success Rate (SR) by 23.7 percentage points over the strongest zero-shot baseline (SPF) on AerialVLN and reduced Navigation Error (NE) by 12.4m.
- Real-World: Fly0 improved SR by 21.4 points on Fly0-Real and reduced the collision rate from 0.13 to 0.07.
Ablation Studies:
- Interface vs. Planner: When the planner (Ego-Planner) and MLLM (Qwen2.5VL-32B) were held constant, replacing a transient point goal with Fly0's persistent relational anchor increased SR from 62.1% to 70.4%. This confirms gains arise from the interface, not just the planner.
- Sensing: While LiDAR improves performance, the uncertainty-aware bridge provided significant gains even before LiDAR integration.
- Frequency: Increasing semantic update frequency from 0.5 Hz to 1.0 Hz yielded only marginal improvements, validating the sufficiency of low-frequency updates once a stable anchor is formed.
Robustness: The system demonstrated resilience to network delays (up to 1s) and cloud outages (up to 10s) via cached anchors and fallback policies, though performance degraded if the anchor age exceeded the hold horizon.

5. Significance and Claims

The paper argues that reliable UAV vision-language navigation depends not only on stronger multimodal models but on interfaces that preserve uncertainty, relation semantics, and metric persistence across the semantic-control boundary.

Modest Claims: The authors explicitly state that Fly0 is an interface contribution rather than a novel low-level planner or a new general-purpose VLM. The components (grounding, memory, planning) have precedents; the novelty lies in the contract that joins them.
Failure Analysis: The paper identifies that while the interface solves metric stability, remaining errors are dominated by semantic ambiguity (repeated objects) and nested spatial relations, which current rule-based offsets cannot fully resolve.
Conclusion: The results suggest that separating semantic reasoning from control allows for safer, more interpretable, and more robust zero-shot navigation, particularly in complex, real-world environments where targets may be occluded or instructions are spatially complex.