Openfly: A comprehensive platform for aerial vision-language navigation

To address the lack of benchmarks for outdoor aerial Vision-Language Navigation, this paper introduces OpenFly, a comprehensive platform featuring a multi-engine simulation environment, an automated data collection toolchain, a large-scale 100k-trajectory dataset, and a specialized keyframe-aware agent model.

The Gist

Imagine you want to teach a drone to be a delivery driver or a search-and-rescue hero. You can't just tell it, "Go to the red building." You need to give it a full set of instructions like, "Fly up, turn left past the park, and stop when you see the big clock tower."

The problem? Teaching a drone this way is incredibly hard, expensive, and slow. Usually, humans have to manually fly the drone, record the path, and then write down the instructions for every single trip. It's like trying to teach a child to ride a bike by manually pushing them around the block for 100,000 different routes.

Enter "OpenFly." Think of OpenFly as a massive, automated "Drone School" simulator that solves all these problems at once.

Here is how it works, broken down into simple concepts:

1. The "World Builder" (The Rendering Engines)

To teach a drone, you need a place for it to fly. Most previous projects only had one or two digital worlds (like a single video game map).

• The Analogy: Imagine trying to teach a driver to navigate only in a small, empty parking lot. They'd fail in the real city.
• The OpenFly Solution: OpenFly is like a universal travel agency. It combines four different "worlds" into one platform:
  • Unreal Engine: High-tech, realistic city simulations.
  • GTA V: The famous video game world (Los Santos), which is surprisingly realistic.
  • Google Earth: Real satellite maps of cities like New York and Tokyo.
  • 3D Gaussian Splatting: A fancy new tech that takes real photos of actual university campuses and turns them into 3D digital twins.
• The Result: The drone gets to practice in 18 different "cities," ranging from digital skyscrapers to real-world parks, ensuring it doesn't get confused when it sees something new.

2. The "Robot Teacher" (The Automatic Toolchain)

This is the magic part. Instead of humans manually flying the drone and writing instructions, OpenFly has a robotic assembly line.

• Step 1: The Map Maker: The system scans the digital world and builds a 3D map of everything (buildings, trees, roads).
• Step 2: The Path Finder: It automatically picks a starting point and a destination (like a "landmark"), then calculates a safe, collision-free flight path. It's like a GPS that instantly draws a line through a maze without hitting the walls.
• Step 3: The Storyteller: This is the coolest part. The system takes the flight path and the images the drone "sees," then feeds them to an AI (like a super-smart chatbot). The AI looks at the path and writes the instructions: "Fly up, turn right toward the blue building, then go straight to the tower."
• The Analogy: It's like having a robot director who films a movie, writes the script, and edits the scenes all by itself in seconds. In the past, humans had to do all three jobs.

3. The "Library" (The Dataset)

Because the "Robot Teacher" works so fast, OpenFly has built the largest library of drone flight data in the world.

• The Scale: It contains 100,000 flight paths. Previous libraries only had about 10,000.
• The Variety: It covers 15,600 different words and phrases, so the drone learns to understand "the tall glass building" just as well as "the skyscraper with the blue roof."

4. The "Star Student" (OpenFly-Agent)

Having a library is great, but you need a student to learn from it. The authors created a new AI model called OpenFly-Agent.

• The Problem: When a drone flies, it sees thousands of images. If you show a brain every single frame of a video, it gets overwhelmed and forgets the important stuff (like the landmark it's looking for).
• The Solution: OpenFly-Agent is like a smart camera operator. Instead of recording every single second, it knows exactly when to hit "record."
  • It ignores the boring parts where the drone is just flying straight.
  • It focuses intensely on the Keyframes—the moments when the drone sees a landmark or needs to turn.
• The Result: This makes the AI faster, smarter, and much better at following instructions without getting confused by visual clutter.

Why Does This Matter?

Before OpenFly, teaching drones to navigate was like trying to learn a language by reading a dictionary one word at a time. It was slow and limited.

OpenFly is like a full immersion language camp.

• It provides the environment (the world).
• It provides the lessons (the data).
• It provides the teacher (the AI model).

The result? Drones that can actually understand complex human instructions and navigate real-world cities safely. The authors tested their system in real life, and it worked significantly better than any previous method, proving that this "automated school" approach is the future of flying robots.

In short: OpenFly built a massive, automated factory that creates millions of flight lessons for drones, and a smart student that learns from them, so we can finally have drones that can fly themselves to deliver packages or find lost hikers without crashing.

Technical Summary

1. Problem Statement

Vision-Language Navigation (VLN) is a core task in Embodied AI, where agents navigate to a target based on natural language instructions and visual observations. While significant progress has been made in ground-based and indoor VLN (e.g., R2R, REVERIE), Aerial VLN (guiding Unmanned Aerial Vehicles or UAVs) remains underdeveloped due to three critical bottlenecks:

• Limited Data Diversity: Existing aerial datasets rely heavily on a single simulator (AirSim/Unreal Engine), restricting the variety of environments and photorealistic assets.
• High Collection Costs: Generating trajectories and instructions traditionally requires human pilots to fly UAVs in simulators followed by manual annotation, a process that is labor-intensive and difficult to scale.
• Small Data Scale: Current aerial VLN datasets contain only ~10k trajectories, which is insufficient compared to the millions of episodes available in ground-based manipulation datasets (e.g., Open X-Embodiment), hindering the training of robust foundation models.

2. Methodology

The authors propose OpenFly, a comprehensive platform designed to solve these issues through a multi-engine integration strategy and a fully automated data generation pipeline.

A. Multi-Engine Rendering Integration

To maximize environmental diversity and realism, OpenFly integrates four distinct rendering engines and techniques:

1. Unreal Engine (UE4/UE5): Provides 8 urban scenes with rich assets (buildings, vehicles, pedestrians) covering over 100km².
2. GTA V: Offers a highly realistic, dynamic cityscape modeled after Los Angeles.
3. Google Earth: Supplies four real-world urban regions (Berkeley, Osaka, Washington D.C., St. Louis) covering 53.6km².
4. 3D Gaussian Splatting (3D GS): Used for Real-to-Sim reconstruction. The authors capture real-world drone footage of five university campuses and reconstruct them using hierarchical 3D GS, enabling photorealistic simulation of real environments.

B. Automatic Data Generation Toolchain

A key innovation is the fully automated pipeline that eliminates manual annotation:

1. 3D Point Cloud Acquisition:
   • For UE/GTA V: Uses rasterized sampling to stitch local point clouds into a global map.
   • For 3D GS: Exports sparse point clouds directly from the COLMAP framework.
2. Scene Semantic Segmentation: Identifies navigation landmarks using three methods:
   • 3D Scene Understanding (Octree-Graph).
   • Point Cloud Projection & Contour Extraction (segmenting ground-contacting structures).
   • Manual Annotation (for low-quality scenes).
3. Automatic Trajectory Generation:
   • Constructs a global voxel map.
   • Selects random landmarks as targets and defines start/end points.
   • Uses the A algorithm* to generate collision-free trajectories based on a discrete action space (Forward, Turn Left/Right, Ascend/Descend).
4. Automatic Instruction Generation:
   • Instead of feeding all images to a Vision-Language Model (VLM), the system splits trajectories into sub-segments based on action transitions.
   • It extracts keyframes (images near landmarks) and the last three images of each sub-segment.
   • A VLM (e.g., GPT-4o) generates sub-instructions for these keyframes, which are then fused by an LLM into a coherent, natural language instruction.

C. OpenFly-Agent Model

The paper introduces OpenFly-Agent, a VLN model based on the OpenVLA baseline, optimized for aerial navigation:

• Keyframe-Aware Selection: Unlike uniform sampling, the model selects frames based on trajectory changes (turns/ascents) and a Landmark Grounding Module. This module predicts bounding boxes for instruction-referenced landmarks; frames containing these landmarks are prioritized.
• Visual Token Pruning (VTM): To reduce computational load, visual tokens from adjacent frames are merged if they are highly similar (cosine similarity). The current frame remains uncompressed to capture the latest observation, while historical tokens are compressed via grid pooling.
• Architecture: Takes a sequence of images and text instructions as input, outputting discrete UAV actions (6 types: Forward, Turn Left/Right, Move Up/Down, Stop).

3. Key Contributions

1. OpenFly Platform: A versatile system integrating 4 rendering engines (UE, GTA V, Google Earth, 3D GS) to support diverse, high-fidelity aerial environments, including Real-to-Sim capabilities.
2. Automated Toolchain: A pipeline for automatic point cloud acquisition, semantic segmentation, trajectory planning, and instruction generation, removing the need for human pilots and annotators.
3. Largest Aerial VLN Dataset: A dataset comprising 100,000 trajectories across 18 scenes, with a vocabulary size of 15.6K. This is an order of magnitude larger than previous aerial datasets.
4. OpenFly-Agent: A novel model incorporating keyframe selection and visual token merging, achieving state-of-the-art performance while reducing computational redundancy.

4. Experimental Results

The authors evaluated OpenFly-Agent against existing methods (e.g., Navid, NaVila, AerialVLN) on both simulated and real-world benchmarks.

• Simulation Performance (Test Seen):
  • Success Rate (SR): 34.3% (vs. 20.3% for the next best, NaVila).
  • Success Weighted by Path Length (SPL): 24.9%.
  • Navigation Error (NE): 93m (lowest among all methods).
  • Improvement: Outperforms the baseline by 14.0% in SR on seen scenarios and 7.9% on unseen scenarios.
• Real-World Experiments:
  • Tested on 23 real outdoor scenes using a Q250 drone with an onboard Jetson Xavier NX.
  • Achieved 26.09% SR and 34.78% Oracle Success Rate (OSR), significantly outperforming other VLN methods.
  • Demonstrated strong Sim-to-Real transfer capabilities, validating the quality of the generated data.
• Ablation Studies:
  • Removing Keyframe Selection (KS) or Visual Token Merging (VTM) resulted in significant performance drops, confirming the necessity of focusing on critical observations and reducing token redundancy.

5. Significance

• Scalability: OpenFly solves the "data bottleneck" in aerial VLN by providing a scalable, automated method to generate massive, diverse datasets without human intervention.
• Realism: The integration of 3D Gaussian Splatting allows for high-fidelity simulation of real-world environments, bridging the gap between simulation and reality.
• Benchmarking: By releasing the toolchain, dataset, and code, OpenFly establishes a new standard benchmark for the community, facilitating future research in aerial robotics and embodied AI.
• Efficiency: The proposed Keyframe-Aware model demonstrates that focusing on critical visual cues and compressing redundant tokens can significantly improve both performance and inference efficiency in long-horizon navigation tasks.