HUGE-Bench: A Benchmark for High-Level UAV Vision-Language-Action Tasks

This paper introduces HUGE-Bench, a novel benchmark featuring real-world digital twin scenes and a 3D Gaussian Splatting-Mesh representation to evaluate UAV agents' ability to interpret high-level language commands and execute safe, complex, multi-stage trajectories, revealing significant gaps in current state-of-the-art models' performance.

The Gist

Imagine you are the boss of a fleet of high-tech drones. In the past, if you wanted a drone to do a job, you had to be a pilot and a programmer at the same time. You'd have to give it a long, boring list of instructions: "Fly 50 meters north, turn 15 degrees right, drop 10 meters, hover for 3 seconds, then move left." If you missed a step, the drone might crash or get lost.

HUGE-Bench is a new "driving test" for drones that changes the rules. Instead of giving the drone a long manual, you just give it a simple, human-like command, like: "Go check out that building on the left."

The big question is: Can the drone figure out the rest? Can it understand what "left" means from the sky, decide how to get there, figure out how to circle the building safely without hitting it, and then come back home?

Here is a breakdown of how the paper solves this problem, using some everyday analogies.

1. The Problem: The "GPS vs. Human" Mismatch

Current drone tests are like a GPS navigation app that tells you exactly which turn to take at every single intersection. But in the real world, humans don't talk like GPS. We say, "Drive to the park," and we expect the driver to know how to get there, avoid potholes, and park the car.

The researchers found that existing drone tests were too easy and too specific. They didn't test if the drone could handle a vague command and figure out the complex steps on its own. They also didn't test if the drone would crash into a tree while trying to follow the order.

2. The Solution: The "Digital Twin" Playground

To test this fairly, the researchers built a massive, super-realistic video game world called HUGE-Bench.

• The World: They took real photos of four different places (an office park, a city block, a swamp, and a construction site) and turned them into a Digital Twin.
• The Magic Mix (3DGS + Mesh): Think of this like a hybrid car engine.
  • One part (3D Gaussian Splatting) is like a photographer. It makes the world look so real you can't tell it's fake. This helps the drone "see" things clearly.
  • The other part (Mesh) is like a construction blueprint. It has invisible walls and solid geometry. This helps the drone know, "If I fly here, I will hit a wall."
  • Why it matters: Most simulations are either pretty but floaty (no collisions) or solid but ugly. HUGE-Bench is both pretty and solid, so it can test if the drone crashes.

3. The Exam: 8 New "Driving Tests"

Instead of just "fly from A to B," the test includes 8 different scenarios that require the drone to think like a human pilot:

• The Landing: "Go land on that roof." (The drone has to find the roof, line up, and gently touch down).
• The Inspector: "Check the road." (The drone has to fly low, follow the road's curve, and look at it).
• The Orbit: "Circle that building." (The drone has to fly in a perfect circle at a safe distance).
• The Spiral: "Go down in a spiral." (Like a helicopter landing in a tight spot).
• The Obstacle Course: "Fly through that area." (The drone must dodge trees and wires while moving).

4. The Grading System: It's Not Just About the Finish Line

In old tests, if the drone reached the destination, it got an "A," even if it crashed halfway there or took a weird, inefficient path.

HUGE-Bench uses a smarter grading system:

• Process Fidelity: Did the drone actually inspect the building, or did it just fly past it? (Like a teacher checking if you showed your work, not just the final answer).
• Safety Score: Did it hit anything? If the drone crashes, it gets a failing grade, no matter how close it got to the goal.
• Efficiency: Did it take a direct route, or did it fly in circles for no reason?

5. The Results: The "Smart" Drones Are Still Learning

The researchers tested the latest, most advanced AI drone brains (like OpenVLA and $\pi_0$) on this new test.

The verdict? They struggled.

• When given a short command, many drones got confused. They didn't know how to break the task down into steps.
• Some drones tried to fly through walls because they couldn't "see" the invisible geometry.
• The best performers were models that had been trained on lots of robot data, but even they had a hard time with the "safety" part.

The Big Takeaway

HUGE-Bench is a reality check for the drone industry. It shows us that while AI is getting good at following long, detailed instructions, it's still bad at taking a simple, human command and turning it into a safe, complex flight plan.

It's like teaching a child to drive. You can't just say "Drive to the store" and expect them to know how to steer, brake, and avoid traffic on the first try. HUGE-Bench is the training ground that helps us teach drones to be safe, independent pilots who can understand us without needing a manual.

Technical Summary

1. Problem Statement

Current Unmanned Aerial Vehicle (UAV) Vision-Language Navigation (VLN) benchmarks suffer from a significant mismatch with real-world operational requirements:

• Instruction Style: Existing benchmarks rely on long, step-by-step route descriptions (e.g., "turn left, then go straight"). In contrast, real-world operators issue brief, high-level, and often ambiguous commands (e.g., "inspect the building on the left").
• Task Abstraction: Current evaluations are goal-centric (focusing only on whether the agent reached the destination), ignoring the process fidelity required for complex missions (e.g., maintaining a specific orbit radius, covering a mapping area, or avoiding obstacles during transit).
• Safety & Representation: Many simulators lack physically executable collision checking (neural rendering like 3DGS is photorealistic but lacks collision geometry), while traditional mesh-based simulators often lack the photorealistic perception needed for robust vision-language grounding.
• Evaluation Gap: Standard metrics like Success Rate (SR) and Success weighted by Path Length (SPL) fail to penalize agents that reach a goal but skip critical intermediate steps or collide with obstacles.

2. Methodology

A. HUGE-Bench Framework

The authors introduce HUGE-Bench, a benchmark designed for High-Level UAV Vision-Language-Action (HL-VLA) tasks. It consists of four main components:

1. Task Suite: Eight representative high-level tasks requiring multi-stage execution:

   • Landing: Target identification, approach, descent, and hover.
   • Inspection-R/B: Road and Building inspection involving approach, altitude adjustment, directional alignment, and return.
   • Mapping: Area coverage with specific altitude constraints.
   • Orbiting (H/R): Circular loitering at specific heights or radii.
   • Spiral Down: Multi-turn descending loiter.
   • Traversal: Obstacle-aware region traversal.

2. Real-to-Sim Pipeline & Digital Twin:

   • Data Collection: Real-world data captured via DJI M400 UAVs across four diverse scenes (offices, urban blocks, swamps, construction roads), covering ~6.45 km².
   • Hybrid Representation: The core innovation is an aligned 3D Gaussian Splatting (3DGS)–Mesh digital twin.
     • 3DGS: Provides high-fidelity, photorealistic rendering for visual perception inputs.
     • Mesh: Provides collision-ready geometry for physics-based queries and safety evaluation.
   • Annotation: Uses Large Language Models (LLMs) to generate spatial referring expressions (e.g., "upper-left building") and human annotators for landmark masks.

3. Trajectory Generation:

   • Trajectories are generated in Isaac Sim using task-specific rules and RRT-based planners for obstacle avoidance.
   • The system records synchronized multimodal streams (RGB, Depth, 4-DoF state, collision signals) at high frequency.
   • Dataset Scale: 2.56 million meters of trajectory data across 4 scenes, split into Train, Test-Seen, and Test-Unseen (novel landmarks and paraphrased instructions).

4. Evaluation Metrics:
   The benchmark proposes a multi-dimensional evaluation protocol:

   • Trajectory Coverage Rate (TCR): Measures how well the predicted trajectory covers the ground-truth process (intermediate steps), not just the endpoint.
   • Success Rate (SR): For goal-oriented tasks (e.g., landing).
   • Collision Rate (CR): Fraction of episodes with at least one collision.
   • Collision-aware SPL (CSPL): Combines success, path efficiency, and collision avoidance.

3. Key Contributions

1. HL-VLA Task Formulation: Defined a new benchmark setting where agents must interpret short, underspecified commands and decompose them into structured, multi-stage semantic behaviors.
2. Real-to-Sim Benchmark Construction: Built a scalable pipeline using an aligned 3DGS-Mesh digital twin, enabling both realistic perception and rigorous safety/collision evaluation.
3. Process-Oriented Evaluation: Introduced metrics (TCR, CSPL) that specifically assess procedural fidelity and safety, addressing the limitations of goal-only metrics in complex UAV missions.

4. Experimental Results

The authors evaluated state-of-the-art Vision-Language-Action (VLA) models: OpenVLA, FastVLM, $\pi_0$, and $\pi_0.5$.

• Performance Gaps: All models struggled significantly with process completion.
  • $\pi_0$ and $\pi_0.5$ (policies pre-trained on large-scale robot data) performed best, particularly on unseen data, suggesting that large-scale pretraining helps transfer skills to aerial control.
  • OpenVLA consistently underperformed, highlighting the difficulty of current VLA systems in handling multi-stage, process-oriented tasks.
• Task Difficulty: "Landing" was the easiest task, while "Traversal" and "Spiral Down" were the most challenging.
• Safety Analysis:
  • Models exhibited high Collision Rates (CR) (e.g., FastVLM had a CR of ~0.73 on unseen data).
  • $\pi_0$ demonstrated significantly better obstacle avoidance compared to FastVLM, reducing CR and improving CSPL.
  • Key Finding: Photorealistic perception alone is insufficient; successful high-level UAV execution requires explicit 3D geometric reasoning and safety-aware planning.

5. Significance

• Diagnostic Tool: HUGE-Bench serves as a critical diagnostic testbed, revealing that current SOTA models fail to bridge the gap between high-level semantic intent and safe, multi-stage physical execution.
• Safety Standardization: By integrating collision-aware evaluation into a photorealistic benchmark, it sets a new standard for safety-critical UAV autonomy research.
• Future Direction: The paper highlights the need for models that can perform implicit subtask decomposition and maintain situational awareness in dynamic, cluttered 3D environments, moving beyond simple point-to-point navigation.

In summary, HUGE-Bench shifts the paradigm from "can the drone get there?" to "can the drone get there safely while correctly executing the complex procedure required by a brief human command?"