💻 computer science

VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision

VEGA introduces a method for training navigation Vision-Language-Action models from unlabeled egocentric videos by reconstructing local scene geometry to generate obstacle-aware trajectories for supervision, achieving significant improvements in collision avoidance and success rates compared to existing baselines.

Original authors: Gershom Seneviratne, Yohan Abeysinghe, Jianyu An, Vaibhav Shende, Dinesh Manocha

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gershom Seneviratne, Yohan Abeysinghe, Jianyu An, Vaibhav Shende, Dinesh Manocha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to walk through a crowded room without bumping into chairs, people, or tables. Usually, to teach a robot this, you need to either:

Record a human walking through that exact room while the robot watches, or
Manually program the robot to avoid specific obstacles.

Both of these are slow, expensive, and hard to scale. You can't record a human walking through every possible room in the world.

Enter VEGA.

The researchers behind this paper came up with a clever way to teach robots to navigate using billions of "in-the-wild" videos found on the internet (like people walking around their houses, hiking trails, or busy streets). These videos are great because they show real, messy, cluttered environments, but they have a big problem: they don't tell the robot where to go or how to avoid the furniture. They are just "action-free" videos.

Here is how VEGA turns those random videos into a navigation teacher, using a few creative steps:

1. The "Mental Map" Trick

First, VEGA looks at a single frame of a video and uses a special AI to build a 3D mental map of the room. It figures out where the floor is, where the walls are, and where the obstacles (like a coffee table or a dog) are standing. It creates a "safety map" that knows exactly how much space is available to walk through.

2. The "What If?" Game

Once the map is built, VEGA plays a game of "What if?"

What if the robot wanted to go to that red chair?
What if it wanted to go to the doorway?
What if it just wanted to walk to an empty spot on the rug?

For every possible destination, VEGA uses a mathematical planner to calculate the perfect, safe path to get there without hitting anything. It does this millions of times, creating a massive library of "Goal + Safe Path" pairs.

3. The "Student" Robot

Now, VEGA trains a robot brain (called a VLA or Vision-Language-Action model). This student robot watches the original video and sees the "Safe Path" VEGA calculated.

The Lesson: "Here is what you see (the video), here is where you want to go (the goal), and here is the safe path you should take."
The Result: The robot learns to look at a scene, understand a goal (like "go to the kitchen" or "go to that picture"), and instantly figure out a safe path, just by looking at the camera feed. It doesn't need the 3D map anymore once it's trained; it just uses its eyes and its brain.

Why is this a big deal?

Think of it like learning to drive.

Old Way: You only learn to drive by having a driving instructor sit next to you in a specific car on a specific street, telling you exactly when to turn.
VEGA Way: You watch millions of hours of dashcam footage from around the world. You use a computer to figure out the "perfect driving lines" for every possible destination in those videos. Then, you train your brain to mimic those perfect lines.

The Results

The researchers tested this on a real robot in messy, cluttered rooms with moving obstacles. Compared to other top-tier robot navigators:

Success: The robot reached its destination much more often (at least 150% better).
Safety: It crashed into things 66% less often.
Space: It gave itself more room to maneuver, avoiding tight squeezes by 60% more.

They also created a giant test called VEGA-Bench (with 250,000 scenes and 5 million goals) to prove that their method works better than the competition at avoiding collisions and reaching specific targets.

In short: VEGA takes the chaos of the internet's video library, turns it into a structured "safety manual" using geometry, and teaches robots to navigate the real world without needing expensive, hand-recorded training sessions.

Technical Summary: VEGA – Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision

Problem Statement

Vision-Language-Action (VLA) models have shown significant promise in robot manipulation by leveraging large-scale, diverse datasets. However, scaling these models to ground mobile robot navigation remains a critical bottleneck. Existing navigation datasets are sparse, typically providing only a single trajectory per scene toward one implicit or explicit goal. This lack of dense supervision fails to teach policies how feasible trajectories vary across different goals, obstacle configurations, and start locations within the same environment. Consequently, learned policies often exhibit weak goal grounding, poor close-range obstacle avoidance, and unsafe behaviors in cluttered scenes.

While autonomous driving datasets offer scale, they target structured environments and do not capture the close-range, object-centric clutter typical of general mobile robots. Conversely, collecting sufficient teleoperated robot data to cover the vast space of environments, goals, and obstacle layouts is not scalable. Although large-scale egocentric videos (e.g., walking tours, household recordings) are abundant and contain rich visual observations of real-world navigation, they lack the explicit robot actions, goal conditioning, and obstacle-aware trajectories required for policy learning.

Methodology: VEGA

The authors propose VEGA (Vision-language navigation from Egocentric Video with Geometric Action supervision), a framework that transforms unlabeled, action-free egocentric videos into a scalable supervision signal for training navigation VLAs. The core innovation is the use of reconstructed scene geometry exclusively during training to generate obstacle-aware trajectories, which are then distilled into a vision-based policy that requires only RGB observations and goals at inference.

1. Data Generation Pipeline

VEGA processes unlabeled egocentric videos through the following steps:

Monocular Geometry Reconstruction: Using the MoGe-2 foundation model, the system reconstructs a dense metric 3D point map ( $P_t$ ) from each RGB frame. A local ground plane is fitted via RANSAC, and points are rotated to align with the robot's coordinate frame.
Visibility-Aware ESDF Construction: The system projects the leveled point map onto a Bird's-Eye-View (BEV) domain. It distinguishes between known free space ( $X_{free}$ ), occupied space ( $X_{occ}$ ), and unobserved space ( $X_{unk}$ ) using raytracing. A Euclidean Signed Distance Field (ESDF) is computed, where positive values indicate safe clearance and negative values mark obstacles or unknown regions.
Multimodal Goal Anchoring:
- Object Goals: Florence-2 detects objects and generates bounding boxes. These are projected onto the ground plane to create spatial waypoints ( $g_{wp}$ ), aligned with text and image-region goals.
- Free-Space Goals: Auxiliary waypoints are sampled from observed free space to provide supervision for arbitrary traversable locations.
Trajectory Generation: A Model Predictive Path Integral (MPPI) planner generates obstacle-aware waypoint trajectories toward these goals. The planner optimizes for goal-reaching, collision avoidance, clearance, control effort, and smoothness using the ESDF. If a goal is unreachable, the best partial trajectory is selected.

2. Model Architecture and Training

VEGA trains a flow-matching VLA to predict local 2D waypoint chunks.

Architecture: Based on the $\pi_0.5$ VLA architecture, it combines a pretrained vision-language backbone with a flow-matching action expert. It supports text, image, and waypoint goals via specific encoders.
Training Objective: The model learns to predict a vector field that transforms noise samples into target trajectories. Unlike methods that predict a single deterministic output, flow matching allows the policy to model multimodal action distributions, capturing the fact that multiple distinct paths may be feasible for the same observation-goal pair.
Inference: At deployment, the policy requires only RGB observations and a goal specification. It does not need online depth sensors, LiDAR, or explicit geometric mapping, as the geometric reasoning has been distilled into the policy weights during training.

Key Contributions

Geometry-Supervised VLA Training: A paradigm that distills obstacle-aware navigation behavior from in-the-wild, action-free egocentric video without manual action labels or robot demonstrations.
Large-Scale Multimodal Trajectory Dataset: A constructed dataset containing approximately 5 million goal-conditioned trajectories aligned with text, image-region, and spatial-waypoint annotations, accompanied by reconstructed 3D point clouds and ESDFs.
VEGA-Bench: A new evaluation suite comprising 250k scenes and 5 million multimodal targets, designed to rigorously assess goal progress, collision rates, and obstacle clearance.
Real-World Validation: Empirical evidence demonstrating the transferability of the approach to physical robots.

Results and Evaluation

The authors evaluated VEGA against state-of-the-art baselines (OmniVLA, NaVILA, and $\pi_0.5$ ) on VEGA-Bench and in real-world trials.

VEGA-Bench Performance: VEGA achieved comparable goal progress to baselines but significantly reduced collision rates and improved obstacle clearance. Specifically, it reduced collisions by 33.0% and improved obstacle clearance by 17.9% over the strongest baseline.
Real-World Robot Trials: Deployed on a quadruped robot in cluttered environments with static and dynamic obstacles, VEGA demonstrated substantial improvements:
- Success Rate: Improved by at least 150.0%.
- Collision Avoidance: Reduced collisions by at least 66.7%.
- Obstacle Clearance: Improved by at least 60.0%.
Goal Grounding: Qualitative and quantitative analysis showed that VEGA successfully conditions on specific goals (text, image, or waypoint), avoiding the "mode collapse" seen in baselines where policies generate similar paths regardless of the goal.

Significance and Claims

The paper claims that video-derived geometric supervision provides a scalable and effective signal for training obstacle-aware navigation VLAs. By leveraging the abundance of egocentric video and using geometry only as a training-time teacher, VEGA bridges the gap between the semantic reasoning capabilities of VLAs and the geometric safety requirements of classical planning. This approach eliminates the need for expensive, large-scale robot data collection while enabling policies that generalize to complex, cluttered real-world environments.

Limitations and Future Work

The authors acknowledge two primary limitations:

Memorylessness: The current policy is memoryless, relying only on the current RGB observation and goal. This can be brittle when goals or obstacles leave the camera's field of view. Future work may incorporate temporal attention or latent memory.
Static Geometry Assumption: The local ESDF supervision does not explicitly model dynamic obstacle motion. While replanning handles newly observed obstacles, the planner does not predict future motions of moving objects. Future iterations could integrate trajectory prediction or time-indexed distance fields.