💻 computer science

From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning

This paper introduces the Seeing-to-Experiencing (S2E) framework, which enhances navigation foundation models by combining offline pretraining on web-scale videos with reinforcement learning in simulation to improve interactive reasoning and safety, supported by a new evaluation benchmark called NavBench-GS.

Original authors: Honglin He, Yukai Ma, Brad Squicciarini, Wayne Wu, Bolei Zhou

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Honglin He, Yukai Ma, Brad Squicciarini, Wayne Wu, Bolei Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Tourist" vs. The "Local"

Imagine you want to teach a robot how to walk through a busy city.

The Old Way (Just "Seeing"):
Currently, most navigation robots are trained like a tourist who has watched thousands of hours of YouTube videos about walking in cities. They have seen every type of street, crowd, and obstacle on screen. This is called Offline Pre-training.

The Flaw: Watching a video of someone dodging a skateboarder is very different from actually dodging one. The robot knows what a collision looks like, but it doesn't understand the physics of falling or the split-second timing needed to stop. If the robot tries to walk in the real world, it might freeze or crash because it has never actually "felt" the consequences of a wrong turn. It lacks causality (cause and effect).

The New Way (The "S2E" Framework):
The authors propose a new method called Seeing-to-Experiencing (S2E). Think of this as taking that tourist and sending them to a highly realistic, safe video game simulator where they can actually walk, trip, and learn from their mistakes without getting hurt.

The goal is to combine the broad knowledge of the video-watching tourist with the street smarts of someone who has actually practiced in the real world.

How It Works: The Two-Step Recipe

The S2E framework uses two main "tricks" to make this work efficiently.

1. The "Anchor" System (During the Video Phase)

The Challenge: When a robot watches a video, it sees that sometimes people walk straight, sometimes they turn left to avoid a dog, and sometimes they stop for a red light. All these actions are valid for the same situation. If the robot tries to guess just one average path, it will fail.

The Solution: The authors use something called Anchor-Guided Distribution Matching.

The Analogy: Imagine the robot is a chef trying to guess a recipe. Instead of guessing a single flavor, they place several "Anchors" (like specific flavor profiles: "Spicy," "Sweet," "Savory") on the table.
How it helps: The robot learns that for a specific situation, the answer might be "Spicy" (go straight) OR "Savory" (turn left). By using these anchors, the robot learns to predict a menu of possible good actions rather than just one average guess. This makes the robot much more flexible and ready for different scenarios.

2. The "Residual" Brain (During the Practice Phase)

The Challenge: Once the robot starts practicing in the simulator, we want it to learn new tricks (like dodging a moving pedestrian). But if we let the robot relearn everything from scratch, it might forget how to walk straight or recognize a sidewalk. This is called "catastrophic forgetting." Also, the simulator looks slightly different from the real world (different lighting, textures), which can confuse the robot.

The Solution: They use a Residual-Attention Module.

The Analogy: Imagine the robot has a "Base Brain" (the part trained on videos) that is excellent at recognizing streets. When it enters the simulator, we don't replace the Base Brain. Instead, we attach a small, lightweight "Add-on Brain" (the Residual Module) on top of it.
How it helps: The Base Brain stays frozen and keeps its general knowledge safe. The Add-on Brain is the only part that learns. It focuses only on the new, interactive stuff: "Oh, that person is moving, I need to slow down."
The Benefit: This is like adding a new app to your phone without deleting your contacts. The robot gains new reactive skills (dodging, stopping) without losing its old general knowledge (recognizing a road).

The Test Drive: NavBench-GS

To prove this works, the authors built a new testing ground called NavBench-GS.

What is it? Instead of testing on flat 2D images (like looking at a photo of a street), they built a 3D world that looks exactly like the real world but has real physics. You can throw a ball at the robot, and it will react.
The Results:
- Robots trained only on videos (the "Tourists") crashed often when faced with moving people or obstacles.
- Robots trained with the S2E method (the "Locals") were much better. They reached their destination more often and crashed far less.
- The Efficiency Surprise: The S2E robot learned faster with less data. A robot trained on just 100 hours of data + simulator practice performed better than other robots trained on over 2,000 hours of video data. This proves that interactive practice is more valuable than just watching more videos.

Real-World Proof

The team didn't just stop at simulations. They put their robot on two real machines:

A wheeled robot (like a delivery bot).
A quadruped robot (a dog-like robot).

They tested these robots in real city streets with real obstacles and people. The S2E robots successfully navigated these complex environments without prior specific training on those exact streets (Zero-Shot Generalization). They could keep to the sidewalk and avoid pedestrians, proving that the skills learned in the simulator transferred perfectly to the real world.

Summary

The paper argues that to make robots truly smart navigators, we can't just feed them more videos. We must let them practice. By combining a stable "video-based" foundation with a "practice-based" learning module that doesn't overwrite their original knowledge, robots can learn to navigate safely and efficiently in the chaotic, real world.

Technical Summary: Seeing-to-Experiencing (S2E)

Problem Statement

Navigation foundation models trained on massive web-scale offline data have demonstrated strong generalization across diverse environments. However, these models rely primarily on passive visual imitation, capturing statistical correlations rather than grounded causality. Consequently, they lack the capacity to reason about the consequences of actions, adapt to counterfactual scenarios, or exhibit reactive behaviors necessary for safe urban navigation (e.g., avoiding dynamic obstacles or pedestrians). While Reinforcement Learning (RL) offers a paradigm for interactive learning, prior attempts to scale RL for navigation have struggled due to poor sampling efficiency, a lack of inductive priors, and the difficulty of training agents from scratch in narrow synthetic environments. Furthermore, simply scaling offline supervised data often leads to diminishing returns in performance.

Methodology: The S2E Framework

The authors propose Seeing-to-Experiencing (S2E), a hybrid learning framework that combines large-scale offline pre-training with online reinforcement learning in simulation. The framework aims to retain the generalizable visual representations learned from real-world videos while acquiring interactive capabilities through RL.

1. Pre-training: Anchor-Guided Distribution Matching (AGDM)

To address the multimodal nature of navigation (where multiple valid actions exist for a single observation), the authors introduce an Anchor-Guided Distribution Matching strategy.

Architecture: Instead of using discrete bins or unimodal Gaussians, the model predicts a Gaussian Mixture Model (GMM) where the mixture components are guided by "anchors."
Mechanism: $M$ representative intention points (anchors) are sampled via K-Means on the unified dataset. These anchors serve as queries in a cross-attention mechanism. Each anchor corresponds to a Gaussian mode with learned scores, means, standard deviations, and correlations.
Benefit: This approach provides a structured, multi-modal distribution that balances expressiveness with training stability. It allows the model to capture diverse behaviors (e.g., going straight, turning, yielding) while maintaining a reliable backbone for subsequent adaptation.

2. Post-training: Residual-Attention Module (RAM)

To inject interactive capabilities without erasing the pre-trained knowledge or suffering from domain shift (sim-to-real), the authors propose a Residual-Attention Module for RL fine-tuning.

Motivation: Full-parameter fine-tuning with RL risks "Forgetting of Pre-trained Capabilities" (FPC) and causes the visual encoder to overfit to simulator textures, degrading real-world performance.
Mechanism: The original pre-trained cross-attention parameters are frozen. A parallel, trainable copy of the attention block is introduced, gated by zero-initialized linear layers.
Training Dynamics: The zero-initialization ensures that at the start of fine-tuning, the residual branch contributes nothing ( $Q' = \psi_{pre}(Q)$ ). As training progresses, the gate weights gradually activate, allowing the model to selectively learn reactive behaviors (e.g., obstacle avoidance) from the RL signal while preserving the generalizable visuomotor representations from the offline pre-training.
Efficiency: This approach is parameter-efficient, updating only a lightweight plugin rather than the entire model, and avoids updating the vision encoder to prevent domain shift.

3. Evaluation Benchmark: NavBench-GS

The authors establish NavBench-GS, a comprehensive end-to-end evaluation benchmark.

Features: Built on photorealistic 3D Gaussian Splatting reconstructions of real-world scenes, it incorporates accurate physical interactions and dynamics.
Scope: It includes 26 scenarios with four task variations: empty environments, static obstacles, moving pedestrians, and combined obstacles/pedestrians.
Metrics: Success Rate (SR), Route Completion (RC), and Collision Times (CT).

Key Results

Scaling Performance

Experiments demonstrate that S2E mitigates the diminishing returns associated with scaling offline data alone.

RL vs. Data Scaling: Increasing offline data from 250k to 750k samples yielded only a 2% improvement in success rate. In contrast, applying RL fine-tuning to a model trained on 100 hours of data resulted in a 15% improvement in success rate without additional offline data.
RL vs. SFT: Compared to Supervised Fine-Tuning (SFT), which suffers from severe overfitting as training costs increase, RL maintains or improves success rates and demonstrates higher sample efficiency.

Benchmark Performance (NavBench-GS)

S2E outperformed state-of-the-art navigation foundation models (including GNM, ViNT, NoMaD, MBRA, and CityWalker) across all test scenarios.

Efficiency: S2E, trained on only 100 hours of data, surpassed CityWalker, which was trained on over 2000 hours of data.
Safety: S2E achieved the highest Success Rates and lowest Collision Times, particularly in complex scenarios involving both static obstacles and moving pedestrians.

Real-World Deployment

The framework was deployed on two distinct robotic platforms: a wheeled robot and a Unitree GO2 quadruped robot.

Zero-Shot Generalization: S2E achieved zero-shot transfer to unseen real-world urban scenarios.
Performance: In real-world tests, S2E-Full achieved a success rate of 0.51 (wheeled) and 0.55 (quadruped), significantly outperforming baselines like NoMaD and CityWalker, which struggled with collision avoidance.

Significance and Claims

The paper claims that the integration of interactive online experiences via Reinforcement Learning is crucial for scaling foundation models in robotics. The key contributions are:

Bridging the Gap: S2E successfully bridges the gap between the broad generalization of offline pre-training and the reactive safety of online interaction.
Technical Innovation: The Anchor-Guided Distribution Matching stabilizes learning of complex multimodal distributions, while the Residual-Attention Module enables safe, incremental learning of interactive behaviors without catastrophic forgetting or domain shift.
New Paradigm: The work challenges the notion that scaling offline data is the primary driver of performance, showing that RL post-training offers a more efficient path to robust, safe navigation policies.
Reproducibility: The authors provide a new benchmark (NavBench-GS) and commit to releasing code and models to facilitate community verification.

The authors acknowledge limitations, noting that current systems rely on 2D visual inputs and lack explicit 3D perception, which can still lead to collisions in complex 3D structures. They suggest future work could integrate depth or occupancy prediction.