Can Vision Foundation Models Navigate? Zero-Shot Real-World Evaluation and Lessons Learned

1. Problem Statement

Visual Navigation Models (VNMs) are foundation models designed to enable robots to navigate using only camera images (without precomputed geometric maps or expensive sensors) by learning from large-scale visual demonstrations. While these models show promise for generalization, current evaluations rely almost exclusively on Success Rate (SR)—a binary metric indicating whether a robot reached a goal within a distance threshold.

The authors argue that SR is insufficient because it:

• Obscures trajectory quality and collision frequency.
• Fails to capture robustness to environmental changes (distribution shift).
• Does not reveal why models fail (e.g., geometric misunderstanding vs. goal confusion).

There is a critical gap in understanding how state-of-the-art VNMs perform in real-world, zero-shot deployments across diverse environments and robot embodiments.

2. Methodology

The authors conducted a comprehensive real-world evaluation of five state-of-the-art VNMs across two robot platforms and five distinct environments.

Models Evaluated

1. GNM: Uses MobileNetV2 with a shared action space.
2. ViNT: Transformer-based with goal fusion tokens.
3. NoMaD: Diffusion-based with goal masking for exploration.
4. NaviBridger: Diffusion-based with learned action priors.
5. CrossFormer: Transformer with embodiment-specific heads.

Experimental Setup

• Robotic Platforms:
  • Spot: A quadruped robot (Nvidia AGX Orin) with built-in collision avoidance.
  • Bunker: A tracked mobile robot (Nvidia AGX Xavier) without collision avoidance, making it a strict test of the VNM's geometric reasoning.
• Environments (5 Total):
  • Indoor (Familiar): Corridor (straight line), Office Loop (cluttered, long-horizon), Arena (visual discrimination of similar doors), Stairs (perceptually identical staircases).
  • Outdoor (Out-of-Distribution): Snow (snowy parking lot, significant distribution shift).
• Perturbations: Controlled visual perturbations (motion blur, sunflare) applied to the topological map to test robustness.

Evaluation Metrics

The study moves beyond SR to include a multi-dimensional metric suite:

1. Navigation Metrics: Path length, distance to goal, collision occurrence, topological node error, and goal prediction accuracy.
2. Vision-Based Metrics: Comparing the final observation to the goal image using LPIPS (Learned Perceptual Image Patch Similarity), DSSIM, and PSNR.
3. Robustness: Performance under distribution shift (snow) and visual perturbations.

3. Key Contributions

1. Comprehensive Real-World Benchmark: The first large-scale evaluation of five diverse VNMs on two different robot embodiments across five real-world environments, including out-of-distribution scenarios.
2. Novel Metric Suite: Introduction of a combined evaluation framework that integrates path-based metrics (collision, length) with vision-based goal-recognition scores (LPIPS, DSSIM) to diagnose failure modes.
3. Public Release: Commitment to releasing the evaluation codebase and dataset to facilitate reproducible benchmarking.
4. Systematic Analysis of Limitations: Identification of three specific failure modes common across architectures.

4. Key Results & Findings

A. Geometric Understanding is Limited

• Collision Frequency: Even sophisticated Transformer and Diffusion-based models (NoMaD, NaviBridger, CrossFormer) exhibited frequent collisions in the "Bunker" (no collision avoidance) environment.
• Finding: Architectural complexity does not guarantee geometric understanding. Models can follow visual paths but fail to recognize obstacles, suggesting training datasets lack sufficient examples of collisions and recovery.

B. Goal Prediction Failures in Repetitive Environments

• Perceptual Confusion: In environments with semantically similar locations (e.g., the "Arena" with multiple doors, "Stairs"), models frequently predicted the goal prematurely.
• Metric Insight: While models like ViNT and NoMaD followed trajectories well, they achieved high "Topological Node Errors" and poor goal discrimination.
• Surprise: Simpler models (GNM with MobileNetV2) often outperformed complex models in goal prediction, suggesting that encoding quality matters more than architectural depth when data is limited.

C. Robustness to Distribution Shift

• Snow Environment: Performance degraded significantly for most models in the snowy outdoor setting.
• Visual Perturbations: Diffusion-based models proved fragile under motion blur and sunflare. GNM and ViNT showed better resilience.
• Counter-intuitive Finding: In the Snow environment, visual metrics (LPIPS/PSNR) actually improved for some models compared to indoor settings. The authors hypothesize that outdoor environments have more distinct spatial structures, reducing the visual ambiguity that plagues repetitive indoor scenes.

D. Architecture vs. Data

• Data Insufficiency: The authors argue that the comparable performance of simple models (GNM) against complex ones (ViNT, NoMaD) indicates that training data has not scaled with architectural complexity. The models are likely under-trained rather than architecturally limited.

5. Significance and Lessons Learned

• Redefining Success: The field must move beyond binary success rates. A robot that reaches the goal but collides 10 times or takes a 30m detour is not a successful deployment.
• Training Data Gaps: Current datasets lack "rare" scenarios (collisions, recovery, extreme weather). To achieve true robustness, future datasets must include these edge cases.
• Geometric Reasoning: Vision-only navigation is insufficient for safety-critical applications without supplementary geometric reasoning or reactive collision avoidance layers.
• Simplicity vs. Complexity: In the current state of VNM development, simpler architectures trained on sufficient data may outperform complex, data-starved foundation models.

Conclusion

The paper concludes that while Vision Foundation Models show promise for generalization, substantial gaps remain before they can be reliably deployed in the real world. The primary bottlenecks are not necessarily the model architectures (Transformers/Diffusion) but rather the quality and diversity of training data and the lack of explicit geometric reasoning. The authors call for a shift in research focus toward data curation for edge cases and the integration of test-time adaptation mechanisms.