Technical Summary: NaviTrace: Evaluating Embodied Navigation of Vision-Language Models

Problem Statement

While Vision-Language Models (VLMs) have demonstrated unprecedented generalization across various tasks, their integration into robotic navigation systems remains hindered by a lack of rigorous, scalable evaluation methods. Existing approaches face three primary limitations:

1. Real-world trials: Closed-loop rollouts on physical robots are expensive, time-consuming, and lack reproducibility across diverse environments.
2. Simulations: While reproducible, simulators often rely on simplified dynamics, static scenes, and limited semantics, failing to capture critical factors like terrain properties or social norms that influence navigation.
3. Existing Benchmarks: Current Visual Question Answering (VQA) datasets for navigation typically constrain outputs to text descriptions rather than actionable path plans, focus exclusively on human navigation (ignoring cross-embodiment challenges), or lack high-quality human annotations for complex real-world scenarios.

Consequently, there is no standardized framework to evaluate how well VLMs understand the specific physical constraints and navigation strategies required by different robot embodiments (e.g., legged vs. wheeled) in real-world settings.

Methodology: The NaviTrace Benchmark

To address these gaps, the authors introduce NaviTrace, a high-quality VQA benchmark designed to evaluate embodiment-specific navigation performance.

Data Collection and Structure

• Scenarios: The dataset comprises 1,000 diverse real-world images paired with high-quality language instructions. The images are sourced from crowdsourced consumer devices and the GrandTour dataset, covering urban/rural settings, various lighting conditions, and weather.
• Instructions: Tasks are formulated as goals (e.g., "Go to the red car") or directional instructions, specifically targeting scenarios where different embodiments would behave differently.
• Embodiments: The benchmark evaluates navigation for four distinct embodiment types:
  1. Human: A pedestrian unable to climb tall obstacles.
  2. Legged Robot: A quadruped with human-like but shorter stature.
  3. Wheeled Robot: A wheelchair-like delivery robot favoring ramps and walkways.
  4. Bicycle: A cyclist adhering to traffic rules, avoiding stairs.
• Annotations: Each scenario includes 2D navigation traces (sequences of image coordinates) generated by human experts. Multiple valid traces are provided where alternatives exist (e.g., passing an obstacle on the left or right).
• Task Categories: Scenarios are tagged with navigation-relevant attributes, including Geometric Terrain, Semantic Terrain, Accessibility, Visibility, Social Norms, and Obstacle Avoidance.

Semantic-Aware Scoring Metric

The paper proposes a novel scoring function to evaluate predicted traces against ground truth, balancing three factors:

1. Trace Similarity: Measured using Dynamic Time Warping (DTW) with Euclidean distance to align sequences regardless of speed or length.
2. Goal Reaching: Measured by Final Displacement Error (FDE), the Euclidean distance between the predicted endpoint and the ground-truth endpoint.
3. Semantic Penalty: A penalty term derived from automated semantic segmentation (using Mask2Former). This penalizes traces crossing unsafe or irrelevant regions (e.g., grass for a wheeled robot, stairs for a bicycle) based on embodiment-specific penalty maps.

The raw score is scaled to a 0–100 range, where 100 represents the ground-truth performance and 0 represents a "Straight Forward" baseline (a vertical line through the image center). The authors validated that this metric correlates strongly (Spearman correlation ~0.87) with human pairwise rankings.

Experimental Setup and Results

The authors evaluated eight state-of-the-art VLMs (including Gemini 2.5 Pro, GPT-5, o3, Qwen 3 VL, and others) against the NaviTrace test set.

Key Findings

1. Significant Performance Gap: Human experts achieved a score of 75.04, while the best-performing VLM, Gemini 2.5 Pro, scored 34.55. All models significantly underperformed compared to human capabilities.
2. Goal Localization as the Primary Failure Mode: Ablation studies revealed that when models were given the correct goal location and only asked to predict the path shape, their performance improved significantly (56.55), approaching the performance of an "Oracle-Goal Straight Line" baseline. However, when asked to predict the goal location from scratch, performance dropped drastically. This indicates that localizing the goal is the dominant challenge, though path shaping also lags behind human performance.
3. Embodiment Robustness: Performance differences across the four embodiment types (Human, Legged, Wheeled, Bicycle) were minimal. This suggests that current VLMs suffer from general limitations in spatial grounding rather than specific blind spots regarding particular robot morphologies.
4. Reasoning vs. Execution: Qualitative analysis of models like o3 showed a disconnect between linguistic reasoning and spatial execution. Models could correctly reason about the correct path in text but failed to translate this reasoning into accurate 2D image coordinates.
5. Baseline Competitiveness: The "Straight Forward" baseline (drawing a vertical line) performed surprisingly well (scoring near o3 and GPT-5 in some categories), highlighting the difficulty of the task and the current limitations of VLMs in complex navigation planning.

Significance and Contributions

The paper claims three main contributions:

1. NaviTrace Benchmark: A novel, high-quality benchmark for evaluating VLMs on embodiment-specific navigation across 1,000 real-world scenarios and four embodiment types. It bridges the gap between high-level VLM reasoning and low-level robotic control by extending "pointing" tasks to sequential navigation prediction.
2. Semantic-Aware Score: A new, computationally efficient metric for evaluating 2D traces that combines DTW, endpoint error, and embodiment-conditioned semantic penalties. The authors demonstrate its strong alignment with human preferences.
3. Comprehensive Evaluation: A systematic assessment of state-of-the-art VLMs, revealing that while these models possess basic scene understanding, they lack the spatial grounding and goal localization capabilities required for reliable embodied navigation.

The authors position NaviTrace as a scalable and reproducible testbed to inform model development, reveal specific limitations in current foundation models, and establish a standard for comparing approaches in real-world robotic navigation. The benchmark and leaderboard are made publicly available to facilitate future research.

Limitations

The authors acknowledge several limitations:

• Geographic Bias: The dataset is heavily concentrated in Switzerland, which may limit generalizability to regions with different infrastructure.
• Static Scenarios: The benchmark uses single images, preventing the evaluation of temporal reasoning or multi-step planning in dynamic environments.
• Embodiment Scope: It is limited to ground vehicles and excludes aerial drones.
• Scoring Nuance: While the semantic score aligns with human preferences, it relies on automated segmentation and may not capture all nuanced aspects of human preference or precise robot kinematic constraints.