💻 computer science

AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness

AgenticNav introduces a lightweight, zero-shot vision-and-language navigation framework that reimagines the VLM-environment interaction as a tool-calling harness, enabling direct pixel-based action selection, on-demand depth queries, and selective memory retrieval to achieve state-of-the-art performance without learned waypoint predictors.

Original authors: Yijian Li, Changze Li, Hantian Shi, Jiaying Luo, Jiyuan Cai, Ming Yang, Tong Qin

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yijian Li, Changze Li, Hantian Shi, Jiaying Luo, Jiyuan Cai, Ming Yang, Tong Qin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to navigate a brand-new house. You give it a simple command: "Go to the kitchen, turn right, and stop by the sofa." This is the world of Vision-and-Language Navigation (VLN). It's a branch of robotics where an agent must understand human speech and visual clues to move through a 3D space it has never seen before. For a long time, robots needed to be trained on thousands of specific houses to learn how to move, which meant they would get lost if you took them to a different building. But recently, we have developed "Large Vision-Language Models" (VLMs)—super-smart AI brains that can already understand pictures and words without needing to be retrained for every single task. The big question for scientists is: How do we connect this smart brain to a robot's legs? How do we let the AI decide exactly where to step without getting confused by the messy, continuous reality of the real world?

This is where the paper AgenticNav comes in. The authors argue that the old way of connecting AI to robots was like giving a driver a map with only three pre-drawn roads to choose from. If the destination wasn't on one of those three roads, the driver was stuck. The new method, AgenticNav, changes the game entirely. Instead of forcing the robot to pick from a limited list of pre-approved moves, it gives the AI a set of "tools" it can call upon whenever it needs them. Think of the AI as a detective in a mystery game. Instead of the game forcing the detective to choose between "Go Left," "Go Right," or "Go Straight," the detective can now say, "I need to check the depth of that specific pixel on the wall," or "Show me the map of where I've been," or "I want to walk directly to that chair I see."

The paper introduces a system called AgenticNav, which acts as a "harness" or a control panel for these smart AI brains. The researchers found that by giving the AI three specific tools, it could navigate unseen environments much better than before, even without any extra training. The first tool is the Action Tool. Instead of choosing from a small list of suggested spots, the AI can point directly at any pixel in the camera's view and say, "Go there." The system then does the math to turn that pixel into a safe walking path. The second tool is the Depth Tool. Sometimes the AI needs to know exactly how far away something is. Instead of showing the AI a giant, confusing 3D map of depth, this tool lets the AI ask, "How far is the wall at this specific spot?" and get a precise number. The third tool is the Memory Tool. As the robot walks, it builds a compact map. If the robot gets confused or needs to remember a sign it saw earlier, it can ask the system to "recall" that specific moment, rather than trying to remember every single frame of the entire journey, which would overwhelm its brain.

The authors tested this system in a computer simulation called R2R-CE (using the Habitat simulator) and also on a real robot. The real-world validation involved 30 specific episodes spanning laboratory, office, and outdoor yard scenes, stressing tasks like reading signs, long-range navigation, and precise target arrival. They found that when using a powerful AI brain (GPT-5.5), their new method achieved a 55% success rate in reaching the goal, which was a significant jump from the previous best method (SmartWay) that only reached 44%. In terms of efficiency (how well it balanced success with taking a short path), they improved from 35.04% to 48.41%. The paper suggests that the bottleneck isn't just how smart the AI brain is, but how well we give it the tools to interact with the world. By letting the AI choose its own targets and ask for specific information, it becomes much more capable of navigating the real world, even in places it has never visited before. The researchers also noted that this approach works well with different types of AI brains, suggesting that this "tool-calling" style is a promising way forward for making robots that can truly understand and move through our complex world.

Technical Summary: AgenticNav

Problem Statement

Zero-shot Vision-and-Language Navigation in Continuous Environments (VLN-CE) aims to enable agents to follow natural language instructions in unseen 3D scenes without relying on predefined navigation graphs or task-specific training. While Large Vision-Language Models (VLMs) have made zero-shot reasoning feasible, existing state-of-the-art methods (e.g., Open-Nav, SmartWay, EvoNav) face three critical bottlenecks in the interface between the VLM and the environment:

Restricted Action Space: Current methods rely on learned waypoint predictors that propose a small, fixed set of navigable candidates. This limits the VLM's ability to select instruction-relevant locations if they are not included in the predictor's output.
Ineffective Depth Utilization: Although depth data is available, it is often fed as raw input or hidden within the waypoint predictor. This prevents the VLM from accessing precise metric distances for specific spatial reasoning tasks (e.g., comparing distances to obstacles).
Inefficient Memory Management: Existing approaches either accumulate long textual/visual histories (introducing irrelevant context and distracting the model) or retrieve cross-episode experiences (violating the strict zero-shot assumption by relying on prior episode data).

Methodology: AgenticNav

The authors propose AgenticNav, a lightweight navigation harness that reframes zero-shot VLN-CE as an agentic tool-calling process. Instead of forcing the VLM to choose from pre-computed waypoints or parse dense depth maps, AgenticNav exposes three specific, grounded tools that the VLM can call on demand:

1. Action Tool (Pixel-Level Selection)

Mechanism: The VLM directly selects a target pixel $(u, v)$ in an RGB observation rather than choosing from a list of predicted waypoints.
Execution: The harness converts this 2D pixel selection into a 3D executable motion $(\theta, \rho)$ using depth unprojection and camera calibration.
Safety: Before execution, a deterministic safety check verifies that the path is free of obstacles (using body-height point filtering) and that the depth is valid. If the target is unsafe, the tool returns a Reselect signal, prompting the VLM to choose again.
Benefit: This removes the dependency on trained waypoint predictors, allowing the VLM to navigate to any visible, accessible location.

2. Depth Tool (On-Demand Metric Queries)

Mechanism: Instead of providing full depth maps, the VLM can call query_depth(P) on a batch of specific pixels.
Output: The tool returns metric depth values, local depth statistics (min, mean, median from a $5\times5$ neighborhood), and an action preview (estimated yaw and forward distance).
Benefit: This allows the VLM to perform targeted spatial comparisons (e.g., "Is the doorway closer than the obstacle?") without the cognitive load of parsing a full depth image, providing precise local evidence only when needed.

3. Agentic Memory and Recall Tool

Mechanism: The system maintains a compact Map Image summarizing the trajectory and a bounded cache of past observations.
Recall: When the VLM determines that visual details from a past decision point are necessary, it calls recall(p, k) to retrieve a specific cached RGB image.
Benefit: This ensures that the context window only grows with task-relevant information, avoiding the accumulation of irrelevant historical frames while maintaining the strict episodic zero-shot setting (no cross-episode data retrieval).

Key Contributions

Agentic Tool-Calling Framework: The paper introduces AgenticNav, which rethinks the VLM-environment interface as a tool-calling process, eliminating the need for learned waypoint predictors.
Agentic Depth Interface: A novel mechanism allowing VLMs to request metric depth at precise locations, shown to be more effective for spatial reasoning than raw depth inputs or waypoint-based proxies.
Selective Memory Mechanism: A hybrid memory system combining a compact trajectory map with a selective recall tool, significantly improving long-horizon decision-making without context bloat.
State-of-the-Art Performance: The method achieves new SOTA results in zero-shot VLN-CE on the R2R-CE benchmark, demonstrating that the harness design is as critical as the VLM backbone itself.

Experimental Results

Simulation (R2R-CE Val Unseen)

Evaluated on the standard 100-episode protocol without policy training:

Performance: Using GPT-5.5 as the backbone, AgenticNav achieved 55% Success Rate (SR) and 48.41% Success Weighted by Path Length (SPL).
Comparison: This outperforms the same-backbone reproduction of SmartWay (44% SR, 35.04% SPL) by 11 percentage points in SR and 13.37 points in SPL.
Generalization: Using Gemini-2.5-Pro, AgenticNav also surpassed Open-Nav and EvoNav under the same benchmark, confirming the approach is not tied to a single VLM.
Ablations:
- Removing the Action Tool (reverting to waypoint predictors) dropped SR by 5 points.
- Removing the Depth Tool dropped SR to 42%; simply adding depth images as input only recovered part of the loss, highlighting the value of structured queries.
- Removing the selective recall mechanism dropped SR to 41%, proving the value of targeted memory retrieval over static maps alone.

Real-World Validation

Deployed on a four-wheel robot with an RGB-D camera in laboratory, office, and yard environments:

AgenticNav (4-view) achieved 46.7% SR and 2.67m Navigation Error (NE), significantly outperforming SmartWay (23.3% SR, 3.57m NE).
Qualitative analysis showed that AgenticNav successfully handled cases where SmartWay's waypoint predictor failed to propose the correct direction, and effectively used depth tools to time turns and recall tools to maintain long-horizon constraints.

Significance and Claims

The paper argues that for foundation-model-based navigation, the design of the action, perception, and memory harness is as important as the choice of the VLM itself.

Shift in Paradigm: The limiting factor in zero-shot VLN-CE is no longer solely the reasoning capability of the model, but how the model is grounded to the environment.
Zero-Shot Integrity: By avoiding learned predictors and cross-episode memory, AgenticNav maintains a strictly episodic zero-shot setting, making it more robust for deployment in novel scenes where no prior training data exists.
Efficiency: The tool-calling interface allows the VLM to decide what evidence to inspect (depth, memory) and where to move, keeping safety checks deterministic while maximizing the model's reasoning freedom.

The authors conclude that while VLM decision quality remains the primary source of failure (accounting for ~89% of simulation failures), the AgenticNav harness successfully unlocks the potential of foundation models by providing a direct, grounded, and continuous interface to the physical world.