🤖 AI

RTNav: Towards Real-Time Zero-Shot Object Navigation

The paper introduces RTNav, a real-time navigation architecture that explicitly accounts for inference latency and asynchronous environment stepping to overcome the performance degradation of existing zero-shot object navigation methods under realistic timing constraints, achieving significant improvements in success rates on HM3D benchmarks.

Original authors: Easop Lee, Lingyu Zhang, Boyuan Chen

Published 2026-08-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Easop Lee, Lingyu Zhang, Boyuan Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot sent into an unfamiliar house to find a specific object, like a red mug, without any prior training on that particular room. To succeed, the robot must look around, understand what it sees, decide where to go next, and move its wheels or legs, all while the world around it continues to change. This is the challenge of zero-shot object navigation: using powerful artificial intelligence to guide a machine through the unknown. For years, researchers have tested these systems in computer simulations where the virtual world pauses while the robot's brain thinks. In this frozen state, the robot can take as long as it needs to process an image or make a decision, and the clock does not tick. This setup has allowed scientists to build increasingly complex and capable navigation agents, but it hides a critical flaw. In the real world, time never stops. While a robot is calculating its next move, seconds pass, the robot stands still, and the task budget—the time allowed to finish the job—shrinks. If a robot takes too long to think, it simply cannot finish the task, no matter how smart its reasoning is.

A team of researchers at Duke University has now exposed this hidden cost and proposed a new way to build navigation systems that respect the ticking clock. They found that the most advanced navigation agents, which rely on large, powerful AI models to understand language and vision, suffer a dramatic drop in performance when forced to operate in real-time. In their study, they created a new testing environment where the simulation runs continuously, just like a real room, while the robot's computer works to decide its next step. When they ran standard navigation methods in this setting, the agents became sluggish and inefficient. The time spent waiting for the computer to finish its calculations meant the robot spent a significant portion of its time standing still, often missing opportunities to reach its goal before the time ran out. The researchers measured this using a new metric that rewards not just finding the object, but doing so quickly, penalizing any wasted seconds.

To solve this, the team introduced a new architecture called RTNav, which treats the flow of time as a fundamental part of the design rather than an afterthought. Instead of forcing the robot to stop and wait for every part of its brain to finish a single thought before moving, RTNav lets different parts of the system work at their own natural speeds. The part of the system that detects objects runs constantly, scanning the environment many times a second. The part that plans the route runs less frequently, only updating when necessary. The part that controls the wheels moves continuously, reacting to the latest plan without waiting for a new one to be fully formed. This is similar to how a human might walk through a crowded street: you do not stop completely to think about every step; you keep moving while your eyes scan for obstacles and your mind updates your path. By allowing these processes to happen simultaneously rather than in a strict line, the robot stays in motion and uses its time efficiently.

The results of this approach were striking. When tested on standard navigation benchmarks, the new system outperformed previous methods significantly. In tests where the robot had to find objects in complex, multi-room environments, the new system improved the success rate by up to 11 percent compared to the best existing methods. More importantly, it completed tasks much faster. The researchers measured a metric called success weighted by completion time, which combines whether the robot found the object with how long it took. The new system scored up to 5.1 points higher on this metric, indicating that it was not only more successful but also far more efficient. The study showed that older methods, which were designed for the frozen simulation world, wasted a large amount of time waiting for calculations to finish. In contrast, the new system kept the robot moving, reducing the time it spent idle by more than 14 percent compared to its competitors.

The researchers also tested how robust this system is by running it on different types of computer hardware, from powerful desktop graphics cards to smaller, more energy-efficient chips designed for edge devices. The system performed consistently well across all of them, maintaining high success rates even when the computing power was reduced. This suggests that the design is flexible enough to work on various robots without needing the most expensive hardware available. The team also experimented with different sizes of the artificial intelligence models used for reasoning. They found that while a very large model was not strictly necessary, a medium-sized model provided the best balance of speed and accuracy, allowing the robot to make good decisions without getting bogged down by slow processing.

This work highlights a crucial shift in how we should build robots for the real world. The old way of testing, where the world waits for the robot to think, no longer reflects the reality of deployment. The study demonstrates that for robots to be practical in everyday environments, their software must be designed to handle the constraints of real-time execution. By decoupling the different tasks of perception, planning, and movement, and letting them run in parallel, robots can become much more responsive and effective. The researchers conclude that ignoring the cost of computation time leads to systems that look good in simulation but fail in practice. Their new approach offers a path forward, showing that with the right architecture, robots can navigate the unknown world quickly and reliably, turning the theoretical promise of artificial intelligence into a practical reality.

Technical Summary: RTNav Towards Real-Time Zero-Shot Object Navigation

1. Problem Statement

Current state-of-the-art zero-shot object navigation agents rely heavily on large vision-language foundation models (VLMs) to interpret goals, recognize objects, and plan actions without task-specific training. However, these models introduce non-negligible inference latency (ranging from hundreds of milliseconds to several seconds).

The core problem identified is a mismatch between evaluation protocols and real-world deployment:

Synchronous Benchmarks: Standard navigation benchmarks (e.g., Habitat) operate in a synchronous, step-based manner. The simulator pauses and waits for the agent to compute an action before advancing the environment. In this setting, inference time is effectively "free," and metrics like Success Rate (SR) and Success weighted by Path Length (SPL) do not penalize slow decision-making.
Asynchronous Reality: In real-world deployment, the environment evolves continuously while the agent computes. Inference latency consumes wall-clock time, directly reducing the agent's ability to cover ground within a fixed task budget.
Consequence: Agents designed for synchronous environments suffer consistent performance degradation when evaluated under realistic, time-constrained conditions. The "pause-and-go" behavior caused by sequential execution of perception, reasoning, and action modules renders many foundation-model-based agents impractical for real-time use.

2. Methodology

A. Real-Time Evaluation Framework

To quantify the impact of inference latency, the authors introduced a real-time stepping interface built on a ROS2 layer atop the Habitat simulator.

Decoupled Execution: The simulator advances at a fixed wall-clock rate (30 Hz) independently of the agent's computation.
Asynchronous Interaction: The agent processes observations and computes actions asynchronously. If an action takes $\delta$ seconds to compute, the simulator continues to advance under the previous command during that interval.
Metrics: In addition to standard SR and SPL, the authors utilize Success weighted by Completion Time (SCT), which penalizes agents for taking longer to complete a task, even if the path is efficient.

B. RTNav Architecture

The authors propose RTNav, a modular, asynchronous architecture designed to operate under bounded compute and latency constraints. Unlike traditional sequential loops where the slowest module dictates system reactivity, RTNav allows modules to run at their natural frequencies.

Key Architectural Components:

Shared State Bus: Perception, mapping, planning, and navigation modules run in independent threads, communicating via a shared state without blocking each other.
Perception (High Frequency):
- Uses Owlv2 for high-frequency (approx. 15 Hz) open-vocabulary object detection to capture potential targets.
- Uses Qwen3.5-9B for semantic verification of detected candidates to reduce false positives.
- Uses MobileSAM for segmentation to clean point cloud projections.
Mapping: Maintains a 2D metric map (obstacles, explored regions, frontiers) by back-projecting RGB-D observations and target masks.
Planning (Event-Driven):
- Utilizes a semantically-grounded frontier exploration strategy.
- The VLM is queried only when necessary (e.g., when a frontier needs selection or a new candidate is proposed), rather than in every decision cycle.
- The VLM selects the most promising frontier based on the target description and candidate views.
Navigation: A PointNav policy executes continuous velocity commands. Because it runs independently, the robot continues moving toward the latest available goal while other modules process new data or perform heavy reasoning.

3. Key Contributions

Real-Time Benchmarking Protocol: The introduction of a ROS-based real-time evaluation interface that decouples environment progression from agent computation, allowing existing methods to be evaluated under realistic timing constraints with minimal modification.
Empirical Evidence of Latency Impact: A large-scale reproducibility study demonstrating that zero-shot navigation methods suffer significant performance degradation (up to 38% drop in SR for some methods) when moved from synchronous to real-time evaluation.
RTNav Architecture: A novel asynchronous design that treats inference latency and wall-clock time as explicit design constraints, enabling concurrent execution of perception, reasoning, and control.
Performance Gains: Demonstration that RTNav significantly outperforms prior state-of-the-art methods in real-time settings, improving Success Rate by up to 11% and Success weighted by Completion Time (SCT) by up to 5.1 points.

4. Experimental Results

The authors evaluated RTNav and seven baseline methods (including VLFM, L3MVN, OpenFMNav, Trihelper, GAMap, and BeliefMapNav) on three benchmarks: HM3D-v1, HM3D-v2, and HM3D-OVON.

Performance Degradation in Baselines: All baseline methods showed performance drops in real-time mode compared to synchronous mode. Methods relying on heavier foundation models (e.g., BeliefMapNav) suffered the most severe degradation (e.g., SR dropped from 73.5% to 36.7% on HM3D-v2).
RTNav Superiority:
- Success Rate (SR): RTNav achieved the highest SR across all benchmarks, outperforming the best synchronous baseline by 6.4% (OVON), 10.5% (HM3D-v1), and 9.1% (HM3D-v2).
- Efficiency (SCT): RTNav showed the most significant advantage in SCT, scoring 2.45 to 5.17 points higher than baselines, indicating significantly faster task completion.
- Idle Time: RTNav reduced idle time (time spent stationary due to computation or waiting) by over 14% compared to other baselines, as shown in execution timeline analysis.
- Visual Processing: RTNav processed over 20x more unique detection frames per second than synchronous baselines, reducing "blind gaps" between detections.
Robustness: Sensitivity analysis showed RTNav maintains consistent performance across different hardware tiers (RTX A6000, L40S, Jetson Thor) and with different VLM sizes (Qwen3.5-9B vs. Gemma4-12B), though smaller models (4B) significantly degraded performance.

5. Significance and Claims

The paper argues that the current reliance on synchronous benchmarks creates a "design gap" that overestimates the practicality of foundation model-driven navigation agents.

Architectural Shift: The authors claim that for real-world deployment, navigation systems must adopt continuous-time execution and asynchronous modularity as first-class design principles. The slowest module should not dictate the reactivity of the entire system.
Practicality: By treating wall-clock time as a critical resource, RTNav demonstrates that it is possible to leverage powerful foundation models for zero-shot navigation without sacrificing real-time responsiveness.
Future Direction: The work motivates a shift in the community toward benchmarking algorithms under realistic timing conditions and suggests that future research should extend these asynchronous principles to more tightly coupled embodied tasks like locomotion and manipulation.

The authors remain modest regarding limitations, noting that their current study focuses on static environments and that extending these designs to dynamic scenes and complex manipulation tasks remains an open challenge.