LiveVLN: Breaking the Stop-and-Go Loop in Vision-Language Navigation

1. Problem Statement: The "Stop-and-Go" Bottleneck

Despite strong benchmark performance in Vision-Language Navigation (VLN), real-world deployment often suffers from a "stop-and-go" behavior. This is not merely a computational bottleneck but a structural issue inherent in the standard deployment architecture:

• Blocking Interface: Traditional VLN systems follow a serialized three-stage loop: Sense $\rightarrow$ Inference $\rightarrow$ Execution. The robot must pause execution completely while the system senses the environment, transmits data, and infers the next action sequence.
• Latency Mismatch: If the time required for sensing and inference ($\ell_{infer}$) exceeds the time the current action buffer can sustain motion, the robot idles.
• Empirical Evidence: In native deployments (e.g., using the NaVIDA system on a Unitree G1 robot), the controller waits for an average of 10.64 seconds per episode, resulting in a 30.5% waiting ratio and frequent pauses (9.25 pauses per episode). This leads to jerky, inefficient motion that degrades real-world usability.

2. Methodology: LiveVLN Framework

The authors propose LiveVLN, a training-free runtime framework designed to decouple execution from the sense-and-inference cycle. It enables continuous motion by overlapping execution with background inference.

Core Mechanisms

LiveVLN introduces a dual-thread runtime and a short-horizon action state composed of three parts:

1. Executed Actions ($e_t$): Actions already performed.
2. Guard Buffer ($g_t$): A committed prefix of actions currently being executed by the controller.
3. Revisable Tail ($r_t$): A hidden suffix of the predicted action sequence that has not yet been released to the controller.

Key Processes

• Guarded Handoff:

  • Thread A (Execution): Continuously executes the current Guard Buffer.
  • Thread B (Inference): In parallel, uses the newest observation to generate a refreshed continuation based on the current state and the committed guard buffer.
  • Handoff Logic: If Thread B finishes before Thread A exhausts the Guard Buffer, the system performs a handoff: the current buffer becomes "Executed," the refreshed prefix becomes the new "Guard Buffer," and the remaining suffix becomes the new "Revisable Tail."
  • Revisability: Because the tail is not yet executed, it can be overwritten by the next inference round if new observations suggest a different path, allowing for online correction without stopping.

• Real-Time Adaptation:

  • Instead of using a fixed number of actions for the guard buffer, LiveVLN dynamically sizes the buffer based on wall-clock time.
  • It estimates the latency of the next sense-and-inference pass ($\psi_{t+1}$) using an exponential moving average of recent latencies.
  • The system selects the shortest prefix of actions whose predicted execution time covers $\psi_{t+1}$. This ensures the robot never runs out of actions while waiting for the next inference.

3. Key Contributions

1. Diagnosis of Runtime Latency: The paper provides a rigorous analysis showing that stop-and-go behavior is caused by the exposed latency of the sense-inference loop, not just policy quality. It quantifies that native deployments waste ~30% of episode time waiting.
2. Training-Free Runtime Framework: LiveVLN requires no retraining of the underlying Vision-Language Model (VLM). It acts as a wrapper for any compatible pretrained VLM navigator that supports multi-step action continuations.
3. Dual-Thread Architecture: It introduces a novel runtime structure that overlaps execution with inference, utilizing a "Guard Buffer" to hide latency and a "Revisable Tail" to maintain adaptability.
4. Comprehensive Evaluation: The framework is validated across simulation benchmarks (R2R, RxR) and, crucially, in real-world deployments on a Unitree G1 robot.

4. Experimental Results

The authors evaluated LiveVLN using StreamVLN and NaVIDA navigators on the Unitree G1 robot.

• Benchmark Performance (Simulation):

  • LiveVLN preserves the navigation quality of the base models. On R2R and RxR val_unseen, metrics like Success Rate (SR) and Success Weighted by Path Length (SPL) remained comparable to the native checkpoints (e.g., NaVIDA SR dropped slightly from 61.4% to 59.9%, within statistical noise).

• Real-Robot Deployment (Continuity & Efficiency):

  • Waiting Time: Reduced by 77.7% (StreamVLN) and 72.8% (NaVIDA).
  • Pause Count: Drastically reduced from 9.25 pauses/episode to **1.20 pauses/episode**.
  • Wall-Clock Time: Total episode duration shortened by 12.6% (StreamVLN) and 19.6% (NaVIDA).
  • Visible Gap: The time the robot waits between actions (visible gap) dropped from ~1.0s to ~0.16s.
  • Task Success: The rate of stopping within 2m of the goal remained comparable, proving that continuity improvements did not compromise task completion.

• Ablation Studies:

  • Removing the Revisable Tail increased the pause count and reduced task stability, proving the tail is essential for online correction.
  • Removing Real-Time Adaptation increased the waiting ratio, proving that dynamic buffer sizing is crucial for hiding latency.
  • Simply increasing inference frequency without LiveVLN's overlapping mechanism increased the waiting ratio, confirming that the solution is architectural, not just computational.

5. Significance and Implications

• Redefining VLN Evaluation: The paper argues that standard benchmarks (SR, SPL) are insufficient for real-world deployment. Continuity metrics (waiting time, pause count, wall-clock efficiency) must be treated as first-class objectives.
• Runtime vs. Policy: It demonstrates that significant improvements in embodied AI behavior can be achieved by optimizing the runtime execution loop rather than solely focusing on training better policies.
• Scalability: As a training-free framework, LiveVLN can be immediately integrated with existing and future VLM-based navigators, accelerating the deployment of smooth, continuous robotic navigation.

In summary, LiveVLN solves the fundamental latency mismatch in embodied navigation by keeping the robot moving while the "brain" thinks, effectively turning a blocking, stop-and-go system into a continuous, fluid control loop.