Technical Summary: MemVLN – Episodic and Procedural Memory for Vision-and-Language Navigation

Problem Statement

Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires embodied agents to navigate unseen 3D spaces based on natural language instructions and egocentric RGB observations, without relying on prior maps or auxiliary sensors like depth or odometry. The core challenge lies in satisfying two conflicting demands simultaneously:

1. Long-Horizon Consistency: Agents must maintain a long-term visual history to ensure trajectory consistency and prevent drift over extended paths.
2. Real-Time Latency: Agents must execute low-level actions with minimal latency (high FPS) to function effectively in continuous, dynamic environments.

Existing approaches struggle to balance these needs. Video-based methods often compromise temporal density by sampling fixed, sparse frames, while others compress visual tokens via pooling or merging, which sacrifices spatial details and conflicts with advanced positional encodings (e.g., M-RoPE) and modern transformer architectures. Furthermore, the auto-regressive nature of Large Vision-Language Models (LVLMs) introduces significant inference latency, making real-time control difficult.

Methodology: MemVLN Framework

MemVLN is a novel VLN framework designed to reconcile long-form video context with real-time responsiveness by mimicking human memory systems. It integrates a visual encoder for continuous observation processing and a Large Language Model (LLM) for instruction interpretation and action generation. The architecture relies on two core bio-inspired mechanisms:

1. Episodic Memory with Pyramidal Resolution

To manage extensive visual histories efficiently, MemVLN employs a pyramidal resolution strategy rather than uniform sampling or token merging.

• Mechanism: Historical observations are categorized into three temporal tiers: immediate, short-term, and long-term.
  • Immediate frames (most recent) are processed at high resolution to capture precise local cues.
  • Short-term and long-term frames are progressively downsampled (spatially compressed) to retain global path consistency while minimizing token count.
• Rationale: Attention analysis reveals that recent observations dominate the model's focus, while distant history provides necessary context but requires less perceptual fidelity.
• Advantage: Unlike token merging, which destroys the uniform 2D spatial grid and conflicts with positional encodings like M-RoPE, pyramidal resolution performs downsampling in the raw pixel space before tokenization. This maintains a strict spatial topology, ensuring compatibility with modern LVLM backbones (e.g., Qwen3-VL) and DeepStack.

2. Procedural Memory for Fast Action

To overcome the latency bottleneck of auto-regressive decoding, MemVLN introduces a fast action mechanism inspired by procedural memory.

• Mechanism: Instead of generating action sequences token-by-token (e.g., "turn left 30 degrees"), the model maps continuous control signals to a compact vocabulary of atomic mid-level action tokens.
• Implementation: These augmented actions (e.g., <a1> for "move forward", <a2> for "turn left") are single tokens within the LLM's vocabulary.
• Benefit: This allows the model to predict complex navigational decisions in a single forward pass (one-shot decoding), bypassing the iterative decoding loop that typically causes delays.

Key Contributions

1. MemVLN Architecture: A novel VLN model that effectively leverages long-form visual observations while boosting inference speed to 14 FPS.
2. Bio-Inspired Memory Mechanism:
   • Episodic Memory: A pyramidal resolution strategy that balances high-fidelity local navigation with compressed global history, optimizing observation management without increasing token budgets.
   • Procedural Memory: A fast action module using a single-token vocabulary for mid-level actions, reducing inference latency by bypassing auto-regressive decoding.
3. State-of-the-Art Performance: Validation on standard VLN-CE benchmarks (R2R and RxR) using exclusively monocular RGB inputs, demonstrating that superior generalization can compensate for the lack of depth or panoramic sensors.

Experimental Results

The authors evaluated MemVLN on the R2R-CE and RxR-CE benchmarks (Val-Unseen splits) using the Qwen3-VL backbone.

• Performance:
  • MemVLN-8B achieved state-of-the-art results with 65.3% Oracle Success (OS) and 58.4% Success Rate (SR) on R2R, and 66.0% SR and 57.3% Success weighted by Path Length (SPL) on RxR.
  • MemVLN-4B surpassed the baseline Qwen3-VL-4B by 5.8% SR on R2R and 9.7% SR on RxR.
  • The model outperformed existing waypoint-based and continuous control methods, as well as other VLM-based approaches (e.g., NaVILA, Uni-NaVid, StreamVLN), despite using only single-view RGB inputs.
• Efficiency:
  • The procedural memory mechanism achieved a 7× speedup in inference latency compared to standard auto-regressive decoding, increasing throughput from 2 FPS to 14 FPS.
  • Ablation studies confirmed that the fast action module improved SR by 1.8% on R2R and 2.1% on RxR while drastically reducing latency.
• Ablation Insights:
  • Pyramidal Resolution: The "Ascending" configuration (prioritizing recent high-resolution frames) yielded the best performance, improving SR by 1.2% and SPL by 1.9% on R2R compared to non-pyramidal baselines.
  • Training Data: Performance scaled with data volume. The inclusion of DAgger-generated synthetic data significantly improved OS and SR, highlighting the value of interactive, on-policy data.

Significance and Claims

The paper claims that MemVLN establishes a scalable and robust path for deploying VLN models in continuous environments. By decoupling the need for long-horizon context from computational cost through pyramidal resolution, and by eliminating auto-regressive latency through procedural memory, MemVLN demonstrates that real-time, high-performance navigation is achievable using only monocular RGB inputs and standard LVLM backbones. The work suggests that future embodied agents can rely on unified vision-language frameworks that are both temporally consistent and reactively fast, without requiring specialized sensors or complex modular pipelines.