💻 computer science

MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learning

This paper introduces MM-Nav, a multi-view Vision-Language-Action model that achieves robust visual navigation by leveraging a teacher-student framework to distill diverse capabilities from privileged-depth RL experts across reaching, squeezing, and avoiding tasks, ultimately outperforming its teachers in both synthetic and real-world environments.

Original authors: Tianyu Xu, Jiawei Chen, Jiazhao Zhang, Wenyao Zhang, Zekun Qi, Minghan Li, Zhizheng Zhang, He Wang

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianyu Xu, Jiawei Chen, Jiazhao Zhang, Wenyao Zhang, Zekun Qi, Minghan Li, Zhizheng Zhang, He Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to walk through a busy, messy house without bumping into anything. This is the challenge of visual navigation. The problem is that cameras (like our eyes) only see pictures; they don't naturally "know" how far away a chair is or how thin a wire is, unlike a laser scanner (LiDAR) which measures distance directly.

The paper introduces MM-Nav, a new "brain" for robots that solves this by combining three powerful ideas: learning from expert teachers, practicing in a video game, and reading real-world books.

Here is how it works, broken down into simple concepts:

1. The Three "Specialist Coaches" (RL Experts)

Instead of trying to teach one robot to do everything at once, the researchers first created three separate "expert coaches" inside a computer simulation (a video game). Each coach is a master of one specific skill:

The Reaching Coach: Teaches the robot how to walk straight to a target while avoiding big, static furniture.
The Squeezing Coach: Teaches the robot how to wiggle through tiny, crowded gaps between walls and objects.
The Avoiding Coach: Teaches the robot how to dodge people or moving objects that are running around randomly.

These coaches are "privileged," meaning they have super-vision (they can see exact distances) that the final robot won't have. They generate millions of perfect examples of how to move.

2. The "Student" Robot (The VLA Model)

The main character, MM-Nav, is a "student" robot. It is a Vision-Language-Action (VLA) model. Think of this as a robot with a brain that can:

See: It looks at 360-degree video (front, back, left, right) like a human turning their head.
Read: It understands language and can answer questions about what it sees.
Act: It doesn't just say "turn left"; it calculates the exact speed and direction to move smoothly.

3. The Training Strategy: "Practice Makes Perfect"

The researchers used a two-step training method to turn the student into a master:

Step 1: The Crash Course (Offline Learning)
The student robot first studies the millions of perfect examples generated by the three specialist coaches. It learns the basics of reaching, squeezing, and avoiding.
Step 2: The "Balanced" Tutoring (Online Iteration)
Here is the clever part. The student robot goes back into the simulation to practice. If the robot is good at "Reaching" but terrible at "Squeezing," the system automatically gives it more practice data on squeezing. It's like a tutor who notices you are struggling with math but good at history, so they give you extra math homework to balance things out. This ensures the robot becomes good at all skills, not just the easy ones.

4. Bridging the Gap: The "Real-World Library"

There is a big problem with training robots in video games: the real world looks different (different lighting, messy objects, weird textures). This is called the "Sim-to-Real Gap."

To fix this, the researchers didn't just use game data. They also fed the robot 300,000 real-world Visual Question Answering (VQA) examples.

The Analogy: Imagine the robot is a student who has only ever studied in a cartoon world. To prepare for the real world, they are also given a library of real-world photos and questions like, "How is the pedestrian moving?" or "Where is the clear path?"
By learning to answer these questions about real photos, the robot's brain learns to understand real-world textures and lighting, making it much less likely to get confused when it leaves the video game and enters a real hallway.

5. The Results: Beating the Teachers

When they tested MM-Nav in both the simulation and the real world (using a Unitree robot dog with cameras all around it), the results were impressive:

It works in the real world: The robot successfully navigated through narrow corridors, avoided thin wires (which are hard for lasers to see), and dodged moving people.
It beat the experts: Surprisingly, the "student" robot eventually performed better than the individual "coach" robots it was trained on. By combining the skills of reaching, squeezing, and avoiding into one brain, it became more robust and adaptable than any single specialist.
Speed: It thinks and moves fast enough (7 times per second) to react instantly to obstacles.

Summary

MM-Nav is a robot brain that learns to navigate by watching three different expert teachers in a video game, but it also reads a library of real-world photos to understand what the real world actually looks like. By balancing its practice so it doesn't get lazy at any one skill, it becomes a master navigator that can handle messy, crowded, and moving environments better than its teachers ever could.

Technical Summary: MM-Nav

Problem Statement

Visual navigation is a critical research direction for robotics, aiming to emulate human navigation by leveraging egocentric visual observations. However, unlike LiDAR point clouds or depth maps, raw visual observations lack explicit geometric information, making navigation in cluttered or dynamic environments particularly challenging. Existing learning-based methods face significant limitations:

Data Scarcity and Bias: Real-world data collection is expensive, often restricted to limited camera configurations (e.g., front-view only), and rarely includes hazardous scenarios like collision avoidance due to safety risks.
Sim-to-Real Gap: Synthetic data, while flexible, suffers from non-photorealistic rendering, creating a gap that hinders transfer to real-world deployment.
Capability Limitations: Current models are often confined to spacious environments and struggle with complex tasks such as squeezing through narrow gaps or avoiding dynamic obstacles.
Action Space Constraints: Many existing Vision-Language-Action (VLA) models rely on discrete action spaces (e.g., "turn left"), which limits agility compared to continuous velocity control required for robust real-world navigation.

Methodology: MM-Nav

The authors propose MM-Nav, a 7B-parameter multi-view VLA model designed to learn diverse navigation capabilities from synthetic expert data while mitigating the sim-to-real gap through co-training with large-scale real-world Visual Question Answering (VQA) data.

1. Multi-Capability RL Experts

To address the lack of challenging scenarios in existing datasets, the authors construct tailored simulation environments using IsaacLab and train three distinct Reinforcement Learning (RL) experts using Proximal Policy Optimization (PPO). These experts utilize privileged information (depth maps) to master specific skills:

Reaching: Approaching a point goal while avoiding static obstacles of various shapes and textures.
Squeezing: Navigating through densely packed pillars and narrow, randomized gaps.
Avoiding: Actively avoiding crowded dynamic obstacles moving at random speeds.

2. Student VLA Model Architecture

The student model is a multi-view VLA policy that processes 360° RGB observations from four cameras (Front, Right, Back, Left).

Visual Encoding: A sliding window of history (8 steps) is encoded using a visual foundation model (SigLIP). The architecture employs a hybrid tokenization strategy: fine-grained tokens for the current observation and coarse-grained tokens for history, resulting in a fixed sequence length of 192 tokens to ensure consistent inference speed.
Language Integration: Point goals are formatted as textual prompts and encoded as language tokens using a Large Language Model (Qwen2-7B). This allows for co-training with open-world VQA data.
Action Head: The model outputs continuous omnidirectional velocity commands ( $v_x, v_y, v_{yaw}$ ) via a two-layer MLP, achieving an inference speed of 7 Hz.

3. Training Strategy: Capability-Balanced Iteration

The training pipeline consists of two stages:

Initial Fine-Tuning: The VLA model is initialized on 500k successful expert demonstrations (collected from the three RL teachers) and 100k real-world VQA data.
Online Teacher-Student Iteration (DAgger-style): The VLA model is deployed in simulation to collect trajectories. A capability-balancing module dynamically adjusts the data ratio for the next fine-tuning step. It calculates a performance gap metric (Weighted Travel Time) between the student and the RL teachers for each capability. Data from capabilities where the student underperforms are up-weighted to ensure stable convergence and prevent capability imbalance.

Key Contributions

Navigation Data: Construction of tailored multi-capability simulation environments and training of privileged RL experts to provide diverse supervision for reaching, squeezing, and dynamic obstacle avoidance—scenarios difficult to capture in the real world.
Training Strategy: Introduction of a capability-balanced iterative refinement procedure that mitigates policy distribution shifts and ensures balanced improvement across different navigation skills.
Model Design: Development of an efficient 7B multi-view VLA policy that utilizes 360° RGB observations to directly predict continuous velocity commands, enabling robust zero-shot sim-to-real transfer.

Experimental Results

Simulation Benchmarks

InternVLA-N1 System-1 Benchmark: MM-Nav achieved a Success Rate (SR) of 88.1% on the point-goal navigation benchmark, outperforming the best previous method (72.2%) by +15.9 points. It significantly outperformed prior RGB-D and depth-based approaches despite relying solely on RGB inputs.
Custom IsaacLab Environments: In tests across Reaching, Squeezing, Avoiding, and Mixed scenes, MM-Nav achieved the highest SR and lowest Collision Rate (CR) among baselines (including NavDP, ViPlanner, and iPlanner). Notably, it achieved an SR of 71% in the challenging "Squeezing" scene, compared to 18% for NavDP.
Student vs. Teacher: The VLA student model eventually outperformed the specialized RL teachers, demonstrating the synergistic effect of integrating multiple capabilities.

Real-World Deployment

Extensive real-world experiments on a Unitree GO2 robot confirmed robust zero-shot transfer in diverse scenarios:

Thin Wire Avoidance: Successfully avoided thin suspended wires, which are difficult for LiDAR to perceive.
Cluttered Corridor: Navigated narrow passages with transparent glass walls and obstacles.
Pedestrian Avoidance: Adjusted trajectories online to avoid moving pedestrians.
Outdoor Alley: Navigated low-light environments with unseen obstacles (e.g., stone bollards).

Ablation Studies

VQA Co-training: Removing VQA data resulted in a performance drop of over 20% in simulation and a failure to generalize to real-world thin/irregular obstacles. UMAP analysis showed VQA data significantly reduced the latent distribution gap between simulation and reality.
Multi-view vs. Single-view: The 360° multi-view input provided a +4.8 SR gain over a single-view variant, validating the importance of surround-view perception for spatial awareness.
Balanced Data: The capability-balanced aggregation method led to faster and more stable training compared to unbalanced data collection.

Significance and Claims

The paper claims that MM-Nav provides a scalable and effective blueprint for training a new generation of general-purpose visual navigation agents. Its primary significance lies in:

Overcoming the Sim-to-Real Gap: By leveraging the generalization capabilities of VLA models and co-training with real-world VQA data, the model successfully transfers skills learned in synthetic, privileged environments to real-world, depthless RGB navigation.
Synergistic Learning: The work demonstrates that a unified VLA policy can learn and integrate multiple distinct navigation capabilities (reaching, squeezing, avoiding) more effectively than specialized RL policies trained in isolation or mixed-scene RL policies.
Efficiency: The model achieves robust performance at 7 Hz inference speed using a 7B parameter model, making it viable for real-time deployment on standard robotic platforms.

The authors conclude that their method effectively addresses the limitations of current visual navigation policies by combining specialized expert knowledge with large-scale semantic understanding, enabling robust navigation in complex, dynamic, and cluttered environments.