💻 computer science

MiniVLA-Nav v1: A Multi-Scene Simulation Dataset for Language-Conditioned Robot Navigation

The paper introduces MiniVLA-Nav v1, a publicly available simulation dataset comprising 1,174 episodes across four photorealistic Isaac Sim environments that pairs natural language instructions with synchronized visual and expert action data to benchmark language-conditioned object approach navigation for the NVIDIA Nova Carter robot.

Original authors: Ali Al-Bustami, Jaerock Kwon

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Al-Bustami, Jaerock Kwon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to walk into a room, find a specific object (like a "red chair" or a "fire extinguisher"), and stop right in front of it, all while you only give it a simple spoken command like, "Go to the chair."

This is exactly what the paper MiniVLA-Nav v1 is about. The authors have created a massive digital training ground (a simulation dataset) to help robots learn this specific skill without needing to crash real robots thousands of times in the real world.

Here is a breakdown of what they built, using simple analogies:

1. The "Video Game" Training Ground

Instead of using a real robot in a real office or hospital, the researchers built four different virtual worlds inside a powerful computer program called Isaac Sim.

The Worlds: They created photorealistic versions of an Office, a Hospital, a Full Warehouse, and a Warehouse with many shelves.
The Robot: They use a digital twin of a real robot called the Nova Carter (a two-wheeled robot that moves like a Roomba but steers like a car).
The Goal: The robot gets a text instruction (e.g., "Drive to the trash can") and must navigate the virtual room to find that object and stop within 1 meter of it.

2. The "Perfect Teacher" (The Expert)

To teach the robot, you need someone who knows exactly how to do the task perfectly. In this dataset, the "teacher" is a computer program (a proportional controller) that acts like a perfect GPS.

It sees the object, calculates the shortest path, and tells the robot exactly how fast to go and how sharply to turn at every single moment.
The dataset records 1,174 successful trips where this "perfect teacher" guided the robot to the target.
Why this matters: Just like a student learns best by watching a master chef cook, a robot learns best by mimicking these perfect, recorded movements.

3. The "Training Menu" (Variety is Key)

If you only practice walking in a straight line on an empty hallway, you won't learn to navigate a crowded kitchen. The authors made sure the training data was diverse:

Different Distances: The robot starts at three different distances from the target: Near (just a few steps away), Mid (across the room), and Far (all the way across the building).
Different Words: They used 30 different sentence templates. Some say "Go to the chair," others say "Drive to the chair and stop," or "Find the chair." This teaches the robot that different words can mean the same thing.
Different Objects: There are 12 types of objects (chairs, tables, fire extinguishers, barrels, etc.). Some are used for training, and some are "hidden" to test if the robot can guess what to do with objects it has never seen before.

4. The "Sensory Package"

Every time the robot takes a step in the simulation, the dataset saves a complete "snapshot" of what the robot experiences, all synchronized together:

Eyes: A 640x640 color photo (RGB).
Depth Sense: A map showing exactly how far away everything is (Metric Depth).
Focus: A mask that highlights exactly which pixels belong to the target object (Instance Segmentation).
The Lesson: The exact speed and turn angle the "perfect teacher" commanded at that exact moment.

5. The "Final Exam" (Evaluation)

The researchers didn't just dump the data; they organized it into five different test groups to see how well a robot learns:

Standard Test: Can it find objects it knows using sentences it has heard before?
Paraphrase Test: Can it understand "Head to the chair" if it was only trained on "Go to the chair"?
New Object Test: Can it find a "barrel" if it was only trained on "chairs" and "tables"?

What the Paper Actually Found

Efficiency: The "perfect teacher" is very efficient. The distance the robot travels is almost exactly the same as the distance it started from the target (a straight line). This confirms the training data is high-quality and not full of unnecessary detours.
Difficulty: The "Warehouse" scenes are harder than the "Office" scenes. Robots take longer and more steps to navigate the cluttered warehouses, proving the dataset captures real-world difficulty.
Limitations:
- Color Blindness: The current version of the simulation doesn't know the colors of the objects (everything is "unknown"). So, instructions like "Go to the red chair" were removed from the training data for now.
- Selection Bias: The "teacher" struggles in very narrow hallways (like in the hospital scene), so the dataset has fewer examples of navigating tight corners. It mostly shows open paths.
- Simulation vs. Reality: Because this is a video game, the lighting and textures are perfect. A robot trained here might get confused when it sees the messy, imperfect lighting of a real room.

In Summary

MiniVLA-Nav v1 is a digital library of 1,174 perfect driving lessons for robots. It teaches them how to listen to a command, look around a virtual room, and drive straight to a specific object. It is designed to help researchers build better "Vision-Language-Action" models—robots that can see, understand language, and move all at the same time.

The paper explicitly states that this is a dataset release and a pipeline description. The actual training results (how well a specific AI model performed on this data) are saved for a future "companion paper."

1. Problem Statement

The paper addresses a critical gap in Embodied AI: the lack of large-scale, high-quality datasets for Language-Conditioned Object Approach (LCOA) tasks involving continuous differential-drive control.

The Gap: Existing datasets either focus on discrete panoramic actions (e.g., R2R, ObjectNav), tabletop manipulation (e.g., ALFRED), or lack systematic Out-of-Distribution (OOD) evaluation splits.
The Challenge: Collecting real-world robot demonstrations is expensive and logistically difficult. While simulation offers a solution, existing simulators often lack the combination of multi-scene diversity, continuous action spaces, dense per-step expert labels, and multi-modal observations (RGB + Depth + Segmentation) required to train robust Vision-Language-Action (VLA) models.

2. Methodology

The authors developed a scalable, reproducible data generation pipeline using NVIDIA Isaac Sim 5.1 and the NVIDIA Nova Carter differential-drive robot.

A. Task Definition (LCOA)

Goal: Given a natural language instruction and front-facing RGB-D observations, the robot must navigate to a named object and stop within 1.0 meter.
Termination Conditions:
- Success: Robot is within 1m and stationary for $\ge$ 5 steps.
- Collision/Stall: No progress for $\ge$ 16 steps near an obstacle.
- Timeout: 1000 steps reached.
Action Space: Continuous $(v, \omega)$ velocity commands, quantized into a 7 $\times$ 7 token vocabulary (49 tokens) for VLA-style training.

B. Data Collection Pipeline

Simulation Environment: Four photorealistic indoor scenes (Office, Hospital, Full Warehouse, Warehouse with Multiple Shelves) loaded from Isaac Assets.
Robot & Sensors: Nova Carter with a front-facing stereo camera (640 $\times$ 640) providing synchronized RGB, metric depth (float32), and instance segmentation masks at 60 Hz.
Expert Controller: A vision-based proportional controller using instance segmentation masks to track the target.
- Visible Target: Adjusts heading based on mask centroid and speed based on median depth.
- Occluded Target: Uses bearing-only control based on known goal coordinates.
- Obstacle Avoidance: Clamps velocity if foreground depth < 0.25m.
Spawn Strategy: To ensure difficulty diversity, episodes use three spawn-distance tiers:
- Near: 1.5–3.5 m (30% of episodes).
- Mid: 3.5–7.0 m (40%).
- Far: Global curated points (30%, currently underrepresented).
Language Instructions: Generated from 18 training templates and 12 paraphrase-OOD templates. (Note: Color-slot templates were suppressed in v1 due to lack of color metadata in assets).

C. Dataset Structure

Total Episodes: 1,174 validated successful episodes.
Modalities: Synchronized RGB, Depth, Segmentation, Continuous Actions $(v, \omega)$ , and Discretized Action Tokens.
Evaluation Splits: Five splits designed to test:
1. In-distribution accuracy (train_id, val_id, test_id).
2. Template-paraphrase robustness (test_paraphrase_ood).
3. Object-category generalization (test_ood_obj).

3. Key Contributions

First Multi-Scene LCOA Dataset: The first publicly available dataset combining multiple indoor scene types, continuous differential-drive expert actions, language instructions, and multi-modal per-step observations.
Scalable Generation Pipeline: A fully automated, resumable pipeline built on Isaac Sim 5.1 that ensures bit-for-bit reproducibility via a single random seed.
Structured OOD Evaluation: Five evaluation splits specifically designed to benchmark model robustness against paraphrased instructions and unseen object categories.
Rich Annotation: Includes 12 object categories, 3 spawn-distance tiers, and synchronized multi-modal data (RGB, Depth, Segmentation) at 60 Hz.
Comprehensive Statistics: Detailed analysis of trajectory lengths, navigation errors, and spawn distance correlations.

4. Results and Statistics

Trajectory Diversity: There is a strong linear correlation ( $r=0.94$ $r = 0.94$ ) between spawn distance and trajectory length, confirming the expert controller follows efficient, near-straight paths.
- Near-tier: Mean length 1.56 m (129 steps).
- Far-tier: Mean length 4.46 m (415 steps).
Navigation Accuracy: All retained episodes are successful, with a mean Navigation Error (NE) of 0.967 m, tightly clustered just below the 1.0 m success threshold due to the "stop-hold" criterion.
Scene Difficulty: Warehouse scenes produce significantly longer episodes (median 189–207 steps) compared to Office/Hospital scenes (156–160 steps) due to clutter and corridor constraints.
Expert Success Rates: Vary by scene complexity (Office: ~80–90%; Hospital/Warehouse: ~40–60%), introducing a selection bias toward unobstructed paths in the current dataset.

5. Significance and Future Work

Significance: MiniVLA-Nav v1 fills a critical void in the VLA research landscape by providing a benchmark for continuous wheeled navigation with language grounding. It enables the training of end-to-end policies that can generalize to new object categories and instruction phrasings, moving beyond discrete action spaces.
Limitations & Future Work:
- Far-tier Underrepresentation: Only 1.2% of episodes are "far-tier"; future updates will expand global spawn points.
- Color Annotation: Currently suppressed; future versions will add mesh-based color extraction.
- Expert Bias: The proportional controller struggles in narrow corridors; future work will integrate path-planning experts (e.g., RRT*) for better coverage.
- Sim-to-Real Gap: Differences in rendering, depth noise, and floor dynamics require domain randomization before real-world deployment.
- Dynamic Obstacles: Current scenes are static; future versions will include moving obstacles.

The dataset is publicly available on Hugging Face, accompanied by the generation pipeline and evaluation tooling, with baseline training results promised in a companion publication.