RVN-Bench: A Benchmark for Reactive Visual Navigation

The paper introduces RVN-Bench, a new collision-aware benchmark built on Habitat 2.0 and HM3D scenes that enables the training and evaluation of safe, robust indoor visual navigation policies for mobile robots in unseen, cluttered environments.

The Gist

Imagine you are teaching a robot dog to navigate a messy living room. The room is full of coffee tables, scattered toys, and fragile vases. Your goal is simple: tell the robot, "Go to the kitchen," and it needs to get there without knocking anything over.

This is the problem the paper RVN-Bench is trying to solve.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Blind" Robot

Currently, most robot navigation tests are like playing a video game where the walls are invisible.

• The Old Way: Existing benchmarks (tests) ask robots, "Did you reach the kitchen?" If the robot crashed through a wall or smashed a vase but still ended up in the kitchen, the old tests would say, "Great job! 100% success!"
• The Reality: In the real world, crashing is bad. Robots need to be "collision-aware." They need to see the vase and stop, not just drive through it.
• The Gap: Most tests are designed for cars driving on open highways (outdoors) or robots that ignore obstacles. There wasn't a good "driving school" for indoor robots that actually penalizes them for bumping into furniture.

2. The Solution: RVN-Bench (The "Obstacle Course")

The authors created RVN-Bench, which is like a high-tech, virtual obstacle course specifically for indoor robots.

• The Simulator: They built this inside a video game engine called Habitat, using 3D scans of real houses (HM3D). It looks and feels like a real home.
• The Rules: The robot is given a series of goals (e.g., "Go to the sofa," then "Go to the door"). It can only use its camera (eyes) to see. It has no map in its head.
• The Twist: If the robot bumps into a wall, it gets a big "penalty." The test measures two things:
  1. Did it get there? (Success)
  2. Did it break anything? (Safety)

3. The Secret Weapon: The "Crash Dataset"

One of the coolest parts of this paper is how they teach the robot to avoid crashing.

• The Problem with Real Life: If you want to teach a robot what a crash feels like, you have to actually crash it. In the real world, this is expensive (broken robots) and dangerous.
• The RVN-Bench Trick: They created a special tool that generates "Negative Trajectories."
  • Imagine a video game where you intentionally drive the car into a wall on purpose, over and over, to record exactly what the camera sees right before the crash.
  • They call this the Negative Dataset. It's a library of "crash videos" that the robot can study without actually breaking anything.
  • They also have an Expert Dataset (videos of perfect, crash-free navigation).
  • By showing the robot both the "perfect path" and the "crash path," the robot learns much faster what not to do.

4. The Experiments: Who Won the Race?

The authors tested several different "brains" (algorithms) in this new obstacle course:

• The Imitation Learners (The Copycats): These robots tried to learn by watching the "Expert Dataset" (like a student copying a teacher's homework). They were okay, but they struggled when the room looked slightly different.
• The Reinforcement Learners (The Trial-and-Errorers): These robots learned by trying things, getting punished for crashing, and rewarded for moving forward. They were much better.
• The Depth-Enhanced Robot: The best performer was a robot that didn't just use its camera (RGB) but also used a "depth sensor" (like a 3D eye) to understand how far away objects were. This robot was the champion, reaching goals more often and crashing less.

5. The Real-World Test: Does it Work Outside the Game?

Finally, they took the robot trained in the virtual house and put it in a real house.

• The Result: The robot trained in the simulation (the video game) actually did better than robots trained only on real-world data!
• Why? The simulation gave the robot thousands of hours of practice, including thousands of "crash lessons" that would have been too expensive to film in real life.
• The Hybrid Winner: The absolute best robot was one trained on both real-world data and simulation data. It combined the "feel" of the real world with the "volume" of the simulation.

The Big Takeaway

This paper introduces a new standard for testing robots. It says: "Don't just ask if the robot can find the goal; ask if it can find the goal without breaking the house."

By creating a safe, virtual place to crash and learn, they are helping us build robots that are ready to safely roam our messy, cluttered homes. It's like moving from teaching a driver in an empty parking lot to teaching them in a busy city with traffic, pedestrians, and potholes.

Technical Summary

Here is a detailed technical summary of the paper "RVN-Bench: A Benchmark for Reactive Visual Navigation."

1. Problem Statement

Reactive Visual Navigation (RVN) involves an agent navigating to sequential goal positions in previously unseen environments using only visual observations (RGB images) without relying on prior maps or task-specific knowledge.

The core challenge addressed by this paper is safety in cluttered indoor environments. While existing benchmarks (e.g., Habitat Challenge, GOAT-Bench) evaluate goal-reaching capabilities, they largely ignore collisions or focus on outdoor/autonomous driving scenarios (e.g., CARLA, MetaUrban).

• Limitations of current state-of-the-art:
  • Benchmarks often disregard collisions, leading to policies that succeed in simulation but fail in real-world indoor settings due to safety risks.
  • Collecting real-world collision data is costly, time-consuming, and risky for hardware.
  • Existing simulators lack high-fidelity indoor scenes with explicit collision-aware evaluation metrics.

2. Methodology: RVN-Bench Framework

The authors introduce RVN-Bench, a simulation-based benchmark built on Habitat 2.0 and utilizing high-fidelity HM3D indoor scenes. It is designed specifically for indoor ground mobile robots with forward-facing RGB cameras.

Key Functionalities

RVN-Bench provides three core components:

1. Standardized Evaluation Environment:

   • Task: The agent must reach $N_{goal}$ sequential targets using only visual inputs ($I_t$) and relative goal coordinates ($P_t$).
   • Collision Awareness: The environment uses a precomputed Navigation Mesh (NavMesh) padded with the agent's radius. A collision is registered if the agent's displacement is blocked by a boundary, effectively modeling static obstacles (walls, furniture).
   • Episode Termination: Episodes end upon success (reaching all goals), collision, or timeout.

2. Reinforcement Learning (RL) Environment:

   • Supports online training with a reward structure that penalizes collisions ($r_{collision} = -0.1$) and rewards goal reaching ($r_{goal} = 1.0$).
   • Includes a cost signal for Safe-RL methods (e.g., PPO-Lagrangian).

3. Trajectory Image Dataset Generator:

   • Expert Dataset: Generates collision-free paths by computing the shortest path on a NavMesh padded with a margin larger than the agent's radius.
   • Negative Dataset: A novel feature that generates collision-inducing trajectories. By padding the NavMesh with a margin smaller than the agent's radius, the system forces collisions. It records pre-collision states and post-collision "stuck" states. This allows for the creation of large-scale negative datasets that are expensive to collect in the real world.

Baseline Models Evaluated

The paper evaluates several state-of-the-art approaches:

• Imitation Learning (IL): ViNT-PointGoal and NoMaD-PointGoal (adapted to accept relative goal coordinates).
• Safe-RL: PPO-Lagrangian.
• Standard RL: PPO, DD-PPO, and DDPPO-DAV2 (DD-PPO augmented with depth maps estimated by the Depth Anything V2 foundation model).
• Proposed Baseline (NoMaD-Neg): A framework training two separate NoMaD models (one on expert data, one on negative data) and selecting the optimal trajectory based on a Constrained Reward (CoR) metric to avoid collisions.

3. Key Contributions

1. RVN-Bench Benchmark: The first standardized benchmark for indoor reactive visual navigation that explicitly prioritizes collision avoidance alongside goal reaching.
2. Negative Trajectory Generation: A pipeline to automatically generate large-scale datasets of collision events, enabling the training of safety-aware policies without real-world risk.
3. Comprehensive Baseline Evaluation: Extensive testing of RL, Safe-RL, and IL methods, establishing a performance ceiling for the field.
4. Real-World Validation: Demonstration that models trained on RVN-Bench simulation data generalize effectively to real-world indoor environments (offices and houses).

4. Experimental Results

Experiments were conducted on 800 training scenes, 50 validation, and 50 test scenes (HM3D dataset).

Performance Metrics

• SR1: Success rate of reaching the first goal.
• E(G): Average number of goals reached per episode.
• CPK: Collisions per kilometer traveled (lower is better).

Key Findings

• RL vs. IL: RL-based methods significantly outperformed Imitation Learning (IL) methods.
  • Best Performer: DDPPO-DAV2 (DD-PPO + Predicted Depth) achieved the highest SR1 (0.928 on test) and lowest CPK (3.6).
  • IL Performance: NoMaD-PointGoal achieved an SR1 of ~0.75, while ViNT-PointGoal struggled significantly (SR1 ~0.09).
  • Insight: Despite having fewer parameters, the ResNet-LSTM architecture used in RL outperformed larger Transformer-based IL models, suggesting that interaction-based learning is crucial for collision avoidance.
• Impact of Depth: Adding depth information (predicted or ground-truth) drastically improved performance. DDPPO-DAV2 reduced CPK by ~60% compared to RGB-only DD-PPO.
• NoMaD-Neg: Incorporating negative datasets improved NoMaD's performance over the standard NoMaD-PointGoal, proving the utility of collision data, though it still lagged behind RL methods.
• Generalization: Models trained on RVN-Bench showed strong generalization to unseen test scenes (SR1 dropped only ~1.5% from training to test).

Real-World Evaluation

The authors tested the NoMaD-PointGoal model on a Clearpath Jackal UGV in real indoor environments:

• Real-Only Data: Poor performance (SR1 = 0.30, high collisions).
• Simulation-Only Data: Significantly better (SR1 = 0.60).
• Hybrid (Real + Sim): Best performance (SR1 = 0.75, CPK = 191.4).
• Conclusion: Large-scale simulation data from RVN-Bench complements limited real-world data, enhancing generalization and safety.

5. Significance and Future Work

• Significance: RVN-Bench shifts the paradigm from "can the robot reach the goal?" to "can the robot reach the goal safely?" It provides the necessary infrastructure (simulator, metrics, and negative data generation) to develop robust, safety-critical navigation policies for indoor robots.
• Future Work: The authors plan to extend the benchmark to include dynamic obstacles (moving people/objects), support for broader robot platforms, and continuous action spaces.

In summary, RVN-Bench establishes a new standard for safe visual navigation, demonstrating that collision-aware training in high-fidelity simulation is essential for deploying reliable mobile robots in cluttered indoor environments.