Technical Summary: NavOL – Navigation Policy with Online Imitation Learning

Problem Statement

Robotic visual navigation in open-world, dynamic, and cluttered environments remains a significant challenge. Existing approaches face distinct limitations:

• Offline Imitation Learning: While efficient, these methods suffer from distribution shift and compounding errors during rollout because they rely on static, pre-collected datasets that may not cover the states an agent encounters during deployment.
• Reinforcement Learning (RL): Although RL allows for interaction with the environment, it heavily depends on carefully designed reward functions, often encounters sparse rewards, and suffers from low sample efficiency, making large-scale training in complex environments difficult.
• Data Scarcity: Collecting large-scale, high-quality real-world trajectories is costly and limits model capacity. Conversely, purely synthetic data generation (e.g., NavDP) improves efficiency but remains an offline pipeline that cannot correct for distribution drift during training.

The core bottleneck is the lack of a scalable paradigm that combines the sample efficiency of imitation learning with the interactive benefits of online learning to mitigate distribution shift without requiring reward engineering.

Methodology: NavOL

NavOL proposes an online imitation learning paradigm that bridges the gap between offline imitation and online RL. It operates within a closed-loop rollout–update framework using the IsaacLab simulator.

1. Model Architecture

NavOL builds upon a pretrained multimodal diffusion policy (specifically from NavDP) and consists of two cooperating components:

• Trajectory Generator ($f$): A Diffusion Transformer (DiT) that maps RGB-D observations and goal instructions to a short-horizon sequence of relative waypoints. It uses DepthAnythingV2 ViT backbones for visual encoding and a transformer decoder to fuse features.
• Critic Network ($V$): Shares the visual tokens and DiT backbone with the generator. It scores candidate trajectories for safety and feasibility, outputting a scalar safety score. This allows the system to rank sampled trajectories and execute the safest plan.

2. Online Imitation Learning Pipeline

The training process alternates between two stages:

• Rollout Stage:
  • The agent navigates in the simulator using the current policy.
  • Expert Supervision: A global planner with privileged access to the environment map (NavMesh) computes the optimal path from the current pose to the goal. This path is densified and smoothed to serve as ground-truth trajectory labels.
  • Data Aggregation: At each step, the agent executes the policy's action with probability $\rho$ or the expert's action with probability $1-\rho$. This "DAgger-style" interaction ensures the policy learns from states it actually visits, mitigating covariate shift.
  • Safety Labeling: A target safety score is derived based on the distance to obstacles, supervising the critic network.
• Update Stage:
  • The diffusion policy and critic are trained on the newly collected observation-trajectory pairs.
  • The policy is optimized using a standard denoising objective (MSE between predicted and target noise) conditioned on observations, goals, and expert trajectories.
  • The critic is optimized using an MSE loss against the target safety scores.
  • Crucially, the system discards data from previous iterations, training only on the most recent rollout to maximize update efficiency.

3. Implementation & Scaling

• Simulation: Built on IsaacLab with high-fidelity rendering (global illumination, reflections, shadows) and domain randomization (camera pose, lighting, agent dimensions).
• Efficiency: The system runs 50 scenes in parallel on 8 RTX 4090 GPUs, collecting over 2,000 new trajectories per hour (averaging 400+ steps each).
• Initialization: The policy is initialized with weights from the pretrained NavDP model to provide a strong behavioral prior, though ablation studies show it can be trained from scratch with reduced performance.

Key Contributions

1. NavOL Framework: A novel online imitation learning framework for navigation that iteratively updates a policy using expert demonstrations gathered online, eliminating the need for reward engineering.
2. Scalable Pipeline: A highly parallelized rollout–update pipeline capable of collecting and learning from thousands of high-quality trajectories per hour across diverse 3D scenes.
3. New Benchmark: Introduction of an indoor visual navigation benchmark based on the 3D-Front dataset, featuring 8 out-of-domain scenes with predefined start-goal pairs to rigorously evaluate zero-shot generalization.
4. Empirical Validation: Extensive experiments demonstrating consistent performance gains over state-of-the-art baselines in both simulation and real-world zero-shot deployments.

Results

Simulation Benchmarks

• NavDP Benchmark: NavOL outperforms baselines (DD-PPO, iPlanner, ViPlanner, and the original NavDP) across all metrics. It achieves a mean Success Rate (mSR) of 80.4% and mean Success-weighted Path Length (mSPL) of 76.3%, surpassing NavDP (77.8% SR, 74.8% SPL).
• NavOL Benchmark: On the new, more challenging benchmark, NavOL achieves an mSR of 69.0% and mSPL of 63.7%. This significantly outperforms the strongest baseline, NavDP (42.2% SR, 37.7% SPL), and other methods like ViPlanner (33.0% SR) and iPlanner (18.7% SR).
• Stability: Qualitative analysis shows NavOL generates stable, collision-free trajectories even near obstacles, whereas NavDP exhibits high variance and shaking behaviors that lead to failures.

Real-World Deployment

• Zero-Shot Sim-to-Real: The policy, trained on a Dingo robot in simulation, was deployed on a Unitree Go2 robot equipped with a RealSense D435i camera.
• Performance: In three cluttered real-world scenarios (Office, Gym, Corridor), NavOL achieved success rates of 80%, 70%, and 100% respectively. In contrast, NavDP achieved only 40%, 20%, and 20%.
• Cross-Embodiment: The results demonstrate strong generalization across different robot embodiments, validating the robustness of the learned spatial representations.

Efficiency Analysis

• NavOL initialized from NavDP required only 16 GPU-days (8 GPUs for 2 days) to achieve an mSR of 69.0%.
• In comparison, the original NavDP required approximately 1,024 GPU-days (32 A100s for 32 days) to achieve an mSR of 42.2%.
• Even the "from scratch" variant of NavOL (16 GPU-days) outperformed the original NavDP, highlighting the data efficiency of the online supervision approach.

Significance and Claims

The paper claims that NavOL successfully addresses the limitations of both offline imitation learning and reinforcement learning by:

1. Mitigating Distribution Shift: By training on the policy's own explored rollouts and incorporating expert corrections (DAgger-style), the method prevents the compounding errors typical of offline policies.
2. Eliminating Reward Engineering: The use of a privileged global planner provides direct trajectory supervision, removing the need for sparse or manually designed reward functions.
3. Scalability: The integration with IsaacLab allows for massive parallelization, making high-fidelity online training feasible and efficient.
4. Robustness: The framework produces policies that generalize well to unseen environments and different robot embodiments, as evidenced by the zero-shot real-world experiments.

The authors conclude that incorporating online planner supervision into imitation learning significantly enhances navigation performance, offering a more data-efficient and scalable path toward robust embodied agents.