💻 computer science

Joint On-and-Off Policy Learning for Vision-and-Language Navigation

This paper introduces JOP-VLN, a novel framework that synergistically integrates off-policy imitation learning with on-policy exploration through a three-stage training pipeline to achieve state-of-the-art performance on Vision-and-Language Navigation benchmarks.

Original authors: Qingrong He, Lin Zhao, Kevin Zheng, Liang Lin

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qingrong He, Lin Zhao, Kevin Zheng, Liang Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be your personal tour guide. You want it to listen to your voice commands like, "Walk to the kitchen, turn left at the blue vase, and stop in front of the fridge," and then actually do it without bumping into walls or getting lost. This is the world of Vision-and-Language Navigation (VLN). It's a branch of robotics where an "embodied agent" (a robot with a body and eyes) has to understand the messy, real world through a camera and make sense of human speech to move around.

The big challenge here is that robots are terrible at learning from their own mistakes. If you teach a robot by showing it perfect videos of how to walk, it learns to copy those videos. But the moment it steps into a new room or sees a chair in a different spot, it panics because it never learned how to recover when things go wrong. This is like a student who memorizes the answers to a practice test but freezes when the teacher changes the numbers. To fix this, scientists usually try two different things: either they show the robot more examples of perfect paths (Imitation Learning), or they let the robot wander around and give it a "treat" (reward) when it gets closer to the goal (Reinforcement Learning). For a long time, these two methods were like two separate schools that never talked to each other.

Enter JOP-VLN, a new framework that decides to throw a joint party for these two schools. The researchers built a system that teaches the robot in three distinct stages, blending the safety of copying experts with the bravery of exploring on its own. Think of it as a training camp where the robot first learns the basics from a master coach, then practices on a "training wheels" version of the world where a safety net catches it when it falls, and finally, it goes into the wild to explore, but with a special rule: it only pays attention to the times it was confused or made a mistake, using those moments to get smarter.

The paper introduces this three-stage pipeline to solve the problem of robots getting stuck when they encounter something new. In the first stage, the robot learns basic navigation skills and how to summarize what it sees, just like a student learning the alphabet and how to write a sentence. In the second stage, the robot tries to navigate, and whenever it starts to drift off course, a "god-mode" safety system (called the oracle) steps in to correct its path. The robot then learns from these corrected paths, effectively practicing how to recover from errors. This is the "off-policy" part, where it learns from data collected by a mix of its own attempts and the safety net's corrections.

The magic happens in the third stage, where the paper combines everything. The researchers realized that if they let the robot explore randomly, it might just wander in circles or get too confident in bad habits. So, they added a clever filter: they only let the robot learn from the moments where it was truly unsure or "high-entropy" (a fancy way of saying it was guessing wildly). This prevents the robot from wasting time on easy tasks it already knows and forces it to focus on the hard stuff. Furthermore, they sorted the training data so that the robot spends the most time practicing the specific paths where it previously failed, ensuring it learns to fix its own mistakes.

The results of this approach are impressive. When tested on standard navigation benchmarks, JOP-VLN achieved a success rate of 69.9% on the R2R dataset and 68.0% on the RxR dataset. These numbers represent a new state-of-the-art, meaning it outperformed previous methods that relied on just one type of learning. The researchers also tested the robot in the real world, using a four-legged robot dog in both indoor offices and outdoor gardens. Even with the messy reality of real-world lighting and textures, the robot could follow complex instructions, proving that this "joint" learning style helps it generalize better than before. The paper suggests that by carefully mixing expert guidance with smart, error-focused exploration, we can build robots that are not just good at following rules, but are also resilient enough to handle the unexpected twists and turns of the real world.

Technical Summary: JOP-VLN

Problem Statement

Vision-and-Language Navigation (VLN) requires embodied agents to navigate physical environments based on natural language instructions. While recent advancements in Vision-Language Models (VLMs) have improved navigation decision-making, existing approaches suffer from a significant gap in integrating two dominant training paradigms:

Imitation Learning (IL): Typically trained on expert demonstrations, often augmented with the Dataset Aggregation (DAgger) algorithm to improve error recovery. However, these methods rely heavily on expert data, limiting generalization to unseen environments due to covariate shift.
Reinforcement Learning (RL): Driven by verifiable rewards (e.g., using Group Relative Policy Optimization or GRPO) to enhance reasoning and exploration. While effective, standard on-policy RL often fails to adequately address covariate shift when relying solely on in-distribution data during inference.

The core problem addressed is the lack of a unified framework that synergistically combines the error-recovery capabilities of off-policy IL with the exploration strengths of on-policy RL.

Methodology: JOP-VLN

The paper proposes JOP-VLN, a novel framework that employs a three-stage progressive training pipeline to jointly learn off-policy imitation and on-policy reinforcement.

Model Architecture

The system is built upon the Qwen3-VL large vision-language model. It adopts a multi-turn dialogue paradigm where the agent processes:

A language instruction ( $I$ ).
Long-term memory: Frames uniformly sampled from distant history.
Short-term memory: The most recent interleaved sequence of observations and actions.
The model predicts "action chunks" (sequences of up to four discrete actions: forward, left, right, stop) to enable efficient KV cache reuse and near-linear inference.

Three-Stage Training Pipeline

Stage 1: Multi-task Training
The model is co-trained on two tasks to acquire fundamental skills:

Action Prediction: Predicting action chunks within the multi-turn dialogue paradigm.
Trajectory Summarization: Synthesizing textual instructions conditioned on sampled frames from navigation videos. This enhances the model's visual understanding and alignment with language.

Stage 2: Imitation Learning with DAgger Trajectories
To improve error recovery, the DAgger algorithm is employed. A heuristic strategy is used where the learned policy is prioritized, but an "oracle" intervenes when the agent deviates from the ground truth beyond a threshold.

Refinement: Unlike standard DAgger, the authors replay all visited states to obtain corresponding oracle action chunks, filtering out erroneous actions generated by the model during the initial rollout. These corrected trajectories are used for fine-tuning the model via imitation learning.

Stage 3: Joint On-and-Off Policy Learning
This stage integrates the CHORD framework, jointly optimizing on-policy RL and off-policy IL on the DAgger-collected trajectories. Two key techniques are introduced to adapt this framework to VLN:

High-Entropy Trajectory Sampling (for RL):
- During DAgger collection, the entropy of model-predicted tokens is recorded.
- GRPO Training: Only the top 20% of samples with the highest entropy are used for Group Relative Policy Optimization (GRPO).
- Rationale: High-entropy tokens represent uncertain or exploratory states. Training on these prevents the model from entering a "zero-gradient" state caused by low-entropy (overconfident) samples, thereby enhancing RL efficiency.
- Reward Functions: The GRPO objective utilizes four rewards: Format (adherence to action structure), Exact Match (matching oracle actions), Position (cosine similarity of pose changes), and Distance (Euclidean distance of displacement).
Error-Correction-Prioritized Trajectory Sorting (for IL):
- Trajectories are sorted based on their "learning value," defined by the proportion of action tokens that were corrected by the oracle (i.e., where the model made a mistake).
- Strategy: The model performs imitation learning on these trajectories in descending order of learning value. This ensures the model frequently revisits and learns from its specific failure points, facilitating a seamless transition from IL to RL.
- Loss Function: A combined loss $L_{CHORD} = (1-\mu)L_{GRPO} + \mu L_{IL}$ is used, where $\mu$ follows a cosine decay schedule (starting at 0.5 and decreasing to 0.05) to shift emphasis from balanced learning to on-policy exploration over time.

Key Contributions

JOP-VLN Framework: A novel VLN model that synergistically combines off-policy imitation learning and on-policy exploration via a three-stage pipeline.
VLN-Specific Adaptations of CHORD:
- High-Entropy Sampling: A strategy to retain high-entropy trajectories for GRPO, preventing model overconfidence and improving RL training efficiency.
- Error-Correction Prioritization: A trajectory sorting strategy that prioritizes samples where the model previously erred, enhancing error recovery capabilities.
Comprehensive Evaluation: Extensive experiments in both simulation and real-world environments, demonstrating state-of-the-art performance.

Experimental Results

Simulation Benchmarks (VLN-CE)

Evaluated on the R2R and RxR datasets (Val-Unseen splits) using the Habitat simulator.

R2R: JOP-VLN achieves a 69.9% Success Rate (SR), setting a new state-of-the-art. It outperforms the previous best method (CorrectNav) by 4.8% in SR, 8.9% in Oracle Success (OS), and 2.6% in Success Weighted by Path Length (SPL). Notably, it surpasses methods using panoramic views using only single-view RGB inputs.
RxR: Achieves 68.0% SR and 59.3% SPL, demonstrating strong long-horizon decision-making capabilities.

Real-World Deployment

Setup: Tested on a Unitree Go2 quadruped robot with an Intel RealSense camera, operating on a remote workstation.
Adaptation: A few-shot adaptation strategy (10-shot fine-tuning per environment) was used to bridge the sim-to-real domain gap.
Performance: The system achieved an average inference time of 0.3s per action chunk. Qualitative results showed consistent instruction following in diverse indoor and outdoor settings. Failures were primarily attributed to ambiguous referents in long instructions (e.g., multiple similar landmarks).

Ablation Studies

Trajectory Summarization: Improved visual understanding and navigation performance.
DAgger Fine-tuning: Significantly boosted SR from ~50% to ~64%, highlighting the importance of off-policy data for error recovery.
Joint Learning (CHORD-ec): The combination of high-entropy sampling and error-correction prioritization yielded the best results, outperforming pure IL, pure RL, and the standard CHORD baseline without these specific VLN adaptations.

Significance and Claims

The paper claims that JOP-VLN represents a significant advancement by structurally fusing on-policy RL exploration with off-policy IL supervision. This allows agents to learn simultaneously from expert demonstrations and environmental exploration signals. The proposed techniques specifically address the limitations of covariate shift and overfitting to expert data. The authors assert that their method achieves a new state-of-the-art on the R2R Val-Unseen split and demonstrates robust adaptability in real-world scenarios, validating the efficacy of the joint on-and-off policy learning paradigm for embodied navigation.