🤖 AI

Policy-Guided World Model Planning for Language-Conditioned Visual Navigation

The paper presents PiJEPA, a two-stage framework that enhances language-conditioned visual navigation by using a fine-tuned policy to warm-start MPPI planning over a JEPA world model, thereby achieving superior goal-reaching accuracy and instruction-following fidelity compared to existing methods.

Original authors: Amirhosein Chahe, Lifeng Zhou

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amirhosein Chahe, Lifeng Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to walk through a house to find a specific object, like "the red mug on the kitchen counter," based on a voice command. This is a tricky job because the robot has to see what's in front of it, understand the words, and figure out the best path forward without bumping into walls or getting lost.

This paper introduces a new system called PiJEPA (Policy-Guided JEPA Planning) that solves this problem by acting like a team of two experts working together: a Fast Instinctive Driver and a Strategic Navigator.

Here is how it works, broken down into simple concepts:

1. The Problem: Two Flawed Approaches

Before PiJEPA, robots usually tried one of two ways to navigate, and both had big weaknesses:

The "Fast Instinctive Driver" (Reactive Policy):
Think of this like a taxi driver who is great at making quick turns and reacting to traffic lights right now. If you say "turn left," they do it immediately. However, if you ask them to drive 20 miles to a specific address, they might get lost because they aren't thinking about the whole trip, just the next second. They lack "long-term vision."
The "Strategic Navigator" (World Model Planning):
This is like a GPS that simulates thousands of possible routes in its head before you even start moving. It's great at planning the long trip. But, it has a major flaw: it doesn't know where to start looking. Imagine asking a GPS to find a coffee shop, but it starts by randomly guessing every street in the city. It wastes so much time guessing wrong paths that it never finds the coffee shop before the battery dies.

2. The Solution: PiJEPA (The Perfect Team-Up)

PiJEPA combines these two into a super-team. It uses the Fast Driver to give the Strategic Navigator a "head start."

Here is the step-by-step process using an analogy:

Step 1: The "Warm Start" (The Instinctive Driver)

First, the robot listens to your command ("Go to the stairs") and looks at the room. It asks its Fast Instinctive Driver (a pre-trained AI called Octo) for advice.

The Driver doesn't just pick one random move; it says, "Based on what I see, there's a 90% chance the best first move is to go forward and slightly left."
It gives the Navigator a "shortlist" of promising directions.

Step 2: The "Strategic Simulation" (The Navigator)

Now, the Strategic Navigator (the World Model) takes that shortlist. Instead of wasting time guessing random directions (like "go backward into the wall"), it only simulates paths that start with the Driver's suggestions.

It runs a mental simulation: "If I go forward-left, what will the room look like in 5 seconds? Will I hit the stairs?"
Because it started with a good guess, it can quickly refine the path to be perfect, rather than searching the whole universe of possibilities.

Step 3: The Execution

The robot takes the very first step of this refined, perfect path, moves forward, and then repeats the whole process for the next step.

3. The Secret Sauce: "Frozen Eyes"

The paper also mentions using special "eyes" (Vision Encoders) for the robot. They tested two types:

DINOv2: Like a high-resolution camera that is amazing at spotting shapes and objects in a single photo.
V-JEPA-2: Like a video camera that understands how things move over time.

The researchers found that while the video camera (V-JEPA-2) was better at understanding the flow of movement, the team worked best when the "Driver" and the "Navigator" were using the same type of eyes. This ensures they are speaking the same language about what they see.

4. Why This Matters

The results showed that this team-up approach was much better than using either expert alone.

Accuracy: The robot reached its goal much more often.
Speed: It didn't waste time guessing wrong paths.
Reliability: Even when the instructions were vague (like "follow the building" when there are three buildings), the "Driver's" initial guess helped the "Navigator" avoid getting completely stuck.

The Bottom Line

PiJEPA is like hiring a local guide (the policy) who knows the neighborhood well to point you in the right direction, and then hiring a tour planner (the world model) to map out the perfect, obstacle-free route from there. By letting the local guide give the planner a "warm start," the robot navigates faster, smarter, and with fewer mistakes.

1. Problem Statement

The paper addresses the challenge of embodied visual navigation where an agent must navigate to a goal specified by an image and a natural language instruction (e.g., "move towards the stairs").

Current Limitations:
- Reactive Policies (e.g., Octo, NoMaD): These Vision-Language-Action (VLA) models excel at short-horizon control and instruction following but lack long-term reasoning. They predict actions greedily in a single forward pass, often failing to account for long-term consequences or avoiding local minima in complex environments.
- World Model Planning (e.g., JEPA-WM): These models predict future states in a latent space to enable explicit planning. However, sampling-based optimizers (like MPPI or CEM) used with these models often suffer from slow convergence and entrapment in local minima because they initialize the search from an uninformed Gaussian prior over high-dimensional action spaces.

2. Methodology: PiJEPA Framework

The authors propose PiJEPA, a two-stage framework that bridges reactive policies and latent world model planning. The core insight is to use a fine-tuned VLA policy to provide an informed action prior that "warm-starts" the world model planner, rather than searching the entire action space from scratch.

The pipeline consists of three main components:

A. Shared Vision Encoder

Both the policy and the world model operate on top of a frozen pretrained vision encoder ( $E_\phi$ ). The paper evaluates two backbones:

DINOv2: A self-supervised image encoder (ViT-S) strong in spatial and object segmentation.
V-JEPA-2: A self-supervised video encoder (ViT-L) capturing temporal dynamics.
Note: The encoder weights remain frozen to ensure a consistent latent representation space for both components.

B. Stage 1: Policy Prior Generation

A fine-tuned Octo-based policy ( $\pi_\theta$ ) takes the current observation ( $o_t$ ) and language instruction ( $\ell$ ) as input.

It generates $N_\pi$ action chunk samples via its diffusion head.
These actions are transformed from the policy's global frame to the world model's local body frame.
The system computes the empirical mean ( $\mu_\pi$ ) and standard deviation ( $\sigma_\pi$ ) of these samples. This distribution serves as the warm-start initialization for the planner.

C. Stage 2: Policy-Guided MPPI Planning

A JEPA World Model ( $P_\psi$ ) predicts future latent states ( $\hat{z}_{t+1}$ ) given the current latent state and candidate actions.

Planning Objective: Minimize the Euclidean distance in the latent space between the predicted future state ( $\hat{z}_{t+H}$ ) and the encoded goal image ( $z_g$ ).
MPPI Algorithm: The Model Predictive Path Integral (MPPI) optimizer is initialized with the policy-derived distribution $(\mu_\pi, \sigma_\pi)$ $(μ_{π}, σ_{π})$ instead of a generic Gaussian.
- It iteratively samples action sequences, rolls them out through the world model, and updates the sampling distribution based on "elite" trajectories (those with the lowest cost).
- The language instruction only influences the process via the initial policy prior; the world model itself remains language-agnostic, focusing purely on visual-action dynamics.

3. Key Contributions

Unified Framework: Introduces PiJEPA, which effectively combines the semantic understanding of a fine-tuned VLA policy with the long-horizon reasoning of a JEPA world model.
Systematic Encoder Analysis: Provides a comprehensive comparison of image-based (DINOv2) vs. video-based (V-JEPA-2) representations for both policy learning and world modeling, analyzing how temporal features impact navigation.
Performance Validation: Demonstrates that policy-guided planning significantly outperforms standalone reactive policies and uninformed world model planning in real-world navigation tasks, specifically improving goal-reaching accuracy and instruction fidelity.

4. Experimental Results

Experiments were conducted on the CAST dataset (a large-scale visual navigation dataset with counterfactual instructions).

Metrics: Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) for position (XY) and heading.
Key Findings:
- PiJEPA vs. Baselines: PiJEPA achieved the best positional accuracy across both encoder configurations.
  - With V-JEPA-2, PiJEPA achieved the lowest ATE XY RMSE (1.65m) and Final error (2.88m), outperforming the reactive Octo policy (3.02m Final) and uninformed MPPI (3.29m Final).
- Role of Components:
  - Reactive Policy: Excels at local control and heading prediction (low RPE).
  - World Model Scoring (Octo-WM): Filtering policy proposals via the world model without iterative optimization captures much of the benefit, but full MPPI refinement yields the best trajectory coherence.
  - Uninformed MPPI: Struggles with position (high ATE) but surprisingly performs well on heading, suggesting the embedding objective implicitly regularizes orientation even when translation fails.
- Encoder Comparison: V-JEPA-2 (video encoder) generally yielded stronger overall performance than DINOv2, likely due to richer temporal dynamics. However, uninformed MPPI struggled more with V-JEPA-2, highlighting the critical need for the policy prior when the latent space is complex.
- Efficiency: The MPPI planning stage adds negligible latency (~~0.35s) compared to the diffusion sampling of the policy (~~2.13s), making the approach practical for real-time deployment.

5. Significance and Future Work

Significance: PiJEPA solves the "cold start" problem in world model planning. By leveraging a policy to narrow the search space to promising regions, it enables efficient long-horizon planning without requiring the world model to be conditioned on language (which is difficult to train). It effectively decouples semantic grounding (policy) from dynamics simulation (world model).
Limitations & Future Directions:
- Stagnation: In failure cases with ambiguous instructions (e.g., "Follow the building"), the world model can get stuck predicting static latent states regardless of actions, causing the planner to fail.
- Future Work: The authors suggest improving dynamics architectures to prevent stagnation, introducing intermediate waypoint costs, exploring richer action spaces, and implementing closed-loop replanning mechanisms.

In summary, PiJEPA represents a significant step forward in embodied AI by demonstrating that hybrid architectures—combining the speed of reactive policies with the foresight of latent world models—are superior to either approach used in isolation.