🤖 AI

FF-JEPA: Long-Horizon Planning in World Models with Latent Planners

This paper introduces FF-JEPA, a hierarchical world model that employs an action-free latent planner to predict subgoals, thereby enabling efficient long-horizon planning without requiring explicit goal images and overcoming the limitations of traditional computationally expensive methods.

Original authors: Sergi Masip, Jonathan Swinnen, Yutong Hu, Renaud Detry, Tinne Tuytelaars

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sergi Masip, Jonathan Swinnen, Yutong Hu, Renaud Detry, Tinne Tuytelaars

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Long Trip" Dilemma

Imagine you are trying to teach a robot to push a block across a table to a specific spot.

Current AI methods (called World Models) are like a robot that can "imagine" the future. It looks at the current picture, guesses what will happen if it pushes left, then guesses what happens if it pushes right, and so on. It tries to find a path that leads to a specific goal picture (e.g., a photo of the block in the final spot).

However, this approach has two major flaws:

The "Too Far" Problem: If the trip is short, the robot can imagine the whole path easily. But if the trip is long (many steps), the robot's imagination gets fuzzy. Small mistakes in the first step get bigger and bigger until the robot is imagining a completely wrong future. It's like trying to predict the weather for the next month; the further out you go, the less accurate you are.
The "Goal Photo" Problem: To plan, the robot usually needs a photo of the exact final destination. In the real world, you often don't have a photo of the future. You just know you want the block "over there," but you don't have a picture of what "over there" looks like yet.

The Solution: FF-JEPA (The "Stop-and-Check" Strategy)

The authors propose a new system called FF-JEPA. Instead of trying to imagine the whole long trip at once, they break it down into a hierarchy. Think of it like a road trip with a GPS.

Instead of driving from New York to Los Angeles in one giant mental leap, the system uses two layers:

1. The "Latent Planner" (The GPS)

This is the new "brain" added to the system. It doesn't look at the raw pixels of the camera; it looks at a simplified, abstract map (called "latent space").

What it does: It looks at where you are now and predicts a sub-goal (a checkpoint) a short distance ahead.
The Magic: It does not need a photo of the final destination. It just predicts, "Okay, in 25 steps, the block should be here." It acts like a GPS saying, "Head to the next exit," rather than "Drive to the final hotel."

2. The "World Model" (The Driver)

This is the existing AI that knows how to move the block.

What it does: Once the "GPS" (Planner) says, "Go to this checkpoint," the "Driver" (World Model) figures out the specific moves to get there.
The Loop: Once the robot reaches that checkpoint, the Planner picks a new checkpoint, and the Driver figures out how to get there.

Why This is Better (The Analogy)

Imagine you are trying to walk through a dense fog to a mountain peak.

Old Way (Flat Planning): You try to visualize the entire path to the peak in your mind at once. Because of the fog, you get confused after 10 steps and end up walking in a circle. Also, you need a photo of the peak to start, which you don't have.
FF-JEPA Way: You have a guide (the Planner). The guide says, "Walk until you see that big rock." You walk to the rock. Then the guide says, "Now walk to that tree." You walk to the tree.
- You never have to imagine the whole mountain at once (solving the "too far" problem).
- You don't need a photo of the peak; you just need to see the next landmark (solving the "goal photo" problem).

The Results: Pushing the "T" Block

The researchers tested this on a task called PushT, where a robot must push a T-shaped block to a target.

Short Trips: The old methods worked okay.
Long Trips: The old methods failed miserably (success rate dropped to almost 0%). They got lost in the fog.
FF-JEPA: It succeeded in over 90% of long trips. Even when they started the robot in random, messy positions (where it didn't know if the target was reachable), FF-JEPA still succeeded about 82% of the time.

Two Types of "GPS"

The paper tested two different ways to build the "Planner" (the GPS):

Deterministic Planner: A straightforward, fast predictor. It's like a simple map. It's very lightweight and fast but slightly less accurate on very short trips.
Diffusion Planner: A more complex, "creative" predictor (similar to how AI generates images). It's slower and uses more computer power, but it is the most accurate, especially for long, difficult trips.

The Bottom Line

The paper shows that by adding a "Planner" that breaks big problems into small, manageable chunks, robots can plan much further into the future without needing a picture of the finish line. It turns a difficult, long-distance navigation problem into a series of easy, short-distance hops.

Key Takeaway: You don't need to see the whole journey to plan it; you just need to know the next step, and then the next, and the next.

Technical Summary: FF-JEPA: Long-Horizon Planning in World Models with Latent Planners

Problem Statement

Recent advances in visual world models, particularly Joint Embedding Predictive Architectures (JEPAs) like LeWorldModel (LeWM), have demonstrated the ability to predict future observations and support model-based control via sampling-based planners such as the Cross-Entropy Method (CEM). However, the authors identify three critical limitations preventing the deployment of these "flat" world models in real-world, long-horizon tasks:

Computational Prohibitions: Planning over long horizons requires repeated rollouts and increasingly large action sequences, making CEM optimization computationally intractable.
Error Compounding: Autoregressive prediction errors accumulate over many steps, causing the planner to diverge for complex trajectories.
Goal Dependency: Existing methods typically require an explicit goal image or state ( $o_g$ ) to drive planning. In many real-world scenarios, such a concrete goal specification is unavailable or impractical.

While hierarchical methods and generative approaches have attempted to address these issues, they often still operate in pixel space or rely on externally specified goal images/trajectories, limiting their applicability in open-ended environments.

Methodology: Forward-Forward JEPA (FF-JEPA)

The paper proposes FF-JEPA, a hierarchical framework that extends JEPA-style world models by introducing a latent planner ( $G$ ) operating alongside a standard action-conditioned forward model ( $P$ ). This approach decomposes long-horizon planning into a sequence of tractable, short-term optimization problems without requiring explicit goal images.

Core Architecture

The system consists of two primary components trained within the latent space defined by a frozen, pre-trained world model encoder ( $E$ ):

The World Model ( $P$ ): A standard action-conditioned forward dynamics model (based on LeWM) that predicts the next latent state given a history of states and an action: $\hat{z}_{t+1} = P(z_{t-W_P+1:t}, a_t)$ .
The Latent Planner ( $G$ ): An action-free forward dynamics model that predicts the next subgoal latent ( $\hat{z}_{sg}$ $\overset{z}{^}_{s g}$ ) based solely on a history of previous subgoal latents. It operates at a higher temporal resolution (every $H$ $H$ environment steps).
- Deterministic Planner ( $G_{Det}$ ): A transformer architecture (similar to the world model's predictor but without action conditioning) trained to minimize the mean-squared error between predicted and true future subgoals.
- Diffusion Planner ( $G_{DM}$ ): A DiT (Diffusion Transformer) backbone trained with a denoising score-matching objective to predict a sequence of future subgoals.

Planning Inference

The inference process is hierarchical:

Subgoal Prediction: The latent planner $G$ predicts the next subgoal state $\hat{z}_{sg, m+1}$ given the current subgoal history.
Action Optimization: The world model $P$ uses CEM to find an action sequence ( $a_{1:H}$ ) that drives the predicted state toward the predicted subgoal $\hat{z}_{sg, m+1}$ within a short horizon $H$ .
Iteration: This cycle repeats, effectively chaining short-horizon plans to achieve long-horizon objectives.

Crucially, this method removes the need for a target goal image at inference time; the planner generates the trajectory toward an implicitly defined objective.

Key Contributions

Goal-Free Long-Horizon Planning: The framework enables planning in world models without explicit goal images by using a latent planner to generate intermediate subgoals.
Hierarchical Decomposition: It addresses the "long-horizon collapse" of flat world models by breaking complex trajectories into short, solvable subproblems, mitigating error compounding and computational costs.
Unlabeled Training: The latent planner can be trained on unlabeled successful demonstrations (subsampled from the world model's latent space), reducing the reliance on action-labeled data.
Unified Framework: It reinterprets the world model as an inverse dynamics module operating over imagined latent trajectories, unifying predictive modeling and control within a single coherent latent space.

Empirical Evaluation

The authors evaluated FF-JEPA on the PushT task using three experimental setups: Short-horizon (25 steps), Long-horizon (75 steps), and Random Initialization (300 steps with randomized start positions).

Results

Long-Horizon Performance: The baseline flat LeWM collapsed on long-horizon tasks (3.52% success rate at $t=75$ ) and failed completely under random initialization (0.00%). In contrast, FF-JEPA (Diffusion) achieved 91.80% success at $t=75$ and 82.42% under random initialization.
Goal-Free Advantage: Unlike baselines (LeWM, DINO-WM) which require a goal image for inference, FF-JEPA does not. Despite this, it outperformed hierarchical DINO-WM baselines in both short-horizon (96.09% vs. 89.0%) and long-horizon (91.80% vs. 61.0%) settings.
Data Efficiency: Ablation studies showed that training the diffusion planner on only 200 high-quality demonstrations (a 40x reduction in data) still yielded competitive performance (76.17% vs. 82.42% with full data), suggesting high data efficiency.
Computational Overhead: The deterministic planner added negligible inference time overhead (~~2.1ms) compared to CEM (~~926ms), while the diffusion planner added ~242ms. Both are significantly faster than the repeated rollouts required for long-horizon flat planning.

Failure Analysis

Failure cases typically occurred when the agent drifted too far from the predicted subgoal position or when the latent planner generated a low-quality subgoal state (e.g., missing the agent).

Significance and Claims

The paper positions FF-JEPA as a promising direction for scalable, goal-free robot learning. By demonstrating that action-conditioned world models paired with latent planners trained on unlabeled data can overcome the limitations of flat planning, the authors suggest this approach enables agents to operate in open-ended environments where explicit goal specifications are unavailable. The work highlights that decomposing planning into short-term subgoals is a viable strategy to mitigate the computational and error-compounding issues inherent in current world model approaches.