💻 computer science

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

Drive-JEPA establishes a new state-of-the-art in end-to-end autonomous driving by integrating Video Joint-Embedding Predictive Architecture (V-JEPA) for predictive representation learning with a proposal-centric planner that distills multimodal simulator-generated trajectories to overcome the limitations of single-trajectory supervision.

Original authors: Linhan Wang, Zichong Yang, Chen Bai, Guoxiang Zhang, Xiaotong Liu, Xiaoyin Zheng, Xiao-Xiao Long, Chang-Tien Lu, Cheng Lu

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Linhan Wang, Zichong Yang, Chen Bai, Guoxiang Zhang, Xiaotong Liu, Xiaoyin Zheng, Xiao-Xiao Long, Chang-Tien Lu, Cheng Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a self-driving car not just by showing it a few examples of how a human drives, but by giving it a massive library of driving movies to study, and then letting it practice in a super-advanced video game simulator. That is essentially what Drive-JEPA does.

Here is a breakdown of how this new system works, using simple analogies:

1. The Problem: The "One-Path" Trap

Most self-driving cars today learn by watching a human driver. But there's a catch: in any given situation (like approaching a yellow light), a human usually only takes one specific action. However, in the real world, there are often many safe ways to handle a situation (e.g., you could slow down gently, or you could stop completely).

If a car only learns from that single human path, it gets "stuck" thinking there is only one way to drive. It becomes rigid and might miss other safe, comfortable options. This is called mode collapse—the car forgets that there are multiple ways to solve a problem.

2. The Solution: Drive-JEPA

The researchers built a three-part system to fix this, acting like a master driving instructor with a time machine and a video game console.

Part A: The "Movie Buff" Pre-training (V-JEPA)

Instead of just memorizing specific turns, the car first watches thousands of hours of raw driving videos.

The Analogy: Imagine a student who watches thousands of hours of driving movies but isn't told what to do yet. They learn the "grammar" of driving: how cars move, how traffic flows, and how the world changes over time.
The Tech: They use a technique called V-JEPA. Think of this like a "fill-in-the-blanks" game for videos. The computer hides a chunk of the video and asks the AI to predict what happens next. By doing this millions of times, the AI builds a deep, intuitive understanding of how the driving world works without needing to label every single object. This gives the car a strong "brain" before it even starts learning to steer.

Part B: The "Simulator Coach" (Multimodal Trajectory Distillation)

Once the AI has its "brain," it needs to learn to make decisions. Usually, it only sees the one path a human took. Drive-JEPA changes the game by bringing in a Simulator Coach.

The Analogy: Imagine a driving student practicing in a video game. The game can generate thousands of different "what-if" scenarios. Maybe in one version of the simulation, the car swerves left; in another, it brakes hard; in a third, it weaves through traffic. All of these are safe and legal.
The Tech: The system uses a rule-based simulator to generate many different safe paths (trajectories) for every scene. It then teaches the AI to recognize and value all these different options, not just the single path the human took. This is called Multimodal Trajectory Distillation. It's like telling the AI, "There isn't just one right answer; here are five different safe ways to handle this intersection."

Part C: The "Smoothness Filter" (Momentum-Aware Selection)

Now the AI has a bunch of different paths to choose from. How does it pick the best one?

The Analogy: Imagine you are walking down a hallway. If you suddenly jerk your body left, then right, then left again, you feel dizzy and uncomfortable. You want to move smoothly.
The Tech: The system includes a Momentum-Aware Selection module. It looks at the path the car took a split second ago and checks the new options. If an option requires a sudden, jerky change in direction, the system penalizes it. It chooses the path that is not only safe but also feels smooth and natural to the passengers, avoiding "jerky" driving.

The Results

When they tested this system on standard driving benchmarks (like NAVSIM and Bench2Drive), the results were impressive:

Smarter Decisions: It outperformed previous state-of-the-art methods, setting new records for safety and smoothness.
Less is More: Even when they removed all the fancy "perception" sensors (like LiDAR) and only used a single front-facing camera, the system still performed better than others. This proves that the "Movie Buff" pre-training gave the car a really strong foundation.
Smoother Rides: By using the momentum filter, the car drives more comfortably, avoiding sudden, jarring movements.

In Summary

Drive-JEPA is like taking a self-driving car student, giving them a PhD in driving theory by watching thousands of movies, letting them practice in a video game where they see every possible safe outcome, and then teaching them to pick the smoothest, most comfortable path. The result is a car that drives more like a skilled human who understands the flow of traffic, rather than a robot just copying a single line on a map.

Technical Summary: Drive-JEPA

Problem Statement

End-to-end autonomous driving aims to map raw sensor observations directly to driving actions using a unified neural model. However, the field faces two primary bottlenecks:

Limited Utility of Video Pretraining: While self-supervised video pretraining is increasingly used to learn transferable representations, existing approaches (e.g., pixel-level generative models or latent world models) have yielded only limited improvements. Pixel-level objectives are computationally expensive and may focus on irrelevant visual details, while latent approaches often suffer from representation collapse or fail to demonstrate clear benefits from scaling.
Multimodal Ambiguity and Supervision Bottleneck: Driving scenarios are inherently multimodal, yet standard datasets provide only a single human trajectory per scene. This scarcity limits the learning of diverse behaviors. Existing solutions either rely on fixed discrete vocabularies (limiting generalization to out-of-vocabulary scenarios) or diffusion-based methods that remain constrained by single-trajectory supervision, leading to mode collapse or limited diversity.

Methodology: Drive-JEPA

The authors propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation to address these bottlenecks. The framework consists of three core components:

1. Driving Video Pretraining (V-JEPA Adaptation)

To learn planning-aligned predictive representations, the authors adapt V-JEPA for the driving domain.

Architecture: A Vision Transformer (ViT) encoder is pretrained on a large-scale dataset of driving videos (curated from CoVLA, DrivingDojo, and OpenScene).
Objective: The model learns to predict the latent representation of a masked target view from a masked source view. This self-supervised objective includes a stop-gradient operator and an exponential moving average (EMA) encoder to prevent representation collapse.
Scaling: Unlike prior latent world models, this approach successfully scales pretraining to 330 hours of video data with lower computational cost, producing features that are inherently aligned with trajectory planning.

2. Multimodal Trajectory Distillation

To overcome the single-trajectory supervision bottleneck, the framework distills knowledge from rule-based simulators into a proposal-centric planner.

Proposal Generation: Instead of a fixed vocabulary, the model generates online trajectory proposals using waypoint-anchored deformable attention. These proposals are iteratively refined by aggregating Bird's-Eye-View (BEV) features around predicted waypoints.
Pseudo-Teacher Distillation: A trajectory vocabulary is constructed via k-means clustering of historical data. For each scene, a rule-based simulator (based on NAVSIM v2 protocols) evaluates trajectories in this vocabulary against safety and comfort constraints (EPDMS score). High-scoring, diverse trajectories are selected as "pseudo-teachers."
Training Loss: The proposal generator is trained to minimize the distance to both the ground-truth human trajectory and the set of diverse simulator-generated pseudo-teacher trajectories, thereby encouraging multimodal output without relying on a single human demonstration.

3. Momentum-Aware Trajectory Selection

To select the final trajectory from the diverse proposal set, a neural scorer evaluates candidates based on collision risk, traffic rule compliance, and comfort.

Comfort Mechanism: To mitigate temporal inconsistency (jitter) often introduced by diverse proposals, the selection module incorporates a momentum-aware penalty. It compares the current proposals against the trajectory selected in the previous frame ( $t-1$ ) and penalizes large distortions.
Final Selection: The trajectory with the highest recalibrated score (combining the learned scorer and the comfort term) is selected for execution.

Key Contributions

V-JEPA Pretraining for Driving: The paper introduces the application of V-JEPA to end-to-end driving, demonstrating that large-scale video pretraining with predictive objectives significantly boosts performance in both perception-based and perception-free settings.
Multimodal Trajectory Distillation: A novel supervision strategy that distills simulator-generated, diverse trajectories into a proposal-centric framework, effectively breaking the limitation of single-human-trajectory supervision.
Momentum-Aware Selection: A lightweight selection module that enhances driving comfort and temporal stability by penalizing frame-to-frame trajectory distortion.
State-of-the-Art Performance: The method achieves new state-of-the-art results on major benchmarks without relying on complex perception annotations.

Experimental Results

The framework was evaluated on NAVSIM v1, NAVSIM v2, and Bench2Drive.

NAVSIM v1 (Perception-Free): Using only a front-view camera and a lightweight transformer decoder with V-JEPA pretrained features, the model achieved 89.0 PDMS, outperforming prior perception-free methods (e.g., Epona at 86.1, LAW at 83.8) by a significant margin.
NAVSIM v1 (Full Framework): With the complete Drive-JEPA framework (including distillation and selection), the model achieved 93.7 PDMS (ViT/L backbone), surpassing the previous best (DriveSuprim at 93.5).
NAVSIM v2: The method achieved 87.8 EPDMS, setting a new state-of-the-art. Notably, it performed well on Extended Comfort (EC) and other rule-compliance metrics where prior methods struggled.
Bench2Drive: In closed-loop CARLA simulations, Drive-JEPA achieved a Driving Score (DS) of 64.52, outperforming other proposal-centric methods like iPad (60.52) by 4 points, validating the effectiveness of the multimodal distillation.

Significance and Claims

The paper claims that Drive-JEPA successfully unifies self-supervised video pretraining with multimodal trajectory generation to address the fundamental limitations of current end-to-end driving systems.

Effectiveness of Pretraining: The results highlight that V-JEPA pretraining provides strong, transferable representations that allow even simple decoders to outperform complex perception-based pipelines in perception-free settings.
Diversity vs. Stability: The work demonstrates that simulator-guided distillation can generate diverse, safe trajectories without sacrificing temporal stability, provided a momentum-aware selection mechanism is used.
Scalability: The approach proves that scaling video pretraining with efficient latent prediction (V-JEPA) is a viable path forward, avoiding the computational costs of pixel-level generation while avoiding the collapse issues of naive latent models.

The authors emphasize that their method sets a new benchmark for end-to-end driving, particularly in scenarios where reliance on explicit perception annotations is minimized, and that the integration of simulator knowledge offers a robust path to learning multimodal driving behaviors.