🤖 machine learning

TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval

TraVEL is a motion-aware fine-tuning framework that enhances driving-video retrieval by leveraging ego-trajectory similarity as a reward within Group Relative Policy Optimization, enabling general-purpose multimodal models to effectively distinguish motion-centric events without relying on auxiliary perception data or expert rules.

Original authors: Yi-Chung Chen, Philip Jacobson, Tom Lampo, Yiren Lu, Jin Yao, David I. Inouye, Jing Gao, Danhua Guo, Burhan Yaman

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi-Chung Chen, Philip Jacobson, Tom Lampo, Yiren Lu, Jin Yao, David I. Inouye, Jing Gao, Danhua Guo, Burhan Yaman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Video Hunt

Imagine you are a detective trying to solve a mystery, but instead of a single crime scene, you have a library containing millions of hours of video footage. This is the daily reality for companies that build self-driving cars. They don't just drive; they record everything. But finding the specific moment a car almost hit a pedestrian, or the exact second a driver made a tricky turn, is like finding a needle in a haystack made of other needles.

To solve this, scientists use something called "multimodal embedding." Think of this as a magical translator that turns both a video clip and a sentence of text into a single, secret code (a vector). If the code for a video of a "red car turning left" is very similar to the code for the text "red car turning left," the computer knows they match. It's like giving every video a unique fingerprint and every search query a matching key. The goal is to make the computer so good at matching these keys that it can instantly pull up the right video from millions of options without needing a human to watch them all.

However, there's a catch. Current "magic translators" are great at recognizing static things like trees, buildings, or the color of the sky. But they often get confused by movement. To a standard AI, a car turning left might look just like a car turning right if the background scenery is the same. They rely too much on the "where" and not enough on the "how." This paper asks: Can we teach these AI models to care more about the motion itself, so they can tell the difference between a car speeding up and a car slowing down, even if the scenery looks identical?

The Solution: Teaching AI to Feel the Road

The researchers behind this paper, working with Uber AV Labs and Purdue University, introduced a new method called TraVEL (Trajectory-Guided Video Embedding Learning). Their main finding is that by using the actual physical path a car takes (its "ego-trajectory") as a special training tool, they can make the AI much better at understanding driving actions.

Here is how they did it, using a simple analogy:

The Problem: The "Context" Trap
Imagine you have two students taking a test about driving.

Student A (the old AI) looks at a picture of a car turning left on a sunny day and a picture of a car turning right on a sunny day. Because the sky and the trees look the same, Student A thinks, "These are basically the same picture!" They get confused because they are focused on the background.
Student B (the new AI) is taught to ignore the background and focus on the car's wheels and steering.

The researchers found that standard AI models were acting like Student A. They could tell the difference between a "daytime clip" and a "nighttime clip," but they struggled to tell the difference between "accelerating" and "decelerating" if the scene looked similar.

The Fix: The "Privileged" Coach
To fix this, the team used a two-step training process.

Step One: Reading the Manual. First, they taught the AI using text descriptions (captions) paired with videos. This helped the AI learn the language of driving, but it wasn't enough. The AI still didn't "feel" the motion deeply.
Step Two: The Trajectory Reward. This is where TraVEL shines. They introduced a "privileged" coach. During training, the AI was shown the video and the exact GPS path the car took. The coach would say, "If the text says 'turn left,' and the video shows a 'right turn' path, you get a bad score. If the path matches the text, you get a high score."

Crucially, this "coach" (the GPS path) is only used during training. Once the AI is finished learning, it doesn't need the GPS path anymore. It just needs the video and the text query. The AI has learned to internalize the feeling of motion so well that it can find the right video using just a single search code, without needing extra sensors or complex rules at the moment of the search.

What They Found
The results were quite promising. When they tested their new method on a dataset of driving clips called nuReasoning:

Better at Motion: The AI became significantly better at finding videos based on movement. For example, when searching for "accelerating" or "turning left," the accuracy jumped.
The Numbers: At a model size of 2 billion parameters, the new method improved the ability to find "longitudinal" (forward/backward) motion by 9.8 points and "lateral" (side-to-side) motion by 4.7 points compared to the previous best method. Even at a larger 8 billion parameter size, they saw gains of 7.2 and 1.5 points respectively.
Visual Proof: In one test, the old AI tried to find a video of a car slowing down for a pedestrian and accidentally picked a video of a car speeding up at night. The new TraVEL model correctly picked the video of the car slowing down, matching both the action and the context.

What They Ruled Out
The paper explicitly argues against the idea that simply making the AI model bigger (adding more parameters) will solve the problem. They tested models ranging from small to huge, and without their new training method, the bigger models were still just as confused about motion as the smaller ones. They also ruled out the need for complex, multi-step systems that require extra sensors or human-defined rules every time a search is made. TraVEL keeps things simple: one video, one code, one search.

How Sure Are They?
The authors suggest that this approach is a strong improvement for "motion-centric" retrieval, but they are careful to note that it is not a perfect solution for every driving scenario. They found that while the AI got much better at big movements like turning and stopping, it still found subtle lane changes (like "slightly moving left in the lane") a bit tricky. They conclude that while physical motion signals are a powerful tool, there is still room for improvement, perhaps by teaching the AI to notice the movement of other cars, not just the one it is riding in.

In short, TraVEL teaches self-driving car computers to pay attention to the dance of the vehicle, not just the stage it's dancing on, making it much easier to find the specific moments that matter for safety and learning.

Technical Summary: TraVEL – Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval

Problem Statement

Efficiently retrieving specific driving scenarios from large-scale autonomous vehicle logs is critical for data curation, model development, and safety analysis. While structured, rule-based retrieval systems can target specific events, they require expert-defined rules, auxiliary data (e.g., object detections, maps), and multi-stage perception pipelines, making them costly to extend to new domains. Conversely, multimodal embedding models offer a simpler, efficient alternative by representing videos as single searchable vectors. However, general-purpose vision-language models (VLMs) often fail in the driving domain because they rely on static scene context shortcuts rather than fine-grained motion. They struggle to distinguish operationally opposite events (e.g., accelerating vs. decelerating, turning left vs. right) when the visual background is similar.

Methodology

The authors propose TraVEL (Trajectory-Guided Video Embedding Learning), a framework designed to adapt general-purpose multimodal embedding models to driving-video retrieval by incorporating physical motion signals during training without requiring them at inference time. The approach consists of a two-stage training pipeline:

1. Driving-Domain Supervised Fine-Tuning (SFT)

The authors first address the domain gap between general pretraining data and driving videos. They fine-tune the Qwen3-VL-Embedding model using paired driving video clips and reasoning-trace captions from the nuReasoning dataset. This stage employs the InfoNCE objective to align text and video representations, significantly improving overall retrieval performance. However, the authors note that caption supervision alone provides only a sparse signal for fine-grained vehicle dynamics, leaving motion-centric queries (like specific acceleration profiles) challenging.

2. Trajectory-Aware Group Relative Policy Optimization (GRPO)

To address the limitations of caption-only supervision, TraVEL introduces a reinforcement learning stage using Group Relative Policy Optimization (GRPO).

Privileged Supervision: During training, each video clip is paired with a synchronized ego-pose sequence (lateral displacement and speed). These poses are privileged information: they are used solely as a reward signal during training and are not provided as inputs to the embedding model or required during retrieval.
Motion Representation: Ego-pose sequences are resampled and transformed using a logarithmic warping function to emphasize changes near zero (e.g., stopping to accelerating). The similarity between motion signals is calculated using Dynamic Time Warping (DTW) to handle temporal misalignments.
Reward Construction: The training process constructs two types of candidate groups:
- Text-to-Video ( $G_{t2v}$ ): Includes the ground-truth video and hard negatives. The reward is binary (1 for the correct pair, 0 otherwise) to preserve cross-modal alignment.
- Video-to-Video ( $G_{v2v}$ ): Includes high-scoring video candidates based on the current model. The reward is a graded value derived from the ego-trajectory similarity (DTW distance). Clips with similar motion receive higher rewards.
Optimization: The GRPO objective updates the model to maximize the advantage of correct pairs and motion-similar videos, effectively reshaping the embedding space to be sensitive to vehicle motion while maintaining the single-vector retrieval efficiency.

Key Contributions

TraVEL Framework: A simple, effective method for adapting general-purpose multimodal embedding models to driving-video retrieval by integrating physical motion signals into the training loop.
Motion-Aware Fine-Tuning: The introduction of a trajectory-derived reward within a GRPO framework. This makes video embeddings sensitive to ego-motion (longitudinal and lateral) without altering the inference pipeline (no need for ego poses or auxiliary perception at retrieval time).
Driving-Video Retrieval Benchmark: The construction of a retrieval benchmark derived from the nuReasoning dataset, specifically evaluating motion-centric queries.

Experimental Results

The authors evaluated TraVEL on the nuReasoning dataset using Qwen3-VL-Embedding models at 2B and 8B parameter scales.

Baseline Performance: Off-the-shelf models (including CLIP4Clip, InternVideo2, and pretrained Qwen3-VL) exhibited a significant domain gap, with R@1 scores generally below 2.5% and median ranks near random chance.
SFT Impact: Supervised fine-tuning on paired clips improved retrieval significantly but remained insufficient for fine-grained motion distinctions.
TraVEL Performance:
- Instance-Level Retrieval: TraVEL substantially improved recall. At 2B, R@1 increased from 1.3% (Pretrained) to 8.7%, and median rank dropped from 266 to 23. At 8B, R@1 reached 10.1%, with a median rank of 17.
- Motion-Centric Retrieval: TraVEL showed the most significant gains on motion-specific queries.
  - 2B Model: Longitudinal mAP improved by 9.8 points (from 45.9% with SFT to 55.7% with TraVEL). Lateral mAP improved by 4.7 points (from 31.0% to 35.7%).
  - 8B Model: Longitudinal mAP improved by 7.2 points (to 57.5%), and lateral mAP improved by 1.5 points (to 39.9%).
- Qualitative Analysis: Visualizations showed that TraVEL successfully reduced similarity between visually similar clips with different motions (e.g., accelerating left turn vs. decelerating right turn) and increased similarity between clips sharing the same trajectory despite different visual contexts (e.g., day vs. night).

Significance and Claims

The paper claims that TraVEL successfully bridges the gap between the efficiency of single-vector embedding search and the need for motion-sensitive retrieval in autonomous driving. By using ego-trajectory similarity as a privileged training signal, the method organizes the video embedding space according to vehicle dynamics without requiring complex perception pipelines or auxiliary data during inference.

The authors emphasize that while TraVEL yields clear gains on motion-centric queries (particularly stopping and turning behaviors), challenges remain for subtle within-lane shifts. They position TraVEL not as a replacement for structured rule-based systems but as a complementary, efficient approach that leverages readily available physical motion signals to enhance the representational power of general-purpose multimodal models. The work suggests that combining physically grounded supervision with embedding-based search is a viable path for improving downstream driving workflows such as data curation and safety validation.