🤖 AI

OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models

The paper introduces OTT-Vid, a training-free temporal token compression framework for Video Large Language Models that leverages optimal transport with non-uniform token mass and locality-aware costs to dynamically allocate compression budgets based on frame-pair compressibility, achieving state-of-the-art performance while retaining only 10% of visual tokens.

Original authors: Minseok Kang, Minhyeok Lee, Jungho Lee, Minjung Kim, Donghyeong Kim, Dayeon Lee, Heeseung Choi, Ig-jae Kim, Sangyoun Lee

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minseok Kang, Minhyeok Lee, Jungho Lee, Minjung Kim, Donghyeong Kim, Dayeon Lee, Heeseung Choi, Ig-jae Kim, Sangyoun Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Too Much Video, Not Enough Brainpower

Imagine you are trying to watch a 10-minute movie, but your brain (the AI model) can only hold a tiny amount of information at once. To understand the movie, the AI breaks every single frame into thousands of tiny puzzle pieces called "tokens."

If you have a long video, the number of these puzzle pieces explodes. It's like trying to carry a mountain of bricks in a wheelbarrow; the AI gets overwhelmed, slows down, and costs a fortune to run.

The Old Solution: "If it looks the same, throw it away"

To fix this, researchers tried to throw away the extra bricks. Their old rule was simple: "If a brick in Frame 5 looks exactly like a brick in Frame 6, delete the one in Frame 6."

The Flaw: This is like watching a video of a person standing still and saying, "They haven't moved, so I'll delete them." But that person is the main character! If you delete them, the AI forgets they exist.

The Result: The AI loses track of things that stay the same (like a stationary object or a person waiting) because it thinks they are boring duplicates. It also struggles to answer questions about how long something lasted or when it happened.

The New Solution: OTT-Vid (The Smart Librarian)

The authors of this paper propose OTT-Vid, a new way to compress video that doesn't just look at whether things look similar, but asks: "Is this piece important?"

They use a mathematical concept called Optimal Transport (think of it as a super-smart logistics planner) to decide what to keep and what to toss. Here is how it works in three steps:

1. The "Highlighter" (Spatial Pruning)

First, the AI looks at a single frame and highlights the most interesting parts.

Analogy: Imagine a teacher grading a test. They don't read every single word with equal attention. They highlight the key sentences. OTT-Vid does this for every video frame, keeping only the "highlighted" tokens and ignoring the boring background noise.

2. The "Importance Score" (Mass Assignment)

This is the secret sauce. The AI assigns a "mass" (a weight) to every remaining token.

The Twist: In this system, Important = Light Weight and Unimportant = Heavy Weight.
Why? Think of it like a delivery truck. You want to keep the fragile, valuable items (the important tokens) safe. You don't want to crush them. So, you give them a "light" status so the system knows not to throw them away. The boring background stuff gets a "heavy" status, meaning it's easy to drop or merge.
Result: Even if a person is standing still for 10 seconds, the AI knows they are important (light weight) and refuses to delete them, even if they look identical to the previous frame.

3. The "Logistics Plan" (Optimal Transport)

Now the AI has to decide how to compress the video between Frame 1 and Frame 2, Frame 2 and Frame 3, etc.

The Analogy: Imagine you are moving houses. You have a limited number of boxes (a budget).
- Old Way: You just throw away duplicates.
- OTT-Vid Way: The AI calculates a "Transport Difficulty."
  - If two frames are very similar and full of boring stuff, the "difficulty" is low. The AI says, "Great, we can pack these into one box and save space!"
  - If two frames have important changes (like a car starting to move), the "difficulty" is high. The AI says, "Don't touch this! We need to save our boxes for this."
Dynamic Budgeting: The AI doesn't give every pair of frames the same number of boxes. It gives more boxes to the exciting parts of the video and fewer boxes to the boring parts.

Why It's Better (The Results)

The researchers tested this on six different video challenges, including:

Video Question Answering: "What happened in the video?"
Temporal Grounding: "Exactly when did the person cut the cake?"

The Outcome:

They threw away 90% of the video data (keeping only 10% of the tokens).
Video Questions: The AI still got 95.8% of the answers right compared to watching the full video.
Timing Questions: The AI kept 73.9% of its accuracy on timing tasks.

The Key Takeaway:
Previous methods failed at timing tasks because they deleted the "boring" stationary parts that actually proved how long an event lasted. OTT-Vid keeps those parts because it understands their importance, not just their similarity.

Summary

OTT-Vid is like a smart editor who doesn't just cut out repeated scenes because they look the same. Instead, the editor asks, "Is this scene important to the story?" If a character is standing still but is crucial to the plot, the editor keeps them. If the background is boring and repetitive, the editor cuts it out. This allows the AI to watch long videos quickly without forgetting the important details.

Technical Summary: OTT-Vid

Problem Statement

As Video Large Language Models (Video-LLMs) scale to handle longer and more complex videos, they face a critical inference bottleneck: the accumulation of visual tokens across frames drastically increases sequence length and computational cost. While training-free token compression offers a practical solution, existing temporal compression methods suffer from two primary limitations:

Over-reliance on Similarity: Current methods primarily rely on cross-frame token similarity or segmentation heuristics. They fail to distinguish between uninformative background recurrence and the persistence of semantically important content (e.g., stationary objects). Consequently, high similarity often leads to the aggressive compression of essential temporal evidence, degrading performance on tasks requiring duration, continuity, and event boundary detection, such as temporal grounding.
Static Budget Allocation: Existing approaches do not explicitly treat compression as an allocation problem. They apply uniform compression thresholds or hard boundaries that fail to adapt to the varying compressibility of different frame pairs. This results in the under-compression of highly redundant regions and the over-compression of frames containing critical temporal evidence.

Methodology: OTT-Vid

The authors propose OTT-Vid, a training-free framework that formulates temporal token compression as an Optimal Transport (OT) problem with adaptive budget allocation. The method operates in two coupled stages:

1. Spatial Pruning and Mass Assignment

Before temporal compression, each frame undergoes independent spatial pruning to remove intra-frame redundancy.

Selection: A saliency-weighted coverage objective greedily selects representative tokens. This balances semantic importance (saliency) with representativeness (coverage) to prevent token collapse into redundant clusters.
Mass Assignment: A "leave-one-out" strategy quantifies the irreplaceable contribution of each retained token. If removing a token significantly degrades the representation of the original frame, it is deemed important.
Inverse Mass: Crucially, the transport mass $m_t$ is assigned inversely to token importance. Semantically critical tokens receive smaller mass (resisting compression), while less informative tokens receive larger mass (facilitating transfer). This encodes preservation priority directly into the OT marginal constraints.

2. Optimal Transport for Temporal Compression

Temporal compression between neighboring frames ( $t$ and $t+1$ ) is solved as an entropically regularized OT problem.

Cost Function: The transport cost combines semantic dissimilarity (cosine distance) and spatial distance. A mixing coefficient $\alpha_t$ adapts dynamically based on scene motion: in static scenes, spatial locality is weighted higher to prevent matching visually similar but spatially distant objects; in dynamic scenes, semantic dissimilarity dominates to allow matching across larger displacements.
Transport Plan: Solving the OT problem yields a coupling matrix $P_t$ that ranks token correspondences. High coupling values indicate candidates for compression (merging or pruning).
Transport Difficulty: The total cost of the transport plan ( $W_t$ ) serves as a metric for the "compressibility" of a frame pair. Low $W_t$ implies high redundancy (easy to compress), while high $W_t$ implies the presence of unique temporal evidence.

3. Adaptive Budget Allocation

The framework distributes the global compression budget non-uniformly across all adjacent frame pairs based on $W_t$ .

Inverse Allocation: Using a negative softmax, pairs with lower transport difficulty (high redundancy) receive larger compression budgets, while pairs with high difficulty (low redundancy) receive smaller budgets.
Execution: Compression is executed in parallel. Selected source tokens are either merged with their destination (if the cost is below a threshold $\tau_c$ ) or pruned. Chains of merges across consecutive frames are resolved via a union-find algorithm to ensure global consistency.

Key Contributions

Importance-Aware Temporal Compression: Unlike prior similarity-driven methods, OTT-Vid integrates intra-frame token importance into the temporal compression process. By assigning inverse mass to important tokens, it preserves evidence of duration and continuity that similarity-based methods typically discard.
Transport-Derived Budget Allocation: The paper introduces a principled method to estimate the compressibility of each frame pair via transport difficulty ( $W_t$ ). This allows for dynamic, non-uniform budget allocation that adapts to the specific redundancy characteristics of the video content.
Unified Framework: OTT-Vid unifies token-level compression decisions (via the transport plan) and pair-level budget allocation (via transport difficulty) within a single Optimal Transport formulation.

Experimental Results

The authors evaluated OTT-Vid on six benchmarks: four Video Question Answering (VQA) datasets (MVBench, VideoMME, LongVideoBench, MLVU) and two Video Temporal Grounding (VTG) datasets (Charades-STA, ActivityNet-Captions with TimeLens annotations).

Performance Retention: On the Qwen2.5-VL-7B backbone, OTT-Vid retains 95.8% of uncompressed VQA performance and 73.9% of VTG performance while keeping only 10% of the original tokens.
Comparison: It consistently outperforms state-of-the-art training-free baselines (HoliTom, FastVID, FlashVID, UniComp). The performance gap is most significant in VTG tasks, where similarity-driven methods suffer severe degradation (dropping to ~50-60% retention) due to the loss of temporal evidence.
Generalization: The method demonstrates robustness across different Video-LLM backbones (LLaVA-OneVision, LLaVA-Video) and various token retention ratios.
Efficiency: OTT-Vid achieves a 2.0× speedup in Time-to-First-Token (TTFT) with peak memory usage comparable to other efficient methods, confirming that the compression overhead does not negate the inference gains.

Significance and Claims

The paper claims that the primary significance of OTT-Vid lies in its ability to decouple semantic persistence from visual similarity. By treating temporal compression as a transport problem with importance-aware mass, the method successfully preserves the "evidence" required for complex temporal reasoning tasks, which previous similarity-based heuristics failed to do.

The authors emphasize that their approach is training-free, ensuring compatibility with existing pretrained Video-LLMs without the need for retraining. They note that while their method currently computes transport plans in parallel from initial spatially-pruned sets (ignoring representational changes induced by earlier compression), this design choice prioritizes low overhead and single-pass efficiency. Future work may explore sequential execution to refine transport plans, though this would increase computational costs.

In summary, OTT-Vid addresses the critical bottleneck of long-video inference by providing a theoretically grounded, adaptive mechanism that balances token importance with cross-frame redundancy, significantly outperforming existing methods in preserving temporal fidelity under aggressive compression.