💻 computer science

EarlyTom: Early Token Compression Completes Fast Video Understanding

The paper proposes EarlyTom, a training-free framework that performs early-stage visual token compression inside the vision encoder to significantly reduce time-to-first-token latency and computational costs while maintaining accuracy comparable to full-token baselines.

Original authors: Hesong Wang, Xin Jin, Lu Lu, Chenhaowen Li, Jian Chen, Qiang Liu, Huan Wang

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hesong Wang, Xin Jin, Lu Lu, Chenhaowen Li, Jian Chen, Qiang Liu, Huan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a 30-minute movie to a very smart, but very busy, friend (the AI). Your friend needs to watch the whole movie first before they can answer your questions.

The Problem: The "Slow Start"
Currently, when AI tries to understand a video, it acts like a student who insists on reading every single page of a textbook before answering a single question.

The Old Way: The AI watches every single frame of the video, breaks it down into thousands of tiny pieces (tokens), and then hands that massive pile to its "brain" (the Large Language Model) to process.
The Bottleneck: The paper discovered that the biggest delay isn't the AI's thinking time; it's the time spent just watching and organizing the video in the first place. It's like spending 40 minutes sorting a deck of cards before you even start playing the game. Even when other methods tried to throw away some cards later, the initial sorting was still slow.

The Solution: EarlyTom (The "Smart Editor")
The authors created a new method called EarlyTom. Think of EarlyTom as a super-efficient film editor who steps in while the video is being watched, not after.

Instead of waiting until the video is fully processed to decide what to keep, EarlyTom edits the video during the viewing process. It uses two main tricks:

1. The "Merge" Trick (Time Compression)
Imagine watching a video where a person is standing still talking for 10 seconds. The video has 300 frames of that same person.

Old Way: The AI processes all 300 frames individually.
EarlyTom: It notices, "Hey, these 300 frames are almost identical." Instead of processing them all, it merges them into one single, high-quality summary frame. It's like summarizing a 10-minute monologue into a single sentence before passing it on. This happens inside the camera lens (the vision encoder), so the heavy lifting is done much faster.

2. The "Spotlight" Trick (Space Selection)
Now imagine the AI has to look at a crowd of people in a video.

The Trap: The paper found that AI has a weird habit called "Attention Sinking." It's like a spotlight that gets stuck on a few specific spots (like a blank wall or a logo) and shines too brightly on them, ignoring the actual action happening elsewhere. If you just pick the "brightest" spots, you might miss the important action.
EarlyTom's Fix: It splits the crowd into two groups:
- The "Moving" Group: For parts of the video where things are changing (action), it picks the most important details globally.
- The "Still" Group: For parts where things are static, it uses a local "window" approach to make sure it doesn't get distracted by the stuck spotlight. It ensures the AI sees a balanced view of the scene without getting biased by those stuck spots.

The Result: Speed Without Losing Smarts
By doing this editing early in the process, EarlyTom achieves two amazing things:

Super Fast Start: It cuts the time it takes to get the first answer (Time-to-First-Token) by up to 2.65 times. If the old way took 900 milliseconds, this takes about 336 milliseconds.
Less Work: It reduces the total computing power needed by up to 61%.

The Bottom Line
The paper claims that EarlyTom allows video AI to be incredibly fast and efficient without needing to be retrained or losing its ability to understand the video accurately. It proves that you don't need to watch every single frame to understand the story; you just need to know when to stop watching and start thinking.

In short: EarlyTom is a training-free tool that edits the video while it's being watched, making video AI faster and cheaper to run, while keeping its answers just as smart.

Technical Summary: EarlyTom

Problem Statement

Video Large Language Models (Video-LLMs) have demonstrated strong capabilities in video understanding but face significant barriers to practical deployment due to the computational inefficiency of processing massive amounts of visual tokens. While recent token compression methods achieve low token retention ratios with minimal accuracy loss, they predominantly operate at the late stage of the prefilling process (after the vision encoder) or within the LLM itself.

Latency profiling reveals a critical bottleneck: the vision encoding stage consumes a substantial portion of the Time-to-First-Token (TTFT). In baseline models, vision encoding accounts for 36.3% of TTFT. In state-of-the-art methods optimized for LLM prefill latency (e.g., HoliTom, VisionZip), this proportion rises to 55.8% and 68.4%, respectively, because these methods leave the vision encoder unoptimized. Furthermore, existing compression strategies often introduce non-trivial computational overhead during token processing, further increasing latency. Additionally, standard Top-K token selection methods are susceptible to "attention sinks"—specific tokens that consistently attract disproportionately high attention scores regardless of content—leading to biased compression and potential loss of semantic information in dynamic frames.

Methodology: EarlyTom

The authors propose EarlyTom, a training-free token compression framework designed to perform compression early within the vision encoder and utilize a decoupled spatial selection strategy. The framework consists of two core stages:

1. Inner Vision Encoder Frame Merging (Temporal Compression)

This stage compresses redundant visual information during the encoding process to reduce the number of tokens before they reach the projector or LLM.

Streaming Frame Segmentation: The input video is adaptively segmented based on frame similarity. Cosine similarity between consecutive frames is calculated and smoothed using an Exponential Moving Average (EMA). A segment boundary is defined when the smoothed similarity drops below a threshold ( $\tau_{seg}$ ).
Middle Frame Merge: Within each segment, middle frames (excluding the first and last) are merged if they are highly similar to their neighbors and more similar to each other than to the subsequent pair.
Weighted Frame Merge: Merged frames are generated via a weighted fusion of the frame features, where weights are determined by the similarity scores. This ensures the merged representation remains concentrated around semantically important content while reducing temporal redundancy.

2. Decoupled Spatial Token Selection

To address the bias introduced by attention sinks and further reduce token count, the method employs a system-level co-designed selection strategy after frame merging.

Dynamic vs. Static Decomposition: Merged frames are divided into dynamic (head and tail frames of segments, possessing high discriminative power) and static (middle frames) sets.
Global Top-K Selection (Dynamic): For dynamic frames, a global Top-K selection is performed based on per-token attention scores to preserve motion-sensitive tokens across the temporal dimension.
Local Window Top-K Selection (Static): For static frames, to avoid the bias of attention sinks dominating the selection, the frames are divided into local windows. Within each window, the token with the maximum attention score is selected. This preserves the original spatial distribution and mitigates the negative effects of structural attractors.
System Co-Design: To enhance efficiency, the computationally lighter static token selection is offloaded to the CPU, while the GPU handles the more intensive dynamic token selection, leveraging heterogeneous computing resources.

Key Contributions

Inner-Vision Encoder Frame Merge: A mechanism that compresses redundant visual information during the encoding phase, effectively reducing visual tokens with negligible overhead and significantly lowering TTFT.
Decoupled Token Selection: A strategy that separates frames into dynamic and static sets, applying distinct sampling schemes (Global Top-K vs. Local Window Top-K) to reduce tokens without introducing bias from attention sinks.
State-of-the-Art Performance: Extensive experiments on LLaVA-OneVision-0.5B and 7B models demonstrate that EarlyTom achieves superior acceleration while maintaining accuracy comparable to full-token baselines.

Experimental Results

Evaluated on a single NVIDIA A100 GPU using LLaVA-OneVision-7B and four benchmarks (MVBench, EgoSchema, LongVideoBench, VideoMME):

Efficiency: EarlyTom reduces TTFT by up to 2.65× (from 889 ms to 336 ms at 10% retention) and reduces FLOPs by up to 61% compared to the full-token baseline. It consistently outperforms other training-free methods (e.g., VisionZip, HoliTom, FastVID) in throughput and latency across various retention ratios (10%–25%).
Accuracy: The method maintains competitive downstream quality. At a 10% retention ratio, EarlyTom achieves an average benchmark score of 56.2 (96.2% of the full-token baseline), outperforming other methods that suffer significant accuracy degradation under aggressive pruning.
Generalizability: The approach is validated on the LLaVA-Video-7B and Qwen2.5-VL-7B architectures, showing consistent improvements in FLOPs reduction and TTFT speedup while preserving accuracy.
Ablation: Removing either the frame merging or the decoupled selection strategy results in performance degradation, confirming the necessity of both components for balancing efficiency and fidelity.

Significance

The paper claims that EarlyTom addresses a fundamental bottleneck in Video-LLM deployment by shifting the compression paradigm from the late stage (post-encoder) to the early stage (inside the vision encoder). By directly optimizing the most time-consuming component of the inference pipeline (vision encoding) and introducing a bias-aware spatial selection strategy, EarlyTom enables significantly faster inference (lower TTFT) and higher throughput without sacrificing model performance. This work lays a foundation for the practical deployment of Video-LLMs in real-world, latency-sensitive production scenarios where computational resources are constrained.