💻 computer science

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

This paper introduces \name, an extreme video compression framework that combines learnable token-level and question-conditioned frame-level compression with local attention mechanisms to enable vision-language models to process significantly more frames with improved accuracy while using minimal supervised fine-tuning data.

Original authors: Zheyu Zhang, Ziqi Pang, Shixing Chen, Xiang Hao, Vimal Bhat, Yu-Xiong Wang

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zheyu Zhang, Ziqi Pang, Shixing Chen, Xiang Hao, Vimal Bhat, Yu-Xiong Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to explain a three-hour movie to a friend who has a very short attention span and can only remember a few key details at a time.

If you try to describe every single second of the movie, your friend's brain (the "Large Language Model" or LLM) will get overwhelmed. They will forget the beginning by the time you get to the middle, or they will just give up because there's too much information.

This is the exact problem computers face when trying to understand long videos. Current AI models are like that overwhelmed friend: they can't process thousands of video frames without losing the plot.

The paper you shared, "XComp," proposes a brilliant new way to solve this. Here is how it works, broken down into simple concepts and analogies.

1. The Problem: The "Token" Traffic Jam

In the world of AI, images and videos are broken down into tiny pieces of data called tokens.

The Old Way: To understand a video, the AI usually turns every single frame into 16 to 100 tokens. If you have a 1-minute video with 30 frames per second, that's 1,800 frames. Multiply that by 100 tokens, and you have 180,000 tokens.
The Result: The AI's "brain" (the LLM) has a limit on how many tokens it can hold in its "working memory" at once. It hits a wall, drops information, and misses crucial details.

2. The Solution: XComp (The Extreme Compression)

The authors created a system called XComp. Think of it as a super-efficient movie editor that doesn't just cut scenes; it rewrites the entire script into a single, perfect sentence for every scene.

They do this in two clever steps:

Step A: The "Smart Summarizer" (Token-Level Compression)

The Analogy: Imagine you have a 10-page document for every single frame of the video. The old AI tries to read all 10 pages. XComp teaches the AI to read those 10 pages and instantly write down one single, perfect sentence that captures the entire meaning of that page.

How it works: Instead of using a rigid, pre-set rule (like "delete every other word"), the AI learns how to compress information. It practices this skill during a short training session.
The "Progressive" Trick: It doesn't try to go from 10 pages to 1 sentence in one giant leap (which would lose details). Instead, it does it gradually, layer by layer, like peeling an onion. Each layer of the AI removes a little bit of fluff until, by the very end, one single token represents one entire video frame.

Step B: The "Relevance Filter" (Frame-Level Compression)

The Analogy: Imagine you are watching a 2-hour movie, but your friend only asks, "What did the villain do in the kitchen?"
The old AI would try to remember the whole movie. XComp is smarter. It looks at the question and says, "I don't need to remember the car chase or the beach scene. I only need to remember the kitchen scene."

How it works: The AI looks at its own internal "attention" (what it's focusing on). It asks, "Which frames are actually relevant to the question?"
The "Lost in the Middle" Fix: AI often gets confused in the middle of long stories (it remembers the start and end best). XComp splits the long video into small, manageable "chapters" (segments). It analyzes each chapter separately to make sure it doesn't miss the important stuff in the middle.

3. The Magic Result: "One Token Per Frame"

By combining these two tricks, XComp achieves something previously thought impossible: It can process a video where every single frame is represented by just ONE token.

Before: 1,000 frames = 100,000 tokens (Too heavy!).
With XComp: 1,000 frames = 1,000 tokens (Light as a feather!).

Why This Matters

Because the AI is now so efficient, it can:

Watch More: It can process videos that are hours long without crashing.
See More: Because it doesn't have to skip frames to save space, it can look at every frame (dense sampling), catching tiny details it used to miss.
Learn Faster: The authors showed that you can teach this system to be this smart using only 2.5% of the data usually required. It's like teaching a student to be a genius by showing them just a few key chapters instead of the whole library.

Summary

Think of XComp as a super-efficient librarian.

Old Librarian: Tries to carry the entire encyclopedia to answer one question. Gets tired, drops books, and forgets facts.
XComp Librarian: Knows exactly which page contains the answer. They don't carry the whole book; they just bring the one specific sentence needed. They can handle a library of millions of books without breaking a sweat.

This allows AI to finally understand long, complex videos (like movies, lectures, or security footage) with the same ease as looking at a single photo.

1. Problem Statement

Long video understanding poses a significant bottleneck for Vision-Language Models (VLMs) due to the sheer volume of visual data.

Token Explosion: Standard VLMs encode each video frame into tens or hundreds of tokens. Processing long videos (minutes to hours) results in token sequences that exceed the context length limits of Large Language Models (LLMs) and exceed computational budgets.
Information Loss: To fit within context limits, existing methods often resort to sparse frame sampling (e.g., 1 frame per second) or heuristic token compression (e.g., pooling, fixed selection). These approaches discard critical temporal information and fine-grained visual details.
Limitations of Heuristics: Previous compression methods rely on training-free heuristics (e.g., selecting tokens based on attention scores without supervision). The authors argue these fail at extreme compression ratios because the LLM layers are not explicitly trained to consolidate information from dropped tokens into retained ones.
Position Bias: LLMs exhibit a "lost in the middle" phenomenon, where attention mechanisms disproportionately focus on the beginning and end of a sequence, causing critical information in the middle of long videos to be overlooked.

2. Methodology: XComp

The authors propose XComp, a framework designed to achieve extreme compression (reducing each video frame to a single token) while maintaining high understanding accuracy. The framework operates on two levels:

A. Token-Level Compression: Learnable & Progressive Compression (LP-Comp)

Instead of relying on heuristics, XComp trains the LLM layers to actively learn how to compress visual tokens.

Supervised Compression Tuning (SCT): The LLM is fine-tuned on a small subset (2.5%) of supervised data to learn the compression behavior.
Progressive Schedule: Compression is not performed in a single step. Instead, it occurs gradually across all $L$ $L$ layers of the LLM. The number of tokens per frame, $N^{(\ell)}$ $N^{(ℓ)}$ , decreases smoothly from the initial count (e.g., 16) to 1 at the final layer.
- The schedule follows a cosine curve to ensure a smooth reduction, preventing abrupt information loss.
Suffix Preservation: To align with the causal attention mechanism of decoder-only LLMs, the compression strategy preserves the suffix tokens (the last $N$ tokens) of each frame. This allows later tokens to absorb features from earlier ones, which is more compatible with how LLMs process sequences than uniform dropping.

B. Frame-Level Compression: Question-Conditioned Compression (QC-Comp)

To further reduce redundancy, XComp selects only the frames most relevant to the user's query.

Attention-Based Selection: The model calculates relevance scores for frames based on the internal attention scores between the question tokens and video tokens within the LLM.
Segmented Local Attention: To mitigate the "lost in the middle" position bias, long videos are split into short, overlapping segments (e.g., 64 frames). Attention scores are computed locally within each segment rather than globally across the entire video.
Selection: Frames with the highest average attention scores across segments are retained, while irrelevant frames are discarded. This allows the model to process a denser set of frames (e.g., 1024+ frames) without exceeding context limits.

3. Key Contributions

Extreme Token Compression: The paper demonstrates the feasibility of compressing every video frame into a single token at the final LLM layer, a significant reduction from the typical 16–100+ tokens per frame.
Learnable Compression (LP-Comp): It introduces a paradigm shift from heuristic-based compression to learnable, progressive compression. By supervising the LLM layers, the model learns to condense information effectively, outperforming training-free methods.
Question-Conditioned Selection (QC-Comp): It proposes a frame selection mechanism that utilizes internal LLM attention scores within local segments to overcome position bias and select the most informative frames for specific queries.
Data Efficiency: The method achieves state-of-the-art performance with a highly data-efficient fine-tuning stage, requiring only 2.5% of the standard supervised fine-tuning (SFT) data used by the baseline model (VideoChat-Flash).

4. Experimental Results

The model was evaluated on four major long-video benchmarks: LongVideoBench, MLVU, VideoMME (Long), and LVBench.

Performance Gains:
- XComp (based on VideoChat-Flash-2B) achieved 46.2% accuracy on LVBench, surpassing the baseline VideoChat-Flash-2B (42.9%) and outperforming several 7B-9B scale models.
- It showed consistent improvements across all benchmarks, demonstrating that extreme compression does not degrade performance but rather enhances it by allowing denser frame sampling.
Scalability: Unlike baselines that suffer performance drops as the number of input frames increases, XComp's performance improves with more input frames, proving its ability to utilize dense temporal information.
Efficiency:
- Training: Reduced training time and resource usage due to shorter token sequences.
- Inference: Achieved a 53–58% reduction in TFLOPs and 45–58% reduction in latency compared to the baseline when processing long videos (1k–4k frames).
Ablation Studies:
- Learnable vs. Heuristic: Learnable compression significantly outperformed heuristic token selection.
- Progressive vs. Step-wise: Smooth, progressive compression yielded better results than abrupt, step-wise reduction.
- Suffix vs. Uniform: Preserving suffix tokens was crucial for performance.
- Local vs. Global Attention: Segmented local attention effectively mitigated position bias, outperforming global attention strategies.

5. Significance and Impact

Redefining Long Video Understanding: XComp challenges the notion that long videos require massive context windows. It proves that with intelligent, learned compression, models can process hours of video with extreme token efficiency.
Bridging the Gap: It enables 2B-scale models to compete with much larger proprietary models (e.g., GPT-4o, Gemini-1.5) on long-video tasks, democratizing access to high-performance video understanding.
Future Directions: The work suggests that future VLMs could integrate learnable compression directly into pre-training and jointly optimize frame selection with LLM layers to achieve even higher compression ratios.

In summary, XComp provides a robust solution to the token bottleneck in long video understanding by combining learnable progressive token compression with query-aware frame selection, enabling models to process videos with unprecedented density and efficiency.