🤖 AI

Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere

The paper proposes **ONCE**, a plug-in video token compression framework that shifts the costly compression process offline by learning a frequency-aware global codebook once, enabling efficient, model-agnostic online compression that significantly reduces inference latency while maintaining competitive performance across video understanding benchmarks.

Original authors: Jiayang He, Tianling Xu, Diancheng Kang, Huaide Jiang, Junyan Bai, Shaoming Zheng, Xuan Song

Published 2026-08-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiayang He, Tianling Xu, Diancheng Kang, Huaide Jiang, Junyan Bai, Shaoming Zheng, Xuan Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to understand movies. To do this, you break every frame of the film down into thousands of tiny digital "pixels" of meaning, called tokens. The robot then reads this massive list of tokens like a sentence to figure out what's happening. The problem is, movies are long, and they are full of repetition. If a character is standing still in a room for ten seconds, the robot is forced to read the exact same "standing in a room" token thousands of times in a row. It's like reading a book where the word "the" is repeated a million times just to fill space; it wastes the robot's brainpower and makes it incredibly slow to answer questions. Scientists have been trying to fix this by teaching the robot to skip the boring parts or merge similar ones while it's watching the movie, but this usually requires the robot to do extra math for every single movie it sees, which is still very slow and expensive.

This paper introduces a new way to handle this called ONCE. Instead of making the robot do the hard work of sorting and skipping every time it watches a new video, the researchers decided to do the heavy lifting before the robot ever sees a movie. They created a "Global Codebook," which is essentially a giant, pre-made index of the most common visual patterns found in thousands of hours of video. Think of it like a library where every book is already sorted into the perfect shelf. When a new movie arrives, the robot doesn't need to figure out how to organize it from scratch; it just looks up the patterns in its pre-made index, grabs the relevant "shelves," and averages them out. This allows the robot to process videos much faster because it's just doing a quick lookup instead of a complex calculation every time. The researchers found that this method works incredibly well, especially when they are forced to use very few tokens, keeping the robot fast without making it forget what it's watching.

The Problem: The Robot's Overwhelmed Brain

Video Large Language Models (Video-LLMs) are like super-powered detectives that can watch a video and answer questions about it. To do this, they turn the video into a long string of digital tokens. But here's the catch: if you feed the detective a two-hour movie, the string of tokens becomes so long that it overwhelms the detective's memory.

Worse yet, movies are full of redundancy. If a scene shows a cat sitting on a couch for five minutes, the video encoder generates thousands of tokens that all say "cat on couch." The detective has to read every single one of them, even though they are all the same. Existing methods try to fix this by having the detective "prune" (cut out) or "merge" (combine) these tokens while it's watching the video. But this is like asking the detective to stop and reorganize its notes for every single movie it sees. It takes extra time and energy every single time, and the rules for organizing change depending on which detective (model) you are using.

The Solution: ONCE and the "Global Codebook"

The authors of this paper, Jiayang He and their team, proposed a different approach. They asked: "What if we did the organizing once for everyone, instead of asking the detective to do it every time?"

They built a system called ONCE (Offline-to-Online Video Token Compression). Here is how it works, using a simple analogy:

Imagine you are a librarian who has to sort millions of books.

The Old Way (Existing Methods): Every time a new shipment of books arrives, you have to stop, read every single book, decide which ones are similar, and then physically group them together before you can shelve them. This takes a long time for every single shipment.
The ONCE Way: Before any new books arrive, you spend time studying a massive collection of books you already have. You create a "Master Index" (the Global Codebook) that lists every common type of book you've ever seen (e.g., "Action Hero," "Cooking Show," "Nature Documentary"). You also create a rule: "If a book looks like a 'Nature Documentary,' put it in Bin A."

Now, when a new shipment of books (a new video) arrives, you don't need to read them all carefully. You just quickly glance at each book, check your "Master Index" to see which bin it belongs to, and drop it in. You then take all the books in Bin A and summarize them into one single note. This is incredibly fast because you aren't re-inventing the sorting rules for every new shipment; you are just using the rules you learned once.

How It Works in Detail

Offline Learning (The "Once" Part): The researchers took a huge dataset of videos (LLaVA-Video-178K) and analyzed the visual patterns. They didn't just pick random patterns; they made sure to capture patterns from different times in the video (temporal stratification) and weighted them by how often they appeared. They then used a mathematical technique called "optimal transport" to fit a Global Codebook of 8,192 representative "prototypes" (or codewords). This codebook acts as a map of the visual world.
Online Compression (The "Everywhere" Part): When a new video comes in, the system doesn't train anything new. It simply looks at the video's visual tokens and asks, "Which of my 8,192 prototypes does this look like?" It picks the most frequent prototypes that appear in the video, reassigns all the video's tokens to these prototypes, and then averages them out. This turns thousands of tokens into a much smaller, manageable number (like 256 or 512) without losing the core meaning.

What They Found

The team tested ONCE on four different video understanding benchmarks (MVBench, EgoSchema, Video-MME, and LongVideoBench) using two different AI models (LLaVA-OneVision-7B and Qwen3.5-9B).

Speed: The results were striking. ONCE reduced the amount of work the language model had to do by about 96% in terms of pre-computation (prefill FLOPs). In real-world terms, this made the system run 2.96 times faster on average compared to running without compression.
Accuracy: Usually, when you make a system faster by cutting out data, it gets dumber. But ONCE was surprisingly smart. When the token budget was very tight (only 32, 64, or 128 tokens allowed), ONCE actually got higher scores than other compression methods. It was particularly good at tasks like counting events or recognizing state changes.
Efficiency: The system achieved these speedups without needing to retrain the main AI model. The codebook was learned once and then reused for everything.

What They Ruled Out

The paper explicitly argues against the idea that compression must be done "online" (during the video playback) for every single video. They showed that doing the heavy lifting during inference (while watching) is inefficient and redundant. They also demonstrated that simply picking random tokens or using a fixed set of tokens for every video doesn't work as well as their method, which dynamically selects the right prototypes for the specific video while keeping the grouping rules fixed.

Furthermore, they found that the "Global Codebook" doesn't just store training examples; it learns a reusable structure. Even when tested on videos it had never seen before (a public-video stress test), the codebook performed better than random groupings, suggesting it truly captured the underlying "grammar" of visual patterns.

The Bottom Line

The authors suggest that ONCE offers a new way to think about video AI: instead of making the AI work harder for every new video, we can teach it a universal language of visual patterns once, and then let it speak that language fluently and quickly whenever it needs to. While the method still requires a separate codebook for different AI models (because they "see" things differently), the results suggest that this "learn once, compress everywhere" approach is a powerful way to make video AI faster and more efficient without sacrificing its ability to understand what's happening on screen. The paper concludes that this is a promising step toward making long-video understanding practical for everyday use.

Technical Summary: Rethinking Video Token Compression with a Global Codebook (ONCE)

Problem Statement

Video Large Language Models (Video-LLMs) represent video inputs as dense sequences of visual tokens, where sequence length scales with temporal and spatial resolution. This density creates two primary bottlenecks:

Computational Redundancy: Neighboring frames and patches often contain repeated visual information, leading to unnecessary computation in the subsequent language model processing.
Inefficiency of Existing Methods: Current token compression techniques (pruning and merging) operate online during inference. They generate dense visual features first and then compress them independently for every input video. This approach incurs repeated per-video computation to estimate token importance or similarity from scratch, fails to share compression knowledge across different videos, and often relies on model-specific designs (e.g., attention-based importance) that limit generalizability.

Methodology: ONCE Framework

The authors propose ONCE, a plug-in video token compression framework that shifts the paradigm from online, instance-specific compression to an offline-to-online approach. The core idea is to learn a reusable, frequency-aware global codebook once in a frozen visual feature space and reuse it for lightweight online compression.

1. Offline Phase: Learning Once

ONCE learns a global visual codebook $\mathcal{C} = [c_1, \dots, c_K]^T$ from a large-scale video corpus (e.g., LLaVA-Video-178K) without updating the Video-LLM parameters.

Temporally Stratified Corpus Sketching: Instead of uniform sampling, the training corpus is stratified along the timeline to preserve temporal diversity. Videos are divided into bins, and spherical Lloyd updates partition normalized tokens into local groups. Each group is represented by a weighted sketch token ( $r_u$ ) and a source-token count weight ( $w_u$ ).
Balanced Global Codebook Fitting: To prevent dominant visual patterns from overwhelming the codebook, ONCE employs a balanced assignment strategy. It defines an effective weight $\omega_u$ that incorporates source-token mass and long-tail correction for category frequency.
Optimization: The codebook assignment is formulated as an entropy-regularized optimal transport problem (solved via Sinkhorn iterations). This ensures balanced utilization of codewords while preserving the importance of high-mass representatives. The resulting codebook is fixed after training.

2. Online Phase: Compressing Everywhere

During inference, the frozen codebook serves as a grouping prior for arbitrary input videos.

Active Codeword Selection: For a new video, dense visual tokens are matched to the nearest codewords in the global codebook. The system counts the frequency of assignments ( $n_k$ ) and selects the top- $B$ active codewords ( $K_B$ ) with the highest frequencies. This adapts the global geometry to the specific content of the current video.
Reassignment and Token Pooling: Source tokens are reassigned to the selected active codewords. Each group of tokens is then compressed into a single continuous token via mean pooling of the original source embeddings.
Output: The output is a compact sequence of continuous tokens (not discrete indices), ordered by the frequency of their corresponding active codewords. The sequence length $M$ satisfies $M \le \min(N, B)$ , where $N$ is the original token count and $B$ is the budget.

Key Distinction: Unlike discrete tokenizers (e.g., VQ-VAE), ONCE does not replace features with discrete codes. Instead, it uses the codebook to define grouping structures, while the compressed representation remains a continuous aggregation of the input video's features.

Key Contributions

Offline-to-Online Paradigm: Introduction of a framework that learns a corpus-level grouping prior once and reuses it for unseen videos, eliminating the need for per-video compression strategy optimization.
Global Codebook Learning: A method for learning a balanced global codebook from source-count-weighted sketches, utilizing budget-aware selection and source-token mean pooling.
Plug-and-Play Efficiency: The method is model-agnostic regarding the compression mechanism (as long as the feature space is frozen) and achieves competitive accuracy with significantly reduced inference latency.

Experimental Results

The authors evaluated ONCE on four benchmarks (MVBench, EgoSchema, Video-MME, LongVideoBench) using two backbones (LLaVA-OneVision-7B and Qwen3.5-9B).

Accuracy-Efficiency Trade-off:
- Under tight token budgets ( $B \in \{32, 64, 128\}$ ), ONCE achieved the highest accuracy among all compared compression baselines (including ForestPrune and FastVID) across all benchmarks.
- At larger budgets ( $B=512, 1024$ ), ONCE remained competitive, though some input-adaptive baselines occasionally surpassed it in specific scenarios.
Inference Latency:
- ONCE achieved the lowest inference latency among compared methods.
- At $B=512$ , ONCE reduced visual-token prefill FLOPs by approximately 96% and achieved an end-to-end speedup of (2.96 ± 1.21)× (mean ± standard deviation) compared to dense inference.
- Memory usage saw modest reductions (~15%), primarily due to reduced sequence length, though model weights remained resident.
Generalization: The approach successfully transferred to the Qwen3.5-9B backbone using a single codebook trained on the same corpus, demonstrating robustness across different model architectures.

Significance and Claims

The paper claims that ONCE addresses the fundamental inefficiency of current Video-LLM compression, which treats compression as an additional, repetitive inference-stage computation. By moving the heavy lifting of grouping logic to an offline phase, ONCE:

Decouples compression knowledge from specific input videos, allowing for knowledge reuse.
Reduces the computational overhead of compression decisions during inference.
Maintains high task performance, particularly when token budgets are constrained, by leveraging corpus-level statistics to guide token aggregation.

The authors note that while the method significantly reduces prefill costs, the end-to-end speedup is bounded by the visual encoding and decoding stages, which remain unchanged. They also acknowledge that the fixed codebook is specific to the visual feature space of the target backbone and that the method currently does not reduce the cost of the initial visual encoder.