💻 computer science

A Survey of Token Compression for Efficient Multimodal Large Language Models

This paper presents the first systematic survey of token compression techniques for efficient multimodal large language models, categorizing existing methods by both their target modalities (image, video, and audio) and their underlying mechanisms to consolidate current progress and guide future research.

Original authors: Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, Huan Wang

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, Huan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart assistant (a Multimodal Large Language Model, or MLLM) who can read text, look at pictures, watch videos, and listen to audio. This assistant is incredibly talented, but it has a major problem: it gets overwhelmed when you give it too much information at once.

Think of the information the assistant receives as a stream of "tokens" (little chunks of data).

A text prompt is like a short letter.
A high-resolution photo is like a letter that has been blown up into a giant, detailed mural.
A video is like a library of thousands of those murals, changing every second.
Audio is like a continuous, high-speed stream of sound waves.

When you feed a 90-minute movie to this assistant, it doesn't just see "one movie." It sees 54 million tokens. That's like trying to read a library of books in a single breath. The assistant's brain (its "self-attention" mechanism) has to compare every single token with every other token. The math for this gets so heavy, so fast, that the computer runs out of memory or takes forever to answer.

The Solution: Token Compression
This paper is a massive guidebook (a survey) on how to teach this assistant to be more efficient without losing its smarts. The authors call this Token Compression.

Think of token compression like packing a suitcase for a trip.

The Problem: You have a suitcase full of clothes, but 80% of them are duplicates (like 50 identical white t-shirts) or things you don't need (like a heavy winter coat for a beach trip).
The Goal: You want to fit everything important into a smaller bag so you can travel faster, without leaving behind the things you actually need.

The paper organizes all the current methods for "packing" this data into two main ways of looking at the problem: What kind of data is it? and How do we pack it?

1. Packing by Data Type (The "What")

Different types of data have different kinds of "clutter" (redundancy).

Images (The Static Photo):
- The Clutter: A photo of a blue sky has millions of blue pixels that are all exactly the same.
- The Fix: Instead of sending every single blue pixel, the computer groups them together. It says, "This whole area is just blue sky," and sends one token to represent the whole patch.
Video (The Moving Picture):
- The Clutter: In a video of a person talking, the background (a wall or a tree) stays exactly the same for 10 seconds while the person moves slightly.
- The Fix: The computer realizes, "We don't need to send the background 30 times a second." It keeps the background once and only sends updates for the moving parts. It's like sending a "change log" instead of re-sending the whole scene every frame.
Audio (The Sound Wave):
- The Clutter: A recording of a voice often has long pauses, silence, or background hum that doesn't add meaning.
- The Fix: The computer cuts out the silence and merges similar sounds, keeping only the parts where the voice is actually speaking or the music is changing.

2. Packing by Method (The "How")

The paper groups the techniques used to do this packing into four main strategies:

The "Shrink Ray" (Transformation-based):
Imagine taking a high-resolution photo and simply shrinking it down. You lose some detail, but you keep the general shape and colors. This is done by mathematically squashing the data (like pooling or averaging) to make the token list shorter.
The "Grouping Game" (Similarity-based):
Imagine you have a pile of 1,000 red Lego bricks. Instead of listing all 1,000, you say, "Here is one red brick, and there are 999 more just like it." The computer finds tokens that look or sound very similar and merges them into a single "representative" token.
The "Spotlight" (Attention-based):
Imagine a teacher looking at a classroom. The teacher only cares about the students who are raising their hands (the important tokens) and ignores the ones sleeping in the back. The computer looks at its own "attention scores" (how much it cares about each piece of data) and throws away the tokens it's ignoring anyway.
The "Question Guide" (Query-based):
Imagine you are looking for a specific needle in a haystack. Instead of looking at every piece of hay, you ask, "Where is the needle?" The computer uses your question (the query) to filter out everything that doesn't match what you are asking for, keeping only the relevant tokens.

Why This Matters

The authors explain that this isn't just about making computers run faster. It's about making them usable.

Without compression, a 90-minute movie is impossible for current models to process in real-time.
With compression, the model can "watch" the movie, understand the plot, and answer questions about it, all while using a fraction of the memory.

The Catch (Challenges)

The paper also warns that this isn't magic. If you pack the suitcase too tight:

You might lose the details: If you compress a photo too much, you might miss a tiny but important sign in the background.
It breaks the flow: In a video, if you merge too many frames, the movement might look jerky or confusing.
It's hard to fit in: Some of these "packing" tricks are hard to use with the fastest computer chips available today because they require the computer to stop and calculate things differently.

In Summary:
This paper is a map for researchers. It says, "We have a problem: our AI is drowning in too much data. Here are all the different ways we are trying to teach it to filter, group, and shrink that data so it can actually function in the real world." It organizes these methods by whether they are looking at pictures, videos, or sound, and by the specific math tricks they use to do the job.

Technical Summary: A Survey of Token Compression for Efficient Multimodal Large Language Models

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved remarkable performance by processing complex contexts, including high-resolution images, extended video sequences, and long audio inputs. However, this capability introduces a significant computational bottleneck: the quadratic complexity ( $O(N^2)$ ) of self-attention mechanisms relative to the number of input tokens ( $N$ ).

Unlike text, where token counts are manageable, visual and auditory data generate sequences orders of magnitude longer. For instance, a 90-minute video can produce approximately 54 million tokens, while a single high-resolution image or a 2-hour audio file generates tens of thousands to millions of tokens. This disparity causes severe inference latency and memory consumption, hindering the practical deployment of MLLMs. Furthermore, multimodal data contains substantial redundancy (spatial in images, spatio-temporal in video, and temporal/spectral in audio), with studies indicating that over 50% of tokens in typical MLLM sequences receive minimal attention during inference.

2. Methodology and Taxonomy

This paper presents the first systematic survey and synthesis of token compression techniques specifically for MLLMs. The authors define token compression strictly as methods that explicitly reduce the number of tokens passed to subsequent layers ( $M < N$ ), distinguishing it from input-level sampling (e.g., frame extraction) and attention sparsity mechanisms that maintain sequence length but reduce computational pathways.

The survey categorizes existing approaches through two primary lenses:

A. Modality-Centric Categorization

Recognizing that redundancy patterns differ across modalities, methods are grouped by their primary data focus:

Image-Centric Compression: Addresses spatial redundancy in static visual data. Neighboring patches often represent similar textures or colors.
Video-Centric Compression: Targets spatio-temporal redundancy in dynamic sequences, where consecutive frames share extensive background elements and limited motion.
Audio-Centric Compression: Handles temporal and spectral redundancy in acoustic signals, where salient information is often concentrated in sparse segments amidst silence or noise.

B. Mechanism-Centric Categorization

Beyond modality, the paper dissects methods based on their underlying algorithmic principles, which often transcend specific data types:

Transformation-Based: Directly transform cross-modality information to compress tokens by altering scale or representation (e.g., Pixel Unshuffle, Spatial/Temporal Pooling, Convolution). These methods often preserve structural representation but offer limited flexibility in compression ratios.
Similarity-Based: Identify and merge tokens based on distance or similarity in an implicit space (e.g., K-Nearest Neighbors, clustering). While effective, they risk losing fine-grained spatial or temporal information if tokens are over-generalized.
Attention-Based: Exploit the sparsity of attention mechanisms to prune tokens with low attention scores. These can be applied within the Encoder (vision/audio encoders) or the Decoder (LLM backbone). While powerful, they face challenges in deployment due to incompatibility with optimized acceleration libraries (e.g., FlashAttention) that compute attention implicitly.
Query-Based: Use external queries (prompts) to guide compression, either through Token Distillation (learning to represent visual/audio info with fewer tokens) or Cross-Modal Selection (filtering tokens based on relevance to text). These are highly task-relevant but may struggle with multi-turn conversations where queries change.

3. Key Contributions

First Structured Survey: This work provides the first comprehensive taxonomy of token compression specifically for MLLMs, bridging the gap between text-centric prompt compression and vision/audio-specific techniques.
Unified Framework: It demonstrates that despite modality-specific redundancies, fundamental algorithmic principles (importance identification, redundancy quantification, and token merging/pruning) are consistent across visual, temporal, and auditory domains.
Critical Analysis of Trade-offs: The paper details the pros and cons of each mechanism, such as the parameter-free nature of pooling versus the learnable abstraction of convolution, and the efficiency of attention-based pruning versus its incompatibility with current hardware accelerators.
Identification of Challenges: The survey highlights critical hurdles, including performance degradation at high compression ratios (especially in larger models), the loss of fine-grained details required for tasks like OCR or temporal grounding, and the difficulty of integrating dynamic pruning into multi-turn conversational systems.

4. Results and Observations

The paper synthesizes findings from numerous state-of-the-art models and methods (e.g., InternVL, Qwen2-VL, FastV, ToMe, HoliTom, SpeechPrune):

Efficiency vs. Performance: Significant token reduction is achievable with minimal performance loss. For instance, retaining only 10-25% of visual tokens often maintains competitive performance on benchmarks like VQA2 and VideoMME. However, aggressive compression leads to sharp performance declines, particularly in models with higher representational capabilities (e.g., Qwen2.5-VL, LLaVA-OV-72B).
Task Sensitivity: Compression strategies are not universally applicable. Tasks requiring high information density (e.g., Optical Character Recognition) or precise temporal reasoning suffer more from token merging or pruning than general visual question answering.
Deployment Gaps: Many attention-based methods, while theoretically efficient, are difficult to deploy in real-world scenarios due to their reliance on explicit attention scores, which are inaccessible in fused, optimized inference pipelines.

5. Significance and Future Directions

The paper claims that token compression is not merely a post-optimization technique but a critical enabler for the scalability of MLLMs. Its significance lies in:

One-Shot Long-Context Understanding: By condensing information into dense tokens, MLLMs can process entire code repositories or long videos in a single pass without requiring architectural overhauls like RingAttention.
Enhanced Modality Alignment: Compression methods, particularly query-based distillation, can improve the alignment between multimodal inputs and language representations by filtering out low-density or irrelevant information.
Orthogonal Synergy: The paper notes that token compression (reducing sequence length $N$ ) is structurally orthogonal to weight compression (reducing model dimension $D$ ), suggesting that combining these approaches (e.g., quantization + token pruning) can yield compounded efficiency gains.

Future Directions Identified:

Joint Multimodal Compression: Moving beyond unimodal strategies to frameworks that exploit cross-modal synergy (e.g., using audio cues to prune video tokens) for Omni-LLMs.
Architecture Design: Shifting from remedial compression to designing models that intrinsically account for data redundancy during feature abstraction.
Evaluation Standards: The paper calls for new benchmarks specifically designed to evaluate token compression efficacy, moving beyond general VQA metrics to tasks requiring fine-grained temporal grounding and high-fidelity detail retention.

In conclusion, this survey establishes a foundational resource for researchers and developers, consolidating current progress and outlining the path toward efficient, scalable, and robust Multimodal Large Language Models.