🤖 AI

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

This paper introduces TOPS, a training-free and model-agnostic visual token pruning method that constructs optimal preservation sets based on task relevance, information coverage, and semantic diversity to significantly improve MLLM inference efficiency while maintaining or even enhancing performance.

Original authors: Tinghao Wang, Yichen Guo, Rui Huang, Zheng Lu, Qizhe Zhang, Chenxi Li, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du, Guangyan Gan, Wenya Wang, Lin William Cong, Shanghang Zhang

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tinghao Wang, Yichen Guo, Rui Huang, Zheng Lu, Qizhe Zhang, Chenxi Li, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du, Guangyan Gan, Wenya Wang, Lin William Cong, Shanghang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of books (the visual tokens) that a super-smart robot (the Multimodal Large Language Model) needs to read to answer a question. The problem is, the library is so huge that the robot gets overwhelmed, takes forever to read, and uses up all its energy just flipping pages.

For a long time, people tried to help the robot by telling it, "Just skip the boring pages!" But the old methods were a bit clumsy:

The "Attention" method was like a robot that only looked at pages with the biggest, boldest fonts. It often skipped important details just because they were written in small text, or it kept reading the same paragraph five times because it looked similar.
The "Diversity" method was like a robot trying to pick pages that looked different from each other. It might pick a page about a cat and a page about a car, but it might accidentally throw away the page that actually answers the user's specific question.

Enter TOPS: The Smart Librarian

The paper introduces a new method called TOPS (Token Optimal Preservation Sets). Instead of just guessing which pages to keep, TOPS acts like a highly organized, first-principles librarian who asks three specific questions before deciding what stays on the shelf:

"Is this page relevant to the question?" (Task Relevance)
- Analogy: If the user asks, "What color is the car?", the librarian immediately knows to keep the page with the car and throw away the page about the sky.
"Does this page cover new ground?" (Information Coverage)
- Analogy: If we already have a page describing the car's red paint, we don't need a second page that says the exact same thing. We need a page that tells us something new, like the car's license plate number.
"Is this page unique from the others we picked?" (Semantic Diversity)
- Analogy: We don't want a stack of five pages that all say "The car is red." We want one page that says "Red," one that says "Fast," and one that says "Old." This ensures we get a full picture without repetition.

How It Works in Real Life

The paper shows that TOPS doesn't need to be taught (it's "training-free"). It can be plugged into different robot brains (like LLaVA, Qwen, or InternVL) and works instantly.

The "Two-Stage" Cleanup: Imagine the robot is reading a long video.
- Stage 1: TOPS does a quick sweep, throwing away the obvious junk (like blank frames or blurry shots) before the robot even starts thinking deeply.
- Stage 2: As the robot reads, TOPS keeps a close eye on the conversation. If the robot starts talking about a specific detail, TOPS makes sure the most relevant visual pages are still there, refining the selection to be perfect for that specific question.

The Results: Less is More

The paper claims that by using this smart, three-question approach, TOPS can throw away up to 90% of the visual pages (tokens) and the robot still answers questions just as well as if it had read everything.

The Magic Stat: On one specific model (LLaVA-NeXT), TOPS removed 77.8% of the visual data but actually improved the robot's performance slightly (100.6%).
Why? The paper suggests that by forcing the robot to ignore the "fluff" and redundant pages, it actually stops the robot from getting confused or making up facts (hallucinations). It's like cleaning a cluttered desk; sometimes, having fewer papers helps you find the right one faster and think more clearly.

In Summary

TOPS is a new way to make AI models faster and smarter by being a stricter, smarter editor. Instead of just picking the "loudest" or "most different" visual clues, it builds a perfect, compact set of clues that are relevant to the question, cover all the necessary information, and don't repeat each other. The result is a robot that thinks faster, uses less memory, and gives better answers.

Technical Summary: TOPS - First-Principles Visual Token Pruning

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant progress in reasoning and understanding but suffer from high computational and memory overhead. This inefficiency stems from the large number of visual tokens generated during tokenization, which are processed through all transformer layers. Since self-attention mechanisms scale quadratically with sequence length, high-resolution images and video inputs create substantial bottlenecks.

Existing visual token pruning methods attempt to mitigate this by removing redundant tokens but rely on heuristic scoring schemes rather than a principled objective. Current approaches generally fall into three categories, each with specific limitations:

Attention-based methods: Retain tokens with high attention scores but often preserve semantically redundant tokens.
Diversity-based methods: Encourage semantic dispersion but are often agnostic to user instructions, potentially discarding task-critical evidence.
Combined methods: While they integrate multiple criteria, they lack a fundamental formulation of the intrinsic objective of token pruning, treating it merely as a ranking problem without theoretical justification for why specific criteria are sufficient.

2. Methodology: First-Principles Formulation

The authors propose TOPS (Token Optimal Preservation Sets), a training-free and model-agnostic pruning module derived from a first-principles perspective using information theory.

2.1 Theoretical Foundation

The core objective of token pruning is defined as finding a subset $S \subseteq V$ (where $V$ is the full set of visual tokens) that maximizes the mutual information between the subset and the joint distribution of the visual tokens and the text query $Q$ :
$\max_{S \subseteq V, |S| \le K} I(S; V, Q)$

By applying the chain rule of mutual information, the authors decompose this objective into three fundamental principles:

Task Relevance ( $I(S; Q)$ ): The subset must be informative regarding the user's query.
Information Coverage ( $I(S; V | Q)$ ): The subset must preserve sufficient information about the original visual set $V$ given the query, minimizing the conditional entropy $H(V | S, Q)$ .
Semantic Diversity: Derived from the upper bound of conditional mutual information, this principle dictates that for a fixed budget, tokens must be semantically distinct from already selected tokens to maximize information gain.

2.2 The TOPS Algorithm

TOPS constructs the optimal token subset greedily at each pruning point. The process involves:

Scoring: For each candidate token $i$ $i$ , the algorithm computes:
- Task Relevance ( $r_i$ ): Estimated via text-rater attention (or CLS attention in early stages).
- Information Coverage ( $c_i$ ): Calculated based on the maximum similarity between the candidate and the remaining unselected tokens, adjusted by the similarity to already selected tokens.
- Semantic Diversity ( $d_i$ ): Measured as $1 - \max_{j \in S} \text{sim}(h_i, h_j)$ , ensuring the new token is distinct from the current set.
Selection: The algorithm combines these scores into an information preservation score:
$\tilde{g}_{info_i} = \tilde{r}_i + \lambda \tilde{c}_i(U) + \alpha \tilde{d}_i(S)$
where $\alpha$ and $\lambda$ are balance factors. The token maximizing this score is added to the subset $S$ .
Multi-Stage Pipeline: To handle fine-grained reduction, TOPS is implemented in two stages:
- Stage I: Applied after the multimodal projector but before the LLM. It uses CLS attention for relevance to remove coarse redundancy.
- Stage II: Applied inside the LLM at designated layers. It leverages the LLM's text-to-visual attention for text-aware refinement.

3. Key Contributions

First-Principles Perspective: The paper redefines token pruning not as a heuristic ranking problem but as an optimal subset selection problem grounded in information theory.
Three Fundamental Principles: The identification and formalization of Task Relevance, Information Coverage, and Semantic Diversity as the necessary conditions for effective pruning.
TOPS Module: A plug-and-play, training-free, and model-agnostic pruning algorithm that dynamically constructs Token Optimal Preservation Sets.
Extensive Validation: Comprehensive experiments across 7 MLLM backbones (including LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, InternVL3, and LLaVA-Video) and 14 diverse benchmarks.

4. Experimental Results

The authors report that TOPS consistently outperforms state-of-the-art methods (e.g., FastV, VisionZip, DivPrune, SCOPE) across various compression ratios and model architectures.

LLaVA-NeXT: On the 7B model, TOPS removes 77.8% of visual tokens while preserving 100.0% of the original performance. At higher compression (88.9% and 94.4% reduction), it retains 99.1% and 96.4% performance, respectively.
LLaVA-1.5: At 77.8% compression, TOPS retains 98.3% performance, outperforming the next best method (SCOPE) by 0.5%. Even at extreme compression (94.4% reduction, 32 tokens remaining), it maintains 94.6% performance.
Advanced Architectures: On Qwen2.5-VL-7B and InternVL3-8B, TOPS achieves the highest accuracy across all tested pruning ratios, demonstrating architecture-agnostic effectiveness.
Video Understanding: On LLaVA-Video with 64 frames, TOPS retains 98.4% performance at 62.1% token reduction and 92.3% at 90.5% reduction, significantly outperforming baselines like FastV.
Efficiency: TOPS achieves nearly a 10x reduction in FLOPs and reduces GPU memory usage by 1.9GB compared to the baseline, with significant improvements in prefill latency.
Hallucination Mitigation: Notably, on the HallusionBench, TOPS shows stronger resistance to hallucination, suggesting that removing redundant tokens can sometimes improve reasoning fidelity.

5. Significance and Claims

The paper claims that effective token pruning should not rely solely on heuristic importance scores but must jointly preserve task-relevant, representative, and non-redundant visual information. By formulating pruning as an information-theoretic optimization problem, TOPS provides a principled foundation for efficient multimodal inference.

The authors suggest that their work offers guidance for future lightweight MLLM design, noting that aggressive pruning can sometimes mitigate hallucinations. However, they acknowledge limitations, including the computational overhead of the greedy construction loop ($O(KN)$), reliance on attention mechanisms for relevance signals, and the use of fixed hyperparameters across layers and tasks. They also note that the framework has not yet been systematically evaluated in highly specialized domains like medical imaging or dense captioning.