🤖 AI

IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMs

IDPruner is a novel visual token pruning framework for Multimodal Large Language Models that leverages the Maximal Marginal Relevance algorithm to achieve a Pareto-optimal balance between token importance and diversity without requiring attention maps, thereby enabling efficient one-shot pruning while maintaining state-of-the-art performance across diverse architectures and tasks.

Original authors: Yifan Tan, Yifu Sun, Shirui Huang, Hong Liu, Guanghua Yu, Jianchen Zhu, Yangdong Deng

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifan Tan, Yifu Sun, Shirui Huang, Hong Liu, Guanghua Yu, Jianchen Zhu, Yangdong Deng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, high-resolution photograph of a busy city street. You want to show this picture to a very smart AI assistant so it can answer questions like, "What is the person in the red coat doing?" or "Is there a dog nearby?"

The problem is that the computer sees this image not as a picture, but as thousands of tiny puzzle pieces (called "tokens"). Processing all these pieces takes a massive amount of time and energy, like trying to read every single word in a 500-page book just to find one specific sentence.

Visual Token Pruning is the art of throwing away the unnecessary puzzle pieces before the AI reads them, keeping only the most important ones. But here's the tricky part: How do you decide what to keep?

The Two Old Ways (and Why They Failed)

Before this paper, researchers tried two main strategies, both of which had flaws:

The "Hype Hunter" (Importance-Based):
- How it works: It looks for the "loudest" or most obvious parts of the image. If there's a bright red car, it keeps that. If there's a person's face, it keeps that.
- The Flaw: It gets so obsessed with the main subject that it throws away the background. If the question is "What kind of trees are in the background?", this AI fails because it deleted the trees to focus on the car. It's like reading a news headline but ignoring the whole article.
The "Scatter Plot" (Diversity-Based):
- How it works: It tries to pick pieces that are all different from each other to make sure it covers the whole image. It spreads its picks out like sprinkles on a donut.
- The Flaw: It might keep a piece of the sky, a piece of the sidewalk, and a piece of a random cloud, but it might miss the actual important object (the red car) because that car looks too similar to other parts of the image. It's like sampling a little bit of everything but missing the main flavor.

The New Solution: IDPruner (The "Smart Curator")

The authors of this paper, IDPruner, realized that the best approach isn't choosing one or the other, but finding the perfect balance. They used a clever mathematical trick called Maximal Marginal Relevance (MMR).

Here is a simple analogy to explain how IDPruner works:

Imagine you are a Curator for a museum, and you have a huge collection of 1,000 paintings, but you can only hang 10 on the wall.

The Old "Hype Hunter" would just hang the 10 most famous paintings. But if they are all landscapes, your museum lacks variety.
The Old "Scatter Plot" would pick 10 paintings that are all totally different (a portrait, a landscape, a still life, an abstract piece). But if the 10 most famous paintings were all landscapes, you might miss the best art entirely.

IDPruner's Strategy:
It uses a two-step rule for every single painting it considers:

Is it famous? (Importance: Does it have high value?)
Is it unique? (Diversity: Does it look too much like the ones I've already picked?)

It calculates a score: "Fame minus Redundancy."

If a painting is super famous but looks exactly like one you already picked, the "Redundancy" penalty knocks its score down. You skip it.
If a painting is a bit less famous but offers a totally new perspective, its score goes up. You pick it.

This way, your final 10 paintings are a mix of the most important subjects and a diverse range of scenes. You get the best of both worlds.

Why This Matters (The "Magic" Parts)

The paper highlights three cool things about IDPruner:

It's a One-Shot Deal: Some other methods try to prune the image slowly, step-by-step, which is slow. IDPruner makes all its decisions at once, very quickly. It's like a chef chopping all the vegetables in one go rather than one by one.
It Plays Nice with Fast Tech: Modern computers use a special speed-boosting tool called FlashAttention. Many pruning methods break this tool. IDPruner was designed to work perfectly with it, meaning it's fast and efficient.
It Works Everywhere: The authors tested this on many different AI models (like Qwen and LLaVA) and on many different tasks (reading charts, spotting objects, understanding videos). In almost every case, IDPruner kept the AI smart even when they threw away 75% to 90% of the image data.

The Bottom Line

IDPruner is like a super-smart editor for visual data. Instead of blindly cutting out the "boring" parts or randomly keeping "different" parts, it uses a balanced formula to ensure the AI sees the main characters of the story while still understanding the setting.

The result? AI models that are much faster and cheaper to run, but still just as smart as the slow, heavy versions. It's the difference between carrying a heavy backpack full of rocks versus carrying a backpack with only the essential tools you need for the journey.

1. Problem Statement

Multimodal Large Language Models (MLLMs) face significant computational bottlenecks due to the massive volume of visual tokens generated when processing high-resolution images or videos. For instance, a single 1080p image can generate nearly 2,700 tokens.

Existing Limitations: Current pruning strategies generally fall into two categories:
- Importance-based: Selects salient tokens (e.g., foreground objects) but often sacrifices global context and background information essential for reasoning.
- Diversity-based: Maximizes semantic coverage to reduce redundancy but risks retaining task-irrelevant noise while missing fine-grained details.
- Hybrid approaches: Recent attempts to combine both criteria often rely on intuitive, heuristic integration (e.g., two-stage filtering) without a principled framework, leading to suboptimal performance.
Goal: To develop a systematic framework that optimally balances token importance (relevance to the task) and semantic diversity (coverage of the visual scene) without requiring attention maps (to ensure compatibility with FlashAttention) or progressive pruning (to support efficient one-shot deployment).

2. Methodology: IDPruner

The authors propose IDPruner, a novel pruning strategy that treats visual token selection as a re-ranking problem in information retrieval, utilizing the Maximal Marginal Relevance (MMR) algorithm.

A. Theoretical Analysis & Trade-off Characterization

Before proposing the solution, the authors conducted a systematic analysis using the Hopkins Statistic to quantify diversity (clustering) and an Importance Retention Ratio to quantify relevance.

Findings: Single-objective methods (Greedy Importance or Greedy Diversity) occupy sub-optimal extremes. Naive hybrid methods (e.g., selecting top-k importance then applying diversity) fail to effectively balance the two, often resulting in a Pareto frontier inferior to the theoretical optimum.

B. The MMR Mechanism

IDPruner adapts the MMR algorithm to iteratively select tokens that maximize a joint objective function:
$v^* = \arg \max_{v_i \in V \setminus S} \left[ \lambda \cdot \text{Imp}(v_i) - (1 - \lambda) \cdot \max_{v_j \in S} \text{Sim}(v_i, v_j) \right]$
Where:

$V$ : Set of all visual tokens.
$S$ : Currently selected subset.
$\text{Imp}(v_i)$ : Normalized importance score (derived from a trainable scorer like VisionSelector).
$\text{Sim}(v_i, v_j)$ : Cosine similarity between tokens (measuring semantic redundancy).
$\lambda$ : Hyperparameter balancing importance and diversity (set to 0.5 in experiments).

Key Operational Features:

One-Shot Pruning: Unlike progressive methods that dynamically change sequence lengths, IDPruner performs pruning at an early stage, making it compatible with inference engines like vLLM.
Attention-Map Free: It does not rely on attention maps from the LLM decoder, ensuring full compatibility with FlashAttention for maximum efficiency.
Efficient Implementation: Instead of recomputing all pairwise similarities at every step, it maintains a vector tracking the maximum similarity for each candidate, reducing complexity from $O(K^2N)$ to $O(KN)$.

3. Key Contributions

Systematic Analysis: Provided the first rigorous theoretical framework characterizing the trade-off between token importance and semantic diversity, demonstrating that heuristic integration is insufficient.
IDPruner Framework: Introduced a method using MMR to achieve a Pareto-optimal balance between importance and diversity, explicitly penalizing redundant tokens while prioritizing salient ones.
State-of-the-Art Performance: Demonstrated superior performance across diverse model architectures (Qwen2.5-VL, LLaVA-1.5, LLaVA-OneVision) and tasks (static images, video understanding).
Practical Efficiency: Achieved high performance while maintaining compatibility with FlashAttention and enabling one-shot pruning, offering a practical solution for real-world deployment.

4. Experimental Results

The authors evaluated IDPruner on multiple benchmarks (MME, MMBench, POPE, OCRBench, VideoMME, etc.) across different models.

Qwen2.5-VL-7B-Instruct:
- 75% Pruning (25% retention): IDPruner retained 95.18% of the baseline performance, outperforming all baselines (e.g., VisionSelector at 94.22%, SCOPE at 92.51%).
- 90% Pruning (10% retention): Even under extreme compression, it retained 86.40% of baseline performance, significantly outperforming competitors.
LLaVA-1.5-7B: IDPruner achieved the best results across all pruning ratios (128, 64, and 32 tokens), showing robustness where other methods (like VisionSelector) degraded significantly.
Video Understanding: On video benchmarks (Vinoground, VideoMME), IDPruner achieved the highest average score (87.13%), effectively handling temporal redundancy where importance-only methods failed.
Efficiency: IDPruner demonstrated the lowest prefill time (1337.76 ms) and end-to-end latency among hybrid strategies due to FlashAttention compatibility.

5. Significance

Theoretical Advancement: The paper moves beyond heuristic "mix-and-match" strategies by providing a mathematically grounded approach (MMR) to multi-objective token pruning.
Generalization: The method proves to be architecture-agnostic, performing consistently well across different MLLM families (Qwen, LLaVA, InternVL), suggesting that the importance-diversity trade-off is a fundamental property of visual token processing.
Deployment Readiness: By eliminating the need for attention maps and supporting one-shot pruning, IDPruner solves critical engineering hurdles for deploying efficient MLLMs in production environments, particularly those utilizing FlashAttention.

In conclusion, IDPruner establishes a new standard for visual token pruning by rigorously harmonizing the need to keep important information with the need to maintain a diverse, non-redundant representation of the visual input.