🤖 AI

Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

This paper proposes Entropy-Aware Dense Pruning (EADP), a novel framework that enhances Visual Language Model efficiency by filtering textual noise via statistical entropy and optimizing visual token selection through submodular maximization to preserve fine-grained cues under dense instructions.

Original authors: Xuehui Wang, Xuankun Yang, Wei Shen

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuehui Wang, Xuankun Yang, Wei Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart but slightly overwhelmed assistant (a Vision-Language Model) who is trying to answer a question about a picture. The picture is made up of thousands of tiny puzzle pieces (tokens). The assistant looks at every single piece to figure out the answer.

The problem? Looking at every piece takes forever, and the assistant gets tired. So, we usually try to throw away the "boring" pieces and keep only the important ones. This is called Token Pruning.

However, the authors of this paper found that the current ways of doing this are broken in two specific, funny ways. They built a new system called EADP (Entropy-Aware Dense Pruning) to fix it.

Here is how they fixed it, using simple analogies:

The Two Big Problems

1. The "Static Noise" Problem (Textual Noise)
Imagine you are trying to find a specific person in a crowded room based on a description like, "Is there a dog in the image?"

Old Method: The assistant listens to the whole sentence as one big block. It hears "dog," but it also hears "is," "there," "a," and "in." Because it treats the whole sentence as one lump, the boring words ("is," "a") create a lot of background static. This static drowns out the important word ("dog"), making the assistant look at the wrong parts of the room (like the empty floor) instead of the dog.
The Paper's Fix: They realized that some words (like "is" or punctuation) are "noisy" because they don't point to anything specific in the picture. They used a math trick called Entropy (which measures chaos or randomness) to detect these noisy words. If a word's attention is scattered all over the room like static, they filter it out. If a word is focused (like "dog"), they keep it. This cleans up the instructions so the assistant knows exactly what to look for.

2. The "One-Point Focus" Problem (Redundancy)
Imagine the assistant finally knows to look for the dog.

Old Method: It uses a "Top-K" rule. It finds the one piece of the puzzle that looks most like a dog's nose and keeps that. Then it finds the piece that looks like the nose again, and keeps that too. It ends up with a pile of 10 puzzle pieces that are all just the dog's nose, and it throws away the ears, the tail, and the body. The assistant has a great nose, but no idea what the rest of the dog looks like.
The Paper's Fix: They realized we need a balanced view, not just the "best" view. They changed the selection process to a game called "Facility Location."
- The Analogy: Imagine you are placing fire stations in a city. You don't just put all 10 stations in the one neighborhood with the most fires (that's the "Top-K" mistake). Instead, you place them so that every part of the city is close to a station.
- EADP does this with the picture. It picks tokens that cover the whole dog (nose, ears, tail) without wasting space on duplicates. It ensures the final, smaller set of tokens tells the whole story, not just a tiny detail.

How EADP Works (The Step-by-Step)

Clean the Instructions: It scans the text prompt, identifies the "noisy" words (like "is" or "?"), and removes them. It keeps the "loud" words (like "oven" or "elephant") that actually point to things in the image.
Make a Map: It creates a heat map of the image showing where the important things are, but it smooths it out so it doesn't just highlight one tiny pixel.
The Smart Selection: Instead of just picking the hottest spots on the map, it plays the "Fire Station" game. It picks a small group of tokens that are spread out enough to cover the whole image, ensuring no important part is left behind.

The Results

The authors tested this on many different AI models and difficult questions (like finding tiny text in images or answering tricky questions about videos).

Speed: By throwing away 80-90% of the useless picture pieces, the AI runs 2 to 5 times faster.
Smarts: Unlike other methods that get confused when the picture is complex or the question is tricky, EADP keeps the AI smart. It actually got better scores than the original, un-pruned AI in some cases because it stopped the AI from getting distracted by noise.

In short: EADP is like a smart editor for AI vision. It cuts out the boring words in the instructions and ensures the AI keeps a diverse, complete set of picture pieces, making it faster to run without losing its ability to see the details.

Technical Summary: Entropy-Aware Dense Visual Token Pruning (EADP)

Problem Statement

Vision-Language Models (VLMs) face significant computational bottlenecks due to the quadratic self-attention costs associated with processing dense grids of visual patch tokens. While visual token pruning is a standard strategy to mitigate this by retaining only a subset of informative tokens, existing methods often fail under strict deployment constraints. The authors identify two critical failure modes in current paradigms:

Textual Noise Dispersion: Standard dense guidance strategies, which compute cross-modal similarity between every text token and visual token, are corrupted by functional words and punctuation. These elements produce broadly dispersed, near-uniform similarity responses that create an indiscriminate "noise floor," drowning out the localized peaks of meaningful semantic entities.
Feature Fragmentation and Redundancy: Standard Top-K selection strategies tend to over-concentrate on a few highly discriminative local regions (e.g., repeatedly sampling the head of an object) while ignoring integral semantic parts. This leads to feature fragmentation and a waste of the limited token budget on redundant local maxima, failing to provide a holistic visual representation.

Methodology: Entropy-Aware Dense Pruning (EADP)

EADP reframes visual token pruning as a structured compression problem consisting of two main stages: robust instruction relevance scoring and non-redundant token selection.

1. Dispersion-Aware Dense Scoring

To address textual noise, EADP introduces an entropy-guided denoising mechanism for instruction relevance scoring:

Dense Similarity Calculation: It computes cross-modal cosine similarities between non-EOS CLIP text tokens and projected visual tokens.
Entropy-Guided Filtering: The spatial similarity distribution of each text token is converted into a probability distribution. The Information Entropy of this distribution is calculated to quantify dispersion. High-entropy tokens (indicating dispersed attention, typical of functional words) are filtered out, while low-entropy tokens (indicating concentrated attention on specific entities) are retained.
Weighted Aggregation: Retained text tokens are weighted based on their entropy (lower entropy receives higher weight) and aggregated to form a dense guidance score ( $S_D$ ).
Score Fusion: This dense score is fused with a global guidance score ( $S_G$ ) derived from the CLIP EOS token to balance fine-grained detail with macroscopic semantic context, yielding a robust instruction relevance score ( $S_I$ ).

2. Spatial Smoothing and Submodular Selection

To address feature fragmentation and redundancy, EADP replaces naive Top-K selection with a structured approach:

Spatial Continuity & Polarization: The relevance score map is reshaped into 2D and smoothed using a Gaussian kernel to propagate activations to adjacent structural parts. A non-linear score polarization step (exponential sharpening) is then applied to re-amplify salient regions and suppress smoothed background noise.
Submodular Maximization: Token selection is formulated as a Facility Location Submodular Maximization problem. The objective function maximizes the sum of weighted similarities, ensuring that every token in the original image is well-represented by its most similar counterpart in the selected subset.
Greedy Optimization: This objective is submodular, allowing for an efficient greedy algorithm with a theoretical $(1 - 1/e)$ approximation guarantee. This ensures the selected subset covers all semantic parts of the image while penalizing local redundancy.

Key Contributions

Identification of Pruning Bottlenecks: The paper systematically analyzes and identifies the dispersion of textual noise in dense guidance and the feature fragmentation inherent in standard Top-K selection as primary causes of performance degradation in existing pruning methods.
Entropy-Aware Dense Scoring: A novel mechanism is proposed to quantify and filter textual noise using statistical entropy without relying on external NLP parsers. This yields a fine-grained, noise-robust instruction relevance score.
Submodular Token Selection with Spatial Priors: The authors reformulate token selection as a facility location submodular maximization problem. Coupled with spatial smoothing and score polarization, this guarantees holistic feature representativeness and explicitly penalizes redundancy, overcoming the limitations of greedy peak-chasing.

Experimental Results

EADP was evaluated across multiple VLM architectures (LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, Qwen2.5-VL, and Qwen3-VL) and diverse benchmarks (VQAv2, GQA, MME, MMBench, MM-Vet, DocVQA, etc.).

Performance: EADP consistently achieves State-of-the-Art (SoTA) performance across all tested token budgets. For instance, on LLaVA-1.5-7B with 128 retained tokens (77.8% pruning), EADP achieves an average accuracy of 63.5, outperforming strong baselines like CDPruner (63.2) and DivPrune (62.5).
Robustness: The method demonstrates superior robustness under extreme compression (e.g., retaining only 32 tokens) and high-resolution inputs (LLaVA-NeXT with 2880 tokens), where it maintains near-lossless performance compared to full-token baselines.
Efficiency: While EADP introduces a marginal computational overhead due to dense scoring and submodular optimization, it delivers significant acceleration in end-to-end inference latency (e.g., 1.5x–2.0x speedup) and reduces FLOPs by up to 66% compared to unpruned models.
Generalization: The approach generalizes effectively across different model scales (from 3B to 13B parameters) and modalities (static images and video), particularly excelling in tasks requiring fine-grained reasoning and document understanding.

Significance and Claims

The paper claims that visual token pruning is not merely a scoring task but a structured compression problem requiring both noise-robust saliency and holistic visual coverage. By addressing the specific bottlenecks of textual noise dispersion and feature fragmentation, EADP significantly advances the accuracy-efficiency trade-off for VLMs. The authors assert that their framework provides a mathematically principled solution that robustly preserves fine-grained visual cues under strict token budgets, making it highly suitable for real-time and resource-constrained multimodal applications. The work suggests that future pruning strategies must move beyond simple Top-K heuristics to incorporate structural priors and noise-aware scoring mechanisms.