EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs

EvoPrune is an early-stage visual token pruning method that performs layer-wise pruning guided by token similarity, diversity, and attention importance during visual encoding, achieving a 2$\times$ inference speedup with minimal performance degradation on high-resolution images and videos.

The Gist

Imagine you have a super-smart robot assistant (a Multimodal Large Language Model, or MLLM) that can look at photos and watch videos to answer your questions. This robot is incredibly talented, but it has a major weakness: it gets overwhelmed by details.

When you show it a high-resolution photo or a long video, the robot tries to look at every single pixel as if it were a separate piece of information. It's like asking a librarian to read every single word in a 1,000-page book, even though you only asked, "What is the main character's name?" The robot spends so much time reading the "boring" parts (like a blank sky or a static background) that it gets slow and expensive to run.

This paper introduces EvoPrune, a clever new way to help this robot work faster without losing its smarts.

The Problem: The "Post-Processing" Mistake

Most previous methods tried to speed up the robot by letting it read the entire book first, and then telling it, "Okay, now forget 80% of what you just read."

Think of it like this: You hire a team of 100 researchers to analyze a massive crime scene. They all run around, take photos, and measure everything. After they've done all that hard work, you tell them, "Actually, we only needed the top 20 researchers; the other 80 were just looking at the same thing."
The problem: You already paid for the time and energy of all 100 researchers. The expensive part (the "Visual Encoding") is already done.

The Solution: The "Early-Stage" Filter

EvoPrune changes the game. Instead of waiting until the end, it acts as a smart filter right at the entrance.

Imagine the robot's brain has a series of checkpoints (layers) as it processes an image. EvoPrune stands at these checkpoints and says:

"Hey, you two look exactly the same. You're both just 'blue sky.' Let's merge you into one person. And you, 'red car,' you're important, so stay. But you, 'distant tree,' you're blurry and not relevant to the question; you can go home."

It does this while the robot is still looking at the image, not after. This means the robot never wastes energy processing the redundant details in the first place.

How Does It Know What to Keep? (The Three Rules)

EvoPrune uses a "Scorecard" with three rules to decide which details to keep and which to throw away:

1. The "Copycat" Rule (Similarity): If two parts of the image look almost identical (like a patch of green grass), merge them. Why keep two copies of the same thing?
2. The "Unique Gem" Rule (Diversity): If a detail is unique and stands out (like a bright red fire hydrant in a gray street), keep it! We don't want to throw away the interesting stuff just because it's rare.
3. The "Spotlight" Rule (Attention): The robot has a natural "spotlight" that focuses on what it thinks is important. If the robot's internal spotlight is shining on a specific part of the image, EvoPrune says, "Definitely keep that!"

The Results: Fast and Furious

The paper tested EvoPrune on images and long videos. Here is what happened:

• Speed: On video tasks, EvoPrune made the robot 2 times faster. It cut the waiting time in half.
• Smarts: Even though it threw away a ton of "junk" data, the robot's answers were almost as good as before (only a tiny, almost unnoticeable drop in accuracy).
• Scalability: The more complex the video (more frames, higher resolution), the more EvoPrune helped. It solved the problem that other methods couldn't handle.

The Big Picture

Think of EvoPrune as a smart bouncer at a club.

• Old methods let everyone in, let them dance for an hour, and then kicked 80% of them out.
• EvoPrune checks the ID at the door, only lets the VIPs and the interesting guests in, and keeps the line moving smoothly.

By pruning (cutting) the visual tokens early in the process, EvoPrune allows these powerful AI models to run on regular devices, handle long videos in real-time, and answer questions instantly, making them much more useful for the real world.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) face a critical efficiency bottleneck when processing high-resolution images and long videos. The number of visual tokens grows exponentially with input complexity, leading to severe computational and memory overhead.

• The Bottleneck: Existing token pruning methods primarily operate after the visual encoding stage (i.e., on the output of the vision encoder).
• The Limitation: As shown in the paper's analysis (Figure 1), the visual encoder itself consumes a significant portion of the inference time, especially as input scales increase (e.g., from single images to 64-frame videos). By pruning only after encoding, existing methods fail to reduce the computational cost of the encoder itself, leading to diminishing returns in acceleration as input size grows. The encoder cost remains dominant, limiting the scalability of MLLMs in latency-sensitive applications like real-time video analysis.

2. Methodology: EvoPrune

EvoPrune proposes a paradigm shift by performing token pruning directly within the visual encoder (early-stage pruning) rather than waiting for full feature extraction.

Core Mechanism

EvoPrune integrates a layer-wise pruning strategy into the visual transformer encoder. Instead of processing all tokens through every layer, it progressively merges redundant or low-importance tokens at selected intermediate layers.

The Score-Guided Merging Strategy

At selected layers, EvoPrune calculates a composite score for token pairs to decide which to merge. This score integrates three complementary criteria:

1. Semantic Similarity (Attraction): Uses cosine similarity between feature embeddings to identify and merge visually redundant tokens.
2. Information Diversity (Penalty): Calculates local density to penalize merging tokens that carry distinct content. This ensures the retained token set remains diverse and representative of the full visual scene.
3. Attention-Based Importance (Preservation): Uses attention weights from the encoder to identify "critical" tokens.
   • Critical Token Ratio (CTR): A mechanism is employed to protect the top $N_{protect}$ tokens (e.g., top 25%) based on their attention scores.
   • Hard Constraint: Tokens identified as critical are assigned an infinite negative score ($-\infty$) for merging, ensuring they are never pruned, thus preserving essential semantic and contextual information.

Algorithm Flow

1. Budget Allocation: A global pruning target is distributed across encoder layers (e.g., using a "Skip" strategy where merging happens at alternating layers).
2. Layer-wise Execution:
   • Tokens pass through standard Multi-Head Self-Attention (MHSA).
   • If the layer is a pruning layer, a bipartite matching algorithm selects the top $r$ token pairs with the highest composite scores.
   • Selected pairs are merged; the rest are propagated to the next layer.
3. Output: The process continues until the final layer, outputting a reduced set of tokens for the LLM.

3. Key Contributions

1. Early-Stage Pruning Paradigm: The first method to integrate token pruning directly into the visual encoding stage, addressing the previously neglected computational cost of the vision encoder.
2. Multi-Factor Guidance: A novel scoring function that balances similarity (for compression), diversity (for information retention), and attention importance (for critical feature preservation).
3. Plug-and-Play Design: The method requires no retraining of the MLLM and can be seamlessly integrated into existing architectures (e.g., LLaVA, LLaVA-Video).
4. Scalability: Demonstrates that pruning effectiveness does not degrade with larger inputs; in fact, it becomes more crucial for high-resolution and video tasks.

4. Experimental Results

The authors evaluated EvoPrune on diverse image and video benchmarks (VQAv2, MME, MMBench, VideoMME, LongVideoBench) using LLaVA-1.5-7B and LLaVA-Video-7B.

• Image Understanding:

  • On image benchmarks, EvoPrune achieved the best trade-off between accuracy and latency.
  • With a 77.8% token reduction (retaining 128 tokens), it achieved 97.9% relative accuracy compared to the native model while reducing overall latency by 16% compared to the strongest baseline (CDPruner).

• Video Understanding (Key Highlight):

  • Performance: On the challenging VideoMME dataset, EvoPrune achieved 2× inference speedup (reducing Time-To-First-Token by 50%) with less than 1% performance degradation (99.7% relative accuracy).
  • Efficiency: Unlike baselines that only speed up the LLM, EvoPrune accelerates the Visual Encoder by 1.8×, intermediate modules by 5×, and the LLM backbone by 2×.
  • Robustness: Even under extreme pruning (90.5% reduction), EvoPrune maintained 95.8% relative accuracy, significantly outperforming other methods which suffered larger accuracy drops.

• Ablation Studies:

  • Removing the Attention Preservation component caused the largest performance drop, confirming that protecting high-importance tokens is critical for spatiotemporal semantics.
  • The Skip strategy (merging at alternating layers) was found to be the most robust allocation pattern, balancing error accumulation and efficiency.

5. Significance

EvoPrune addresses a fundamental scalability issue in MLLMs. By moving pruning to the "early stage" (inside the encoder), it tackles the root cause of the computational bottleneck rather than just the symptom.

• Real-World Impact: It enables the deployment of MLLMs in latency-sensitive environments (e.g., edge devices, real-time video analysis) where high-resolution inputs are common.
• Future Direction: The paper suggests that early-stage pruning is a necessary evolution for handling long-context and high-resolution multimodal tasks, paving the way for more efficient and scalable vision-language systems.