💻 computer science

Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

This paper proposes CLSE, a training-free token pruning framework that quantifies cross-layer spectral evolution in the frequency domain to identify and preserve semantically active visual tokens, thereby accelerating Multimodal Large Language Models while maintaining or improving reasoning performance.

Original authors: Bin Chen, Yuxiang Cai, Yadan Luo, Yi Zhang, Jianwei Yin, Zhi Chen

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bin Chen, Yuxiang Cai, Yadan Luo, Yi Zhang, Jianwei Yin, Zhi Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, but the box contains 1,000 pieces, and 800 of them are just pictures of the blue sky or the empty table. A standard computer (a Multimodal Large Language Model, or MLLM) tries to look at every single one of those 1,000 pieces to figure out the picture. This takes a long time and uses a lot of energy, even though most of those pieces don't actually help solve the puzzle.

This paper introduces a new, "training-free" way to quickly throw away the useless pieces (redundant tokens) so the computer can focus only on the important ones, without needing to be re-taught how to do it.

Here is how the authors' new method, called CLSE (Cross-Layer Spectral Evolution), works, using some simple analogies:

1. The Problem: The "Spotlight" is Biased

Current methods try to decide which puzzle pieces are important by looking at a "spotlight" (called attention scores) inside the computer's brain.

The Flaw: The paper argues this spotlight is a bit glitchy. It tends to shine brighter on pieces that appear later in the list, simply because of where they are sitting, not because they are actually important. It's like a teacher grading a test who accidentally gives higher scores to the answers written at the bottom of the page, regardless of whether they are correct.
The Result: The computer might keep useless background pieces and throw away the crucial ones.

2. The Solution: Watching the "Dance" of the Pieces

Instead of taking a single snapshot of importance, the authors suggest watching how the pieces change as they move through the computer's layers (like passing a ball through a relay race).

The Analogy: Imagine the visual tokens (puzzle pieces) are dancers.
- Background pieces (like the sky) are boring dancers. They stand still and do the exact same move from the start of the song to the end. They don't change.
- Important pieces (like a motorcycle doing a trick) are dynamic dancers. They start with a simple pose (low-level details) and, as the song progresses, they transform into a complex, meaningful routine (high-level meaning).
The Method: The authors use a mathematical tool called Spectral Evolution (think of it as a "change detector") to measure how much a dancer's routine changes from one layer to the next.
- If a token changes a lot (evolves), it's kept because it's doing something important.
- If a token stays the same (static), it's thrown away because it's just background noise.

3. Why "Frequency" Matters

The paper uses a concept called "frequency" (from music or radio waves) to explain this.

Low Frequency: Think of a smooth, slow hum. This represents the boring, unchanging background.
High Frequency: Think of a sharp, rapid drumbeat. This represents the fine details and sudden changes that make up the interesting parts of the image.
The Trick: The new method filters out the "smooth hum" (the boring background) and only pays attention to the "drumbeats" (the changing, important details) as they evolve through the layers.

4. The Results: Faster and Smarter

The authors tested this on many different models and tasks (like answering questions about images and videos).

The Outcome: By throwing away the "boring dancers" and keeping the "dynamic ones," the computer became much faster (using less energy and memory) but didn't lose its ability to understand the image. In fact, in many cases, it performed better than other methods that tried to guess importance using the flawed "spotlight" (attention).
Video: This worked especially well for videos, where there is even more "boring" repetition between frames. The method could cut the number of pieces the computer had to process by over 90% while still understanding the action perfectly.

Summary

In short, this paper says: Don't just look at a piece once to decide if it's important. Watch how it changes as it moves through the system. If it stays the same, it's probably just background noise. If it evolves and transforms, it's the key to understanding the image. This allows AI to be much faster without getting confused.

Technical Summary: Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant progress in visual reasoning and dialogue but suffer from high computational costs due to the processing of hundreds to thousands of visual tokens per sample. This inefficiency is particularly acute in high-resolution image and video scenarios.

Existing token pruning methods attempt to mitigate this by reducing visual token redundancy. However, current approaches typically rely on single-layer signals, such as attention scores, feature magnitudes, or similarity metrics computed within a specific Transformer layer. The authors identify two critical limitations in these methods:

Positional Bias: Attention scores often exhibit a bias toward tokens appearing later in the flattened sequence due to positional encoding effects, rather than reflecting true semantic importance.
Static Representation: Single-layer signals overlook the cross-layer transformation of visual representations. In MLLMs, visual tokens undergo a structured evolution from low-level spatial details to high-level semantic abstractions as they pass through the LLM decoder layers. Instantaneous signals fail to capture this dynamic transition, leading to the potential pruning of semantically active tokens or the retention of redundant background tokens.

2. Methodology: Cross-Layer Spectral Evolution (CLSE)

To address these limitations, the authors propose Cross-Layer Spectral Evolution (CLSE), a training-free token pruning framework. Instead of measuring token importance based on feature magnitude or attention intensity at a single point, CLSE quantifies how token representations evolve across Transformer layers in the frequency domain.

Core Mechanism

The method operates on the observation that semantically active tokens undergo significant spectral redistribution (transitioning from high-frequency structural details to low-frequency semantic abstractions) across layers, whereas redundant background tokens remain relatively stationary in the frequency domain.

The CLSE pipeline involves the following steps:

Frequency Projection: Visual tokens at a specific layer $\ell$ are reshaped into 2D spatial feature maps and transformed into the frequency domain using the Fast Fourier Transform (FFT).
High-Pass Filtering: A Gaussian high-pass filter is applied to the spectrum to suppress low-frequency global patterns and isolate high-frequency structural energy. This isolates the "residual" structural information.
Spectral Evolution Calculation: The method computes the normalized difference between the high-frequency energy distributions of adjacent layers ( $\ell$ $ℓ$ and $\ell+1$ $ℓ + 1$ ).
- Let $\mathbf{S}_\ell$ be the spatial distribution of high-frequency energy at layer $\ell$ .
- The evolution intensity $\mathcal{I}^{(\ell)}$ is calculated as:
  $\mathcal{I}^{(\ell)} = \text{clip}\left( \frac{|\mathbf{S}_{\ell+1} - \mathbf{S}_{\ell}|}{\mathbf{S}_{\ell} + \overline{\mathbf{S}}_{\ell} + \epsilon}, 0, \tau_{\max} \right)$
- Here, $\overline{\mathbf{S}}_{\ell}$ is the mean energy across tokens, acting as a regularizer to suppress minor fluctuations in inactive regions.
Token Selection: Tokens with higher evolution intensity scores are deemed semantically active and preserved. Tokens with low scores (indicating static, redundant representations) are pruned.

Integration

CLSE is integrated into the early layers of the LLM decoder (specifically between the input embeddings and the first decoder layer output). It serves as a plug-and-play module that guides token selection before subsequent layers process the reduced sequence. The framework is orthogonal to token merging strategies and can be combined with them (e.g., CLSE-M).

3. Key Contributions

The paper outlines three primary contributions:

Cross-Layer Spectral Evolution Framework (CLSE): A novel principle for token importance estimation that characterizes the evolution of visual representations from structural details to semantic abstractions across LLM layers, moving beyond single-layer statistics.
Training-Free Pruning Strategy: A practical implementation that identifies and removes redundant tokens during inference without requiring model retraining or weight modification. It leverages spectral dynamics to preserve tokens undergoing meaningful representational changes.
Empirical Validation: Extensive experiments on diverse image and video benchmarks demonstrating that CLSE achieves a superior trade-off between efficiency and accuracy, particularly under aggressive token reduction ratios.

4. Experimental Results

The authors evaluated CLSE on multiple MLLMs (LLaVA-1.5, LLaVA-Next, Qwen2-VL) and benchmarks (GQA, MMB, POPE, SQA, VQAText, etc.).

Image Understanding

Performance: On LLaVA-1.5-7B, CLSE achieved an average accuracy of 99.4% relative to the full-token model when retaining only 192 tokens (66.7% reduction), outperforming state-of-the-art methods like FastV (92.1%) and PDrop (96.9%).
Aggressive Reduction: Even at 88.9% reduction (64 tokens), CLSE maintained 94.8% performance, significantly outperforming competitors which suffered larger degradation.
Generalization: The method demonstrated consistent improvements across different model architectures (LLaVA-Next-7B, Qwen2-VL-7B/72B), maintaining high accuracy where attention-based methods failed.

Video Understanding

Extreme Compression: On Video-LLaVA, compressing tokens from 2048 to 194 (>90% reduction), CLSE achieved 35.7% average accuracy (using GPT-4o-mini evaluation), compared to 26.1% for FastV.
Token Merging Variant (CLSE-M): By recalling pruned tokens and clustering them based on spectral scores, CLSE-M matched the vanilla model's performance (38.7% accuracy) while reducing FLOPs by ~90%.

Efficiency

Computational Cost: CLSE reduces FLOPs and KV cache memory significantly. On LLaVA-1.5-7B, it achieved a 1.50× speedup in prefilling time with the lowest FLOPs (43.9%) among compared methods.
Scalability: Unlike attention-based pruning which requires "Eager" attention computation (quadratic complexity $O(N^2)$ ) to extract weights, CLSE relies on FFT, maintaining near-linear complexity $O(N \log N)$ . This ensures the overhead of the pruning mechanism itself remains low even for long sequences.

5. Significance and Claims

The paper claims that Cross-Layer Spectral Evolution provides a more reliable signal for token importance than instantaneous attention or magnitude-based metrics.

Mitigation of Bias: By analyzing the change in representation across layers rather than the state at a single layer, CLSE mitigates the positional bias inherent in attention scores.
Semantic Fidelity: The method aligns with the theoretical understanding that MLLM decoding involves a transition from spatial details to semantic abstractions. Tokens that actively participate in this transition (showing spectral evolution) are preserved, while static background tokens are removed.
Robustness: The approach is robust across different compression levels and model scales, offering a stable pathway for accelerating MLLMs without retraining.

The authors conclude that modeling representation dynamics in the frequency domain offers a promising direction for efficient multimodal reasoning, encouraging further exploration of spectral representations in MLLMs.