💻 computer science

Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models

This paper proposes Trend-aware Pruning, a training-free framework for Multimodal Large Language Models that improves efficiency by modeling the dynamic evolution of token importance across layers to prevent the premature loss of critical visual cues, achieving over 77.8% token reduction while maintaining competitive performance.

Original authors: Jie Ma, Zhike Qiu, Jie Gao, Jiayi Ji, Qian Chen, Xiaoshuai Sun, Rongrong Ji

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jie Ma, Zhike Qiu, Jie Gao, Jiayi Ji, Qian Chen, Xiaoshuai Sun, Rongrong Ji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to understand the world by showing it pictures and asking questions. This field is called Multimodal Large Language Models (MLLMs). Think of these models as brilliant detectives that can read text and look at images at the same time. To "see" an image, the robot breaks it down into thousands of tiny puzzle pieces called "tokens." Each token is a little snippet of the picture, like a patch of blue sky or a wheel on a car. The problem is, when you show the robot a high-resolution photo, it gets overwhelmed with too many pieces. It tries to pay attention to every single one, which makes the robot slow, expensive to run, and sometimes even confused by all the noise.

To fix this, scientists have tried a simple trick: just throw away the pieces that seem unimportant right at the start. It's like a bouncer at a club who looks at a crowd and immediately kicks out anyone who isn't wearing a specific color shirt. But here's the catch: sometimes the most interesting person in the room is wearing a boring shirt at first, only to start dancing and becoming the life of the party a few minutes later. If the bouncer kicks them out too early, the party is ruined. This paper asks a big question: Is it safe to make a final decision about what to keep based only on a split-second glance, or do we need to watch how things change over time?

The researchers behind this paper, Jie Ma and their team, say that the old "bouncer" method is too rigid. They propose a new way of thinking called Trend-aware Pruning. Instead of just looking at a snapshot of which tokens are important right now, their method acts like a patient observer who watches the tokens over time to see their "tendencies." They realized that some tokens are like late-blooming flowers; they might look quiet and unimportant in the early layers of the robot's brain, but as the robot processes the image deeper, these tokens suddenly become crucial for understanding the scene.

The team built a system that tracks the "momentum" of these tokens. Imagine a river where some rocks are just sitting there, but others are slowly drifting toward the center of the current. The old methods would ignore the drifting rocks because they aren't in the center yet. The new method, however, notices the drift. It keeps an eye on tokens that are showing an "upward trend"—meaning their importance is rising even if they aren't the top priority yet. If a token starts to look more interesting as the robot digs deeper, the system "rescues" it, bringing it back into the conversation before it's too late. They also watch out for tokens that are "fluctuating" (going up and down) or "downward-trending" (fading away), treating each group differently rather than just tossing them all out.

When they tested this idea on popular robot brains like LLaVA and Qwen, the results were impressive. The new method didn't just save time; it actually helped the robot perform better. In their experiments, they were able to cut the number of visual tokens down by more than 77.8%, leaving the final layer with only about 23 tokens to process. Despite throwing away so much data, the robot kept 98.89% of its original performance when cutting the tokens by half, and even maintained a strong 96.03% performance when cutting them by nearly 78%. This is a big deal because other methods that just pick the "top" tokens at the start often lose important details when they get this aggressive.

The authors suggest that this approach works because it respects the fact that understanding an image is a journey, not a single moment. By allowing the robot to change its mind and "reactivate" tokens that were initially overlooked, they prevent the loss of critical clues. They found that this dynamic approach is especially good at tricky tasks like reading text in images (OCR) or spotting small details, where static methods often fail. While they admit their system relies on a fixed window of observation and could be improved to handle even longer sequences, their work shows that watching the trend of attention is a powerful way to make these smart robots faster and sharper without needing to retrain them from scratch.

Technical Summary: Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models

Problem Statement

Multimodal Large Language Models (MLLMs) face a significant computational bottleneck due to the quadratic complexity of the attention mechanism when processing high-resolution images and multi-image inputs. This results in an excessive number of visual tokens, creating a severe redundancy that hinders efficient inference.

Existing training-free token pruning methods attempt to mitigate this by relying on static, instantaneous heuristics (e.g., attention-based Top- $k$ selection) to filter tokens at early layers. The paper identifies a critical flaw in this approach: it treats token importance as a fixed, single-shot decision. However, MLLMs exhibit a hierarchical nature where token importance evolves dynamically across layers. Visual tokens that appear redundant or weakly activated in shallow layers (representing low-level features) may become critical for high-level reasoning in deeper layers. Static pruning irreversibly discards these "late-blooming" tokens based on shallow-layer estimates, leading to a loss of essential semantic cues and degraded performance, particularly in complex reasoning tasks.

Methodology: Trend-aware Pruning

To address the misalignment between static pruning and dynamic semantic evolution, the authors propose Trend-aware Pruning, a training-free framework that reframes token pruning as a temporal trajectory modeling problem rather than a local snapshot decision. The framework consists of four key components:

Layer-wise Token Collector:
Instead of evaluating tokens in isolation, this module aggregates historical attention states across a sliding observation window ( $W$ ). It maintains a short-term memory cache of attention scores, allowing the system to track the "momentum" of token importance rather than just its current value.
Token Flow Modeling:
The method computes a Token Flow ( $F_l$ ) by calculating the step-wise difference in attention scores within the observation window. This decouples the direction of representation change from the single-layer importance score, effectively capturing the dynamic evolution of each token.
Adaptive Flow Identification:
Based on the token flow, tokens are categorized into three evolutionary patterns:
- Upward-trending: Tokens with increasing attention scores, indicating accumulating semantic importance.
- Downward-trending: Tokens with decreasing scores, suggesting irrelevance or representation shifts.
- Fluctuating: Tokens with oscillating patterns, often representing context-relevant cues that vary across layers.
The system employs a distribution-aware adaptive thresholding strategy to select tokens based on these trends, rather than fixed empirical thresholds.
Flow Activation and Dynamic Retention:
This is the core mechanism that enables reversible pruning. It combines a coarse static Top- $k$ pruning (for efficiency) with a fine-grained trend-aware activation. Crucially, it selectively reactivates tokens that were initially filtered out by the static Top- $k$ but exhibit positive semantic trends (Upward or Fluctuating). This ensures that tokens essential for deep-layer reasoning are preserved even if they were undervalued in earlier layers.

Key Contributions

Reformulation of Pruning: The paper shifts the paradigm from static, single-shot token filtering to a dynamic, cross-layer process that explicitly models the semantic evolution of visual tokens.
Trend-aware Pruning Framework: A novel method that jointly guides token discarding and selective reactivation. It introduces a mechanism to dynamically recover tokens whose semantic importance emerges in deeper layers, mitigating the degradation caused by irreversible visual cue loss.
Training-Free Efficiency: The approach is plug-and-play, requiring no additional training or fine-tuning, making it applicable to diverse MLLM architectures.

Experimental Results

The authors evaluated Trend-aware Pruning on multiple mainstream MLLMs (LLaVA series and Qwen2.5-VL) across various benchmarks (MME, GQA, POPE, SQA, MMBench, etc.).

Efficiency-Performance Trade-off: The method achieves superior performance retention while significantly reducing computational costs. At a 50% pruning ratio, it retains 98.89% of the average performance while reducing FLOPs to 55.10%, outperforming state-of-the-art methods like FastV, PDrop, and SparseVLM.
Aggressive Pruning: Under the most aggressive setting (77.8% pruning), the method reduces the number of visual tokens in the final layer to approximately 23 (the lowest among all compared methods) while maintaining competitive performance (96.03% average retention).
Generalization: The approach demonstrates consistent superiority across different model sizes (0.5B, 7B, 13B) and architectures, including the Qwen2.5-VL, where it significantly outperforms static Top- $k$ pruning on tasks requiring fine-grained detail (e.g., OCRBench, InfoVQA).
Ablation Studies: Experiments confirm that modeling all three trends (Upward, Fluctuating, and Downward) is crucial. Specifically, incorporating Upward-trending tokens significantly improves performance on detail-heavy tasks, while Fluctuating and Downward tokens provide further refinement.

Significance and Claims

The paper claims that Trend-aware Pruning offers a robust and reversible solution for high-efficiency multimodal inference. By acknowledging that token importance is not fixed but evolves dynamically, the method prevents the premature loss of critical visual cues that static methods inevitably discard.

The authors emphasize that their approach highlights the importance of modeling token dynamics across layers. They argue that single-layer importance is insufficient for determining the final semantic value of a visual token. By elevating pruning from a local decision to a temporal trajectory problem, the method achieves a more precise and stable pruning decision, enabling aggressive token reduction without sacrificing the representational completeness required for complex reasoning tasks.

The work is presented as a step toward reversible and representation-consistent pruning strategies, offering a new perspective on how to handle the computational bottlenecks in MLLMs without the need for retraining.