Technical Summary: Prior-Corrected Token Reduction for MLLMs

Problem Statement

Multimodal Large Language Models (MLLMs) convert images into long sequences of visual tokens, leading to significant inference costs, particularly for high-resolution images and videos. While existing training-free methods accelerate inference by pruning redundant visual tokens based on text–visual attention scores, this paper identifies a critical flaw in their underlying assumption: that attention magnitude reliably reflects instruction-conditioned semantic relevance.

The authors demonstrate that MLLMs exhibit a strong model-induced prior. Even in the absence of textual instructions, models tend to allocate high attention to certain task-agnostic, visually salient regions (e.g., high-contrast backgrounds or textured areas). When pruning relies on absolute attention scores, this prior dominates the ranking. Consequently, tokens containing crucial instruction-specific evidence may be suppressed and discarded, while irrelevant background tokens are retained. This issue is particularly severe under aggressive token budgets, where incorrect selections can irreversibly remove the visual evidence required for correct reasoning.

Methodology: PriorTR

To address this, the authors propose PriorTR (Prior-Corrected Token Reduction), a training-free method that explicitly separates task-conditioned attention from the model-induced prior.

Core Concept: V-Usable Information

PriorTR reframes token selection as a problem of identifying tokens that contribute additional usable information beyond the model's intrinsic bias. It decomposes the observed posterior attention $P_\theta(v | X)$ (conditioned on instruction $X$) into two components:

1. Model-Induced Prior ($Q_\theta(v)$): The attention distribution when no instruction is present (or a null instruction is used).
2. Semantic Lift ($L(v, X)$): The multiplicative increase in importance induced specifically by the instruction.

The method scores tokens based on Pointwise V-Information (PVI), defined as:
$$\text{PVI}(v) = \log \frac{P_\theta(v | X)}{Q_\theta(v)}$$
To ensure tokens with negligible probability are not selected, the final importance score ($S_{\text{PriorTR}}$) combines the posterior mass with the log-ratio:
$$S_{\text{PriorTR}}(v_i) = P_\theta(v_i | X) \cdot \log \left( \frac{P_\theta(v_i | X)}{Q_\theta(v_i)} \right)$$
Tokens are ranked by this score, and the top-$K$ are retained.

Efficient Single-Pass Implementation

A key innovation of PriorTR is its ability to estimate both the prior and the posterior within a single forward pass, avoiding the computational overhead of running the model twice.

• Null Token as Probe: The method exploits the causal attention structure of autoregressive decoders. A "null token" (e.g., a separator like \n) is placed immediately after the visual tokens but before the instruction tokens.
• Causal Masking: Due to the causal mask, the null token cannot attend to subsequent instruction tokens. Therefore, its attention distribution over the visual sequence naturally represents the instruction-agnostic prior ($Q_\theta$).
• Simultaneous Extraction: In the same attention block, the attention from the actual instruction tokens to the visual sequence provides the task-conditioned posterior ($P_\theta$).
• Physical Pruning: Unlike methods that merely mask tokens, PriorTR physically removes unselected hidden states and KV-cache entries. This ensures that subsequent layers operate on a truly shortened sequence, yielding real reductions in memory and computation.

Key Contributions

1. Identification of Model-Induced Prior: The authors formalize the observation that attention-based pruning is often dominated by instruction-agnostic saliency patterns, leading to the suppression of task-relevant tokens under strict budgets.
2. PriorTR Algorithm: A training-free token reduction method that ranks visual tokens based on "V-usable information" (the semantic lift over the prior) rather than raw attention magnitude.
3. Efficient Single-Pass Design: An implementation that estimates the prior using a null token probe within a single forward pass, enabling physical pruning and real inference acceleration without duplicated propagation.
4. Empirical Validation: Extensive experiments demonstrating that PriorTR consistently improves the accuracy-efficiency trade-off across multiple MLLMs (LLaVA-1.5, Qwen3-VL) and benchmarks (image and video understanding), particularly in low-token regimes.

Experimental Results

The paper evaluates PriorTR on 12 multimodal benchmarks (including GQA, MME, MMB, POPE, and video QA tasks) using LLaVA-1.5-7B, LLaVA-1.5-13B, and Qwen3-VL-8B.

• Accuracy vs. Efficiency: PriorTR consistently outperforms strong training-free baselines (FastV, PDrop, SparseVLM, VisPruner, PruMerge).
  • On LLaVA-1.5-7B with 64 retained tokens (88.9% reduction), PriorTR achieves an average accuracy of 94.5%, significantly higher than FastV (70.7%) and VisPruner (91.7%).
  • On Qwen3-VL-8B with an 11.1% token ratio, PriorTR achieves 87.8% relative performance, outperforming the baseline FastV (75.5%).
• Video Understanding: On Video-LLaVA, PriorTR combined with token merging achieves near-baseline accuracy (38.7% vs. 38.7% uncompressed) under GPT evaluation and exceeds the baseline under Gemini evaluation, surpassing SparseVLM.
• Efficiency Gains: PriorTR achieves substantial speedups (up to 2.21× on LLaVA-1.5-7B) and reduces KV cache memory usage proportionally to the token reduction, confirming that physical pruning effectively lowers computational load.
• Ablation Studies: Comparisons of scoring functions show that the proposed $P \cdot \log(P/Q)$ formulation outperforms alternatives like simple difference ($P-Q$), log-ratio ($\log(P/Q)$), or raw posterior ($P$), validating the necessity of jointly considering relevance magnitude and relative lift.

Significance

The paper claims that PriorTR offers a principled direction for efficient multimodal inference by decoupling intrinsic saliency from task-driven relevance. By correcting the model-induced prior, the method ensures that token selection is driven by instruction-conditioned semantics rather than visual dominance. This approach provides a robust, plug-and-play solution that maintains high accuracy even under aggressive token compression, addressing a fundamental limitation in current attention-based pruning strategies. The authors emphasize that their method is orthogonal to existing approaches and can be seamlessly integrated with them.