Rethinking Token Reduction for Large Vision-Language Models

1. Problem Statement

Large Vision-Language Models (LVLMs) excel in visual reasoning but suffer from high inference costs due to the massive number of visual tokens generated, especially when using multi-scale vision towers (e.g., LLaVA-NeXT, InternLM-XComposer-2.5). The computational complexity of self-attention scales quadratically ($O(n^2)$) with sequence length, making real-time or resource-constrained deployment difficult.

While existing token reduction methods exist, they face critical limitations in Multi-Turn Visual Question Answering (MT-VQA) scenarios:

• Prompt-Dependent Methods (e.g., FastV): These rely on the initial text prompt to determine which tokens to keep. In multi-turn dialogues, subsequent questions are unknown at the start and may reference arbitrary image regions (e.g., background details ignored in the first turn). Consequently, these methods discard crucial information needed for later turns.
• Prompt-Agnostic Heuristics (e.g., PruMerge): These rely on hand-crafted heuristics, such as attention scores to the [CLS] token or prompt tokens, to prune or merge tokens. The authors argue these heuristics are suboptimal because they do not theoretically guarantee the preservation of information necessary for future, unseen queries.

2. Methodology: MetaCompress

The authors propose MetaCompress, a learning-based, prompt-agnostic token reduction framework that treats token reduction as a learnable compression mapping rather than a heuristic selection process.

Core Formulation

The authors unify token pruning and merging into a single linear projection formulation:
$$\tilde{X}_{IMG} = P X_{IMG}$$
Where $X_{IMG}$ is the input visual token sequence, $\tilde{X}_{IMG}$ is the compressed sequence, and $P \in \mathbb{R}^{m \times n}$ is a sparse compression matrix ($m \ll n$).

Key Components

1. Insight from Preliminary Analysis:

   • The authors first trained an optimal compression matrix $P^*$ for individual images to minimize the KL divergence between the LLM's output with compressed vs. uncompressed tokens.
   • Finding: The tokens retained by the optimal $P^*$ showed no significant correlation with heuristic attention scores (e.g., attention to [CLS] or prompt tokens). This empirically proved that existing heuristic criteria are suboptimal for MT-VQA.

2. Meta-Generator Architecture ($P_{meta}$):

   • Instead of learning a unique matrix for every image, MetaCompress learns a generator $P_{meta}$ that produces the compression matrix $P$ conditioned only on the visual sequence $X_{IMG}$.
   • Architecture: It consists of a position embedding layer, a query down-sampling projection ($\tilde{D}_q$), a key projection ($D_k$), and a weighted inner product layer.
   • Mechanism: It computes the inner product between spatially down-sampled queries and keys to generate $P$. This allows the model to adapt to varying image resolutions and token counts dynamically without retraining.

3. Training Paradigm:

   • Objective: Minimize the prediction discrepancy (KL divergence) between the original and compressed outputs.
   • Regularization: To prevent the model from collapsing to trivial solutions (e.g., selecting only one token source), the authors introduce:
     • Entropy Loss ($L_{entropy}$): Encourages diversity in token selection.
     • Collapse Regularization ($L_{collapse}$): Prevents the compression matrix from concentrating all weights on a single column.
   • Data Efficiency: The model is trained on a small subset (~20k samples) of MT-GQA and MT-VQA-v2, demonstrating high data efficiency.

3. Key Contributions

1. Re-evaluation of Token Reduction: The paper is the first to systematically explore token reduction in the MT-VQA scenario, revealing that heuristic methods relying on attention scores are fundamentally suboptimal for multi-turn interactions.
2. MetaCompress Framework: Proposes a novel, learning-based, prompt-agnostic method that unifies pruning and merging into a learnable compression mapping, eliminating reliance on hand-crafted heuristics.
3. Data-Efficient Training: Introduces a lightweight training paradigm that learns optimal compression mappings with minimal computational cost and data requirements, while maintaining compatibility with modern multi-scale LVLMs.
4. Superior Generalization: Demonstrates that the learned compression strategy generalizes well across different dialogue turns, datasets, and model architectures without task-specific fine-tuning.

4. Experimental Results

The method was evaluated on three MT-VQA benchmarks (MT-VQA-v2, MT-GQA, ConvBench) across five LVLM architectures (LLaVA-1.5, LLaVA-NeXT, InternLM-XComposer-2.5).

• Accuracy: MetaCompress consistently outperformed baselines (Random, Sample, FastV, PruMerge) across all models and reduction rates (50%–95%).
  • Example: On LLaVA-NeXT-7b with 90% reduction, MetaCompress achieved 75.18% average accuracy on MT-GQA, significantly beating FastV (50.31%) and PruMerge (not applicable to this architecture).
  • It maintained strong performance even on ConvBench, a benchmark it was not trained on, proving strong transferability.
• Efficiency:
  • Achieved comparable inference speed (Time to First Token and End-to-End time) and memory usage to the most efficient baseline (Random/Sample).
  • Reduced computational FLOPs by approximately 80% (e.g., from 71.4 TFLOPs to 13.3 TFLOPs for LLaVA-1.5-7b at 90% reduction) without the accuracy degradation seen in heuristic methods.
• Ablation Studies: Confirmed that all three loss terms (Prediction Discrepancy, Entropy, Collapse) are necessary for optimal performance. Removing the collapse regularization led to training divergence.

5. Significance

• Paradigm Shift: Moves token reduction from "heuristic selection" (based on human priors like attention scores) to "learnable optimization" (data-driven mapping).
• Practicality for Dialogue: Solves a critical bottleneck for deploying LVLMs in real-world conversational agents where future questions are unpredictable.
• Scalability: The method is compatible with modern, complex LVLMs that use multi-scale vision towers, which previous pruning methods often failed to support.
• Resource Efficiency: Offers a path to deploy high-performance LVLMs on resource-constrained devices by drastically reducing memory and compute overhead while preserving reasoning capabilities across multi-turn contexts.

In conclusion, MetaCompress establishes a new standard for efficient LVLM inference in dialogue scenarios by proving that learned, prompt-agnostic compression is superior to heuristic, prompt-dependent strategies.