💻 computer science

Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Models

The paper proposes DeSAP, a novel token pruning method for Large Vision-Language Models that utilizes decoupled similarity to capture fine-grained cross-modal relevance, enabling task-aware pruning that significantly reduces computational costs while maintaining high performance.

Original authors: Kexin Ma, Jing Xiao, Chaofeng Chen, Geyong Min, Guibo Zhu, Jinqiao Wang, Liang Liao

Published 2026-08-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Kexin Ma, Jing Xiao, Chaofeng Chen, Geyong Min, Guibo Zhu, Jinqiao Wang, Liang Liao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to see and talk at the same time. You give it a picture and ask a question, like "What is the cat doing?" To understand the picture, the robot breaks it down into thousands of tiny puzzle pieces called "tokens." Think of these tokens like individual pixels, but instead of just being colors, each one carries a little bit of meaning. The problem is, for a high-resolution photo, the robot might have to process thousands of these tokens just to answer a simple question. It's like trying to read a whole encyclopedia to find out if it's raining outside; the robot gets bogged down, slow, and uses up a massive amount of energy. This is the world of Large Vision-Language Models (LVLMs), where the goal is to make these giant brains faster and cheaper to run without making them forgetful. The big question scientists are asking is: How can we throw away the boring, useless puzzle pieces before the robot even starts thinking, so it only focuses on the important stuff?

Enter a new method called DeSAP (Decoupled Similarity-Aware Pruning), which acts like a super-efficient editor for these robot brains. The researchers found that the old ways of deciding which puzzle pieces to keep were a bit clumsy. Some methods only looked at the picture itself, keeping the "flashy" parts like bright colors or big shapes, but they often missed the specific details the user was asking about. Other methods waited until the robot started talking to decide what was important, but by then, the robot had already wasted time processing the boring stuff.

DeSAP solves this by doing two clever things at once, right at the start. First, it looks at the picture to find the "salient" parts—the things that naturally stand out, like a bright red ball in a green field. But it doesn't stop there. It also listens to the question before the robot starts its full thinking process. It uses a special trick called "decoupled similarity" to match the specific words in the question with the tiny details in the picture. Imagine you are looking for a "leather bag" in a photo. Old methods might just keep the whole bottom half of the image because that's where the bag usually is, or keep the whole background because it's "important." DeSAP, however, acts like a detective who knows exactly what to look for. It ignores the background and the people, and zooms in specifically on the leather bag, even if it's small or in a weird spot.

The paper suggests that by combining these two clues—the visual "flashiness" and the specific "task guidance" from the question—the robot can throw away up to 88.9% of the puzzle pieces (keeping only about 11% of them) and still answer questions almost as well as if it had seen the whole picture. In tests on models like LLaVA-1.5, this method didn't just save time; it actually made the robot faster (speeding up the initial processing by 2.3 times) and used 9 times less computing power, all while keeping the answers accurate. The researchers argue that previous methods failed because they relied on just one source of information, which often led to bias or missed details. DeSAP, by contrast, uses a dual-source strategy that keeps the robot focused on what matters most, proving that sometimes, seeing less is actually seeing more clearly.

Technical Summary: Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Models

1. Problem Statement

Large Vision-Language Models (LVLMs) have achieved significant progress in bridging visual and linguistic modalities but suffer from substantial computational overhead. This inefficiency stems primarily from the processing of a large number of visual tokens, which often far exceed text tokens, especially in high-resolution images and multi-frame videos.

Existing training-free token pruning methods aim to mitigate this by discarding less informative tokens. However, current approaches generally fall into two paradigms that suffer from inherent limitations:

Visual-Centric Pruning: Relies on attention signals from the visual encoder (e.g., [CLS] attention). These methods tend to focus on global semantics and salient regions, often overlooking fine-grained local details and failing to align with specific task requirements.
Task-Centric Pruning: Relies on cross-modal attention from the Large Language Model (LLM) decoder. These methods often introduce task guidance too late (during the prefill stage) and exhibit spatial biases (e.g., favoring bottom regions of the image due to sequence ordering) or global semantic biases that do not faithfully reflect token importance.

The core problem identified is that existing methods rely on individual attention sources that are biased and incomplete, leading to suboptimal pruning decisions that either lose critical visual details or fail to preserve task-relevant information under aggressive pruning ratios.

2. Methodology: DeSAP

The authors propose DeSAP (Decoupled Similarity-Aware Pruning), a novel method designed to perform precise, task-aware visual token pruning before LLM decoding. The methodology is built on three key components:

A. Empirical Analysis and Decomposition

Through empirical study, the authors observe that:

Global semantic bias in visual encoders corrupts vanilla token-wise similarity estimation, leading to misalignment between visual and text tokens.
Decomposing the hidden state of the visual encoder into residual and attention components reveals that the decoupled attention component preserves local details and fine-grained semantics better than the residual (global) component.

Based on this, DeSAP decomposes the final-layer visual hidden state ( $S^l_v$ ) into:

Decoupled Residual ( $S^d$ ): Captures global semantic patterns.
Decoupled Attention ( $A^d$ ): Captures localized, fine-grained semantic details.

B. Fine-Grained Cross-Modal Alignment

Instead of using standard cosine similarity between hidden states and text embeddings (which inherits global bias), DeSAP computes a Decoupled Similarity. This involves calculating the cosine similarity between the decoupled attention component ( $A^d$ ) and text embeddings ( $S_q$ ).
To further refine this, the method employs a Fine-Grained Cross-Modal Alignment (FCMA) strategy. It weights token-pair similarities based on each token's relevance to the global representation of the complementary modality (using [EOS] for text and global visual tokens for vision). This creates a reliable, task-relevant attention map ( $A_t$ ) that highlights specific regions aligned with the textual query.

C. Dual-Source Guided Token Pruning

DeSAP integrates two information sources to assess token importance:

Task-Relevant Attention ( $A_t$ ): Derived from the decoupled similarity, guiding the selection of tokens relevant to the specific query.
Salient Semantic Attention ( $A_s$ ): Derived from the visual encoder's self-attention (e.g., [CLS] attention), providing holistic visual cues.

The pruning process follows a Rank & Prune strategy:

Select the top $k_1$ tokens based on $A_t$ (task relevance).
Select the top $k_2$ tokens based on $A_s$ (visual salience) from the remaining set.
The final set combines both task-specific and global visual cues.
Token Merging: Discarded tokens are merged into the most similar retained tokens via clustering to mitigate information loss without increasing the token count.

3. Key Contributions

Empirical Analysis: The paper provides an in-depth analysis of existing pruning paradigms, revealing that visual encoder attention is biased toward global semantics and that vanilla similarity is insufficient for fine-grained alignment. It introduces the perspective of decoupling visual features into residual and attention components to preserve local details.
DeSAP Framework: The proposal of a novel pruning method that leverages decoupled attention to derive precise task cues via fine-grained cross-modal alignment. It uniquely integrates these task cues with visual saliency signals to enable robust pruning before LLM decoding.
Performance: Extensive experiments demonstrate that DeSAP consistently outperforms state-of-the-art (SOTA) methods across diverse benchmarks and model architectures.

4. Experimental Results

The authors evaluated DeSAP on three LVLMs (LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B) across eight benchmarks (including GQA, MME, POPE, VQAv2, etc.).

Accuracy: On LLaVA-1.5-7B with an aggressive 88.9% pruning ratio (retaining only 11.1% of tokens, i.e., 64 tokens), DeSAP achieved 98.1% of the original model's performance. Notably, at a 192-token budget, it slightly surpassed the full-model performance by 0.3%.
Comparison: DeSAP outperformed SOTA methods like HoloV, FlowCut, and VisionZip. On the 88.9% pruning ratio, it surpassed the second-best method (HoloV) by an average of 2.3% on LLaVA-1.5-7B.
Efficiency: On the POPE benchmark, DeSAP achieved a 9.1× reduction in FLOPs and a 2.3× prefill speedup compared to the full model, with only a 0.1% drop in accuracy. It also demonstrated superior efficiency-accuracy trade-offs compared to SparseVLM and HoloV under equivalent FLOPs.
Generalizability: The method maintained robust performance on LLaVA-NeXT-7B (high-resolution) and Qwen2.5-VL-7B, consistently outperforming baselines across various pruning ratios.

5. Significance and Claims

The paper claims that DeSAP addresses the critical limitation of existing pruning methods by introducing task awareness earlier in the pipeline (at the encoder side) without relying on biased attention distributions. By decoupling visual features and leveraging fine-grained cross-modal alignment, DeSAP mitigates representational collapse from both visual and task perspectives.

The authors assert that their dual-source guided approach enables robust token pruning even under aggressive ratios, preserving key information that other methods discard. The work demonstrates that integrating task guidance with visual saliency is more effective than relying on bias correction within a single attention source, offering a significant step forward in the efficiency and scalability of LVLMs for practical applications.