🤖 machine learning

AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference

AsymVLM is an efficient Vision-Language Model inference framework that leverages the distinct properties of visual and text modalities by applying aggressive, adaptive pruning to spatially redundant vision tokens and temporal threshold-based eviction to text tokens, achieving up to 54% FLOPs savings while outperforming state-of-the-art methods on document and chart understanding tasks.

Original authors: Yilin Feng, Ahmed Burak Gulhan, Mahmut Taylan Kandemir

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yilin Feng, Ahmed Burak Gulhan, Mahmut Taylan Kandemir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Vision-Language Model (VLM) as a highly intelligent but slightly overwhelmed assistant. You show it a picture (like a complex document or a chart) and ask a question. To understand the picture, the assistant breaks it down into thousands of tiny puzzle pieces called "vision tokens." To understand your question and formulate an answer, it uses a smaller set of "text tokens."

The problem is that the assistant's "working memory" (its GPU memory) gets clogged up. It tries to hold onto every single puzzle piece and every word it generates, which makes it slow and expensive to run.

Existing methods try to fix this by treating the picture pieces and the words exactly the same way: they just throw away a fixed percentage of everything (e.g., "delete 50% of the puzzle pieces and 50% of the words"). The authors of this paper, AsymVLM, argue that this is like trying to organize a library by throwing away half the books and half the bookmarks without thinking about what they actually do.

Here is how AsymVLM works, using simple analogies:

1. The Two Different Problems

The paper points out that picture pieces and words are fundamentally different:

Vision Tokens (The Puzzle Pieces): These are all independent. If you remove a piece of the sky from a photo, the piece of the tree next to it doesn't care. They are "spatially redundant," meaning many pieces are just background noise.
Text Tokens (The Story): These are a chain reaction. Word 2 depends on Word 1, and Word 3 depends on Word 2. If you delete a word in the middle of a sentence, the rest of the story might make no sense.

2. The Solution: A Two-Pronged Strategy

Instead of using one rule for everything, AsymVLM uses two different strategies tailored to each type of token.

Strategy A: The "Smart Editor" for Pictures (Vision Token Pruning)

Before the assistant even starts thinking, AsymVLM acts like a smart editor looking at the photo.

The Old Way: "Delete 50% of the pixels randomly" or "Delete the ones that look least important to the question."
The AsymVLM Way: It uses a Learned Scorer. Imagine a coach who has watched thousands of games and knows exactly which players (tokens) actually help win the game. This scorer doesn't just look at the picture; it looks at how the picture connects to the specific question you asked.
The "Adaptive Budget": This is the cleverest part. The paper notes that some questions need a full scan of the image (e.g., "How many apples are in this orchard?"), while others only need a tiny spot (e.g., "What is the price of the red apple?").
- If the question is specific to one small area, the "gap" between important and unimportant pieces is huge. The system says, "Great, we can be aggressive and throw away 75% of the picture!"
- If the question needs the whole image, the "gap" is small. The system says, "Okay, keep 90% of the picture just to be safe."
- Analogy: It's like packing a suitcase. If you are going to the beach for one day, you pack lightly. If you are going on a month-long trip, you pack more. AsymVLM adjusts the "packing" based on the specific trip, rather than using the same suitcase size for everyone.

Strategy B: The "Memory Janitor" for Text (Text Token Eviction)

Once the assistant starts generating an answer, it writes words one by one. If the conversation gets long, the memory fills up.

The Old Way: Standard methods for text-only AI (like H2O or StreamingLLM) try to delete the oldest or least "important" words to make room for new ones.
The AsymVLM Way: The authors realized that in these visual assistants, the conversation is usually short, and the picture is the most important context.
- They set a fixed budget. The assistant keeps writing until it hits a limit (e.g., 500 words).
- Once it hits the limit, it starts deleting the oldest generated words (the ones that are no longer needed for the immediate next sentence), but it never deletes the original picture or the original question.
- Analogy: Imagine you are telling a story to a friend. You don't need to remember the first sentence you said 10 minutes ago to finish the current sentence. AsymVLM acts like a janitor who clears out the old, irrelevant drafts from the whiteboard to make space for the new sentence, but it leaves the original prompt and the photo pinned to the wall forever.

3. The Results: Faster and Smarter

The paper claims that by treating these two types of data differently, AsymVLM achieves:

Massive Speedups: It saves up to 54% of the computing power (FLOPs) needed to run the model. This is because it physically removes the unnecessary picture pieces before the heavy math begins, rather than just ignoring them during the math.
Better Accuracy: On tasks like reading documents or understanding charts, it actually performs 2–3% better than previous methods. This is because it keeps the specific picture pieces that answer the question and discards the rest, whereas other methods might accidentally delete the crucial piece.
Memory Efficiency: It reduces the memory needed to run the model, allowing it to fit on smaller, cheaper computer chips that couldn't handle the full, unpruned model.

Summary

AsymVLM is a system that realizes "one size does not fit all." It uses a smart, adaptive filter to trim the fat off images based on the specific question, and a careful janitor to manage the text memory without losing the original context. The result is a Vision-Language Model that is faster, uses less memory, and is surprisingly more accurate at reading documents and charts.

Technical Summary: AsymVLM

Problem Statement

Vision-Language Models (VLMs) face significant computational and memory bottlenecks during inference. While existing compression methods often treat visual and textual tokens uniformly, this paper argues that such an approach is suboptimal due to fundamental asymmetries between the two modalities:

Structural and Causal Differences: Vision tokens are spatially redundant patches generated in a single forward pass with no causal dependency, allowing for independent removal. In contrast, text tokens are generated autoregressively with strict causal ordering; evicting a text token risks corrupting all subsequent representations.
Shifting Bottlenecks: In typical single-turn VLM configurations, vision tokens dominate the input (e.g., 2,500 vision tokens vs. ~30 text tokens), making them the primary bottleneck during the prefill phase. However, in free-form generation or multi-turn dialogue, text tokens accumulate rapidly, shifting the bottleneck to the decoding phase.
Variable Input Difficulty: The optimal pruning ratio is not uniform across samples. Some queries require only localized visual information (allowing aggressive pruning), while others demand global scanning (requiring conservative retention). Existing methods using fixed pruning ratios fail to accommodate this spectrum.

Methodology: AsymVLM

The authors propose AsymVLM, an asymmetric token pruning framework that decouples compression strategies for vision and text tokens based on the observations above. The framework operates in two distinct stages:

1. Vision Token Pruning (Pre-Prefill)

Applied before the transformer layers to address the prefill bottleneck, this stage consists of two components:

Learned Importance Scorer: Unlike prior methods relying on visual saliency or raw cross-modal similarity, AsymVLM employs a lightweight, learnable scorer. This scorer re-scales embedding dimensions using a learnable weight vector ( $w$ ) to better align token rankings with the preservation of the model's final output. It is trained to minimize the discrepancy between the hidden states of pruned and unpruned models.
Per-Sample Adaptive Budgeting: Recognizing that importance gaps vary across inputs, the system calculates an "importance gap" (the difference between the 75th and 25th percentile importance scores) for each sample. Based on this gap, a dynamic keep ratio is assigned: samples with a large gap (clear distinction between important and unimportant tokens) receive an aggressive pruning ratio, while samples with a small gap (uniform importance) receive a conservative ratio.

2. Text Token Eviction (During Decoding)

Applied only when the generated sequence exceeds a fixed budget, this stage addresses the decoding bottleneck:

Threshold-Based Eviction: Text tokens are retained until the sequence length exceeds a predefined threshold. Once triggered, the system evicts the lowest-importance generated tokens.
VLM-Specific Scoring: Unlike standard LLM cache compression methods that track cumulative attention over long contexts, AsymVLM leverages the short-context nature of VLMs. It scores tokens based on the attention received at the final transformer layer of the current decoding step. Crucially, it protects all prefill tokens (images and prompts) from eviction, as these carry irreplaceable context for subsequent turns.

Key Contributions

Systematic Analysis: The paper provides a structural analysis revealing the fundamental asymmetry between vision and text tokens in VLMs, arguing that modality-aware compression is essential for efficiency.
Dual-Strategy Framework: AsymVLM introduces a framework combining query-aware vision token pruning (pre-transformer) with budget-constrained text token eviction (during decoding).
Learned Scorer with Adaptive Budgeting: The authors introduce a scorer that optimizes for output preservation rather than just semantic similarity, coupled with a mechanism that dynamically adjusts pruning ratios based on the input's importance distribution.
Comprehensive Evaluation: Extensive experiments demonstrate that the approach achieves superior efficiency and accuracy compared to state-of-the-art baselines.

Experimental Results

The framework was evaluated on Phi-3.5-Vision (vision-dominant) and Gemma 3 (text-dominant in long-generation scenarios) across seven benchmarks (DocVQA, ChartQA, MME, TextVQA, OCRVQA, LLaVA-Bench, MMDU).

Efficiency: AsymVLM achieves the highest FLOPs savings among compared methods, reaching up to 54% at 50% vision token retention. It is the only method that physically reduces the sequence length before the transformer, resulting in proportional reductions in peak memory and KV cache size (up to 75% peak memory reduction).
Accuracy on Localized Tasks: On document and chart understanding tasks (DocVQA, ChartQA), where visual information is spatially localized, AsymVLM outperforms existing methods (FastV, SparseVLM) by 2–3% at 65–75% retention ratios. For instance, on DocVQA at 65% retention, it achieves 85.07% ANLS compared to 81.74% for FastV.
Holistic Tasks: On holistic benchmarks (MME, TextVQA), where token importance is more uniform, AsymVLM maintains competitive accuracy, often slightly outperforming baselines.
Text Eviction Performance: In text-dominated scenarios (Gemma 3), the threshold-based eviction strategy substantially outperforms standard LLM cache methods (H2O, StreamingLLM). On MMDU (multi-turn dialogue), AsymVLM achieves a score of 7.03 at 90% retention, significantly higher than the 5.61 achieved by baselines. The baselines degrade severely because they treat vision and prompt tokens as eviction candidates, whereas AsymVLM protects them.

Significance and Claims

The paper claims that AsymVLM represents a shift from uniform compression to modality-aware strategies. By acknowledging that vision tokens are an input selection problem and text tokens are a cache management problem, the framework achieves the highest efficiency gains (up to 54% FLOPs reduction) while maintaining or improving accuracy on complex, query-specific tasks. The authors emphasize that their learned scorer generalizes effectively across different visual layouts and question types without task-specific tuning, and that their text eviction strategy is specifically tailored to the short-context, image-grounded nature of VLM inference, avoiding the pitfalls of applying long-context LLM compression techniques directly to VLMs.

Limitations: The authors note that the evaluation is currently limited to Phi-3.5-Vision and Gemma 3, and generalization to other architectures (e.g., LLaMA-based VLMs, Qwen-VL) requires further validation. Additionally, the text eviction strategy uses a fixed threshold and has not been tested on very long multi-turn dialogues (50+ turns).