🤖 AI

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

DUET-VLM is a versatile, dual-stage framework that combines vision-only redundancy-aware compression with salient text-guided token dropping to achieve aggressive visual token reduction (up to 89% for images and 93.4% for video) while maintaining or even surpassing baseline accuracy in vision-language models.

Original authors: Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma, Zicheng Liu, Emad Barsoum

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma, Zicheng Liu, Emad Barsoum

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart assistant (a Vision-Language Model) who is great at answering questions about pictures and videos. But there's a problem: this assistant is incredibly hungry.

Every time you show it a picture, it tries to eat every single pixel as a separate bite of food. A high-resolution photo might have thousands of these "bites" (tokens). If you show it a video, the hunger becomes insatiable. This makes the assistant slow, expensive to run, and hard to fit into your phone or laptop.

The paper introduces DUET-VLM, a new way to feed this assistant so it stays full and smart, but eats much less food. Think of it as a two-stage "smart diet" for AI.

The Problem: The "Buffet" of Pixels

Currently, when an AI looks at a photo, it sees a massive buffet of 576 (or even 2,800+) tiny food items. It tries to chew on every single one, even the boring background pixels that don't matter. This wastes energy and time.

Previous attempts to fix this were like a clumsy waiter:

Method A: "Let's just mash all the food together into a big smoothie." (Merging too early). Result: You lose the flavor of specific details.
Method B: "Let's just throw away half the plate randomly." (Dropping tokens uniformly). Result: You might accidentally throw away the most important part of the meal.

The Solution: The "DUET" Diet

The authors propose a Dual-Stage approach, like a two-step filter system.

Stage 1: The "Smart Sorter" (Vision Encoder)

Before the food even reaches the main brain, it goes through a Smart Sorter.

The Analogy: Imagine you have a pile of 500 Lego bricks. Some are the main structure of a castle (the "Dominant" tokens), and some are just loose, identical red bricks scattered on the floor (the "Redundant" tokens).
What DUET does: Instead of keeping all 500, it picks out the 50 most important bricks that hold the castle together. Then, it takes the remaining 450 loose bricks and glues them into 10 small, neat bundles (clusters).
The Result: Instead of 500 individual bricks, you now have 60 items (50 key bricks + 10 bundles). You haven't lost the shape of the castle, but you've reduced the clutter by 88%.

Stage 2: The "Contextual Chef" (Language Backbone)

Now, the reduced food goes to the main brain (the Language Model), which is reading a question like, "What number is on the player's jersey?"

The Analogy: The brain is a chef reading a recipe. It knows it doesn't need to taste the whole soup to know if it's salty; it just needs to taste the spoonful near the salt shaker.
What DUET does: As the AI reads the question, it looks at the 60 items from Stage 1. It asks, "Which of these items helps me answer the question?"
- If the question is about a jersey number, the AI says, "Ignore the background trees and the sky. Keep the player and the jersey."
- It progressively drops the irrelevant bundles and bricks as it digs deeper into the reasoning process.
The Result: By the time the AI gives an answer, it might only be looking at 10 or 20 items, but it has kept the exact ones needed to solve the puzzle.

Why is this a Big Deal?

The paper tested this on famous AI models (like LLaVA and Video-LLaVA) and found amazing results:

It's Super Efficient: They reduced the amount of data the AI had to process by 67% to 89%. That's like cutting your grocery bill by 80% without going hungry.
It's Still Super Smart: Even with so much less data, the AI kept 99% of its accuracy. In some cases, because it wasn't distracted by "junk food" (redundant pixels), it actually performed better than before.
It Works on Videos Too: Videos are huge data monsters. DUET-VLM managed to cut video data by over 90% and still understood the story perfectly.

The Bottom Line

DUET-VLM is like giving your AI a pair of smart glasses. Instead of staring at the entire world with wide, unfocused eyes, it learns to zoom in on the important details and ignore the noise.

Old Way: "I see everything, so I must think about everything." (Slow, expensive).
DUET Way: "I see the important stuff, so I only think about that." (Fast, cheap, and just as smart).

This means we can run powerful AI models on smaller devices, faster, and cheaper, without sacrificing how smart they are.

1. Problem Statement

Vision-Language Models (VLMs) have achieved remarkable reasoning capabilities but suffer from high computational costs due to dense visual tokenization.

The Bottleneck: High-resolution images generate thousands of visual tokens (e.g., LLaVA-NeXT generates >2,800 tokens for a single image), leading to quadratic attention costs ( $O(N^2)$ ) in transformers. This creates memory and latency bottlenecks.
Limitations of Existing Methods: Current efficiency approaches generally fall into two categories, both with significant drawbacks:
- Vision-side compression (Early merging): Methods like VisionZip or PruMerge merge tokens early in the vision encoder. While they reduce redundancy, they often risk information loss by merging too early or using static/heuristic strategies that lack semantic adaptivity.
- Language-side pruning (Late dropping): Methods like PyramidDrop or FastV drop tokens progressively within the language backbone. However, they often drop tokens uniformly or lack a mechanism to jointly optimize the initial visual representation with the linguistic context.
The Gap: There is a lack of a unified framework that jointly optimizes redundancy removal and context-aware retention across both the vision and language backbones.

2. Methodology: DUET-VLM

The authors propose DUET-VLM, a versatile, plug-and-play dual-stage compression framework that operates in two complementary stages to manage visual tokens efficiently.

Stage 1: Vision-to-Vision (V2V) Merging (Redundancy-Aware Compression)

This stage occurs within the Vision Encoder before cross-modal fusion.

Dominant Token Selection: Using the self-attention map from the vision encoder (e.g., CLIP), the model identifies a subset of $k_1$ "dominant" tokens that have high visual-to-visual affinity.
Local Cluster Aggregation (Novelty): Instead of globally averaging the remaining residual tokens (which dilutes fine-grained features), DUET-VLM partitions the residual tokens into $k_2$ $k_{2}$ small clusters.
- Centroid Selection: Cluster centroids are selected based on the ranking of V2V self-attention scores.
- Local Averaging: Neighbors within a fixed cluster width $w$ are averaged to form a single "contextual token."
- Early Dropping: Tokens that do not fall into these local clusters are dropped immediately, reducing the token count before entering the language model.
Goal: To preserve structural diversity and fine-grained cues while removing background noise.

Stage 2: Text-to-Vision (T2V) Pruning (Saliency-Guided Dropping)

This stage occurs within the Language Backbone (LLM) during inference or training.

Salient Token Identification: The model identifies a subset of "salient" text tokens ( $S$ ) from the input prompt. Crucially, the last text token (sink token) is always included to stabilize attention, but the method also experiments with using other salient tokens based on the query.
Layer-wise Progressive Pruning: The visual tokens are processed through the LLM layers. At specific "stages" (layers), the model computes Text-to-Vision (T2V) cross-attention scores between the salient text tokens and the retained visual tokens.
Rank-and-Drop: Visual tokens are ranked by their attention scores. The least informative tokens are progressively dropped ( $\lfloor \lambda \cdot N_l \rfloor$ ) at each stage, ensuring that deeper layers attend only to semantically relevant regions defined by the text query.

3. Key Contributions

Joint Vision-Language Optimization: Unlike prior one-sided methods, DUET-VLM couples the vision and language backbones through a differentiable interface. This allows visual representations and linguistic context to co-adapt during token reduction, maintaining strong cross-modal alignment.
Dual-Stage Visual Compression: The framework unifies early-stage structural merging (V2V) with late-stage semantic pruning (T2V). This ensures low-level details are preserved early on, while redundancy is removed as reasoning deepens.
Local Cluster Aggregation: A novel modification to token merging that replaces global averaging with local neighborhood aggregation, preventing the dilution of semantically informative but low-frequency features.
Training-Aware Efficiency: The method is designed to be effective not just for inference but also for training, enabling robust adaptation to reduced visual inputs without sacrificing accuracy.

4. Experimental Results

The authors evaluated DUET-VLM on LLaVA-1.5-7B, LLaVA-NeXT-7B, Qwen-2.5-VL-7B, and Video-LLaVA-7B across multiple benchmarks (POPE, GQA, TextVQA, MME, SQA, and video QA datasets).

Inference-Only Performance (LLaVA-1.5-7B):
- 67% Token Reduction (192 tokens): Maintains 99.0% of baseline accuracy (outperforming VisionZip at 97.7% and PyramidDrop at 96.4%).
- 89% Token Reduction (64 tokens): Retains 95.4% of baseline accuracy.
- Speedup: Achieves up to 1.16× speedup in inference.
Training Performance:
- When trained with DUET-VLM, the model achieves 99.7% accuracy at 67% reduction and 97.6% at 89% reduction, surpassing prior State-of-the-Art (SoTA) methods.
- Training Time: Reduces training time by 31% (at 128 tokens) while maintaining >99% accuracy.
Video Understanding (Video-LLaVA-7B):
- At 53.1% token reduction, DUET-VLM surpasses the baseline, achieving >100% accuracy.
- At an extreme 93.4% reduction, it retains 97.6% accuracy, significantly outperforming other methods (which drop to ~45-91%).
Generalization: The method generalizes well to newer architectures like Qwen-2.5-VL-7B, maintaining >98% accuracy even at high compression rates.

5. Significance

Paradigm Shift: DUET-VLM demonstrates that smart token management can match or even surpass traditional model scaling in terms of accuracy. It proves that aggressive compression does not require a trade-off with performance if the compression is context-aware and dual-staged.
Efficiency: It offers a practical path to deploying compact, high-performing VLMs within the same computational budget, addressing the memory and latency constraints of current LVLMs.
Versatility: As a "plug-and-play" framework, it can be integrated into various VLM architectures (image and video) without requiring fundamental architectural changes, making it highly scalable for future multimodal systems.

In conclusion, DUET-VLM bridges the gap between visual redundancy and semantic relevance, offering a robust solution for efficient multimodal learning that preserves critical information while drastically reducing computational overhead.