💻 computer science

Hybrid Token Compression for Vision-Language Models

This paper introduces HTC-VLM, a hybrid visual token compression framework that effectively balances the preservation of fine-grained appearance details and high-level semantics by fusing continuous patch features with discrete semantic anchors, achieving superior performance retention and efficiency compared to existing continuous baselines.

Original authors: Jusheng Zhang, Xiaoyang Guo, Tongyu Mo, Qinhan Lv, Wenhao Chai, Jian Wang, Keze Wang, Liang Lin

Published 2026-08-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jusheng Zhang, Xiaoyang Guo, Tongyu Mo, Qinhan Lv, Wenhao Chai, Jian Wang, Keze Wang, Liang Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Visual Overload: Why AI Needs a Better Summary

Imagine you are trying to teach a brilliant but very hungry student how to understand a picture. In the world of Artificial Intelligence, this student is called a "Vision-Language Model" (VLM). It's a type of computer brain that can look at an image and answer questions about it, like "What is the dog wearing?" or "Why is the sky blue?" To do this, the computer breaks the image down into hundreds of tiny squares, called "patches," and turns each one into a long list of numbers (a token) that it can read.

The problem is that the computer gets overwhelmed. If you feed it 576 of these tiny squares, the brain has to compare every single square to every other square to understand the picture. This creates a massive amount of work, like trying to read a library of books all at once. It slows everything down and eats up huge amounts of computer memory. Scientists have been trying to fix this by summarizing the image into just one or a few "summary tokens," but there's a catch. If you just mash all the details into one average number, you lose the important story (like the fact that it's a dog). If you try to keep the story by picking specific "keywords," you lose the fine details (like the dog's fur texture). It's a frustrating trade-off: you either get the gist but miss the details, or you get the details but forget the main point. This paper asks a simple question: Can we have both?

The Hybrid Hero: HTC-VLM

The researchers behind this paper, led by Jusheng Zhang and his team, say the answer is yes, but only if we stop trying to force the computer to do everything with one single tool. They introduce a new method called HTC-VLM (Hybrid Token Compression for Vision-Language Models). Think of it as a clever way to summarize a complex story without losing the plot or the character details.

Here is how their "Hybrid" trick works, using a simple analogy:

Imagine you are describing a chaotic scene at a park to a friend who can only hear one sentence.

The Old Way (Continuous Compression): You try to squeeze everything into one sentence: "There are many things moving with colors and shapes." Your friend hears the noise but has no idea what is actually happening. They know it's busy, but they don't know what is busy. This is what happens when AI tries to squash an image into one average number; the "high-level meaning" (like "it's a dog") gets washed out by the "noise" (like the grass and the sky).
The Other Old Way (Discrete Quantization): You try to be precise and say, "Dog. Grass. Tree." Your friend knows the main actors, but they have no idea what the dog looks like, if it's running, or what color its fur is. They miss the texture and the action.
The HTC-VLM Way (The Hybrid Solution): The researchers suggest a two-step approach. First, you give your friend four special "anchor" cards that say exactly what the main things are (e.g., "Dog," "Grass," "Tree"). These are the discrete tokens. They act as a skeleton or a map. Then, you give them one single "summary" token that holds all the messy, fine-grained details (the fur texture, the wind in the grass, the pose of the dog).

The magic happens because the computer is trained to use the "anchor cards" to guide how it reads the "summary token." The anchors tell the computer, "Hey, look for a dog here!" so the summary token doesn't get confused by all the other details. This way, the AI keeps the big picture (the semantics) and the tiny details (the appearance) separate until the very last moment, where they are fused together perfectly.

What They Found

The team tested this idea on seven different challenging tests, ranging from answering questions about images to understanding science diagrams. They compared their new method against the best existing ways of compressing images.

The Results: When they squeezed the entire image down to just one single token (the ultimate compression), their hybrid method kept 87.2% of the original performance. The previous best method (which only used the "average" approach) only kept 81.0%. While that might sound like a small difference, in the world of AI, keeping that extra 6% of intelligence when you've thrown away almost all the data is a huge deal.
Why It Works: The researchers analyzed how the computer's "attention" worked. They found that the single summary token was actively looking at the four "anchor" cards first. It was using those anchors as a guide to make sense of the rest of the image. This proved that the anchors were doing the heavy lifting of keeping the meaning intact.
The Trade-off: The method does require a tiny bit of extra work to generate those four anchor cards, but the researchers showed that the time saved by not having to process hundreds of tokens is so massive that the overall speed is still incredibly fast. It's like spending 5 seconds to write a great outline, which saves you 5 hours of writing a bad essay.

What It's Not

It is important to note what this paper does not claim. The researchers explicitly argue against the idea that you can just "average" everything together and expect it to work well under extreme compression. They showed that trying to force a single continuous number to hold both the "story" and the "details" causes the story to collapse. They also showed that simply picking random pieces of the image or using older "pruning" methods (cutting out pieces) doesn't work as well when you get down to just one or two tokens.

The paper suggests that this hybrid approach is a robust solution for the specific problem of extreme compression, but it doesn't claim to solve every problem in AI. For instance, they note that while it works great for single images, it might need adjustments for longer videos or complex multi-image conversations.

The Bottom Line

In short, HTC-VLM suggests that to make AI smart and fast, we shouldn't just try to shrink the image into a tiny dot. Instead, we should give the AI a map (the discrete anchors) and a briefing (the continuous details) and let them work together. By separating the "what" from the "how," the computer can understand a picture with just one token, keeping the meaning clear and the details sharp, without getting lost in the noise. It's a reminder that sometimes, to save space, you don't need to throw things away; you just need to organize them better.

Technical Summary: HTC-VLM – Hybrid Token Compression for Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) typically rely on hundreds of visual patch tokens (e.g., $N=576$ from a ViT encoder) to provide rich perceptual information to Large Language Models (LLMs). This dense coupling incurs quadratic computational costs, $O((N+L)^2)$ , rapidly exhausting GPU memory and context windows.

Existing compression strategies face a fundamental trade-off:

Continuous Compression: Methods like global pooling or attention aggregation compress all patches into a single dense vector. While efficient, this often leads to semantic dilution, where high-level semantics (object identities, spatial relations) are lost as mutual information $I(v_c; S)$ drops due to the averaging of diverse patch distributions.
Discrete Quantization: Methods using Vector Quantization (VQ) preserve categorical semantics but discard fine-grained continuous cues (textures, poses, deformations), creating a granularity gap that limits detailed reasoning.

The core challenge is whether a VLM can retain semantically useful visual information when the entire image is compressed to a single token without sacrificing either high-level structure or low-level detail.

2. Methodology: HTC-VLM

The authors propose HTC-VLM (Hybrid Token Compression for Vision-Language Models), a framework that disentangles visual information into two complementary channels before fusing them into a single latent token.

2.1 Dual-Channel Architecture

Instead of compressing raw patches directly, HTC-VLM decomposes the visual input $I$ into:

Continuous Detail Channel ( $D$ ): Utilizes a pretrained vision encoder (CLIP ViT-L/14) and a linear projector to generate $N=576$ patch embeddings. This pathway preserves low-level details such as texture gradients and pose variations.
Discrete Semantic Channel ( $S$ ): Utilizes MGVQ (Multi-Group Vector Quantization) to quantize the image into a feature vector, which is then projected via an MLP into a small sequence of 4 discrete semantic anchor tokens ( $v_d$ ). These anchors capture high-level categorical semantics (e.g., "dog," "tree") and spatial layouts.

2.2 Hybrid Fusion and Disentanglement Bottleneck

The discrete anchors are prepended to the continuous patch sequence to form a hybrid sequence $V_{hy} \in \mathbb{R}^{580 \times d}$ (4 anchors + 576 patches). This sequence is fed into a trainable <voco> token.

A critical component is the Disentanglement Attention Mask ( $M_{hy}$ ):

Text-to-Vision Constraint: Text tokens are strictly forbidden from attending to the raw hybrid visual tokens ( $V_{hy}$ ). They can only attend to the compressed <voco> token.
Visual Self-Attention: The visual tokens (both discrete and continuous) are allowed to attend to one another before aggregation. This allows the <voco> token to integrate contextualized visual evidence.
Latent Extraction: The <voco> token compresses the 580-token sequence into a single latent vector $z$ , which serves as the sole visual input to the LLM.

2.3 Theoretical Foundation

The method is grounded in the Information Bottleneck (IB) principle. The authors argue that a single continuous vector suffers from Entropy Domination, where high-frequency details ( $D$ ) saturate the limited capacity of the bottleneck, forcing semantic information ( $S$ ) to vanish. By introducing discrete anchors as a "semantic prior," HTC-VLM lowers the entropy barrier for $S$ , allowing the latent $z$ to focus its capacity on encoding the residual details $D$ . This is formalized via a tighter Evidence Lower Bound (ELBO) compared to continuous-only baselines.

3. Key Contributions

Representational Bottleneck Analysis: The paper identifies the "semantic-detail capacity conflict" in single-token compression, providing information-theoretic proofs (Theorem 1) that explain why purely continuous bottlenecks fail to preserve high-level semantics under extreme compression.
Hybrid Semantic-Disentanglement Framework: A principled architecture that injects minimal discrete semantic anchors prior to compression, enabling the joint optimization of discrete semantics and continuous appearance.
Practical Architecture (HTC-VLM): An instantiation achieving 87.2% average performance retention across seven benchmarks under a one-token budget, outperforming the leading continuous baseline (VoCo-LLaMA) at 81.0%.
Extended Validation: Comprehensive studies on token budgets (1–576), cross-architecture generalization (Qwen-VL, LLaVA-NeXT), inference efficiency, and robustness to hyperparameters (codebook size, masking topology).

4. Experimental Results

4.1 Performance Retention

Evaluated on seven benchmarks (GQA, VQAv2, MMBench, MME, POPE, SEED-Bench, ScienceQA-Image), HTC-VLM achieved an average retention of 87.2% relative to the full 576-token model. This significantly outperforms:

VoCo-LLaMA (Continuous-only): 81.0%
Average Pooling: 64.1%
Q-Former: 57.2%

4.2 Token Budget Robustness

While structured pruning methods (e.g., ToMe, PDrop) perform well at moderate compression (64–192 tokens), they degrade sharply in the ultra-low token regime ( $<32$ tokens). HTC-VLM remains stable down to the 1-token regime, demonstrating that discrete anchors prevent information collapse where continuous methods fail.

4.3 Generalization

The hybrid bottleneck transfers effectively to stronger and structurally different backbones:

Qwen-VL-Chat: 94.5% retention (vs. 86.4% for VoCo-LLaMA).
LLaVA-NeXT (Dynamic Resolution): 95.7% retention (vs. 86.6% for VoCo-LLaMA).

4.4 Efficiency and Latency

On an NVIDIA A100 GPU, HTC-VLM reduces the LLM prefill latency by 94.3% compared to the vanilla 576-token model. While the addition of the MGVQ encoder introduces a modest overhead (~6.1ms), the total system latency (61.0ms) remains close to the continuous-only baseline (51.9ms) and is significantly faster than pruning-based methods that retain multiple tokens.

4.5 Ablation Studies

Discrete Anchors: Removing anchors (Continuous-only) drops performance to 81.0%; using only discrete tokens drops it to ~33%.
Anchor Count: 4 tokens yield optimal performance; 1 or 2 are insufficient, while 8 introduces redundancy.
Masking: The text-bottleneck mask (allowing visual self-attention but blocking text-to-raw-visual) is superior to a strict star-graph mask, confirming the need for visual context before compression.

5. Significance and Claims

The paper claims that HTC-VLM offers a practical solution to the efficiency-fidelity trade-off in VLMs. By explicitly disentangling semantics and details, the hybrid approach allows for extreme compression (1 token) without the semantic collapse observed in continuous-only methods.

The authors emphasize that the performance gains stem from the structural hybrid design rather than fragile hyperparameter tuning. The discrete anchors act as interpretable semantic guides, stabilizing the compression process and enabling the LLM to reason effectively even when the visual input is reduced to a single token. The work suggests a future direction for multimodal compression: designing structured, task-adaptive bottlenecks that preserve semantic interpretability while drastically reducing token costs.

Limitations Acknowledged: The method introduces a lightweight MGVQ encoder, meaning it is not strictly zero-overhead compared to continuous-only compression. Additionally, the current evaluation focuses on image-level VLMs; extending this to long video or dense localization tasks may require further adaptation.