💻 computer science

UniCompress: Token Compression for Unified Vision-Language Understanding and Generation

This paper proposes UniCompress, a lightweight and modular token compression framework that utilizes learnable global meta tokens to significantly reduce visual token counts in unified vision-language models, thereby lowering computational overhead and enabling efficient deployment in resource-constrained scenarios while maintaining high performance in both understanding and generation tasks.

Original authors: Ziyao Wang, Chen Chen, Jingtao Li, Weiming Zhuang, Jiabo Huang, Ang Li, Lingjuan Lyu

Published 2026-03-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ziyao Wang, Chen Chen, Jingtao Li, Weiming Zhuang, Jiabo Huang, Ang Li, Lingjuan Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a massive, high-definition movie to a friend who has a very slow internet connection and a tiny phone. If you send the whole movie, it will take forever to download and might crash their phone. But if you just send a blurry thumbnail, they can't see the details.

This is the exact problem computer scientists are facing with Unified AI models. These are smart AI systems that can both look at an image and describe it (Understanding) and look at a description and draw a picture (Generation).

The Problem: Too Much "Visual Noise"

Currently, these AI models treat images like a giant grid of tiny Lego bricks (called tokens).

To understand a picture, the AI needs to look at all 1,024 bricks.
To draw a picture, it needs to place all 1,024 bricks back together perfectly.

This is like trying to carry a whole library of encyclopedias in your backpack just to read one sentence. It's heavy, slow, and expensive. If you try to simply throw away half the bricks to make it lighter, the AI gets confused: it can still guess the general topic, but the picture it draws becomes a blurry mess.

The Solution: UNICOMPRESS (The "Smart Summarizer")

The paper introduces a new tool called UNICOMPRESS. Think of it as a super-smart travel agent that helps you pack your visual "suitcase" efficiently without losing anything important.

Here is how it works, using a simple analogy:

1. The "Global Meta Tokens" (The Tour Guide)

Imagine you are looking at a complex cityscape. Instead of remembering every single window on every building, you hire a Tour Guide (the Global Meta Token).

This guide looks at the whole city and gives you a quick summary: "It's a sunny day, there's a big wind turbine, and a person in a yellow jacket is climbing it."
This guide captures the big picture and the relationships between objects.

2. The "Compressor" (The Packing Cube)

Next, the AI takes the original 1,024 Lego bricks and squishes them into a smaller box (4x smaller).

Normally, squishing bricks breaks them.
But UNICOMPRESS uses Average Pooling. Imagine taking a 4x4 grid of bricks and melting them down into one single, slightly larger brick that represents the average color and shape of that area. It keeps the general shape but removes the tiny, unnecessary details.

3. The "Decompressor" (The Magic Reconstructor)

This is the magic trick. When the AI needs to draw the picture again, it doesn't just guess.

It looks at the Tour Guide's summary (the Global Meta Tokens) to remember the big rules: "Keep the sky blue, keep the person on the turbine."
Then, it uses the squeezed bricks to fill in the details.
Because it has the Tour Guide's instructions, it knows exactly how to expand the small box back into a full, high-quality image without losing the important details.

Why is this a Big Deal?

1. It's a "Plug-and-Play" Upgrade
You don't need to rebuild the entire AI brain. You just snap this new "packing module" onto the existing system, like adding a new lens to a camera. It works with almost any current AI model.

2. It's 4x Faster and Smaller
By reducing the number of visual bricks from 1,024 to just 256, the AI:

Thinks 40% faster (less data to process).
Uses less memory (fits on smaller, cheaper devices).
Trains cheaper (saves money on electricity and computing power).

3. It Doesn't Sacrifice Quality
Previous attempts to shrink images made the AI bad at drawing. UNICOMPRESS is special because the Tour Guide ensures the AI remembers the "soul" of the image.

Understanding: It can still read the image perfectly.
Generation: It can still draw sharp, detailed pictures, not just blurry blobs.

The Bottom Line

UNICOMPRESS is like a translator that speaks two languages: "Big Picture" and "Fine Details." It allows AI to carry a lighter load while still seeing the world clearly and creating beautiful art. This means we can soon run powerful, creative AI on our phones, robots, and laptops without needing a supercomputer in the cloud.

1. Problem Statement

Unified Vision-Language Models (VLMs) aim to handle both image understanding and image generation within a single autoregressive framework by encoding images into discrete tokens. While this architecture offers simplicity and cross-modal synergy, it suffers from a critical bottleneck: token inefficiency.

High Computational Cost: Standard discrete tokenizers (e.g., VQ-VAE, VQGAN) map a $512 \times 512$ image to $1024$ tokens ( $32 \times 32$ ). Processing these long sequences increases memory footprint, training costs, and inference latency, making deployment on resource-constrained devices (e.g., embodied AI) difficult.
The Compression Dilemma: Naïve compression methods (like uniform downsampling or token pruning) work well for understanding tasks but severely degrade generation performance (often by $>15\%$ ). Generation requires fine-grained, spatially consistent tokens to reconstruct details, making it highly sensitive to information loss.
Retraining Costs: Training a new, more efficient tokenizer from scratch requires retraining the entire downstream language model, which is computationally expensive and impractical.

Goal: Develop a plug-in, modular compression framework that reduces visual token counts significantly (e.g., $4\times$ ) while maintaining high performance on both understanding and generation tasks, without requiring full model retraining.

2. Methodology: UNICOMPRESS

The authors propose UNICOMPRESS, a lightweight, modular framework that augments existing discrete tokenizers with three key components. It operates in a "compress once, reuse for both" manner.

A. Architecture Overview

The framework inserts three modules around a standard discrete tokenizer:

Global Token Extractor: Uses one-way cross-attention to summarize scene-level semantics into a small set of learnable global meta tokens ( $G$ ).
Token Compressor: Aggregates local visual tokens using average pooling over non-overlapping spatial patches (e.g., $2 \times 2$ or $4 \times 4$ ) to create a shorter sequence of compressed local tokens ( $\hat{X}$ ).
Global-Guided Autoregressive Decompressor: A Transformer decoder that reconstructs the dense, high-fidelity token grid from the compressed local tokens, conditioned on the global meta tokens.

B. Workflow

Tokenization: An input image is encoded into a dense grid of tokens ( $X$ ).
Extraction & Compression:
- Global Tokens: Learnable query tokens attend to the full image grid to extract holistic semantics ( $G$ ).
- Local Compression: The grid $X$ is reshaped and downsampled via average pooling to produce a compact sequence $\hat{X}$ .
- Sequence Construction: The final sequence for the Language Model (LLM) is ordered as: [IMG_BOS], Global Tokens ( $G$ ), [IMG_SEP], Compressed Local Tokens ( $\hat{X}$ ), [IMG_EOS].
Dual-Mode Operation:
- Understanding: The LLM consumes the continuous compressed features directly.
- Generation: The LLM autoregressively predicts discrete indices for both global and local tokens. These indices are de-quantized and fed into the Decompressor, which expands them back to a dense grid for the image decoder.

C. Training Pipeline

The method uses a two-stage training process to avoid full retraining:

Tokenizer Training: The LLM is frozen. The tokenizer stack (Extractor, Compressor, Codebook, Decompressor) is trained using a reconstruction loss ( $\mathcal{L}_{recon}$ ) to ensure high-fidelity recovery of the original image from the compressed representation.
LLM Fine-tuning: The compressed tokenizer is frozen. The LLM is lightly fine-tuned on the new compact token sequences for both understanding and generation tasks.

3. Key Contributions

Identification of the Bottleneck: The paper highlights that token efficiency is a major barrier for unified models and demonstrates that naïve compression disproportionately harms image generation.
UNICOMPRESS Framework: A novel, plug-and-play solution featuring global-guided autoregressive decompression. It preserves spatial structure and long-range dependencies crucial for generation while reducing sequence length.
Modularity: The approach integrates seamlessly into existing unified models (e.g., VILA-U, UniTok, VARGPT) without modifying the LLM backbone or requiring full retraining.
Empirical Validation: Extensive experiments show up to $4\times$ token reduction with minimal performance degradation ( $\le 3\%$ drop in understanding, $\le 5\%$ FID increase in generation).

4. Experimental Results

The method was evaluated on multiple unified models (UNITOK, VILA-U, VARGPT, UNIFORK, OPENUNI, BAGEL) across standard benchmarks.

Understanding Performance:
- Compression caused only marginal drops in accuracy. For example, on the GQA benchmark, UNITOK dropped from $55.71$ to $53.07$, and POPE from $82.66$ to $79.36$.
- Performance remained robust across diverse architectures, including those using diffusion heads.
Generation Performance:
- While token compression naturally introduces some detail loss, the global-guided decompression mitigated this significantly.
- UNITOK: FID increased slightly from $16.14$ to $16.33$.
- VILA-U: FID increased from $14.80$ to $16.37$.
- BAGEL: Maintained competitive quality ( $12.73 \to 17.22$ FID).
- Crucially, the method preserved semantic alignment (CLIP scores) much better than naïve downsampling.
Efficiency Gains:
- Inference Latency: Achieved up to $41.8\%$ reduction in inference time. For example, UNITOK generation inference dropped from $32.25$ minutes to $18.96$ minutes.
- Training Cost: Reduced training time by up to $15.4\%$ .
- Token Reduction: Successfully reduced visual tokens by $4\times$ (e.g., $256 \to 64$ tokens).
Ablation Studies:
- Global Tokens: Learnable global meta tokens significantly outperformed mean-pooling or CLS tokens for generation quality, proving the necessity of explicit global guidance.
- Decompression: Autoregressive decompression was essential; non-autoregressive (naïve) decompression resulted in over-smoothed textures and artifacts.
- Compression Ratio: A $4\times$ reduction ( $s=2$ ) offered the best trade-off; further compression ( $1/16$ ) caused sharp declines in generation quality.

5. Significance

UniCompress addresses a critical gap in the deployment of unified multimodal models. By enabling a $4\times$ reduction in visual tokens with negligible performance loss, it makes unified models viable for resource-constrained environments (e.g., edge devices, robotics).

Unlike previous methods that optimized only for understanding or required expensive retraining, UniCompress offers a universal, plug-in solution that accelerates both training and inference for both understanding and generation tasks. This paves the way for scalable, efficient, and high-quality unified AI systems capable of real-world application.