🤖 machine learning

A More Word-like Image Tokenization for MLLMs

This paper proposes Disentangled Visual Tokenization (DiVT), a novel method that clusters image patch embeddings into coherent semantic units and dynamically adjusts token budgets based on image complexity, enabling multimodal large language models to process visual inputs more efficiently and compatibly with significantly reduced memory and latency costs.

Original authors: Hyun Lee, Hyemin Jeong, Yejin Kim, Hyungwook Choi, Hyunsoo Cho, Soo Kyung Kim, Joonseok Lee

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hyun Lee, Hyemin Jeong, Yejin Kim, Hyungwook Choi, Hyunsoo Cho, Soo Kyung Kim, Joonseok Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but very literal robot (a Large Language Model) how to "see" pictures. Currently, the way we do this is a bit clumsy.

The Problem: The "Grid" vs. The "Story"

Think of a standard digital photo as a giant grid of tiny squares (pixels). Right now, when we feed a photo to a robot, we chop the image into fixed-size squares (like a checkerboard) and hand the robot a long, boring list of these squares.

The Robot's Perspective: The robot is used to reading words. Words are distinct, meaningful units (like "cat," "run," or "blue").
The Current Method: The robot is forced to read a long, repetitive list of "square-1," "square-2," "square-3," even if those squares are all just empty sky or part of the same wall.
The Result: The robot gets overwhelmed by a flood of redundant, confusing data. It's like trying to describe a beautiful painting by listing the color of every single brick in the frame, rather than saying "a red rose." This wastes the robot's memory and makes it slow.

The Solution: DiVT (Disentangled Visual Tokenization)

The authors of this paper propose a new method called DiVT. Instead of chopping the image into a rigid grid, DiVT acts like a smart editor that looks at the picture and says, "Okay, what are the actual important things here?"

Here is how it works, using a simple analogy:

1. The "Grouping" Game (Clustering)
Imagine you have a messy room full of scattered toys.

Old Way: You pick up every single toy one by one and put it in a box, regardless of whether it's a Lego, a doll, or a sock. You end up with 500 tiny boxes.
DiVT Way: You look at the toys and group them by what they are. You put all the Legos in one pile, all the dolls in another, and the socks in a third. You only create a "token" (a box) for each distinct group.
- If the room is empty, you only make a few boxes.
- If the room is full of complex toys, you make more boxes.
- The Magic: The number of boxes automatically adjusts to how "busy" the room is.

2. The "Smart Summary" (Token Formulation)
Once the groups are made, DiVT doesn't just dump the toys in the box. It creates a single, perfect summary for each group.

Instead of sending the robot 500 patches of a "cat's fur," it sends one token that says "cat."
Instead of sending 100 patches of "blue sky," it sends one token that says "sky."
Crucially, it keeps the tiny, unique details (like a small bird in the corner) as their own separate tokens, so nothing important is lost.

3. The "Volume Knob" (Granularity Control)
The researchers added a special knob (called a threshold) that lets you decide how detailed you want the summary to be.

Turn it up: You get very detailed descriptions (many small groups), which is great for complex images but uses more memory.
Turn it down: You get broad summaries (fewer, larger groups), which is super fast and saves memory.
The Best Part: You can turn this knob after the robot is already trained. You don't need to re-teach the robot; you just change the setting to fit your needs.

Why This Matters (The Results)

The paper tested this new method on many different tasks, from answering questions about pictures to describing scenes.

Speed and Efficiency: DiVT achieved the same (or even better) results as the old methods while using significantly fewer tokens. In some tests, it used less than 1/10th of the data the old method needed.
Less Confusion: Because the tokens represent actual concepts (like "a cup" or "a tree") rather than random grid squares, the robot understands the image much better.
No Retraining Needed: You can use this method with different types of cameras (vision encoders) and different robot brains (LLMs) without having to rebuild the whole system.

The Bottom Line

The paper claims that by treating images more like a story (grouping meaningful concepts) rather than a spreadsheet (rigid grid squares), we can make AI vision faster, cheaper, and smarter. It's like switching from reading a book letter-by-letter to reading it word-by-word.

Technical Summary: A More Word-like Image Tokenization for MLLMs

1. Problem Statement

Multimodal Large Language Models (MLLMs) typically employ a pre-trained vision encoder and a fixed Large Language Model (LLM), connected by a visual projector. The goal is to map image pixels into a sequence of tokens within the LLM's embedding space. However, the authors identify a fundamental structural mismatch between current visual tokens and the discrete, semantically meaningful tokens LLMs are optimized to process.

The paper highlights three primary discrepancies:

Semantic Entanglement: Unlike text tokens generated by discrete tokenizers (e.g., BPE) which are independent units, visual patch features from Vision Transformers (ViTs) are highly entangled. Repeated self-attention in vision encoders homogenizes embeddings, causing patches to become semantically mixed. Linear projectors preserve this redundancy, resulting in visual tokens that are highly correlated and lack distinct semantic boundaries.
Fixed Token Budget: Existing methods (e.g., fixed grid aggregation, resamplers) produce a fixed number of tokens regardless of image complexity. This leads to redundancy in simple scenes and loss of detail in complex scenes, failing to mimic the variable-length nature of linguistic descriptions.
Lack of Granularity Control: Current visual tokenization relies on spatial operations without a principled mechanism to control the level of detail (granularity) per token, unlike text tokenization which balances sub-words and whole words.

Consequently, visual inputs behave differently from the "word-like" units LLMs expect, leading to inefficient use of the KV cache, increased latency, and suboptimal reasoning capabilities.

2. Methodology: Disentangled Visual Tokenization (DiVT)

The authors propose DiVT, a clustering-based visual tokenization approach designed to align visual representations with the discrete nature of LLM tokens. DiVT operates without modifying the vision encoder or the LLM backbone, serving as a trainable projector.

The process consists of four stages:

2.1. Initial Patch Clustering (Adaptive Centroid Selection)

Instead of imposing a fixed token count, DiVT dynamically determines the number of clusters based on the feature distribution of the image patches.

Mechanism: It computes a pairwise cosine similarity matrix of patch features. Using a similarity threshold $\theta$ , it constructs a neighborhood graph.
Algorithm: A greedy algorithm selects the patch with the highest degree (most neighbors) as a centroid. All neighbors within the threshold are assigned to this cluster and removed from the candidate pool. This repeats until no patches remain.
Outcome: This ensures that centroids are semantically distinct and that the token count adapts to the image's intrinsic complexity (more tokens for complex scenes, fewer for simple ones).

2.2. Cluster Refinement

The initial greedy selection may assign a patch to a suboptimal centroid. To achieve better semantic disentanglement, the method reassigns every patch to the centroid with the highest similarity, ensuring each cluster contains patches sharing coherent semantics.

2.3. Visual Token Formulation

Once clusters are finalized, a single visual token is generated for each cluster.

Cross-Attention: The cluster centroid acts as the query ( $Q$ ), while all patches in the cluster serve as keys ( $K$ ) and values ( $V$ ).
Positional Encoding: Learnable positional embeddings are injected only into the value branch to preserve spatial context without perturbing attention scores.
Masking: A cluster-restricted attention mask ensures that tokens only aggregate information from their own cluster, preventing cross-contamination of unrelated semantic concepts.
Aggregation: The result is passed through an MLP to produce the final token.

2.4. Semantic Granularity Control

The similarity threshold $\theta$ acts as a tunable knob for semantic granularity:

High $\theta$ : Stricter grouping criteria, resulting in more, finer-grained tokens (higher detail, higher compute).
Low $\theta$ : Broader grouping, resulting in fewer, coarser tokens (lower detail, lower compute).
Inference Flexibility: Crucially, $\theta$ can be adjusted at inference time without retraining, allowing practitioners to trade off representational detail against memory and latency.

3. Key Contributions

Semantic Disentanglement: DiVT explicitly clusters patch embeddings into coherent semantic units, addressing the entanglement inherent in standard ViT patch features.
Adaptive Token Budget: The method dynamically allocates token counts based on image complexity, avoiding the redundancy of fixed-budget approaches.
Training-Free Inference Adjustment: The ability to adjust the similarity threshold $\theta$ at inference allows for flexible cost-performance trade-offs without retraining the model.
Encoder and LLM Agnosticism: The approach is designed to work with various vision encoders (CLIP, SigLIP, DINOv2) and scales effectively to larger LLMs (e.g., 13B parameters).

4. Experimental Results

The authors evaluated DiVT on the LLaVA-1.5 7B and 13B backbones across eight benchmarks (MMBench, VQAv2, GQA, MME, MM-Vet, TextVQA, SQA-IMG, POPE).

Performance vs. Token Count: DiVT matches or surpasses baselines (including MLP, ToME, FastV, PruMerge, VisionZip, and TokenPacker) using significantly fewer tokens.
- For example, with ~136.5 tokens (vs. 576 in full MLP), DiVT achieves comparable performance on MMBench (66.7 vs. 64.3) and VQAv2 (78.2 vs. 78.5).
- Under tight budgets (e.g., ~35 tokens), DiVT significantly outperforms other methods, demonstrating robustness where token reduction usually degrades performance.
Comparison with Projectors: DiVT outperforms other visual projectors (Resampler, C-Abstractor, Pixel-Shuffle) on general conversation and VQA tasks while remaining competitive on specialized tasks.
Qualitative Analysis: Attention maps show that DiVT produces more focused and semantically coherent attention patterns compared to the dispersed attention of standard MLP projectors.
Granularity Control: Experiments show that models trained with a specific $\theta$ can be evaluated with different $\theta$ values at inference with minimal performance loss, validating the flexibility of the approach.

5. Significance and Claims

The paper claims that DiVT addresses the "modality gap" between visual and textual inputs by creating visual tokens that behave more like "words." By explicitly disentangling semantic information and adapting the token budget to image complexity, DiVT offers a practical mechanism to:

Reduce Computational Costs: Significantly lower memory usage (KV cache) and latency by reducing the number of visual tokens.
Improve Compatibility: Make visual inputs more compatible with the discrete reasoning capabilities of LLMs.
Enable Flexible Deployment: Provide a training-free knob for practitioners to balance accuracy and compute resources during inference.

The authors emphasize that their method does not require external supervision (like object annotations) or complex multi-stage training, relying solely on the language modeling objective to shape the visual tokens. This makes DiVT a lightweight, plug-and-play solution for cost-efficient and human-aligned multimodal understanding.