🤖 AI

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

OccamToken is a training-free, budget-adaptive framework that improves Vision-Language Model inference efficiency by replacing brittle absolute token ranking with register-anchored relative evidence testing, enabling extreme token compression (e.g., reducing 2,880 tokens to ~40) while preserving over 93% of original accuracy.

Original authors: Geng Li, Guohao Chen, Ting Chen, Shilin Shan, Kuangji Zuo, Bofan Lyu, Tuo An, Gen Li, Jianfei Yang

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Geng Li, Guohao Chen, Ting Chen, Shilin Shan, Kuangji Zuo, Bofan Lyu, Tuo An, Gen Li, Jianfei Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Too Much Noise in the Kitchen

Imagine you are a chef (the AI) trying to cook a meal based on a customer's order (the text question) and a massive pile of ingredients (the image).

In modern AI, when you show an image, the computer breaks it down into thousands of tiny pieces called "tokens." If you have a high-resolution photo, that's like having a pile of 2,880 ingredients. The chef has to look at every single one of them before starting to cook. This takes a huge amount of time and energy (computing power), even if most of those ingredients are just background noise like a speck of dust on the counter or a blurry patch of sky.

The Old Way: The "Top 10" Rule

Previously, to save time, people tried to throw away the "least important" ingredients. They used a rule like: "Keep the top 100 most interesting ingredients and throw the rest away."

The paper argues this rule is broken for two reasons:

The "Screaming Baby" Effect: Sometimes, a boring ingredient (like a blank wall) accidentally gets a very loud "importance score" just because of how the computer calculates things. This loud noise drowns out the quiet, important ingredients (like a tiny detail in a photo), making the computer think the boring stuff is actually the most important.
The "One Size Fits All" Problem: A fixed rule (like "keep 100") doesn't work for every picture.
- If you ask, "Is there a cat?" in a photo of a forest, you only need to check a few spots. Keeping 100 spots is wasteful.
- If you ask, "What is the texture of the fabric?" in a photo of a sweater, you might need to look at many more spots. Keeping only 100 might mean you miss the answer.

The New Solution: OccamToken

The authors created a new method called OccamToken. Instead of asking, "Which are the top 100?", they ask, "Is this ingredient more useful than our 'Reference Jar'?"

Here is how it works, step-by-step:

1. The "Reference Jar" (The Register Token)

Imagine you have a special jar on the counter that contains a generic, average sample of "everything in the kitchen." It doesn't know about specific cats or sweaters; it just holds the "background vibe" of the room.

Why it helps: In the old system, the "Screaming Baby" (the boring noise) would steal all the attention. But in this new system, the "Reference Jar" soaks up that noise. It acts as a sponge for the useless, loud signals.
The Result: Now, the computer can clearly see which ingredients are actually special, because the noise has been muffled.

2. Stage 1: The "Image Cleanup" (Redundancy Pruning)

Before the chef even reads the customer's order, the kitchen staff does a quick sweep.

They compare every ingredient to the Reference Jar.
If an ingredient looks just like the generic background in the jar, it gets thrown away.
Analogy: If the photo is of a busy street, the staff throws away the 2,000 tokens that are just "sky" or "blurry pavement" because the Reference Jar already knows what that looks like. They keep the tokens that look different from the jar (the cars, the people).
Benefit: This happens automatically for every image. A simple photo gets cleaned up heavily; a complex photo gets cleaned up less.

3. Stage 2: The "Customer Order" Check (Relevance Pruning)

Now the chef reads the specific question: "Is the man to the left standing?"

The chef looks at the remaining ingredients.
They compare them to the Reference Jar again, but this time, they ask: "Does this ingredient help answer the specific question better than the generic background?"
Analogy: If the question is about a man, the chef keeps the tokens showing the man and the ground he stands on. If the question is about a cat in the corner, the chef keeps those tokens and throws away the man.
Benefit: The number of ingredients kept changes based on the question. A simple question might only need 10 tokens; a hard question might need 50.

The Results: Less Work, Same Taste

The paper tested this on several AI models (like LLaVA and Qwen).

The Claim: They managed to throw away 98.6% of the ingredients (going from 2,880 tokens down to about 40) and still get the right answer 93% of the time.
The Analogy: It's like realizing you don't need to taste every single grain of rice in a pot to know it's salty. You just need to taste the right spoonfuls.
No Training Needed: The best part is that they didn't have to re-teach the chef how to cook. They just changed the rules of how the chef selects ingredients. It works "out of the box."

Summary

OccamToken is a smart filter that stops AI from wasting time looking at boring background noise. Instead of using a rigid "keep the top 100" rule, it uses a "Reference Jar" to figure out what is actually new and useful for the specific question being asked. This makes the AI faster and cheaper to run without making it "dumber."

Technical Summary: OccamToken

Problem Statement

Vision-Language Models (VLMs) rely on long sequences of visual tokens to encode images, leading to significant computational and memory overhead during the prefill stage and autoregressive decoding. Existing visual token compression methods predominantly follow an absolute-ranking paradigm: they assign importance scores to tokens and retain a fixed top- $K$ subset. The authors argue this paradigm is fundamentally brittle for two reasons:

Attention Sinks: Low-information tokens (often outliers with high norms) can absorb excessive attention mass, distorting the importance ranking of genuinely informative tokens and compressing the score distribution.
Static Budget Mismatch: A fixed top- $K$ cutoff cannot adapt to the per-sample variation in attention-score distributions. It fails to account for image redundancy or query-dependent evidence demands, often retaining redundant tokens in simple cases while discarding critical evidence in complex ones.

Furthermore, existing adaptive methods often require additional training or rely on weakly responsive training-free statistics that do not fully capture query-conditioned evidence needs.

Methodology: OccamToken

The authors propose OccamToken, a training-free, budget-adaptive framework that replaces absolute token ranking with register-anchored relative evidence testing. The core insight is that register tokens (introduced in recent transformer studies to absorb attention sinks) naturally absorb low-information attention patterns, making them a stable, non-discriminative reference for identifying genuinely informative visual evidence.

The framework operates in two stages:

1. Core Mechanism: Relative Comparison

Instead of asking "Is this token in the top- $K$ ?", OccamToken asks "Does this token provide stronger evidence than the register token?"
The pruning rule is defined as:
$S(x) = \{v \in V(x) \mid s_x(v) \ge \lambda \cdot s_x(r(x))\}$
Where $r(x)$ is a sample-dependent reference token (the register), $s_x$ is the scoring function, and $\lambda$ controls pruning intensity. This allows the retained token budget to vary dynamically based on the input's score distribution.

2. Two-Stage Pruning Pipeline

Stage I: Image-Adaptive Redundancy Pruning (AdapRP)
- Location: Output of the Vision Encoder.
- Mechanism: A test-time register token is constructed by redirecting high-norm "sink" activations from sparse neurons to a dedicated register token. The [CLS] token (or a mutual-scoring scheme for architectures without [CLS]) jointly scores all visual tokens and the register token.
- Action: Tokens scoring below $\lambda_1 \cdot s_1(r)$ are pruned. This removes image-level redundancy before the tokens enter the Large Language Model (LLM), creating an image-adaptive intermediate budget.
Stage II: Register-Anchored Relevance Pruning (RegRP)
- Location: Inside the Language Model (at a specific decoder layer, e.g., layer 11).
- Mechanism: Text tokens act as query-conditioned evaluators. For each surviving visual token, the maximum attention received from text tokens is used as its score (to capture localized relevance). The mean attention received by the register token from text serves as the dynamic threshold.
- Action: Tokens scoring below $\lambda_2 \cdot s_2(r)$ are removed. This produces a query-adaptive final budget, where the same image can yield different token counts depending on the specific question asked.

Key Contributions

Register-Anchored Dynamic Thresholding: A training-free mechanism that links attention-sink mitigation to adaptive pruning. By inserting a register token into the attention competition, it absorbs sink-induced distortion and serves as a distribution-adaptive threshold.
OccamToken Framework: A two-stage, plug-and-play pruning system that performs image-level redundancy removal followed by query-level relevance selection, enabling adaptive token allocation without learned predictors or extra training data.
Empirical Validation: Extensive experiments demonstrate that OccamToken outperforms state-of-the-art training-free and training-based baselines across diverse architectures (LLaVA-v1.5, LLaVA-NeXT, Qwen3-VL) and benchmarks (GQA, POPE, MME, etc.).

Experimental Results

Accuracy-Efficiency Trade-off: OccamToken consistently improves the trade-off compared to fixed-budget and other adaptive methods.
Extreme Compression: On LLaVA-NeXT (processing 2,880 visual tokens), OccamToken reduces the token count to approximately 40 tokens (a 1.4% retention regime) while preserving over 93% of the original full-token accuracy.
Robustness: The method maintains high relative accuracy (e.g., 97.2% on LLaVA-v1.5 with 32 tokens) and significantly outperforms fixed-budget baselines in low-budget regimes where preserving task-relevant evidence is critical.
Efficiency: By reducing the visual sequence length, OccamToken significantly lowers KV-cache memory usage (e.g., reducing from 313 MB to 47 MB on LLaVA-v1.5) and prefill latency, though it introduces a minor overhead for the adaptive scoring logic.
Generalization: The method transfers effectively to non-LLaVA architectures (e.g., Qwen3-VL, Mini-Gemini) and video VLMs, demonstrating that the register-anchored principle is not tied to specific token interfaces.

Significance and Claims

The paper claims that reference-based adaptive pruning is a simple yet effective direction for scalable multimodal inference. By shifting from absolute ranking to relative comparison against a register anchor, OccamToken addresses the structural deficiencies of attention sinks and static budgets without requiring model retraining. The authors highlight that this approach enables stable visual token compression even in extreme regimes (1.4% retention) while preserving the evidence required for multimodal reasoning.

The work positions OccamToken not merely as a latency-reduction tool, but as a practical solution for balancing accuracy and efficiency, particularly in resource-constrained deployment scenarios where preserving critical visual evidence is paramount. The authors acknowledge limitations regarding the reliance on constructing reliable test-time register tokens for specific Transformer-based backbones, suggesting future work could extend this to broader multimodal architectures.