🤖 AI

How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A

This paper introduces F^3A, a training-free visual token pruning router that dynamically allocates a fixed token budget by treating pruning as task-conditioned evidence search, thereby reducing inference costs in multimodal models without requiring additional training or disrupting the original decoding pipeline.

Original authors: YiJie Huang, Yiqun Zhang, Zhuoyue Jia, Xiaocui Yang, Junzhao Huang, Zihan Wang, Shi Feng, Daling Wang, Yifei Zhang, Yongkang Liu

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: YiJie Huang, Yiqun Zhang, Zhuoyue Jia, Xiaocui Yang, Junzhao Huang, Zihan Wang, Shi Feng, Daling Wang, Yifei Zhang, Yongkang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: How Much of the Picture Do We Actually Need?

Imagine you are trying to solve a riddle, and someone hands you a giant, high-resolution photo to help you find the answer. The photo is so detailed it has millions of tiny pixels.

In the world of AI, these "pixels" are called visual tokens. Current AI models (Multimodal Large Language Models) are like brilliant detectives who want to look at every single pixel in that photo to make sure they don't miss a clue. But looking at millions of pixels takes a long time and uses up a lot of battery power (computing resources).

The researchers asked a simple question: If we have a limited amount of time and energy, how many of those pixels do we actually need to keep to get the right answer?

The Problem with Current Methods: The "One-Shot" Guess

Right now, most AI systems try to cut down the number of pixels by using a "one-shot" rule. They look at the photo once and say, "Oh, this part is bright, so it's important," or "This part is blurry, so let's throw it away."

The authors argue this is like trying to find a specific needle in a haystack by only looking at the top layer of hay. If the needle is hidden deep down or is a dull color, a simple "brightness" check will miss it. They call this static ranking. It doesn't care what the specific question is; it just guesses based on general rules.

The Solution: F3A (The Fruit-Fly Strategy)

The paper introduces a new method called F3A (Fruit-Fly-Foraging Algorithm). To understand it, imagine a fruit fly looking for a piece of rotting fruit.

The Smell (Odor Field): A fruit fly doesn't just look at the fruit; it smells the air. It creates a "map" of where the smell is strongest.
- In the AI: Instead of just looking at the image, the AI reads the question first. It creates a "smell map" based on what the question is asking. If the question is "What color is the hat?", the AI knows to "smell" for hats, not just bright spots.
The Three-Step Hunt: The fruit fly doesn't just land randomly. It uses a smart strategy:
- Step 1: Coarse Search (The Wide Net): The fly flies high and looks for the general area where the smell is strong. It doesn't pick a single leaf yet; it picks a whole branch.
  - In the AI: The system looks at big chunks of the image to find general areas relevant to the question.
- Step 2: Visual Lock-on (The Zoom): Once it finds a promising branch, it lands and looks closely to confirm the fruit is actually there and not just a leaf that smells like fruit. It avoids picking two leaves right next to each other (redundancy).
  - In the AI: The system zooms in on those specific areas to pick the exact best tokens, making sure it doesn't pick the same thing twice.
- Step 3: Rescue Jump (The Safety Net): Sometimes the fruit is hidden in a weird spot the fly missed. The fly has a rule: "If I haven't found enough fruit yet, I'll jump to a random spot just in case."
  - In the AI: If the system realizes it missed a small but crucial detail (like a tiny text label or a small object), it "rescues" those tokens so they aren't lost.

Why This Matters: The Results

The researchers tested this "Fruit-Fly" strategy on many different AI models, ranging from small ones (2 billion parameters) to massive ones (235 billion parameters).

The "Fixed Budget" Test: They told the AI, "You can only look at 20% of the picture."
- Result: The Fruit-Fly AI (F3A) got the right answer 93.86% of the time compared to looking at the whole picture. Other methods (like the "one-shot" guessers) got it right much less often.
The "Fixed Goal" Test: They told the AI, "You must get 97% of the answers right, but use as few pixels as possible."
- Result: The Fruit-Fly AI only needed 39.9% of the pixels to reach that goal. The next best method needed 50.1% of the pixels.

The Takeaway

The paper claims that how you allocate your attention matters more than just how big your model is.

Instead of blindly cutting down the image based on general rules, F3A treats the image like a treasure map. It uses the question to guide a smart, three-step search (Smell, Lock-on, Rescue) to find the exact clues needed. This allows the AI to be much faster and use less memory without losing its ability to understand the world.

In short: Don't just throw away half the picture because it looks "redundant." Use the question to hunt for the specific evidence you need, just like a fruit fly hunting for fruit.

Technical Summary: How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F3A

1. Problem Statement

As Vision-Language Models (VLMs) scale in size and capability, they increasingly rely on long visual token sequences to achieve fine-grained perception (e.g., OCR, chart understanding, video analysis). However, these extended sequences impose significant computational costs, dominating prefill computation, enlarging KV caches, and increasing end-to-end latency.

The central question addressed by this work is: As multimodal models grow, how many visual tokens are actually needed, and how should they be allocated under a fixed visual token budget?

Existing training-free pruning methods (e.g., FastV, VisionZip, DivPrune, CDPruner) typically treat token pruning as a static ranking or subset selection problem based on one-shot proxy signals such as decoder attention, visual similarity, or conditional diversity. The authors argue that these approaches are incomplete, particularly under aggressive compression. A token's value is not merely a function of its salience or redundancy but depends on its evidence for the current query. Standard methods often fail to allocate tokens dynamically across an image based on the specific task, leading to suboptimal performance when token budgets are tight.

2. Methodology: Fruit-Fly-Foraging Algorithm (F3A)

The paper proposes F3A, a training-free router that reframes visual token pruning as task-conditioned evidence search. Inspired by fruit-fly optimization algorithms, F3A operates after the vision tower but before the language model consumes image tokens. It does not require model training, extra LLM forward passes, or changes to the decoding pipeline.

The method consists of three sequential stages:

A. Building the Task Odor Field

F3A constructs a low-cost global evidence map (the "odor field") over visual tokens.

Odor Cues: Instead of learned attention, F3A uses deterministic templates and the frozen tokenizer/embedding layer of the base MLLM to generate lightweight evidence queries. These include a global cue (full question context), a target cue (queried entity), and a task cue (e.g., OCR, counting, spatial relations).
Sparse Sensing: The cues are matched to visual-grid tokens using frozen sparse sensing heads (random projections). This generates a scalar evidence score ( $a_i$ ) for each token, representing prompt-relevant evidence without requiring gradient updates.

B. Three-Stage Foraging Process

F3A allocates a fixed token budget ( $K$ ) through three operators:

Coarse Search (Exploration):
The visual grid is partitioned into non-overlapping windows. Windows are scored by their average odor value. The top windows are selected to form a candidate pool ( $P_1$ ), and a "scaffold" subset ( $S_1$ ) is kept to ensure spatial coverage. This avoids over-committing to isolated high-score tokens too early.
Visual Lock-on (Local Exploitation):
Within the coarse pool, F3A refines the selection to confirm local evidence and suppress redundancy.
- Local Support: Tokens are scored based on neighborhood support and agreement with task cues.
- Inhibition of Return: A redundancy penalty is applied based on visual similarity and spatial proximity to already selected tokens. This encourages the selection of diverse, locally supported evidence rather than clustered redundant patches.
Rescue Jump (Exploration/Recovery):
To prevent missing small objects, peripheral text, or counter-evidence outside the selected windows, a fraction of the budget ( $\alpha_{jump}$ ) is reserved.
- Uncertainty/Coverage: Tokens are selected based on high uncertainty (e.g., small margins between option scores in multiple-choice tasks) or low coverage by the current subset.
- This step ensures that under-covered regions are recovered, preventing the model from collapsing onto narrow high-score areas.

C. Sequence Reconstruction

The final selected token set replaces the full visual block before the LLM prefill. The original text prefix, prompt template, and model weights remain unchanged. For grid-aware models, token indices are mapped back to original grid coordinates to preserve positional information.

3. Key Contributions

Reframing Pruning: The authors reframe training-free visual token pruning as a cross-scale token-allocation problem rather than simple compression. They evaluate token demand across dense and MoE multimodal models to determine how many tokens are needed to preserve full-token behavior.
F3A Router: They introduce F3A, a single-pass, training-free router that performs task-conditioned evidence search using question-conditioned cues, sparse sensing, and a foraging-inspired three-stage process (coarse search, lock-on, rescue).
Large-Scale Evaluation: The method is evaluated on Qwen3-VL models ranging from 2B to 235B parameters (covering both dense and MoE architectures) across 11 multimodal benchmarks. It is also tested on Qwen2.5-VL and InternVL3.5 backbones to demonstrate transferability.

4. Experimental Results

The evaluation covers 10 models and 3 retention ratios (60%, 40%, 20%).

Performance Retention: Across all model scales, F3A consistently outperforms baselines (FastV, VisionZip, DivPrune, CDPruner).
- At 20% visual token retention, F3A retains 93.86% of full-token performance on average.
- On the largest model (Qwen3-VL-235B-A22B), F3A retains 93.86% of performance even after removing 80% of visual tokens.
Fixed-Fidelity Token Demand: When targeting 97% of full-token performance, F3A requires only 39.9% visual tokens on average, compared to 50.1% for the strongest competing baseline. On the 235B model, F3A achieves this with 41.2% tokens.
Scaling Behavior: Unlike other methods, F3A's advantage persists and often grows as the model scales. The average accuracy gain over the strongest baseline is positive across all 18 model-retention settings tested.
Efficiency: F3A improves the accuracy-latency trade-off. On Qwen3-VL-8B at 20% retention, it reduces end-to-end latency from 354.7 ms to 274.3 ms and KV-cache footprint from 117.7 MB to 29.0 MB, while maintaining the highest accuracy among pruned methods.
Statistical Significance: A paired sign-test across 30 model-retention settings yields a p-value of $1.9 \times 10^{-9}$ , confirming that F3A consistently outperforms the strongest non-F3A baseline.

5. Significance and Claims

The paper claims that scalable multimodal inference depends not only on model size but also on the search-structured allocation of task-relevant visual evidence.

Task-Conditioned vs. Static: The results suggest that treating pruning as a static ranking problem is insufficient for aggressive compression. The "foraging" approach, which dynamically locates, refines, and recovers evidence based on the specific query, is necessary to preserve the capacity of larger models under tight token budgets.
Fixed-Fidelity Perspective: The work argues that the answer to "how many tokens are needed" is not a universal retention ratio but a scale-dependent allocation problem. F3A demonstrates that effective pruning allows larger models to realize their full capacity with significantly fewer tokens than previously thought possible.
Generalizability: The method is shown to be robust across different model families (Qwen3, Qwen2.5, InternVL) and architectures (Dense vs. MoE), suggesting that the principle of task-conditioned evidence search is a general solution for multimodal token pruning.

The authors conclude that F3A offers a practical path to reducing inference costs for multimodal models without sacrificing the fine-grained perception capabilities that drive their performance.