🤖 AI

Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

The paper proposes "Reroute," a training-free plug-in for Vision-Language Models that replaces irreversible token removal with a recoverable routing mechanism, allowing deferred visual tokens to re-enter the processing pool at later decoder stages to improve grounding performance while maintaining efficiency.

Original authors: Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of visual information (an image) that a smart robot (a Vision-Language Model) needs to read to answer a question. The robot doesn't read the image as a whole picture; it breaks it down into thousands of tiny "visual tokens," like individual puzzle pieces.

The problem is that reading thousands of pieces takes a huge amount of energy and memory, slowing the robot down.

The Old Way: "Rank and Remove"

Previously, to speed things up, researchers used a method called "Rank and Remove."
Think of this like a strict librarian who looks at your puzzle pieces and says, "These 100 pieces look important right now. Keep them. Throw away the other 900 pieces forever."

The librarian makes this decision based on how the pieces look at that exact moment. The problem? The robot's brain works in layers, like a multi-story building.

On the ground floor (early layers): The robot is confused and looks at everything vaguely. It might think a piece of the sky is unimportant.
On the top floor (deep layers): The robot suddenly realizes, "Wait! That piece of sky is actually the background for the person I'm looking for!"

But because the librarian on the ground floor threw that piece away, the robot on the top floor can never see it. The decision to throw it away was irreversible. If the robot needed that piece later, it was too late. This caused the robot to fail at tasks requiring precise location, like pointing out exactly where a "man in a green shirt" is standing.

The New Way: "Reroute, Don't Remove"

The authors of this paper propose a new method called Reroute. Instead of throwing pieces away, they put them in a "waiting room."

Here is how it works:

The Checkpoint: At the bottom of the building, the librarian still checks the pieces. They pick the top 10% to go straight to the next floor for detailed work.
The Waiting Room: The other 90% aren't thrown in the trash. They are sent down a bypass hallway (a residual path) that skips the heavy work of the current floor.
The Re-Entry: When the robot reaches the next checkpoint on the next floor, the librarian looks at the entire collection again—including the pieces that were waiting in the hallway.
The Second Chance: If a piece that was ignored on the first floor suddenly looks very important on the second floor, the librarian can pull it out of the waiting room and let it join the work.

Why This Matters

The paper shows that this simple change—keeping the pieces in the system instead of deleting them—makes a huge difference, especially when the robot is forced to be very efficient (keeping only a tiny fraction of the pieces).

The Analogy of the "Man in the Green Shirt": In the paper's examples, when the robot had to find a "man in a green shirt," the old method threw away the shirt token early because it didn't look important yet. The robot failed to find the man. The new method kept the shirt token in the waiting room. When the robot got deeper into its reasoning, it realized the shirt was crucial, pulled it back in, and successfully found the man.
No Extra Cost: The best part is that this doesn't require the robot to learn anything new or use more energy. The "waiting room" pieces skip the heavy lifting on the current floor, so the total energy used remains roughly the same as the old "throw away" method. It's just a smarter way of managing the pieces.

The Bottom Line

The paper argues that we shouldn't view cutting down visual information as a one-time "delete" button. Instead, we should view it as a routing system. By allowing "deferred" information to come back and be re-evaluated as the AI gets smarter, we can make these models much faster without making them "blind" to important details they might discover later.

The authors tested this on several different AI models and found that it consistently improved the ability to locate objects (grounding) without hurting the model's ability to answer general questions, all while keeping the computer running efficiently.

Technical Summary: Reroute, Don't Remove

1. Problem Statement

Vision-Language Models (VLMs) project images into hundreds or thousands of visual tokens, causing decoder inference costs to scale quadratically with attention and linearly with KV-cache memory. To mitigate this, existing visual-token reduction methods (e.g., FastV, PDrop, Nüwa) largely adhere to a "rank-and-remove" paradigm. In this approach, visual tokens are scored (typically via text-to-vision attention), a compact subset is retained, and the rest are permanently discarded from the sequence.

The authors identify a critical flaw in this paradigm: token importance is not stable across decoder depth.

Dynamic Importance: Shallow decoder layers often attend diffusely, while target-relevant regions may only emerge in middle or deep layers.
Irreversibility: A token ranked low at an early pruning stage (e.g., Layer 3) might be essential for a specific query in a deeper layer (e.g., Layer 25). Once removed, it cannot contribute to later computations.
Consequence: Under aggressive token reduction (e.g., 88.9% reduction), this irreversibility leads to a collapse in grounding performance (e.g., IoU dropping below 0.4), even when general VQA performance is partially maintained.

2. Methodology: Reroute

The paper proposes Reroute, a training-free plug-in that replaces irreversible removal with recoverable routing.

Core Mechanism

Reroute reframes decoder-side visual-token reduction as a sequence of stage-wise routing decisions rather than a single pruning event.

Stage-wise Selection: The decoder is partitioned into $S$ stages. At each routing layer $\ell_i$ , a scorer (reusing the existing text-to-vision attention scores) ranks the current candidate set of visual tokens.
Routing vs. Deferral:
- Selected Tokens ( $V_{sel}$ ): A top- $r_i$ fraction of tokens are routed through the full decoder block (Attention + FFN) for the current stage.
- Deferred Tokens ( $V_{def}$ ): The remaining tokens bypass the current stage's Attn+FFN computation via a residual path. Crucially, they are not deleted.
Re-entry: At the next routing layer, the candidate set is restored to include all deferred tokens ( $C_{i+1} = V$ ). Deferred tokens are re-evaluated against the full visual set. If their attention score improves in deeper layers, they can be re-selected and routed through subsequent blocks.

Theoretical Efficiency

Matched Budget: Reroute preserves the theoretical FLOPs and KV-cache budget of the pruning baseline it augments. Only the selected tokens ( $V_{sel}$ ) consume the heavy compute (Attn+FFN) at each stage.
No Training: The method reuses the attention scores already computed by the decoder for the original pruning methods. No new router parameters are trained.
Relation to MoD: It is framed as a training-free instantiation of Mixture-of-Depth (MoD) for VLM visual tokens, where the "router" is the pre-existing attention mechanism.

3. Key Contributions

Recoverable Routing Formulation: The authors recast visual-token pruning as stage-wise routing with deferred-token bypass. This makes conventional irreversible pruning a degenerate case where re-entry is forbidden.
Plug-in Mechanism: Reroute is a drop-in augmentation that requires no retraining, reuses existing scoring rules and schedules, and maintains the theoretical efficiency class (TFLOPs and KV-cache) of the underlying pruning method.
Empirical Validation: The method is validated across three backbones (LLaVA-1.5-7B, Qwen2.5-VL-7B, Qwen3.5-9B-Hybrid) and three pruning baselines (FastV, PDrop, Nüwa).

4. Experimental Results

The paper evaluates Reroute on visual grounding (RefCOCO series) and general VQA benchmarks.

Visual Grounding (RefCOCO):
- Aggressive Reduction: Under an 88.9% token reduction (576 $\to$ 64 tokens), standard pruning (e.g., FastV) causes grounding accuracy to collapse (e.g., IoU < 0.4). Reroute significantly recovers this performance.
- Quantitative Gains: On LLaVA-1.5 (HuggingFace format), PDrop+Reroute improves the average grounding ratio from 22.2% to 34.0% at 64 tokens. On Qwen3.5-9B-Hybrid, PDrop+Reroute improves the ratio from 19.3% to 37.4% at 88.9% reduction.
- Cross-Backbone: The improvements transfer effectively to Qwen-family models, including hybrid architectures (Mamba-Transformer).
General VQA:
- Reroute maintains or slightly improves general VQA performance (GQA, MME, TextVQA) compared to the pruning baselines, indicating that the recoverable routing does not sacrifice general multimodal capability.
Efficiency:
- Reroute achieves comparable TFLOPs and KV-cache reductions to the pruning baselines (e.g., ~80% reduction in TFLOPs and ~86% in KV-cache at 64 tokens).
- Minor TFLOPs overhead exists due to the overhead of additional routing stages compared to one-shot pruning, but the theoretical class remains matched.
Ablation Studies:
- Frequency: Increasing the number of reroute stages generally improves performance up to a point, suggesting repeated re-evaluation is beneficial.
- Location: The placement of routing layers matters; early-to-middle decoder layers are optimal for re-entry decisions.

5. Significance and Claims

The paper argues that visual-token reduction should not be viewed solely as irreversible pruning, but also as recoverable routing.

Paradigm Shift: The limitation of existing methods lies not in the scoring mechanism but in the "remove" action. By replacing removal with deferral, the model can adapt to the shifting importance of visual tokens across decoder depth.
Grounding Sensitivity: The benefits are most pronounced in grounding-heavy tasks and under aggressive token reduction, where the risk of discarding essential visual evidence is highest.
Modest Scope: The authors acknowledge that Reroute inherits the limitations of attention-score routers (poor ranking can still lead to inactive tokens) and that practical latency depends on optimized gather/scatter kernels. They also note that in hybrid architectures (like Qwen3.5), routing is currently applied only to softmax-attention blocks, leaving interactions with linear-attention state dynamics for future work.

In conclusion, Reroute demonstrates that preserving the option to recover deferred evidence allows VLMs to maintain high grounding accuracy even when computational budgets are severely constrained, without requiring additional training or increasing theoretical compute costs.