💻 computer science

Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding

This paper identifies "Relevant Visual Information Shift" (RVIS) as the primary cause of visual token pruning failures in complex reasoning tasks within Multimodal Large Language Models and proposes a training-free, decoding-stage framework called DSTP to dynamically align token selection with shifting reasoning requirements, thereby significantly improving performance across diverse architectures.

Original authors: Jiwan Kim, Kibum Kim, Wonjoong Kim, Byung-Kwan Lee, Chanyoung Park

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiwan Kim, Kibum Kim, Wonjoong Kim, Byung-Kwan Lee, Chanyoung Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Overloaded Librarian" Problem

Imagine you have a brilliant librarian (the AI) who is trying to solve a complex puzzle based on a massive photo album (the image).

The Problem: The photo album has thousands of pages (visual tokens). Reading every single page takes forever and uses up all the librarian's energy. To make things faster, researchers invented a method called Visual Token Pruning. This is like hiring a "Scout" who looks at the photo album before the librarian starts reading. The Scout picks out the 10 most important pages and throws the other 990 away. The librarian then only reads those 10 pages.

The Success: For simple questions like "What color is the cat?", this works perfectly. The Scout picks the page with the cat, throws the rest away, and the librarian answers quickly and correctly.

The Failure: But when the question gets hard—like a complex math problem or a logic puzzle requiring a step-by-step story—the Scout fails. The librarian starts reading the 10 pages, realizes they need to look at a different page to solve step 2, then another page for step 3, and another for step 4. But those pages were already thrown away! The librarian is stuck, confused, and gives the wrong answer.

The Core Discovery: "The Shifting Spotlight" (RVIS)

The authors of this paper discovered why this happens. They call it Relevant Visual Information Shift (RVIS).

Think of the librarian's attention as a flashlight.

Simple Tasks (Visual Understanding): The flashlight stays fixed on one spot the whole time. If the question is "Where is the dog?", the flashlight shines on the dog and never moves. The Scout's job was easy.
Complex Tasks (Visual Reasoning): The flashlight is a moving spotlight. To solve a math problem, the light first shines on the numbers, then jumps to the diagram, then moves to the formula, then back to the units. The "important" part of the image changes constantly as the thinking progresses.

The Flaw: Existing pruning methods act like a rigid security guard who locks the door after the Scout picks the initial 10 pages. Even though the librarian's flashlight moves to a new area that needs light, the guard says, "No, we only have the 10 pages we picked at the start." The librarian is forced to guess, leading to errors.

The Solution: DSTP (The "Smart Re-Scout")

The authors propose a new system called DSTP (Decoding-stage Shift-aware Token Pruning).

Instead of a rigid guard, DSTP acts like a dynamic assistant who stays in the room with the librarian.

The Scout still works: At the very beginning, the Scout still picks the initial 10 pages and puts the rest in a "waiting room" (they aren't thrown away, just set aside).
The Watcher (RISD): As the librarian starts reading and thinking, a "Watcher" monitors the flashlight. If the Watcher sees the flashlight suddenly jump to a new area that isn't in the current 10 pages, it sounds an alarm: "Hey! The focus has shifted!"
The Swap (CPTS): When the alarm sounds, the assistant immediately runs to the "waiting room," grabs the new relevant pages the librarian needs, and swaps them into the reading pile.
Context Preservation: Crucially, the assistant doesn't just throw away the old pages. It keeps the old ones and adds the new ones temporarily. This ensures the librarian doesn't lose the "big picture" context while focusing on the new detail.

Why This Matters

It's Free: This system doesn't require retraining the librarian (the AI model). It's a plug-and-play add-on.
It's Fast: It only swaps pages when absolutely necessary. Most of the time, the librarian keeps working with the small set of pages, so it remains super fast.
It's Smart: It fixes the specific problem where AI fails at hard reasoning tasks.

The Analogy Summary

Old Method (Static Pruning): Like a tour guide who picks 5 stops for a city tour at the start and refuses to let the tourists leave the bus, even if they see a famous landmark they really want to visit later.
New Method (DSTP): Like a tour guide who picks 5 stops, but keeps a map of the whole city. If the tourists point out a new interesting building halfway through, the guide says, "Great idea! Let's hop off the bus, look at that building, and then get back on."

The Result

By using DSTP, the AI can solve complex math and logic puzzles almost as well as if it had read the entire photo album, but it does so with the speed and efficiency of reading just a few pages. It bridges the gap between being fast and being smart.

1. Problem Statement

Multimodal Large Language Models (MLLMs) rely on a vast number of visual tokens generated by vision encoders, creating prohibitive computational and memory overhead. Visual Token Pruning has emerged as a solution to remove redundant tokens, typically by selecting the most relevant tokens during the Prefill Stage (the initial processing of the input) based on attention weights.

However, the authors observe a critical gap: while existing pruning methods perform well on simple Visual Understanding tasks (e.g., standard VQA), they suffer precipitous performance drops on complex Visual Reasoning tasks (e.g., visual math, logical puzzles, STEM).

The Core Issue: Existing methods assume that visual information relevant at the start (Prefill) remains sufficient for the entire generation process.
The Failure: In complex reasoning, the model's visual focus must dynamically shift to different parts of the image as the reasoning steps progress. Static pruning, which permanently discards tokens after the Prefill stage, causes the model to lose access to critical visual cues needed for later reasoning steps.

2. Key Concept: Relevant Visual Information Shift (RVIS)

Through systematic analysis, the authors identify Relevant Visual Information Shift (RVIS) as the primary driver of pruning failure in reasoning tasks.

Definition: RVIS is the phenomenon where the visual regions requiring attention change dynamically throughout the decoding process.
Evidence:
- VQA (Visual Understanding): Attention maps remain static; the model focuses on the same region from start to finish.
- VMR (Visual Math Reasoning): Attention maps fluctuate drastically. As the model generates logical steps (e.g., "calculate side A," then "calculate side B"), it shifts focus to different parts of the diagram.
Correlation: The frequency of RVIS events correlates directly with task complexity. Higher RVIS frequency leads to a sharp decline in the success rate of static pruning methods.

3. Methodology: DSTP (Decoding-stage Shift-aware Token Pruning)

To address RVIS without requiring model retraining, the authors propose DSTP, a training-free, plug-and-play framework that adapts visual token selection during the Decoding Stage.

DSTP consists of two integrated modules:

A. Relevant Visual Information Shift Detect (RISD)

Function: Monitors the model's visual attention in real-time during decoding.
Mechanism:
1. Establishes an Anchor Attention vector from the Prefill stage ( $t_0$ ).
2. At each decoding step ( $t_l$ ), calculates the cosine similarity between the current attention and the anchor.
3. If the similarity drops below a predefined threshold ( $\tau$ ), it signals an RVIS event, indicating the model's focus has shifted away from the initially selected tokens.

B. Context-Preserving Visual Token Swap (CPTS)

Function: Retrieves and integrates newly relevant visual tokens when an RVIS is detected.
Mechanism:
1. Re-evaluation: Instead of discarding pruned tokens, DSTP keeps them in a "Reserved Set." When RVIS is detected, it re-computes attention scores for the entire visual token set (Retained + Reserved) based on the current text query.
2. Context Preservation: To prevent the model from losing global context by swapping tokens entirely, DSTP performs a Union Operation. It merges the original Prefill tokens with the newly identified relevant tokens.
3. Duration: This expanded token set is maintained for a specific duration ( $L$ ) to ensure logical continuity before reverting to the standard budget.

4. Key Contributions

Identification of RVIS: First to identify and quantify "Relevant Visual Information Shift" as the fundamental reason why static token pruning fails in complex reasoning tasks.
DSTP Framework: Proposes a novel, training-free, and lightweight add-on framework that dynamically aligns visual tokens with shifting reasoning requirements.
Generalizability: Demonstrates effectiveness across diverse state-of-the-art MLLM architectures (Qwen3-VL, InternVL3.5) and various pruning baselines (FastV, DivPrune, VisionZip).

5. Experimental Results

The authors evaluated DSTP on benchmarks including MathVerse, WeMath, DynaMath, LogicVista, and MMMU-Pro (Reasoning) and SQA, GQA, VQA-T (Understanding).

Visual Reasoning Performance:
- Existing pruning methods (e.g., FastV) suffered massive accuracy drops (e.g., dropping to ~30-50% of full-token performance on MathVerse).
- DSTP significantly mitigated this degradation. For example, on Qwen3-VL with FastV, DSTP improved MathVerse accuracy from 32.23% to 52.54% (at 33.3% token retention), recovering ~85% of the full-token performance.
- DSTP consistently outperformed baselines across all reasoning benchmarks, with the most significant gains in high-complexity tasks.
Visual Understanding Performance:
- DSTP maintained or slightly improved performance on standard VQA tasks, proving it does not degrade simple tasks while fixing complex ones.
Efficiency:
- Computational Overhead: Minimal. The additional TFLOPs and latency introduced by the detect-and-swap mechanism are negligible compared to the vanilla model.
- Latency: DSTP achieved lower latency per example than static pruning in some cases because the improved reasoning accuracy reduced the need for redundant generation steps.

6. Significance

Paradigm Shift: The paper challenges the prevailing assumption that visual token pruning can be a one-time decision made at the Prefill stage. It argues that for reasoning-heavy tasks, pruning must be dynamic and adaptive.
Practical Deployment: DSTP offers a practical solution for deploying efficient MLLMs in real-world scenarios involving complex logic, math, and science, where current pruning methods are too brittle.
Resource Allocation: It demonstrates that allocating computational resources adaptively (retrieving tokens only when needed) is far more effective than statically increasing the token budget during the prefill stage.

In summary, this work bridges the gap between efficient inference and complex reasoning by introducing a mechanism that allows MLLMs to "look back" at discarded visual information when their reasoning path demands it, ensuring robustness without sacrificing speed.