💻 computer science

Do Vision Language Models Need to Process Image Tokens?

This paper challenges the assumption that Vision Language Models require deep, sustained processing of image tokens by demonstrating that visual representations rapidly converge to a stable, low-complexity state, suggesting that deeper visual layers offer diminishing returns and that the necessity of such processing is highly task-dependent.

Original authors: Sambit Ghosh, R. Venkatesh Babu, Chirag Agarwal

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sambit Ghosh, R. Venkatesh Babu, Chirag Agarwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart assistant (a Vision-Language Model) who can look at a picture and answer questions about it. Usually, this assistant is built like a giant factory with 30 different rooms (layers). When a photo comes in, it gets broken into tiny puzzle pieces (image tokens) and sent through every single room, one after another, to be analyzed, re-analyzed, and polished until the final answer is ready.

This process is slow and uses a lot of electricity (computational power). The big question this paper asks is: "Do we really need to send those picture pieces through all 30 rooms? Or can we stop the analysis halfway through and still get a good answer?"

Here is what the researchers discovered, explained through simple analogies:

1. The "Photo Album" Effect (Stabilization)

Imagine you are looking at a photo of a cat.

The Text: If you are reading a story about the cat, the meaning changes constantly. In the first sentence, it's just "a cat." In the middle, it's "a cat chasing a mouse." By the end, it's "a tired cat sleeping." The meaning is always evolving.
The Image: The researchers found that the "meaning" of the picture stabilizes very quickly. By the time the picture pieces reach the 10th room of the factory, the assistant has already figured out, "Okay, that's a cat." Sending those same picture pieces through rooms 11 to 30 doesn't change the fact that it's a cat. The image representation becomes "frozen" or "stable."

The Analogy: Think of the image analysis like developing a photograph in a darkroom. Once the image is fully developed (around layer 10), putting it in the chemical bath for 20 more minutes doesn't make the photo "more" developed. It's just sitting there. The text, however, is like a writer constantly editing a story; the story keeps changing until the very last second.

2. The "Swappable Parts" Test (Interchangeability)

Because the image analysis stabilizes early, the researchers tried a weird experiment: they took the picture analysis from the 5th room and swapped it with the analysis from the 25th room.

Result: The assistant gave the exact same answer.
Meaning: Once the picture is "seen," the later rooms don't really do anything new to it. The picture tokens are interchangeable. You could stop processing the image after room 10, and the assistant would still "know" what it's looking at.

3. It Depends on What You Ask (Task Dependence)

Here is the catch: Just because the picture is "seen" doesn't mean you can stop looking at it for every task.

Task A: "What is in this picture?" (Single Word Answer)
- Analogy: This is like asking, "Is that a cat?"
- Result: You can stop the image processing early (around layer 10). The assistant can still answer correctly because it just needs a rough idea.
Task B: "Write a poem about the cat's fur texture and the lighting." (Long Description)
- Analogy: This is like asking the assistant to write a whole novel based on the photo.
- Result: If you stop looking at the picture early, the poem becomes garbage. The assistant needs to keep looking at the picture details (the fur, the light) while it is writing every single word of the poem.
- Lesson: Simple questions need a quick glance. Complex stories need a sustained gaze.

4. Can We "Retrain" the Assistant? (Fine-Tuning)

The researchers asked: "If we force the assistant to stop looking at the picture early, can we teach it to get better at guessing?"

Result: Yes, but with limits.
Analogy: If you take away the assistant's glasses after layer 10, they will stumble. But if you give them a crash course (fine-tuning), they can learn to guess the details of the picture based on the context.
The Catch: They can learn to guess the general vibe (e.g., "It's a happy scene") very well. But if you need precise details (e.g., "The cat has a scar on its left ear"), the crash course isn't enough. They can't fully recover the lost details without actually seeing the picture.

5. Does "Thinking Harder" Help? (Reasoning Chains)

Sometimes, we tell AI: "Don't just give the answer; explain your thinking step-by-step first."

The Expectation: Maybe if the AI writes a long reasoning chain, it can compensate for not seeing the picture deeply.
The Reality: No. In fact, it makes things worse.
Analogy: If you are blindfolded (no image processing) and asked to solve a puzzle, writing a long explanation of how you would solve it doesn't help you actually solve it. The more steps you take without the visual data, the more likely you are to hallucinate (make things up). The "thinking" process actually relies more on the picture than a simple "yes/no" answer does.

The Big Takeaway

The current way AI models work is like a factory that processes every single piece of raw material through every single machine, even if the first 10 machines have already done the job.

This paper suggests:

Stop wasting energy: We can stop processing images early for simple tasks (like answering a multiple-choice question).
Keep looking for complex tasks: If you need a long description or a detailed story, you must keep the image processing running deep into the model.
Efficiency: By knowing exactly when to stop looking at the picture, we can make these AI models much faster and cheaper to run without losing their smarts.

In short: The AI sees the picture quickly, but it needs to keep "looking" at it while it writes a long story.

1. Problem Statement

Vision Language Models (VLMs) integrate visual encoders with Large Language Models (LLMs) to process multimodal inputs. Standard architectures process dense sequences of image tokens through deep transformer stacks, incurring significant computational overhead. However, it remains unclear whether this sustained, deep processing of image tokens is necessary for performance or if visual representations evolve meaningfully throughout the network depth.

The authors challenge the current paradigm by asking: Do VLMs need to process image tokens across all layers, or do visual representations stabilize early, rendering deeper processing redundant?

2. Methodology

The authors adopt a representation-centric perspective, analyzing the structural evolution of token embeddings across transformer layers rather than relying solely on task-level accuracy. They investigate five key research questions (RQs) using three primary VLM families (LLaVA-1.5, Qwen2.5-VL, InternVL) and diverse datasets (BLINK, ChartQA, Flickr8k, M3CoT).

Key Metrics for Representation Analysis

To quantify structural changes, the authors employ three complementary metrics:

Matrix Entropy & Spectral Structure: Measures the concentration of energy in the embedding space (via Von Neumann entropy of the Gram matrix). Low entropy implies compression; high entropy implies diversity.
Intrinsic Dimensionality (ID): Estimates the effective degrees of freedom of the local manifold, reflecting representational complexity independent of embedding size.
Trajectory Curvature (TC): Quantifies the angular deviation between successive layer updates. High curvature indicates sharp, local reconfiguration; low curvature indicates smooth, stable evolution.

Experimental Protocols

Layer Substitution: Swapping image tokens from early layers with those from later layers (and vice versa) to test functional interchangeability.
Depth Truncation: Systematically removing image token processing after a specific layer $l_c$ and propagating only textual tokens through the remaining decoder.
Distillation-Based Fine-Tuning: Training truncated models to mimic the outputs of the full base model to test recoverability.
Reasoning Chain Analysis: Comparing single-token predictions vs. multi-step reasoning chains (Summary $\to$ Caption $\to$ Reasoning $\to$ Conclusion) under truncation.

3. Key Findings & Results

RQ1: Representational Evolution

Visual Stabilization: Image tokens rapidly converge to a bounded-complexity regime. After an initial adjustment phase (typically within the first few layers), their entropy stabilizes, intrinsic dimensionality compresses, and trajectory curvature becomes near-constant.
Textual Dynamics: In contrast, textual tokens continue to undergo substantial restructuring, spectral redistribution, and curvature fluctuations throughout the entire depth of the network.
Conclusion: Visual representations stabilize structurally early, while text representations remain dynamically evolving.

RQ2: Functional Interchangeability

Depth Invariance: Once visual representations stabilize, they become functionally interchangeable. Substituting image tokens from deep layers with those from shallow layers (or vice versa) preserves semantic output with high similarity ( $\approx 1.0$ ).
Text Sensitivity: Textual tokens are highly depth-sensitive; substituting them across layers causes significant semantic degradation.
Implication: Deeper layers do not fundamentally restructure visual information; they primarily preserve and refine it.

RQ3: Task-Dependent Utility

Single-Token vs. Multi-Token: The necessity of visual depth is highly task-dependent.
- Single-Token Prediction (e.g., MCQ): Relatively robust to early truncation. Coarse decision boundaries can be formed with limited visual processing.
- Multi-Token Generation (e.g., Captioning, Open-ended VQA): Highly sensitive to early truncation. Performance degrades sharply if visual processing is removed before deep layers, as sequence generation requires repeated visual conditioning.
Conclusion: Visual depth requirements scale with output complexity.

RQ4: Recoverability via Fine-Tuning

Partial Recovery: Fine-tuning truncated models (using distillation from the base model) can partially restore performance, particularly for descriptive tasks (captioning).
Limits of Recovery: Recovery is gradual and limited under aggressive early truncation. Structured tasks requiring precise visual-text alignment (e.g., ChartQA) show lower asymptotic recovery compared to semantic generation.
Stability: Fine-tuning reduces the coefficient of variation across different decoding strategies (greedy, sampling, etc.), making the model more robust to stochastic generation.
Efficiency: Fine-tuned models achieve higher accuracy per unit of compute (GFLOPs) than truncated base models, shifting the compute-performance frontier.

RQ5: Reasoning Chain Sensitivity

Hierarchical Degradation: Under truncation, the quality of reasoning components degrades hierarchically: Caption > Reasoning > Summary. Summaries (coarse abstraction) remain stable, while captions (fine-grained grounding) degrade fastest.
Reasoning Does Not Compensate: Generating explicit reasoning chains does not compensate for reduced visual depth. In fact, structured reasoning is more sensitive to truncation than single-token answers because it requires repeated cross-modal grounding across multiple decoding steps.

4. Key Contributions

Structural Asymmetry Discovery: The paper provides the first systematic evidence that visual and textual representations evolve differently in VLMs, with visual tokens stabilizing early while text tokens continue to reorganize.
Task-Dependent Necessity: It challenges the "one-size-fits-all" approach to token processing, demonstrating that visual depth is critical for multi-token generation but less so for single-token classification.
Recoverability Analysis: It quantifies the limits of distillation-based recovery, showing that while coarse semantics can be redistributed, precise visual grounding is structurally difficult to recover once early layers are removed.
Efficiency Guidelines: The work offers concrete guidance for efficient VLM design, suggesting that visual processing can be truncated in the prefill stage for simple tasks, but must be preserved for complex reasoning and generation.

5. Significance

This study fundamentally questions the assumption that deeper visual processing is uniformly essential in VLMs.

For Researchers: It introduces "representation stabilization" as a measurable signal for identifying redundant depth in multimodal architectures.
For Practitioners: It enables task-aware efficiency trade-offs. For example, in latency-constrained scenarios, one can safely truncate visual processing for short-answer VQA but must preserve it for image captioning or complex reasoning.
Architectural Impact: The findings suggest that future VLM architectures should not treat visual and textual tokens identically; instead, they could employ dynamic depth allocation where visual tokens are processed shallowly for simple tasks and deeply for complex generation, potentially reducing inference costs significantly without sacrificing performance.