💬 NLP

Pathways of Visual Information Flow in Vision-Language Models

This paper reveals that vision-language models utilize two distinct, task-dependent pathways for visual information flow—a direct route and a text-mediated route—that exhibit remarkable flexibility and can switch to a fallback mechanism under specific interventions.

Original authors: Israfel Salazar, Stella Frank, Dan Oneata, Desmond Elliott, Constanza Fierro

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Israfel Salazar, Stella Frank, Dan Oneata, Desmond Elliott, Constanza Fierro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Vision-Language Model (VLM) as a highly skilled detective who is trying to solve a mystery based on a photograph and a written question. The paper asks a simple but deep question: How does the detective actually look at the photo to find the answer? Does the detective look at the photo directly, or do they first describe the photo to a colleague, who then tells the detective the answer?

The researchers discovered that these AI models don't just use one way of thinking. Instead, they have two different "routes" or pathways to get information from the image to the final answer, and they switch between them depending on the situation.

Here is a breakdown of their findings using simple analogies:

1. The Two Pathways

Think of the AI's brain as a busy office with two ways to get a file from the "Image Department" to the "Final Decision Maker" (the last token that types the answer).

The Direct Highway (Direct Pathway):
- How it works: The Final Decision Maker looks straight at the Image Department. They grab the visual information directly, like grabbing a file straight off a desk.
- When it's used: The paper found this is the main route for Object Recognition (e.g., "What is this object?"). It's fast and direct. The model sees a star and says "star" without needing to talk about it first.
The Messenger Route (Text-Mediated Pathway):
- How it works: The Image Department doesn't talk to the Final Decision Maker directly. Instead, they hand the visual information to the "Question Writer" (the text tokens). The Question Writer processes the image, turns it into words, and then passes that information to the Final Decision Maker.
- When it's used: This is the main route for Spatial Relations (e.g., "Where is the cup relative to the plate?"). The model seems to need to "think about" the relationship in words before answering.

2. The Switching Mechanism

The most surprising discovery is that the model doesn't get stuck on just one route. It's like a driver who usually takes the highway but can instantly switch to a back road if the highway is blocked.

Task Dependence: The model chooses the route based on the job.
- Simple ID tasks: Uses the Direct Highway.
- Complex relationship tasks: Uses the Messenger Route.
Data Dependence: The route can change based on the type of picture. For example, when identifying where an object is (Localization), the model uses the Direct Highway on simple, computer-generated shapes but switches to the Messenger Route when looking at complex, real-world photos.
Prompt Dependence: Even the way you ask the question changes the route. If you give the model a multiple-choice list (e.g., "Is it A, B, or C?"), it tends to switch to the Messenger Route. If you let it answer freely, it often takes the Direct Highway.

3. The "Backup Plan" Discovery

This is the most critical part of the paper. The researchers tested what happens if they "break" one of the routes.

The Experiment: They used a technique called "causal patching" (imagine swapping the detective's memory with a different scenario) and "attention knockouts" (imagine putting a blindfold on the detective so they can't look at the photo directly).
The Result:
- When they blocked the Direct Highway, the model didn't fail. Instead, it instantly switched to the Messenger Route and still got the right answer!
- Even for tasks where the model usually looks directly at the photo (like recognizing an object), if you force it to stop looking, it can still solve the problem by relying on the text descriptions it generated along the way.

4. Why This Matters (The "Aha!" Moment)

The paper explains that previous studies got confused because they only looked at one side of the story.

Some researchers blocked the direct path and saw the model still working, so they concluded the model always uses the text route.
Others looked at normal behavior and saw the model looking directly at the image, so they concluded the text route wasn't needed.

The Truth: The model has both routes active at the same time. It usually picks the most efficient one for the specific task. However, if you mess with the model (like blocking a path), it activates its "backup plan."

The Takeaway: You cannot assume you know how an AI thinks just by breaking parts of it. If you break the "Direct Highway," the AI might just take the "Messenger Route" and still succeed, making it look like it never needed the highway in the first place. The paper shows that these models are incredibly flexible, having multiple ways to solve the same puzzle depending on how you ask them and what you allow them to do.

Technical Summary: Pathways of Visual Information Flow in Vision-Language Models

Problem Statement

While Vision-Language Models (VLMs) successfully solve complex multimodal tasks, the specific mechanisms by which visual information is routed and integrated within the language model during text generation remain unclear. Previous research has yielded conflicting conclusions regarding these mechanisms: some studies suggest visual information flows directly from image tokens to the final output token (direct pathway), while others argue that visual information is always mediated through text/query tokens before reaching the final token (text-mediated pathway).

A critical limitation in prior work is the reliance on attention interventions (e.g., attention knockouts). These methods primarily test necessity (whether a model can solve a task without a specific component) rather than usage (whether the model typically uses that component during standard inference). Consequently, it remains uncertain how VLMs actually route information under normal conditions and whether alternative pathways exist as latent fallbacks.

Methodology

The authors employ causal patching with counterfactual examples to distinguish between what models could do versus what they typically do. The study utilizes a controlled experimental design across three visual tasks:

Object Recognition: Identifying entities.
Spatial Relations: Reasoning about positions between entities.
Object Localization: Mapping entities to global positions.

Data: Experiments are conducted on both synthetic datasets (colored shapes with controlled primitives) and natural datasets (e.g., COCO, Visual Genome, What's Up) to test generalization.

Models: The study evaluates Qwen3-VL-4B, InternVL3.5-4B, and LLaVA-1.5-7B.

Interventions:

Causal Patching: Hidden representations from a counterfactual input (different image, same text) are inserted into the forward pass of an original input at specific layers. This measures the "Restoration Score," indicating whether the patched token set contains sufficient information to change the model's prediction.
Attention Knockouts: Specific attention edges are suppressed to test the necessity of a pathway.
Corrupted-Input Patching: Text representations from a valid image are patched into an inference run where the image is replaced by random noise, testing if text tokens alone encode sufficient visual signal.

Pathway Definitions:

Direct Pathway ( $r_{I \to last}$ ): The final token directly attends to image tokens.
Text-Mediated Pathway ( $r_{I \to T} \to r_{T \to last}$ ): Visual information transfers from image tokens to text/query tokens, which are then read by the final token.

Key Results

1. Task-Dependent Pathway Selection

The study identifies that VLMs utilize distinct pathways depending on the specific task:

Object Recognition: Relies on the direct pathway. Causal patching shows that the final token reads visual information directly from image tokens throughout the network; patching text tokens has no effect on the output.
Spatial Relations: Relies on the text-mediated pathway. Visual information is transferred to text tokens at intermediate layers. Once this transfer occurs, patching the image tokens no longer affects the prediction, indicating the final token relies on text representations.
Object Localization: The pathway selection is modulated by data distribution.
- On synthetic data, models use the direct pathway.
- On natural data, models switch to the text-mediated pathway, likely due to the increased visual complexity and need for grounding in distractor-rich scenes.

2. Modulation by Prompt Design

Prompt design influences pathway selection. For object recognition, an "open" generation protocol (free-form text) maintains the direct pathway. However, a "choices" protocol (selecting from a fixed set of candidates) induces a shift toward the text-mediated pathway, suggesting that the constraints of the output space alter the internal routing strategy.

3. Pathways as Flexible Fallbacks

A significant finding is that these pathways are not rigid; they coexist as backup alternatives.

Corrupted-Input Patching: When the image input is replaced with noise, models can still recover correct answers using only the text-mediated pathway. This holds true even for tasks (like object recognition) where the direct pathway is dominant under normal inference.
Attention Knockouts: Blocking the direct pathway ( $r_{I \to last}$ ) has negligible impact on accuracy. The model successfully reroutes through the text-mediated pathway to maintain performance. Conversely, disrupting the text-mediated pathway causes substantial performance drops.

Significance and Claims

The paper claims to provide a mechanistic characterization of visual information flow, unifying previously conflicting findings in the literature. The authors argue that:

Intervention Bias: Standard ablation-based interventions (like attention knockouts) risk activating fallback pathways. A model's ability to solve a task without a component does not imply it normally uses that component. This explains why different studies have reported contradictory mechanisms for the same tasks.
Flexibility of Internal Mechanisms: VLMs possess flexible routing strategies. The text-mediated pathway encodes sufficient visual information to solve tasks even when the direct visual stream is ablated or corrupted.
Methodological Implication: To accurately characterize how models process information, researchers must combine usage-sensitive interventions (causal patching) with necessity tests (knockouts). Relying on a single method can lead to incomplete or misleading conclusions about model behavior.

The authors conclude that visual information integration in VLMs is not a single fixed mechanism but a dynamic system where pathway selection depends on the task, data distribution, and prompt design, with text-mediated routing serving as a robust, latent fallback mechanism.