💻 computer science

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

This paper reveals that Multimodal Large Language Models successfully encode coarse-grained visual evidence but struggle to reliably control their reliance on it versus language priors, a limitation that can be mitigated through supervised fine-tuning and activation steering.

Original authors: Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan, Xi Liu, Serge Belongie, Vésteinn Snæbjarnarson

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan, Xi Liu, Serge Belongie, Vésteinn Snæbjarnarson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a super-smart robot that has read almost every book ever written and watched millions of hours of movies. This robot is a "Multimodal Large Language Model" (MLLM). It's like a genius librarian who knows everything about the world from its training data—its "prior knowledge." But this robot also has eyes. It can look at a picture and combine what it sees with what it knows. The big question scientists have been asking is: When the robot looks at a picture that contradicts what it knows (like a picture of a horse with five legs), does it fail because it can't see the extra leg, or does it fail because it sees the leg but decides to ignore it? This paper dives into that mystery, treating the robot's brain like a detective story to figure out where the breakdown happens: in the "senses" (seeing) or in the "decision-making" (using what it sees).

The researchers, led by Jiaang Li and colleagues, set out to solve this puzzle using a clever two-part investigation. First, they wanted to know if the robot was actually "seeing" the weird stuff in the pictures. To test this, they didn't just ask the robot questions; they tried to rebuild the pictures from the robot's internal brain signals. They found that even when the robot gave the wrong answer, the "blueprint" of the weird picture (like a horse with five legs) was still clearly visible in its final thoughts. It was as if the robot's eyes were working perfectly, but its brain was choosing to close them. This ruled out the idea that the robot was just blind to the details.

Next, the team created a new game called "WhatIfVis" to test how well the robot could switch between trusting its eyes and trusting its memory. They showed the robot pictures that broke the rules of nature (like a red watermelon) and asked it to answer questions in two ways: "Look only at the picture" or "Ignore the picture and use what you know." They found that without extra training, the robot was a bit of a mood swing. Sometimes it ignored the picture and gave the textbook answer (like saying a watermelon is green even when it's red). Other times, it got so stuck on the picture that it couldn't follow instructions to ignore it. The robot wasn't broken; it just didn't have a reliable "switch" to decide when to look and when to think.

However, the good news is that the researchers found the switch. By giving the robot a little bit of practice (called Supervised Fine-Tuning) on just one type of tricky picture, they taught it how to control its attention. Even better, they discovered a specific "knob" inside the robot's brain—a single mathematical direction—that could turn the robot's focus on or off without needing any verbal instructions at all. When they turned this knob, the robot became much better at following the rules, whether it was looking at the picture or ignoring it.

The study also compared how the robot handled pictures versus written sentences. They found that it was much easier to make the robot follow a written sentence that broke the rules than a picture. It's like the robot is a much better listener than a viewer. This gap actually got bigger as the robots got smarter and larger, suggesting that as these models grow, they might get even more stubborn about ignoring what they see.

In the end, the paper suggests that for the big, obvious things we study (like color, size, and how many legs an animal has), these AI models aren't failing because they can't see. They fail because they haven't learned a consistent way to decide when to trust their eyes and when to trust their memory. But since the researchers found a specific "knob" that controls this behavior, it means we might be able to fix it in the future, teaching these digital brains to be more flexible and reliable when the world doesn't look exactly like their textbooks.

Technical Summary: Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

1. Problem Statement

Multimodal Large Language Models (MLLMs) integrate visual inputs with the rich parametric priors of pretrained language models (LLMs). While effective for many tasks, they frequently fail on vision-centric tasks when visual evidence conflicts with their internal world knowledge. The field has traditionally attributed these failures to perception deficits (Hypothesis H1), arguing that vision encoders discard fine-grained details, forcing the model to fall back on priors.

However, the authors observe a dual failure mode:

Prior-dominant failures: The model ignores anomalous visual input (e.g., a horse with 5 legs) and answers based on its prior (4 legs).
Context-dominant failures: The model ignores explicit instructions to ignore the image and answers based on visual evidence even when it contradicts canonical knowledge (e.g., answering "Red" for a watermelon's rind when asked for its usual color).

The paper posits that the core issue is not necessarily that the model cannot see the evidence, but that it lacks a controllable policy for when to rely on visual context versus parametric priors (Hypothesis H2). This is termed Visual Context Sensitivity. The authors aim to disentangle perception (seeing) from utilization (using) to determine where the bottleneck lies and whether it can be controlled.

2. Methodology

The study employs a three-pronged approach to isolate and address the utilization failure:

A. Probing Perception via Image Reconstruction (Testing H1)

To determine if visual evidence is degraded before reaching the LLM backbone, the authors perform a generative probe.

Setup: They freeze MLLMs and attempt to reconstruct counterfactual images solely from the final-layer image tokens.
Design: Using a framework adapted from Metaquery, they pair a frozen MLLM with a trainable connector and an image decoder (SANA). Crucially, the connector and decoder are trained only on real-world images, never seeing the counterfactuals.
Goal: If the decoder can successfully reconstruct the counterfactual attributes (e.g., anomalous limb counts) from the frozen MLLM's tokens, the visual evidence was preserved in the representation, refuting H1.

B. The WhatIfVis Benchmark (Testing H2)

To measure the model's ability to switch between visual context and prior knowledge, the authors introduce WhatIfVis, a benchmark of 3,049 counterfactual samples.

Structure: It covers five coarse-grained dimensions: Spatial-Temporal, Color, Count, Size, and Weight.
Mechanism: Each example pairs a counterfactual image (or text statement) with a question. The model is tested under two strict intent instructions:
1. Context-dominant: "Consider only the image/statement."
2. Prior-dominant: "Ignore the image/statement."
Metric: They use PairAcc (Pair Accuracy), a conjunctive metric requiring the model to answer correctly for both intents on the same example. This isolates the ability to control context sensitivity, rather than just accuracy on a single task.

C. Activation Patching and Subspace Steering

To locate and control the mechanism governing the vision-versus-prior trade-off:

Activation Patching: Using Supervised Fine-Tuned (SFT) models that reliably follow both intents, the authors apply causal tracing (activation patching) to identify the specific layers where the trade-off is computed.
Subspace Steering: They hypothesize that the trade-off is encoded in a one-dimensional subspace (a vector) within the residual stream. They learn a unit vector $u$ that spans this subspace.
Control: Instead of using text instructions, they apply a scalar constant $c(w)$ along this learned vector during inference to steer the model toward either the visual context or the prior.

3. Key Findings and Results

1. Visual Evidence is Preserved (Refuting H1)

Reconstruction Success: Across three distinct model families (Qwen2.5-VL, Qwen3.5, Gemma-4), coarse-grained counterfactual attributes (identity, pose, count, dominant color) were successfully reconstructed from the final-layer image tokens.
Metric Evidence: Reconstructions from counterfactual tokens were metrically closer to the counterfactual reference than to the real-world image (e.g., mIoU > 0.86 for counterfactuals vs. < 0.83 for real images).
Conclusion: The visual evidence is not degraded during perception. The failure to use it is a utilization issue, not a perception issue.

2. Vanilla Models Lack Stable Control (Confirming H2)

Brittleness: Vanilla (non-SFT) models show unstable context sensitivity. They often follow the image for some attributes (e.g., Color) but ignore it for others (e.g., Weight), or vice versa.
PairAcc Collapse: Vanilla models achieve near-zero PairAcc on reasoning-heavy tasks (Count, Weight) and low scores on others, indicating they cannot reliably switch between intents on demand.
SFT Improves Controllability: Supervised Fine-Tuning on a single subset (Spatial-Temporal) significantly lifts PairAcc across all tasks and models, demonstrating that controllability is a learnable, generalizable capability.

3. The Trade-off is Localized and Controllable

Layer Localization: Activation patching reveals that the vision-versus-prior trade-off is resolved in a narrow, recurring band of layers (mid-to-upper portion) specific to each architecture.
Steering Vector: A single learned vector in this subspace acts as a "knob." Applying this vector without any intent instruction improves controllability over the vanilla model (raising macro PairAcc from 14.7% to 37.7%).
Specificity: Random vectors in the same subspace degrade performance, confirming the effect comes from the specific learned direction, not just the rank-1 intervention.

4. Asymmetry Between Modalities

Text vs. Vision: The modality gap is significant. Text-based context sensitivity is far more controllable than visual context sensitivity.
Scaling: As model size increases, the gap between text and vision controllability widens (e.g., from 9.1 pp to 20.6 pp in Qwen2.5-VL).
Root Cause: The deficit is primarily a failure to follow visual evidence on demand, not a failure to override it. The ability to ignore evidence (prior intent) is nearly identical across modalities.

4. Significance and Claims

The paper claims to relocate the bottleneck in MLLM failures from perception to utilization.

Redefining the Problem: For coarse attributes, MLLMs encode visual evidence faithfully but lack a consistent, controllable policy for when to use it. The issue is not "blindness" but "unreliable reliance."
Mechanism Identification: The vision-versus-prior trade-off is a locatable, compact property of the network architecture, residing in specific layers and controllable via a learned vector.
Control without Instructions: The authors demonstrate that visual context sensitivity can be steered effectively without explicit prompt instructions, simply by manipulating a learned subspace vector.
Modality Gap: The work highlights a fundamental asymmetry where MLLMs are significantly better at following textual counterfactuals than visual ones, a gap that persists and grows with model scale.

The authors conclude that separating "seeing" from "using" allows for a firmer footing in building multimodal models that can be explicitly directed to heed visual evidence when it contradicts their priors.