💬 NLP

Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Models

This paper identifies that spatial bias in Large Vision-Language Models stems from attention mismatches between the vision encoder and language model rather than the encoder itself, and proposes a lightweight Adaptive Global Context Injection (AGCI) mechanism to mitigate this bias and enhance spatial robustness without architectural modifications.

Original authors: Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Youcheng Pan, Yongshuai Hou, Weili Guan, Jun Yu, Min Zhang

Published 2026-02-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Youcheng Pan, Yongshuai Hou, Weili Guan, Jun Yu, Min Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Model with "Tunnel Vision"

Imagine you have a very smart assistant who is great at looking at pictures and answering questions about them. You show them a photo of a cat sitting in a box. They say, "Yes, that's a cat."

Now, imagine you take that exact same photo, but you move the cat from the center of the box to the far left corner. You show it to the assistant again. Surprisingly, the assistant might get confused. They might say, "I don't see a cat," or give a completely different answer, even though the picture is identical except for the cat's location.

This paper calls this problem "Spatial Bias." It means the AI's understanding of an image changes depending on where things are located, rather than just what is there.

The Investigation: Where is the Glitch?

The researchers wanted to find out why this happens. They treated the AI like a two-part machine:

The Eyes (Vision Encoder): This part looks at the pixels and turns them into data.
The Brain (Large Language Model): This part takes that data and uses logic to answer questions.

They ran a series of tests (like a "spot the difference" game) to see which part was failing:

Did the Eyes fail? They checked if the "Eyes" could still see the cat clearly when it moved. Result: No, the Eyes worked perfectly. They saw the cat no matter where it was.
Did the Brain fail? They checked if the "Brain" understood the data the Eyes sent. Result: Yes, the Brain got confused.

The Root Cause:
The paper explains that the "Eyes" and the "Brain" speak different languages regarding how they share information.

The Eyes are like a group of friends sitting in a circle, all talking to each other at once. They share the whole picture globally.
The Brain is like a line of people passing a note down a chain. Person A can only talk to Person B, who can only talk to Person C. They cannot look back or see the whole group at once.

When the "Eyes" send the picture to the "Brain," the "Brain" tries to process it in a line. Because of this "one-way street" rule, the information gets distorted depending on where the object sits in the line. If the cat is at the end of the line, the "Brain" might lose track of the context.

The Solution: AGCI (The "Global Context" Injection)

To fix this, the authors created a lightweight tool called Adaptive Global Context Injection (AGCI).

Think of the "Brain" processing the image tokens (the pieces of the puzzle) one by one.

Before AGCI: Each piece of the puzzle only knows about the pieces before it. It's like trying to understand a story by only reading the first sentence of every paragraph.
After AGCI: The researchers added a "summary note" to every single piece of the puzzle. This note says, "Hey, remember, the whole picture is about a cat in a box."

This "summary note" is the Global Context.

If a piece of the puzzle is already clear, the note is light.
If a piece of the puzzle is confused or isolated, the note is stronger, reminding it of the big picture.

Crucially, this doesn't require rebuilding the AI's brain. It's like adding a small plugin or a "cheat sheet" that helps the AI remember the whole image while it's thinking.

The Results: Does it Work?

The researchers tested this on several popular AI models. Here is what happened:

Consistency: The models stopped getting confused when objects moved. Whether the cat was in the corner or the center, the AI gave the same correct answer.
Better Reasoning: The models got better at answering complex questions, not just simple ones.
Fewer Hallucinations: The models started making up fewer fake details (like seeing a dog when there was only a cat).
Reading Text: The models got much better at reading text inside images (like signs or books), which is very sensitive to where the text is located.

Summary

The paper discovered that big AI models are "spatially biased" because their "brain" processes images in a line, losing the big picture. They fixed this by giving every part of the image a reminder of the whole scene. This made the AI more reliable, consistent, and accurate without needing a massive overhaul of its architecture.

Technical Summary: Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Models

1. Problem Statement

Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks but exhibit insufficient robustness regarding spatial variations. Specifically, current LVLMs often produce inconsistent or contradictory outputs when identical key visual information is placed at different locations within an image. This phenomenon, termed spatial bias, suggests that the models' semantic understanding is not position-invariant but rather biased toward specific spatial preferences.

While previous research has linked spatial bias to object hallucination or the long-term decay effects of Rotary Position Embedding (RoPE), this work argues that these explanations lack generalizability or fail to address the root cause in fundamental semantic understanding. The paper posits that the bias stems not from the vision encoder's perception or encoding capabilities, but from a mismatch in attention mechanisms between the vision encoder and the Large Language Model (LLM) backbone.

2. Methodology and Analysis

A. Systematic Probing of Spatial Robustness
To quantify spatial bias, the authors constructed a controlled probing dataset based on image-text matching.

Dataset Construction: 10,000 image-caption pairs were sampled from LAION. For each key image, 8 distractor images were retrieved. The key image and distractors were arranged in a $3 \times 3$ grid to form a composite image.
Experimental Setup: The key image was systematically moved to each of the 9 grid positions ( $n=0$ to $8$) while keeping the caption and distractors constant. The model was asked a binary question to determine if the composite contained the target image.
Findings: Experiments on six representative LVLMs (including Qwen2.5-VL, LLaVA-v1.6, and InternVL3) revealed significant accuracy fluctuations across different grid positions. Notably, performance did not correlate strongly with token distance, suggesting the bias is not primarily caused by RoPE's long-term decay.

B. Root Cause Analysis
The authors decomposed the visual processing pipeline into two stages to isolate the source of bias:

Perception Stage (Eraser Search): Using masking experiments, the study confirmed that LVLMs consistently identify critical regions regardless of their spatial location. Perception is spatially robust.
Semantic Understanding Stage (Vision Encoder): By measuring the cosine similarity between visual features and text embeddings across different positions, the study found that the vision encoder produces stable, semantically aligned representations independent of location.
The LLM Bottleneck: Since both perception and vision encoding are robust, the bias originates in the LLM's cross-modal reasoning. The authors hypothesize that the causal self-attention mechanism in LLMs (which restricts tokens to attending only to previous tokens) disrupts the bidirectional global information flow established by the Vision Transformer (ViT) encoder. This disruption causes image tokens to lose access to holistic global context as they propagate through the LLM layers, making their contribution overly dependent on their position.

C. Proposed Solution: Adaptive Global Context Injection (AGCI)
To mitigate this without architectural modifications, the authors propose AGCI, a lightweight mechanism that dynamically injects shared global visual context into each image token.

Mechanism:
1. Compute the global image context $\tilde{g}$ as the mean representation of all image tokens.
2. Calculate the semantic similarity $w_i$ between each image token $\tilde{v}_i$ and the global context.
3. Inject the global context into each token with an adaptive strength proportional to $(1 - w_i)$ :
  $\tilde{v}'_i = \tilde{v}_i + \lambda(1 - w_i)\tilde{g}$
4. Here, $\lambda$ is a hyperparameter acting as a safety valve to preserve fine-grained details. Tokens with lower global semantic relevance receive stronger supplementation.

3. Key Results

A. Spatial Robustness (Probing Task)

AGCI significantly reduced accuracy variance ( $\Delta$ ) across grid positions for all tested models.
For models with high initial bias (e.g., LLaVA-v1.6), AGCI reduced variance from 14.93 to 8.05 and improved average accuracy from 70.82 to 80.75.
For larger models (e.g., Qwen2.5-VL-72B), AGCI improved average accuracy from 70.12 to 86.11 while maintaining low variance.

B. Downstream Task Performance
AGCI was evaluated on six downstream benchmarks with lightweight LoRA fine-tuning:

Complex Reasoning (MMMU-Pro, ScienceQA): AGCI improved performance, particularly in 10-option settings, indicating better integration of visual cues and global understanding.
Hallucination Benchmarks (CRPE, HallusionBench, POPE): AGCI generally reduced hallucination tendencies and improved prediction reliability. For instance, on POPE, Qwen3-VL's average accuracy improved from 84.51 to 86.96.
OCR Tasks (TextVQA): AGCI showed strong gains (e.g., +4.93 for LLaVA-v1.6), suggesting that global context helps aggregate dispersed textual regions.

C. Information Flow Visualization
Visualizations of attention maps demonstrated that baseline models often focused on irrelevant spatial regions (e.g., the top-left corner) due to positional bias. After applying AGCI, attention became more concentrated on task-relevant regions (e.g., specific OCR text), confirming that AGCI restores balanced contextual representation.

4. Significance and Claims

The paper claims three primary contributions:

Systematic Investigation: It provides empirical evidence that LVLM predictions vary significantly with the spatial location of identical visual content, highlighting a critical weakness in current spatial understanding.
Root Cause Identification: It pinpoints the source of spatial bias to the loss of global image context caused by the causal attention mechanism in LLMs, explicitly ruling out the vision encoder and RoPE decay as the primary culprits.
Effective Mitigation: It introduces AGCI, a simple, architecture-agnostic method that enhances spatial robustness and downstream performance without sacrificing intrinsic capabilities.

The authors emphasize that AGCI works by restoring stable access to global context during cross-modal reasoning, thereby alleviating the inconsistency caused by the mismatch between bidirectional vision encoding and causal language modeling. The work suggests that explicitly reinforcing global visual semantics is crucial for achieving robust, position-invariant multimodal understanding.