Technical Summary: Mechanistic Diagnostics of Spatial Lexical Bias in Multimodal Large Language Model Spatial Reasoning

1. Problem Statement

Multimodal Large Language Models (MLLMs) frequently fail at spatial reasoning tasks, particularly in multiple-choice question (MCQ) formats. While prior work often attributes these failures to weak visual grounding, missing visual evidence, or incorrect visual attention, this paper identifies a complementary and systematic failure mode: Spatial Lexical Bias.

The core phenomenon observed is Binary-Stable but Ternary-Fragile (BSTF) behavior. In BSTF cases, an MLLM correctly answers a binary spatial question (e.g., "Is the object left or right?") but consistently fails when a third, plausible spatial option is added (e.g., "left, right, behind"), even if the correct answer remains the same. The model systematically selects the newly added spatial word as the answer, regardless of the option order. This suggests the failure is not due to a loss of visual information but rather a language-side decision process distortion triggered by the semantic structure of the added spatial vocabulary.

2. Methodology

2.1 Diagnostic Framework

The authors constructed a controlled evaluation pipeline using nine open-weight MLLMs (from LLaVA, Qwen, and InternVL families) and three synthetic datasets (Sphere–Cube, Sphere–Dog, and Sphere–Dog Outdoor).

• Strict Scoring: To eliminate position bias, a sample is considered correct only if the model answers correctly across all permutations of option order (e.g., all 6 permutations for ternary questions).
• BSTF Selection: Cases were filtered where binary robust accuracy was perfect (1.00) but adding a specific spatial option caused a drop of at least 80 percentage points in ternary accuracy. This yielded 55 high-confidence diagnostic cases across 7 models.

2.2 Mechanistic Interpretability

To determine the locus of the failure, the authors employed three diagnostic tools:

1. Visual Attention Analysis: Comparing attention maps between binary-correct and ternary-wrong prompts to verify if the model shifts focus away from relevant objects.
2. Residual Stream Probing: Training linear probes on the last-token residual stream vectors to check if the correct spatial relation remains decodable even when the model outputs a wrong answer.
3. Activation Patching & Sparse Intervention:
   • Patching: Swapping residual stream states between binary-correct and ternary-wrong prompts layer-by-layer to identify where the decision signal switches.
   • Sparse Knockout: Zeroing out specific residual-stream channels and MLP neurons at the position of the added option token to see if the error can be corrected.

2.3 Mitigation Strategy

The authors tested whether the bias could be repaired via LLM-only Direct Preference Optimization (DPO).

• Data: A minimal synthetic preference dataset derived from a single object pair (torus–pyramid) with four relation variants.
• Training: Only the LLM decoder was updated using LoRA adapters; the vision encoder and projector remained frozen.
• Goal: To verify if adjusting the language-side decision process alone could mitigate the bias without new visual supervision.

3. Key Findings

3.1 Visual Evidence is Recoverable

• Attention: Attention heatmaps for binary-correct and ternary-wrong prompts focus on the same object regions. The added spatial option does not redirect visual attention away from the relevant evidence.
• Probing: Linear probes recover the true spatial relation from the last-token residual stream with nearly 100% accuracy (macro accuracy 0.998) even in ternary-wrong cases. This confirms the visual information is present internally but not utilized for the final decision.

3.2 The Bias is Language-Side and Structured

• Specificity: Replacing the added spatial option with an irrelevant word (e.g., "violin") caused only a modest accuracy drop (16.7 pp), whereas adding a spatial option caused a massive drop (97.8 pp). This confirms the bias stems from the semantic interaction between spatial words, not merely the increased number of choices.
• Localization:
  • Layer-wise: Activation patching revealed that the decision transition occurs in the mid-to-late layers (approx. 60–64% depth) of the LLM.
  • Component-wise: Sparse knockout of just 100 residual channels or 200 MLP neurons at the added-option token position (in early-to-mid layers) could restore correct predictions, identifying specific "leverage points" for the bias.

3.3 LLM-Only Repair is Effective

• Synthetic Data: Applying LoRA-DPO to the LLM decoder using the single-pair dataset improved four-way robust accuracy on the synthetic test set by up to 100 percentage points (e.g., from near 0% to ~80-100%).
• Generalization: The repair transferred to broader, noisier public benchmarks:
  • WhatsUp: +68.0 points
  • SpatialMQA-Direct: +32.6 points
  • VSR: +20.1 points
• Preservation: The repair maintained binary accuracy while fixing ternary and four-way failures, indicating the model learned to resist the lexical bias without losing its original spatial reasoning capabilities.

4. Significance and Claims

The paper claims to shift the understanding of MLLM spatial reasoning failures from a purely visual deficit to a language-side lexical bias.

• Mechanistic Insight: It provides causal evidence that specific LLM-side components (residual channels and MLP neurons) modulate spatial option selection, and that the correct visual relation remains available internally during these failures.
• Actionability: The study demonstrates that this bias is actionable. A lightweight, LLM-only update on minimal synthetic data can substantially repair the failure mode, suggesting that spatial reasoning robustness can be improved by tuning the language model's decision process rather than retraining the entire multimodal architecture.
• Limitations: The authors note that their findings are specific to controlled four-direction spatial relations. While the mechanism is localizable, the study does not claim to have mapped a complete circuit, and the repair's efficacy on complex, open-world scenes with richer spatial relations remains an open question.

In summary, the work establishes that MLLMs often "know" the correct spatial relation but are misled by the lexical structure of the answer options, and that this specific failure mode can be diagnosed and mitigated through targeted interventions on the language model side.