Technical Summary: Mechanisms of Object Localization in Vision–Language Models

Problem Statement

Visually-grounded Language Models (VLMs) have demonstrated remarkable success in linking visual and textual information for tasks like visual question answering and captioning. However, they frequently struggle with fundamental vision tasks, specifically the accurate classification and localization of objects. While the mechanisms underlying classification have been studied, the processes supporting object localization remain poorly understood. This gap is critical because VLMs often inherit visual features from CLIP, which relies on global image-text supervision and lacks the pixel-level precision required for detection. Despite this, VLMs can successfully answer queries requiring object localization, suggesting they construct spatial structures from weakly grounded representations. This work investigates how these localization mechanisms emerge within VLM architectures.

Methodology

The authors conduct a mechanistic interpretability study on two representative VLM families: LLaVA-1.5 (a simple, interpretable baseline using CLIP ViT and Vicuna) and InternVL-3.5 (a state-of-the-art model featuring token compression and multi-view processing). The study utilizes a carefully curated subset of the COCO validation split, filtered to remove annotation errors and ambiguous cases. To control for hallucinations driven by contextual cues, an auxiliary "object-removed" dataset was created using inpainting.

The investigation employs a suite of mechanistic tools:

1. Token Ablations: Systematically removing specific visual tokens (object-aligned, register, gradient-based, or random) to assess their contribution to task performance.
2. Containerization Tests: Artificially expanding object masks by duplicating tokens and shuffling internal token arrangements to test if localization relies on token presence or semantic coherence.
3. Multi-View Ablation: Separately ablating global (low-resolution) and local (high-resolution) views in InternVL to determine the distribution of spatial vs. semantic information.
4. Positional Decoding: Training linear classifiers on intermediate model layers to determine how well spatial coordinates can be reconstructed from token embeddings.
5. Attention Knockout: Blocking attention flows from image tokens to object tokens in grouped layers to identify where task-specific processing occurs.
6. Causal Mediation Analysis (CMA): Using activation patching to quantify the causal contribution of specific attention heads to classification and localization tasks, measured via Mediation Fraction (MF).

Key Findings

1. Grounding through Containerization

Localization is driven by a "containerization" mechanism. The model groups object-aligned tokens into a spatial container that defines the object's boundaries. Crucially, the predicted bounding box is largely independent of the internal semantic arrangement of tokens within those boundaries. Experiments showed that shuffling tokens within an object mask or artificially expanding the mask with misplaced but object-related tokens caused the predicted box to scale accordingly, while classification remained robust to such perturbations.

2. Multi-View Integration

In architectures with dual views (global and local), the global view carries the dominant spatial signal for localization, while local high-resolution crops primarily refine classification, particularly for small objects. The two views provide complementary, rather than redundant, evidence. Removing global tokens caused a significant drop in localization accuracy, whereas local removal had a smaller effect, though both views were necessary for robust performance.

3. Implicit Spatial Layout Learning

The LLM reconstructs the 2D spatial structure of the image from the 1D token sequence. While positional information in the visual backbone (ViT) degrades over depth, the multimodal projection retains strong signals for image corners. These corner anchors allow the LLM to infer approximate row boundaries and a grid-like layout, which is sufficient for spatial reasoning.

4. Sparse, Task-Critical Attention Heads

A very small number of attention heads mediate the causal effect for both classification and localization.

• Layer Distribution: In LLaVA, these heads concentrate in early-to-mid layers; in InternVL, they appear in mid-to-late layers.
• Task Specificity: While early processing is shared, the dominant heads for localization and classification are largely disjoint.
• Sequential Mechanism: Localization causally depends on classification-critical heads. Ablating heads critical for classification significantly degraded localization performance, suggesting a sequential mechanism where object identification precedes spatial grounding.

Significance and Contributions

This work provides the first layer- and head-level account of localization in VLMs. By revealing that localization relies on narrow, non-redundant computational pathways rather than broad semantic understanding, the paper challenges the assumption that VLMs process spatial and semantic information uniformly.

The authors claim these findings offer a new perspective on how VLMs build spatial structure from weakly grounded representations. Specifically, the discovery of the "containerization" mechanism and the sequential dependency of localization on classification suggests that future model design and grounding objectives could be optimized by:

• Targeting specific attention heads for fine-tuning.
• Implementing grounding-aware attention supervision.
• Designing architectures that better preserve spatial signals in early layers or leverage the identified "container" mechanism more explicitly.

The study concludes that VLMs do not merely "see" objects but construct spatial boundaries through a specialized, sparse computational process that separates object identification from spatial extent definition.