Technical Summary: Linguistic Context Recodes Visual Representations in Vision-Language Models

Problem Statement

Vision-language models (VLMs) are frequently conceptualized as systems where language queries retrieve or reason over static visual information stored in image tokens. Prior interpretability work has largely focused on how language tokens manipulate these pre-existing visual repositories. However, it remains poorly characterized whether VLMs dynamically recode visual representations in response to goal-directed language, similar to how biological visual systems employ top-down, goal-directed signals to direct attention and alter stimulus processing. This paper investigates the hypothesis that linguistic input does not merely query visual representations but actively modifies them to support specific goals articulated in text.

Methodology

The authors investigate two primary mechanisms of language-induced visual recoding: the addition of an abstract "reference" representation and the modulation of existing visual attributes.

Models and Datasets:
The study utilizes two transformer-based VLMs: Qwen 2.5 VL 7b and InternVL3 8b. Experiments employ:

1. Synthetic Datasets: Images containing multiple objects defined by unique (color, shape) pairs arranged on a grid, allowing for precise object-level token analysis.
2. Natural Image Dataset: A filtered subset of COCO containing well-separated object triplets to test generalization beyond synthetic settings.

Experimental Design:
The authors analyze three task types:

• Counting: Querying the number of objects with a specific attribute.
• Yes/No: Determining the presence of an attribute.
• Spatial: Identifying an object based on its spatial relation to a landmark.

Core Techniques:

1. Linear Probing: Training binary classifiers on hidden states to detect whether a vision token corresponds to a "referent" (goal-relevant) object versus a non-referent.
2. Contrastive Steering Vectors: Computing difference vectors between hidden states of an object when it is a referent versus a non-referent. These vectors are used for causal interventions (adding/subtracting vectors to hidden states) to test if they drive model predictions.
3. Visual Concept Vectors: Constructing vectors for specific shapes and colors by averaging representations of objects possessing those attributes. Projections of object representations onto these vectors measure the strength of attribute encoding.
4. Causal Interventions:
   • Steering: Modifying hidden states with reference vectors to flip model outputs (e.g., changing a "Yes" to "No" or altering the identified object in a spatial query).
   • Freezing: Halting the update of referent object hidden states from a specific layer onward to measure the impact on final answer confidence (using logit lens analysis).

Key Contributions and Results

1. Discovery of Abstract Reference Representations
The authors identify a linearly decodable "reference" representation within visual tokens that signals goal-relevance.

• Decodability: Linear probes can distinguish referent from non-referent objects with high accuracy in middle-to-late layers when goal-directed prompts are present. Control prompts (empty strings or irrelevant descriptions) fail to elicit this signal, confirming the change is driven by the specific query.
• Causality: Contrastive steering vectors derived from these representations causally influence model behavior. Adding a reference vector to a non-referent object (or subtracting it from a referent) successfully flips model predictions across Counting, Yes/No, and Spatial tasks.
• Generalization: These reference representations are abstract; vectors extracted from one task (e.g., Spatial) often generalize to others (e.g., Counting), and vectors derived from synthetic data effectively steer models on naturalistic COCO images.

2. Evidence of Attribute Modulation
Beyond marking relevance, language input selectively amplifies the representation of specific attributes within the referent object.

• Explicit Modulation: When prompted to "focus on shape," projections of object tokens onto shape concept vectors increase significantly relative to a baseline, while color projections remain stable or decrease, and vice versa for color-focused prompts.
• Implicit Modulation: Even when attributes are not explicitly queried, selecting an object (e.g., "the object in front") amplifies that object's specific shape and color representations compared to non-selected objects.
• Temporal Dynamics: Attribute modulation emerges in later layers, following the initial emergence of the reference signal in middle layers, suggesting a two-stage process: identification of the relevant object followed by the sharpening of its features.

3. Causal Role of Modulation
The paper demonstrates that attribute modulation mediates the model's response distribution.

• Freezing Experiments: Freezing the hidden states of referent objects in later layers (where modulation is strongest) significantly reduces the model's confidence (logit gap) in its final answer. This confirms that the continued processing and amplification of these attributes are causally necessary for generating confident responses.

Significance and Claims

The paper argues that visual representations in VLMs are not static repositories of information but are dynamically modulated by linguistic context. The findings support a more dynamic account of cross-modality processing where:

1. Reference Mechanism: VLMs construct abstract representations to denote which objects are relevant to a query, a mechanism that generalizes across tasks and domains.
2. Modulation Mechanism: VLMs selectively amplify goal-relevant attributes (shape, color) within the visual representation of the referent object to support downstream reasoning.

These mechanisms resemble goal-directed recoding observed in biological vision systems, where perceptual representations are modulated to support specific tasks like categorization or search. The authors conclude that linguistic input in pretrained VLMs serves to recode object-level visual representations to support the specific goals or queries specified in the text.

Limitations:
The authors note that their experiments rely heavily on controlled synthetic datasets with constrained spatial layouts. The steering vectors are extracted from these datasets, and modulation experiments currently focus only on shape and color, excluding other attributes like size or brightness. Additionally, the counting task showed lower intervention success rates, suggesting that not all tasks rely entirely on the identified reference signal.