Technical Summary: Real Images, Worse Judgments

Problem Statement

The paper investigates a critical assumption in multimodal AI: that visual inputs inherently improve language understanding. Specifically, the authors examine whether Vision-Language Models (VLMs) can distinguish between useful visual evidence and incidental image context when making lexical judgments.

The study focuses on concreteness (how directly a word denotes something perceptible) and imagery (how easily a word evokes a mental image). These are well-established psycholinguistic properties typically rated by humans on a word-level basis, independent of any specific picture. The core research question is: When a VLM is asked to rate the concreteness or imagery of a word, does the presence of a real image (even if irrelevant to the word's definition) degrade the model's alignment with human judgments? The authors hypothesize that models may treat retrieved images as direct evidence for a word's properties, leading to "shortcut learning" where incidental visual cues override the intended lexical judgment.

Methodology

Datasets and Models

• Datasets: The evaluation uses two standard human-rating datasets:
  • MT40k (Brysbaert et al., 2014) for concreteness (5-point scale).
  • CP2004B (Clark and Paivio, 2004) for imagery (7-point scale).
  • The evaluation set consists of ~2,000 single-word entries from each dataset.
• Visual Contexts: For each word, the models are tested under five conditions:
  1. None: No image.
  2. White: A blank white image.
  3. Noise: A random-noise image.
  4. ImageNet: A retrieved natural image.
  5. Wikimedia: A retrieved natural image.
  • Note: For abstract or low-imagery words, retrieved images are "best-effort" matches and often lack clear semantic grounding.
• Models: The study evaluates instruction-tuned text-only models (LLaMA 2, Mistral, Qwen2.5) and their corresponding VLM counterparts (LLaVA 1.5, InternVL3, Pixtral, Qwen2.5-VL, Gemma 3).

Experimental Design

1. Prediction Evaluation: Models are prompted to assign a numerical rating to a target word. Predictions are compared against human ratings using Root Mean Square Error (RMSE).
2. Probing: Layer-wise hidden representations are probed using ridge regressors to determine how recoverable concreteness and imagery properties are from the model's internal states under different visual contexts.
3. Canonical Correlation Analysis (CCA): Used to measure the similarity between hidden representations generated under "None" (no image) versus real-image contexts. Lower correlation indicates a representational shift.
4. Attribution Analysis: Integrated Gradients (LayerIG) are used to visualize token-level attributions, determining whether the model relies more on text or image tokens during prediction.
5. Human Validation: A matched human study (500-word subset) was conducted to verify if the observed degradation is unique to VLMs or a general pragmatic effect of image salience.
6. Mitigation: A simple inference-time prompt intervention was tested: "The image may be irrelevant to the word; please base your rating solely on the word itself."

Key Results

1. Real Images Degrade Alignment

Contrary to the assumption that visual input enriches understanding, real-image contexts often hurt alignment with human ratings, particularly for abstract and low-imagery words.

• Text-only models (or VLMs with no image/uninformative images) often outperform their VLM counterparts on these subsets.
• Concrete/High-imagery words sometimes see improved performance with real images, suggesting visual grounding is beneficial only when the visual evidence is semantically aligned with the lexical target.
• RMSE Analysis: For abstract/low-imagery words, the RMSE increases significantly when real images are present compared to "None" or "White" contexts.

2. Mechanism of Failure

The degradation is linked to specific representational changes:

• Representational Shifts: CCA reveals that hidden representations under real-image contexts diverge significantly from those under "None" contexts, indicating that the model's internal state is altered by the visual input even when it should be irrelevant.
• Reduced Recoverability: Probing shows that concreteness and imagery properties are less recoverable from hidden states when real images are present. The text-token representations in VLMs are negatively affected by the presence of incidental visual cues.
• Spurious Sensitivity: Attribution analysis (e.g., on the word "nature") shows a shift in reliance from text tokens to image tokens when real images are present. This correlates with an upward bias in ratings (overestimating concreteness/imagery), suggesting the model is "hallucinating" concreteness based on the visual scene rather than the word's definition.

3. Human Comparison

A matched human study confirmed that while humans also show a slight upward shift in ratings when viewing images, the magnitude of degradation is significantly larger in VLMs. Humans do not lose calibration to the same extent, indicating that the VLM failure is not merely a pragmatic effect of image salience but a specific vulnerability in how these models integrate visual context.

4. Mitigation via Prompting

A simple instruction to "base your rating solely on the word itself" partially mitigates the degradation.

• This intervention reduces errors most effectively for abstract and low-imagery words.
• The gains are selective; performance on concrete words sometimes slightly decreases or remains unchanged, indicating a trade-off.
• This suggests the issue is partly a "shortcut learning" behavior at inference time that can be counteracted by explicit instructions.

Contributions

1. Comprehensive Evaluation: The paper provides a systematic evaluation of instruction-tuned VLMs on human concreteness and imagery judgments across multiple visual contexts.
2. Identification of a Failure Mode: It demonstrates that real-image contexts can actively harm lexical judgment alignment, specifically for words where visual evidence is weak or incidental.
3. Human Validation: It validates that this degradation is a VLM-specific sensitivity to incidental visual cues, distinct from general human pragmatic effects.
4. Diagnostic Analysis: Through probing, CCA, and attribution, the paper links performance drops to representational shifts and increased sensitivity to spurious visual cues.
5. Inference-Time Mitigation: It demonstrates that lightweight prompt engineering can reduce these errors, offering a practical guardrail for noisy or incidental visual contexts.

Significance and Claims

The authors claim that current instruction-tuned VLMs lack the calibration to determine when visual context is relevant to a specific linguistic judgment. The study suggests that:

• Visual grounding is not universally beneficial; it can introduce noise that overrides lexical knowledge.
• The models exhibit a form of shortcut learning where the presence of an image triggers a reliance on visual features, even when the task requires ignoring them.
• The findings highlight a need for better mechanisms to control when visual evidence should inform lexical judgments, particularly in scenarios involving abstract concepts or incidental imagery.

The paper concludes modestly, noting that while prompt-based mitigation helps, stronger solutions may require adaptive calibration strategies or model-specific training to better distinguish between relevant and incidental visual evidence.