Technical Summary: Seeing Is Not Sharing

Problem Statement

In collaborative, situated dialogue, shared perception does not guarantee shared interpretation. While dialogue participants incrementally establish mutual understanding (grounding) through interaction, an external observer (or "overhearer") lacks the ability to contribute grounding acts. This creates an "overhearer illusion," where an observer may mistake the mere co-presence of potential referents for established common ground.

This paper investigates whether Vision-Language Models (VLMs), when acting as overhearers of information-asymmetric dialogues, can distinguish between potential common ground (what could be shared based on visual or textual map content) and established common ground (what has been confirmed through dialogue interaction). Specifically, the study asks if VLMs overestimate alignment by conflating the availability of referential overlap with the successful completion of the grounding process.

Methodology

Dataset and Task

The study utilizes a perspectivist annotation of the HCRC MapTask corpus (Li et al., 2026), comprising 13,077 annotated reference expressions (REs) from 128 dialogues. In these dialogues, two participants (a "giver" and a "follower") navigate routes on maps that differ in landmark placement, names, or quantities.

• Task Formulation: The task is framed as a binary interpretation-matching judgment. Given a dialogue excerpt and map information, the model must determine if the speaker's and addressee's interpretations of a specific RE match (YES/Aligned) or diverge (NO/Not Aligned).
• Ground Truth: Labels are derived from separate annotations of the giver's intended landmark and the follower's interpreted landmark.

Experimental Design

The authors evaluate VLMs under systematically controlled manipulations of two independent variables:

1. Text Access (Dialogue Context): Four window sizes ranging from the current transaction line (curL) to the full dialogue history (startT).
2. Map Access (Information Modality):
   • Baseline: Text-only (no map).
   • Authentic Maps: Both maps, Giver-only, or Follower-only images.
   • Textual Descriptions: Lists of landmark names or textual descriptions of discrepancies.
   • Non-Informative Visual Controls: Blank maps (uniform gray) and Shuffled maps (landmarks from unrelated pairs).

Models

The primary analysis focuses on Qwen3-VL-8B-Instruct, which showed the best preliminary performance. Cross-model generalization is assessed on four additional models: Qwen3-VL-2B, Qwen3-VL-4B, Gemma3-4B, and Gemma3-12B. All models are evaluated using greedy decoding with constrained output to valid YES/NO tokens.

Key Results

1. Map Access Drives Over-Prediction of Alignment

Providing authentic map images improves overall macro-F1 scores but systematically shifts the model toward over-predicting alignment.

• Recall Trade-off: With map access, recall for aligned cases (recall_pos) increases significantly (e.g., from 0.590 to 0.822), while recall for non-aligned cases (recall_neg) drops sharply (e.g., from 0.677 to 0.518).
• Yes-Rate: The proportion of YES predictions rises, often exceeding the gold base rate, indicating a bias toward assuming mutual understanding exists.

2. The Bias is Content-Driven, Not Visual

The over-alignment bias is not an artifact of the visual channel itself but is driven by task-relevant map content.

• Textual Reproduction: Textual descriptions of map content (landmark names, discrepancy details) reproduce the bias, yielding similar yes-rates and recall profiles to authentic images.
• Visual Controls: Non-informative visual inputs (blank or shuffled maps) suppress alignment predictions entirely, making the model more conservative (yes-rate ~0.18–0.19). This confirms that the bias arises from the model interpreting the presence of shared referents as evidence of established grounding.

3. Calibration and Confidence

Calibration analysis reveals that map access makes models confidently wrong on non-aligned instances.

• Under map conditions, the model is well-calibrated on gold-YES instances but severely miscalibrated on gold-NO instances (high Expected Calibration Error).
• The model assigns high confidence to YES predictions even when the participants have not established shared understanding, suggesting it relies on static referential cues rather than tracking the dynamic grounding process.

4. Reference-Chain Dynamics

As reference chains (repeated mentions of the same landmark) lengthen, models with map access maintain high accuracy but exhibit steadily increasing yes-rates. This suggests the model increasingly treats repeated mentions as confirmation of alignment, even when the underlying interpretations may still diverge.

5. Cross-Model Variability

The bias is not universal across all architectures.

• Qwen3-VL models generally exhibit the over-alignment bias.
• Gemma3 models (particularly the 12B variant) are extremely conservative and perform poorly, likely due to limitations in parsing dense, hand-drawn map images (fixed-resolution encoding vs. Qwen's dynamic resolution).
• Scaling: Larger model size does not guarantee better performance; in some cases, larger models become more confident without becoming more discerning.

Key Contributions

1. Evaluation Methodology: The paper introduces a framework for probing VLMs' ability to distinguish potential from established common ground in information-asymmetric dialogue, moving beyond standard referential resolution tasks.
2. Empirical Characterization of Failure Modes: It identifies a specific failure mode where VLMs conflate "potential referential overlap" (what is visible on maps) with "communicative alignment" (what has been negotiated).
3. Mechanism Identification: Through modality controls and calibration analysis, the study demonstrates that this bias is driven by the semantic content of the shared environment rather than the visual modality itself, highlighting a structural limitation in how current VLMs simulate the "overhearer" role.

Significance and Claims

The paper claims that current VLMs are structurally prone to the "overhearer illusion." While they can effectively infer what interlocutors could be talking about based on shared visual or textual context, they struggle to distinguish this from what interlocutors have actually established through the incremental process of dialogue.

The authors emphasize that this is not a general failure of multimodality but a specific tendency to treat task-relevant information about potential common ground as evidence that grounding has already occurred. The study suggests that for VLMs to accurately model collaborative dialogue, they must be evaluated not just on their ability to recognize referents, but on their capacity to track the interactional history of grounding, rather than relying on static cues of co-presence.