Grounded Misunderstandings in Asymmetric Dialogue: A Perspectivist Annotation Scheme for MapTask

1. Problem Statement

Collaborative dialogue relies on the incremental establishment of "common ground," where interlocutors assume they agree on the entities being discussed. However, in asymmetric settings (where participants possess different information), speakers and addressees may believe they have successfully grounded a reference while actually referring to different entities.

• The Gap: Traditional reference resolution assumes that once an utterance is acknowledged, mutual understanding is achieved. Existing annotation schemes for the HCRC MapTask corpus (a standard dataset for asymmetric navigation tasks) often fail to capture the personal interpretations of both the speaker and the addressee simultaneously. Consequently, subtle misalignments that persist after apparent grounding are difficult to identify and quantify.
• The Challenge: There is a lack of fine-grained, perspective-dependent annotation schemes that can distinguish between a speaker's intended referent and an addressee's interpreted referent, particularly in scenarios involving landmark discrepancies (e.g., missing landmarks, multiple instances of the same landmark, or lexical variants).

2. Methodology

A. Dataset and Preprocessing

The authors utilized the HCRC MapTask corpus, consisting of 128 dialogues between pairs of participants navigating routes on slightly different maps.

• Landmark Discrepancies: The maps contain systematic differences classified into three types:
  1. Lexical: Same location, different names (e.g., "cliffs" vs. "sandstone cliffs").
  2. Existence: A landmark exists on one map but not the other.
  3. Multiplicity: A landmark appears twice on one map (usually the giver's) but only once on the other.
• Unified Landmark IDs (ULM): The authors introduced a new identifier scheme (<map-id>_<name>#<ordinal>@<side>) to distinguish between specific instances of landmarks on the giver's vs. the follower's map, resolving ambiguities in the original annotation scheme.

B. Perspectivist Annotation Scheme

The core contribution is a hierarchical, RE-level (Reference Expression) annotation scheme that captures two perspectives for every reference:

1. Speaker's Intended Referent: The landmark the speaker intends to discuss.
2. Addressee's Interpreted Referent: The landmark the addressee believes is being discussed.

The scheme uses five binary attributes in a decision cascade to characterize the state of understanding:

1. is_quantificational: Is the expression an existence query (e.g., "Do you have a...?") rather than a definite reference?
2. is_specified: Is there sufficient context to infer the addressee's interpretation?
3. is_accommodated: Did the addressee respond without signaling confusion?
4. is_grounded: Did the addressee successfully link the expression to a specific landmark on their map?
5. is_imagined: Did the addressee mentally project a landmark absent from their map based on the speaker's description?

C. LLM Annotation Pipeline

To scale this complex annotation task, the authors developed a scheme-constrained LLM pipeline using GPT-5 (via OpenAI Batch API).

• Prompt Engineering: The model was guided through a Chain-of-Thought (CoT) workflow using structured JSON prompts. The prompts included the dialogue context, move-level annotations, and the unified landmark ID list.
• Process: The model sequentially determined the speaker's intent, evaluated the five attributes, and assigned the addressee's interpreted landmark ID.
• Quality Control: The pipeline was validated against a human gold standard (504 reference expressions from 3 dialogues), achieving 94.4% accuracy at the RE level and high F1 scores (0.89–1.00) across attributes.

3. Key Contributions

1. Perspectivist Annotation Framework: A novel scheme that explicitly models the divergence between speaker intent and addressee interpretation, operationalizing "grounding" as a graded, perspective-dependent process rather than a binary state.
2. Large-Scale Annotated Dataset: The creation of 13,077 annotated reference expressions across the entire MapTask corpus, providing a rich resource for studying misunderstanding dynamics.
3. LLM-in-the-Loop Pipeline: A validated methodology for using constrained LLMs to perform complex, multi-turn perspective-taking annotations, demonstrating high reliability compared to human annotators.
4. Analytical Insights: A quantitative analysis of how understanding states evolve, specifically identifying multiplicity discrepancies as the primary driver of silent misunderstandings.

4. Key Results and Findings

A. Misunderstanding Rates

• Initial Rate: Before unifying lexical variants, the "misunderstood" rate (where addressee and speaker refer to different entities) was 7.07%.
• Adjusted Rate: After unifying lexical variants (treating "cliffs" and "sandstone cliffs" as the same entity), the rate dropped to 1.82%. This confirms that full, persistent misunderstandings are rare in collaborative dialogue due to active "running repairs."
• The "Silent" Hazard: Despite the low overall rate, multiplicity discrepancies are highly dangerous. They account for 48.5% of all misunderstandings (115 out of 239 cases) and have a misunderstanding rate of 12.0% (6x the corpus average). This occurs because participants assume referential uniqueness when multiple instances exist on the giver's map but only one on the follower's.

B. Understanding State Transitions

• Reference Chains: The authors analyzed sequences of references to the same landmark. Lexical variants required the longest chains (mean 10.7 turns) to resolve, while existence discrepancies were resolved quickly (mean 4.09 turns).
• Turns-to-Ground: Multiplicity discrepancies required the most negotiation turns to reach alignment, highlighting the cognitive load of distinguishing between specific instances of the same named landmark.

C. LLM Performance

• The LLM achieved high accuracy in identifying grounding states but occasionally failed in "Special Cases" (e.g., when a speaker mistakenly starts from the wrong point). The model exhibited an "omniscient bias," underestimating asymmetric knowledge because it could "see" both maps simultaneously, treating contradictions as noise rather than evidence of a fundamental misalignment.

5. Significance and Future Implications

• Theoretical Impact: The study challenges the assumption that acknowledgment equals alignment. It demonstrates that apparent grounding can mask referential misalignment, particularly in asymmetric information settings.
• Benchmark for AI: The annotated dataset serves as a new benchmark for evaluating Vision-Language Models (VLMs) and Large Language Models (LLMs). Current models struggle to track evolving, asymmetric interpretations; this framework provides the ground truth necessary to test a model's ability to perform "perspective-taking" in collaborative tasks.
• Methodological Advance: It proves that LLMs, when constrained by structured schemas and decision cascades, can reliably perform complex, multi-turn pragmatic annotation tasks that were previously limited to human experts.

In summary, this paper provides a rigorous, data-driven lens into the mechanics of collaborative dialogue, revealing that while full misunderstandings are rare, multiplicity discrepancies create a persistent, silent risk of misalignment that current AI models struggle to detect without explicit perspective-taking mechanisms.