Using Perspectival Words Is Harder Than Vocabulary Words for Humans and Even More So for Multimodal Language Models

1. Problem Statement

While Multimodal Language Models (MLMs) have achieved near-human performance in vocabulary recognition (e.g., identifying objects, colors, and verbs), their ability to use perspectival words (indexicals) remains poorly understood. Perspectival words—such as personal pronouns (I, you), possessives (mine, yours), and demonstratives (this, that)—are grammatical forms whose meaning is entirely dependent on the context of use, specifically the speaker's identity, the listener's perspective, and spatial relationships.

The core problem is that current evaluations of MLMs focus heavily on context-independent vocabulary, overlooking the pragmatic reasoning and social cognition (Theory of Mind) required to correctly use perspectival words. It is unclear whether MLMs can ground language in physical and social contexts as humans do, or if they struggle with the cognitive demands of perspective-taking and spatial reasoning inherent in these word classes.

2. Methodology

The authors designed a controlled experimental framework to compare human performance against seven state-of-the-art MLMs across three distinct tasks, hypothesizing a hierarchy of difficulty: Vocabulary < Possessives < Demonstratives.

Experimental Design

• Participants: 360 human participants (native English speakers) and seven MLMs (GPT-4o, Llama-4-Scout, Gemma-3, InternVL 2.5-4B/8B, Ovis 2-4B/8B).
• Stimuli: Visual scenes depicting two characters (a boy and a girl) on opposite sides of a table with objects.
  • Vocabulary Task: Select the correct label for a unique object (e.g., "boat"). No perspective required.
  • Possessive Task: Select between "mine/yours" based on object ownership indicated by proximity to a character. Requires identifying the speaker and the owner.
  • Demonstrative Task: Select between "this one/that one" based on the object's spatial distance relative to the speaker. Requires perspective-taking and fine-grained spatial reasoning.
• Modality Adaptation:
  • Humans: Viewed images with speech bubbles and gaze lines to indicate speaker/gazer identity.
  • MLMs: Viewed adapted images (green squares for targets, no speech bubbles) with text prompts explicitly stating the speaker, gazer, and ownership/distance relationships to avoid visual confounds.
• Evaluation Metrics: Accuracy in selecting the correct referential expression from four multiple-choice options.

Ablation and Prompting Studies

To diagnose the source of model failures, the authors conducted:

1. Ablation Analysis: Systematically removed perspectival cues (speaker identity) and spatial cues (linguistic ownership/distance markers and visual green squares) to test reliance on specific information types.
2. Prompting Strategies: Tested Instruction-based prompting (explicitly guiding the model through pragmatic rules) and In-context learning (providing examples) to see if performance could be bridged to human levels.

3. Key Contributions

• Unified Framework for Perspectival Evaluation: Introduced a novel benchmark specifically targeting word-level pragmatics in multimodal settings, moving beyond sentence-level phenomena like irony or metaphor.
• Hierarchy of Difficulty: Empirically established that perspectival words impose a gradient of cognitive load: Vocabulary is easiest, followed by Possessives, with Demonstratives being the most difficult due to the need for continuous spatial reasoning.
• Identification of Model Deficits: Demonstrated that while MLMs approach human ceiling performance on vocabulary, they exhibit significant deficits in perspectival words, particularly in demonstratives.
• Causal Analysis of Failures: Through ablation, identified that model failures stem from a lack of perspective-taking (inability to adopt the speaker's viewpoint) and spatial reasoning (difficulty integrating visual distance with linguistic proximity terms).
• Modality Imbalance Discovery: Revealed that MLMs rely heavily on linguistic cues for spatial reasoning and struggle when visual cues are the primary source of spatial information, unlike humans who integrate both in parallel.

4. Key Results

Performance Hierarchy

• Humans: Showed the predicted difficulty gradient: Vocabulary (~99%) > Possessives (93%) > Demonstratives (83%).
• MLMs: Followed the same hierarchy but with a significantly wider gap compared to humans.
  • Vocabulary: Most models achieved near-human performance (95-99%).
  • Possessives: Performance dropped significantly. The best model (Llama-4-Scout) reached 85.7%, while humans were at 93%.
  • Demonstratives: The gap was largest here. The best model (GPT-4o) scored only 70.2% compared to 83% for humans. Smaller models performed near chance levels.

Ablation Findings

• Perspectival Cues: Removing speaker identity caused a sharp drop in performance for top models (GPT-4o, Llama-4-Scout), indicating they rely on viewpoint information. Lower-performing models showed no change or slight improvement, suggesting they fail to utilize perspective cues at all.
• Spatial Cues:
  • Removing linguistic spatial cues (e.g., "close to") hurt performance significantly, especially for possessives.
  • Removing visual cues (green squares) had minimal impact or slightly improved performance, suggesting models often ignore visual spatial data unless supported by text.
  • Combined Removal: Performance dropped drastically, confirming models need both modalities, but rely on text as the primary scaffold for spatial reasoning.

Prompting Interventions

• Instruction-based Prompting: Significantly improved Possessive task performance, bringing models close to human accuracy (e.g., GPT-4o improved to ~90%). This suggests explicit pragmatic rules can scaffold ownership reasoning.
• Demonstrative Limitations: Even with explicit spatial instructions, models failed to reach human levels on Demonstratives. Error analysis showed improvements were mostly in personal pronouns (perspective-taking) rather than demonstrative pronouns (spatial reasoning), indicating spatial reasoning remains a fundamental bottleneck.

5. Significance and Implications

• Pragmatic Shortfall: The study reveals a critical gap in current NLP systems: they lack the social micro-processes required for genuine human-like communication. While they can describe the world, they struggle to refer to it from a specific agent's perspective.
• Embodiment and Interaction: The difficulty with demonstratives suggests that true spatial language understanding may require embodied, interactive experiences in physical environments, which current static MLMs lack.
• Future Directions: The findings argue for:
  1. Developing models with explicit spatial reasoning modules.
  2. Training on richer datasets involving physical interaction and face-to-face reference.
  3. Shifting evaluation benchmarks from static vocabulary recognition to dynamic, context-dependent pragmatic reasoning.

In conclusion, the paper establishes that perspectival words are a "harder" cognitive task than vocabulary for both humans and machines, but the difficulty is disproportionately amplified in MLMs due to limitations in perspective-taking and spatial reasoning, highlighting a crucial frontier for achieving truly human-like multimodal communication.