💬 NLP

LVLMs and Humans Ground Differently in Referential Communication

This paper presents a referential communication experiment demonstrating that Large Vision-Language Models (LVLMs) struggle to establish common ground and interactively generate or resolve referring expressions as effectively as humans, thereby limiting their ability to collaborate seamlessly with human users.

Original authors: Peter Zeng, Weiling Li, Amie Paige, Zhengxiang Wang, Panagiotis Kaliosis, Dimitris Samaras, Gregory Zelinsky, Susan Brennan, Owen Rambow

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Peter Zeng, Weiling Li, Amie Paige, Zhengxiang Wang, Panagiotis Kaliosis, Dimitris Samaras, Gregory Zelinsky, Susan Brennan, Owen Rambow

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you and a friend are playing a game where you have to match 12 specific baskets from a pile of 16. You can't see what your friend sees, and they can't see what you see. You have to describe the baskets to each other to get them in the right order.

This is exactly what researchers did in this new study, but they swapped their human friends for AI robots (specifically, advanced Large Vision Language Models, or LVLMs). They wanted to see if AI can learn to "speak the same language" as humans or as other AIs over time, just like real people do.

Here is the breakdown of what they found, using some simple analogies.

The "Secret Handshake" of Conversation

When two humans play this game, something magical happens.

Round 1: You might say, "Pass me the tall, brown basket with the handle that looks like a question mark."
Round 2: You say, "The question-mark handle one."
Round 3: You just say, "The Q-basket."

Humans naturally develop a secret handshake (called a "conceptual pact"). They realize, "Oh, we both know what 'Q-basket' means now, so I don't need to describe the whole thing again." They get faster, shorter, and more accurate. They are building a shared understanding, or common ground.

The AI Experiment: 4 Different Teams

The researchers set up four teams to play this game for four rounds:

Human + Human: Two people.
AI + AI: Two robots talking to each other.
Human + AI: A person talking to a robot.
AI + Human: A robot talking to a person.

The Results: Who Won?

1. The Humans (The Champions) 🏆

The human teams got better and better. By the end, they were speaking in shorthand, making fewer mistakes, and finishing quickly. They were like a jazz band that finally gets into a groove; they stopped over-explaining and started improvising together.

2. The AI vs. AI Team (The Verbose Robots) 🤖🤖

This was the most surprising result. Even when two AIs talked to each other, they did not get better.

The Problem: They kept describing the baskets in excruciating detail every single time.
- Round 1: "The tall, brown, woven basket with a curved handle..."
- Round 4: "The tall, brown, woven basket with a curved handle..." (They said the exact same long sentence again!)
The Analogy: Imagine two people trying to order a pizza. Even after the first time they agree on "Pepperoni," the second time they say, "I want the pizza with the red circles and spicy cheese on the dough." They never learned to just say "Pepperoni."
The Result: They actually got worse at the game over time because they were so busy talking that they lost track of which basket was which. They failed to build that "secret handshake."

3. The Mixed Teams (The Frustrating Partners) 😫

When humans played with AI, it was a disaster for the humans.

Human as Director (Talking to AI): The human tried to be concise, but the AI kept asking for unnecessary confirmations or repeating things. The human had to do all the heavy lifting.
AI as Director (Talking to Human): The AI would describe the baskets in a long, confusing way. Sometimes, the AI would describe the wrong basket entirely (a "hallucination"), and the human would try to fix it, but the AI would just ignore the correction and keep talking.
The Analogy: It's like trying to drive a car with a passenger who keeps changing the destination without telling you. You know where you're going, but the passenger keeps insisting you turn left, even though you're already there.

Why Did the AI Fail?

The paper suggests that AI is like a photocopier of text, not a collaborator.

Humans learn by interacting. They listen to what their partner needs and adjust.
AI (in this study) just looked at its instructions and the history of the chat and said, "Okay, I will describe the object again using the most detailed words I know." It didn't realize, "Hey, my partner already knows what this is!"

The AI failed to understand the concept of "Common Ground." It didn't realize that once something is established, you don't need to explain it again. It violated the basic rule of conversation: Don't say too much if you don't have to.

The Big Takeaway

If we want AI to be a good partner for humans (like a doctor's assistant or a co-pilot), it needs to learn how to listen and adapt, not just talk. Currently, even the smartest AI models are terrible at building that shared understanding. They are like a student who memorized the textbook but has never actually had a conversation with a friend.

In short: Humans get smarter together by building a shared language. AI just keeps repeating the dictionary.

1. Problem Statement

The core problem addressed is the inability of Large Vision-Language Models (LVLMs) to establish "common ground" during collaborative, multi-turn communication. While generative AI agents are increasingly expected to partner with humans, they lack the pragmatic competence to interactively build shared understanding (conceptual pacts) regarding referents.

Unlike humans, who adapt their language to become more efficient and concise over repeated interactions (a process known as lexical entrainment), LVLMs struggle to:

Track what information a partner needs based on conversation history.
Adapt referring expressions to reduce redundancy.
Collaboratively repair misunderstandings in real-time.

The paper investigates whether LVLMs can engage in the grounding process described by Clark and Brennan (1991), where partners converge on shared meanings to increase joint efficiency.

2. Methodology

Experimental Design

The authors conducted a factorial experiment involving a referential communication task (a "director-matcher" game) based on Clark and Wilkes-Gibbs (1986).

Task: A "Director" describes a sequence of 12 visually similar baskets (non-lexicalized objects) to a "Matcher." The Matcher must identify and place the baskets in the correct order.
Conditions: The study tested all four combinations of partner types in both roles (Director/Matcher):
1. Human-Human (HH)
2. Human-AI (HA): Human Director, AI Matcher
3. AI-Human (AH): AI Director, Human Matcher
4. AI-AI (AA): AI Director, AI Matcher
Procedure: Pairs interacted over 4 rounds with the same partner and set of objects. The order of baskets changed each round, but the set remained constant, allowing for the accumulation of common ground.
Platform: The experiment was hosted online using oTree with text-based communication.

Participants and Models

Humans: 103 participants recruited via Prolific, strictly screened for fluency and task performance.
AI: The primary LVLM used was GPT-5.2 (OpenAI, 2025) with the "none" reasoning option to prioritize responsiveness. Follow-up experiments tested GPT-5.2 against Gemini Pro and Claude Opus.
Prompting: AI agents were given system prompts that included:
- Role definitions and task constraints.
- Pragmatic norms (e.g., conciseness, turn-taking).
- Structural Scaffolding: A strict JSON output format enforcing Chain-of-Thought (CoT) reasoning before generating an utterance, ensuring the model tracked state and potential confusions.

Evaluation Metrics

The study quantified grounding capabilities across three dimensions:

Communicative Success: Measured by Accuracy (percentage of correctly matched baskets per round).
Communicative Effort: Measured by the number of words and number of turns per round. Lower values indicate higher efficiency.
Lexical Entrainment: Measured by:
- Length Reduction: Decrease in the number of content words used to describe objects over rounds.
- Lexical Overlap: The proportion of shared content words between the referring expression in round $i$ and round $i-1$ . High overlap combined with length reduction indicates successful entrainment.

3. Key Contributions

First Comprehensive Factorial Study: This is the first study to generate real-time, multi-turn task-oriented dialogues covering all four combinations of human/AI roles (HH, HA, AH, AA).
Empirical Evidence of Grounding Failure: The paper provides rigorous quantitative evidence that even frontier LVLMs (GPT-5.2) fail to form conceptual pacts or adapt communication strategies based on partner feedback, unlike humans.
Corpus Release: The authors released a dataset of 356 dialogues (89 pairs over 4 rounds) along with tools for analyzing accuracy, efficiency, and lexical overlap.
Role-Specific Analysis: The study isolates specific failure modes depending on whether the AI acts as the Director (initiator) or Matcher (responder).

4. Key Results

Human-Human (Baseline)

Performance: Accuracy started high (~80%) and increased to >90% by Round 4.
Efficiency: Significant reduction in words and turns over rounds.
Entrainment: Referring expressions became shorter while maintaining high lexical overlap (>50%), demonstrating successful formation of "conceptual pacts" (e.g., adopting a shared nickname like "scary rodent").

AI-AI

Performance: Accuracy started high (~90%) but declined significantly over rounds.
Efficiency: Remained extremely verbose (3x the word count of HH) with no reduction in turns or words.
Entrainment: Failed to shorten descriptions. While lexical overlap was high, it was due to the AI repeating exhaustive, verbose descriptions rather than refining them.
Behavior: AI Directors often hallucinated descriptions or switched targets; AI Matchers excessively confirmed every turn (violating Grice's Maxim of Quantity) rather than providing concise acknowledgments.

Mixed Pairs (Human-AI & AI-Human)

Human Director / AI Matcher (HA):
- Lowest accuracy of all conditions; no improvement over rounds.
- High communicative effort.
- The human attempted to adapt, but the AI Matcher failed to learn or accept the human's simplified terms, persisting in re-introducing old vocabulary.
AI Director / Human Matcher (AH):
- Accuracy started high but collapsed precipitously in later rounds.
- The AI Director failed to maintain a consistent order of description and introduced new, unnecessary lexical material in later rounds.
- Human matchers often gave up on clarifying questions, realizing the AI was not a capable communicator.

Follow-up Experiments

Varying reasoning levels (Low/Medium/High) or using different models (Gemini, Claude) did not resolve the fundamental lack of grounding.
Simplifying prompts did not improve performance; in fact, accuracy often dropped further.
Conclusion: Prompt engineering alone is insufficient to induce common ground formation in current LVLMs.

5. Significance and Implications

Limitations of Current AI: The study demonstrates that despite high single-turn performance, LVLMs lack the pragmatic competence required for long-horizon collaboration. They treat dialogue as a sequence of independent tasks rather than a cumulative process of building shared understanding.
The Turing Test Revisited: The authors argue that in an interactive collaborative task, humans can easily distinguish AI partners due to the AI's failure to ground meaning. This suggests that current frontier models fail the "collaborative Turing Test."
Risk for Embodied AI: The findings highlight significant risks for deploying AI in roles requiring initiative (e.g., AI Directors). If an AI cannot adapt to a human partner's needs or track shared context, it will degrade task performance and cause user frustration.
Future Directions: The paper suggests that simply prompting models to be "concise" or "human-like" is ineffective. Future research must focus on architectural changes or training regimes that explicitly model the dynamics of common ground and interactive repair, rather than relying on static prompting strategies.

In summary, the paper establishes that while LVLMs can mimic human language patterns, they fundamentally fail to ground meaning interactively, leading to inefficient, verbose, and increasingly inaccurate collaboration compared to human-human pairs.