💻 computer science

Communicating about Space: Language-Mediated Spatial Integration Across Partial Views

This paper introduces COSMIC, a benchmark for evaluating how Multimodal Large Language Models (MLLMs) collaborate via dialogue to build shared spatial understanding from partial views, revealing that while models can identify shared objects, they significantly lag behind humans in relational reasoning and maintaining globally consistent mental models of the environment.

Original authors: Ankur Sikarwar, Debangan Mishra, Sudarshan Nikhil, Ponnurangam Kumaraguru, Aishwarya Agrawal

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ankur Sikarwar, Debangan Mishra, Sudarshan Nikhil, Ponnurangam Kumaraguru, Aishwarya Agrawal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you and a friend are trying to meet up in a massive, unfamiliar park, but neither of you has a map. You are standing on opposite sides of a large fountain.

You can see a tall oak tree and a red bench, but the fountain is hidden behind a hill.
Your friend can see the fountain and a blue slide, but the oak tree is blocked by a fence.

To find each other, you can't just look at your own view. You have to talk. You say, "I see a red bench near a tree." Your friend replies, "I see a fountain near a blue slide." Slowly, by combining your descriptions, you build a shared mental picture of the whole park in your heads. You realize, "Ah! The red bench is right next to the fountain!"

This paper is about teaching AI robots (specifically, advanced "Multimodal Large Language Models" or MLLMs) to do exactly this. The researchers created a test called COSMIC to see if AI can learn to build that shared mental picture just by chatting with each other.

Here is the breakdown of what they found, using some fun analogies:

1. The Test: "The Blindfolded Roommates"

The researchers set up a digital experiment where two AI agents are placed in a 3D room.

Agent A (The Answerer) has a question but can only see half the room.
Agent B (The Helper) has the rest of the view but doesn't know the question.
The Goal: They must chat back and forth to solve the puzzle.

The test had five levels of difficulty, like climbing a ladder:

Spotting the Common Ground: "Do you see the lamp I'm looking at?" (Easy)
Counting Together: "How many chairs are in the whole room?" (Harder, because you have to make sure you aren't counting the same chair twice).
Measuring Distances: "Which object is closer to the lamp?" (Requires combining views to measure).
Changing Perspectives: "From my point of view, where is the door?" (The Helper sees the door; the Answerer has to imagine it from their own angle).
Drawing the Map: "Is this top-down map of the room correct?" (The hardest level: building a complete 3D map in your head from two 2D pictures).

2. The Results: The AI is "Chatty but Clueless"

The researchers compared the AI's performance to real humans. The results were a mix of "not bad" and "disastrous."

The Good News: The AI is pretty good at the easy stuff. If you ask, "Do you see the red chair?", the AI can usually say "Yes." It's like a tourist who can point out landmarks.
The Bad News: As soon as the task gets tricky (like figuring out where things are relative to each other or drawing a map), the AI falls apart.
- Humans: Got 95% of the answers right. They were efficient, using few words to lock onto the right objects and quickly building a shared map.
- The Best AI: Got only 72% right.
- The Worst AI: Was barely better than guessing randomly on the hardest tasks.

3. The Problem: The AI is Like a Tourist with a Broken Compass

The paper found three main reasons why the AI fails:

The "Hallucination" Trap: Sometimes the AI sees a chair and calls it a "sofa." Once it makes that mistake, the whole conversation goes off the rails. It's like trying to navigate a city when your GPS keeps telling you the bakery is a gas station.
The "Double-Counting" Confusion: When counting objects, the AI often gets confused about whether the chair it sees is the same chair its partner sees. It ends up counting the same chair twice, or missing one entirely.
The "Perspective" Failure: This is the biggest issue. Humans are great at saying, "If I'm facing North, and you see the door to your Left, that means the door is behind me." The AI struggles to rotate the world in its head. It gets stuck in its own viewpoint and can't imagine what the other person sees.

4. The "Thinking" Feature Didn't Help Much

The researchers tried turning on the AI's "Thinking Mode" (where the AI pauses to reason before speaking).

Did it help? Yes, for simple things like spotting objects.
Did it fix the hard stuff? No. Even when the AI "thought" hard, it still couldn't build a consistent map of the room. It's like a calculator that is very good at adding numbers but still can't understand the concept of a map.

5. The Verdict: We Have a Long Way to Go

The study concludes that while AI is getting smarter at seeing and talking, it is still terrible at collaborating to build a shared reality.

Humans are like expert detectives: They quickly agree on the facts, stop guessing, and solve the case.
AI is like a nervous tourist: It talks a lot, keeps suggesting new possibilities, gets confused about what the other person is seeing, and rarely settles on a single, correct picture of the world.

In short: If you want an AI to help you navigate a new building or coordinate a team of robots, you can't just ask it to "chat." Right now, the AI is too likely to get lost in its own head and fail to build the shared map needed to get the job done. The researchers hope this test (COSMIC) will help engineers fix these specific blind spots so AI can truly work with us, not just near us.

1. Problem Statement

The paper addresses a critical gap in Multimodal Large Language Models (MLLMs): the ability to build shared spatial understanding through natural language communication. While humans can seamlessly integrate partial, viewpoint-dependent observations from different agents to form a coherent, allocentric (world-centered) mental model, current MLLMs are largely untested in this distributed setting.

Existing benchmarks focus on:

Single-view reasoning: A single model analyzing one image.
Centralized multi-view: A single model analyzing multiple images simultaneously.

The paper argues that real-world collaboration (e.g., robotics, AR/VR) requires distributed agents who possess only partial views of an environment and must rely on dialogue to reconcile their observations, resolve conflicts, and solve spatial queries. The core challenge is whether MLLMs can effectively communicate to bridge the gap between egocentric (self-centered) views and a shared global understanding.

2. Methodology: The COSMIC Benchmark

To systematically evaluate this capability, the authors introduce COSMIC (Collaborative Spatial Communication), a diagnostic benchmark with the following components:

A. Task Setup

Agents: Two static MLLM agents, an Answerer (A) and a Helper (H), are placed in a 3D indoor environment at different locations.
Input: Each agent receives a distinct egocentric RGB view ( $I_A$ and $I_H$ ). The views have controlled partial overlap (shared anchor objects) and exclusive regions.
Process: The agents engage in a multi-turn natural language dialogue. The Answerer receives a question and must collaborate with the Helper to integrate information from both views to select the correct answer.
Constraints: No parameter sharing; communication is strictly via natural language. The dialogue is limited to 10 turns.

B. Benchmark Composition

Scale: 899 procedurally generated diverse indoor scenes (Living Rooms, Kitchens, etc.) and 1,250 unique question-answer pairs.
Task Hierarchy: The benchmark evaluates three levels of spatial reasoning:
1. Object-Level:
  - Anchor Recognition: Identifying objects visible in both views.
  - Global Counting: Aggregating object counts across views while avoiding double-counting shared instances.
2. Relation-Level:
  - Relative Distance: Determining which object is closest/farthest from a target, requiring metric reasoning across views.
  - Relative Direction: Inferring the direction of an object (visible only to the Helper) relative to the Answerer's egocentric frame (requiring perspective transformation).
3. Map-Level:
  - Cognitive Mapping: Judging whether a provided top-down 2D map accurately represents the combined 3D layout of the environment.

C. Data Curation & Filtering

Scene Generation: Uses Infinigen for photorealistic procedural generation.
Question Generation: Uses templates and LLM paraphrasing to ensure linguistic diversity.
Necessity Filtering: Questions that can be answered via common sense or single-view priors are removed to ensure cross-view integration is strictly required.
Human Baseline: 250 human-human dialogues were collected to establish a performance ceiling and analyze communication patterns.

3. Key Contributions

First Distributed Spatial Benchmark: COSMIC is the first benchmark to test MLLMs on building shared spatial models through dialogue between agents with partial observability.
Diagnostic Hierarchy: It isolates failure modes across a hierarchy of complexity: from simple object grounding to complex geometric integration and cognitive mapping.
Comprehensive Evaluation: The authors evaluated state-of-the-art open-source (Qwen3-VL, InternVL3.5, Gemma-3) and closed-source (GPT-5.2, Gemini-3) models, including "thinking" variants.
Human-Model Comparative Analysis: A systematic comparison of human vs. model dialogue strategies, revealing fundamental differences in convergence and efficiency.

4. Key Results

A. Performance Gap

Humans vs. Models: Humans achieve 95.22% aggregate accuracy. Even the best model, Gemini-3-Pro-Thinking, achieves only 71.82%.
Capability Hierarchy: Performance degrades significantly as task complexity increases:
- Anchor Recognition: Models perform reasonably well (e.g., Gemini-3-Pro: ~92%).
- Global Counting/Distance: Performance drops.
- Relative Direction: Significant drop (e.g., Gemini-3-Pro: ~46%).
- Cognitive Mapping: Models perform near chance level (~50%), indicating a fundamental inability to construct globally consistent maps from dialogue.

B. Impact of "Thinking"

Explicit reasoning ("Thinking" modes) improves performance on Anchor Recognition and Relative Distance (metric reasoning).
However, thinking does not help with Relative Direction or Cognitive Mapping, suggesting the bottleneck is not reasoning depth but a lack of geometric understanding required for perspective transformation and map integration.

C. Communication Efficiency

Verbosity vs. Accuracy: MLLMs are significantly more verbose (avg. ~438 words) than humans (avg. ~200 words) but achieve lower accuracy. There is no correlation between dialogue length and success.
Convergence: Humans rapidly converge on shared anchor objects and narrow the hypothesis space. MLLMs tend to explore new possibilities continuously without converging, failing to build a stable shared mental model.
Repair: Humans have a high dialogue repair rate (79%) when errors occur; models rarely recover from flawed reasoning trajectories (best model: ~28%).

D. Failure Mode Analysis

The primary bottlenecks identified are:

Cross-View Grounding Failures (46% of errors): Agents fail to uniquely identify shared objects (Referential Ambiguity, Instance Merging, or Instance Duplication).
Geometric & Relational Failures: Agents struggle to transform allocentric descriptions (from the Helper) into their own egocentric frame (Perspective-Taking Failure) or to understand 3D layout from 2D views.

5. Significance and Future Directions

Real-World Implications: The findings highlight that current MLLMs are not yet ready for deployment in collaborative robotics, warehouse automation, or AR guidance systems where agents must share spatial context.
Beyond Chain-of-Thought: The paper suggests that linguistic "thinking" is insufficient for spatial tasks. Future work requires explicit visual reasoning mechanisms (e.g., internal geometric verification, mental rotation).
Structured Protocols: The authors propose that structured communication protocols (e.g., enforcing anchor-first grounding, explicit reference frame agreement) could mitigate current failures.
New Standard: COSMIC establishes a rigorous standard for evaluating the "collaborative spatial intelligence" of AI agents, moving beyond single-agent perception to multi-agent communication.

In summary, the paper demonstrates that while MLLMs are improving at visual perception, they fundamentally lack the collaborative spatial reasoning and dialogue management skills necessary to build shared mental models of the physical world, a capability that humans possess naturally.