Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry

This paper introduces the Distributed Partial Information Puzzle (DPIP) and its associated multimodal dataset to evaluate how well current large language models and logic-based systems can construct common ground under epistemic asymmetry, revealing that modern LLMs struggle to accurately track task progression and belief states compared to axiomatic approaches.

The Gist

Imagine you and three friends are trying to build a specific, complex castle out of Lego bricks. But here's the catch: none of you can see the whole picture.

• You (the Builder) are the only one who can touch the bricks. You have no instructions and no reference image.
• Three friends (the Directors) are standing around you. Each of them has a tablet showing a different side view of the finished castle (one sees the front, one the left, one the right). They don't know which side they are looking at, and they can't touch the bricks.

Your goal is to work together to build the castle. The Directors have to describe what they see using words, pointing, and gestures, while you try to figure out where to put the blocks. If they say "put a red block here," but you put it in the wrong spot because you misunderstood their perspective, the castle fails.

This paper introduces a new game called DPIP (Distributed Partial Information Puzzle) based on this exact scenario. The researchers recorded real people playing this game to see how well they could work together and, more importantly, to test if Artificial Intelligence (AI) can do the same.

Here is a breakdown of what they found, using some simple metaphors:

1. The Challenge: "The Blind Orchestra"

In a normal conversation, everyone usually hears the same song. In this puzzle, everyone is playing a different instrument, looking at a different sheet of music, and trying to create one symphony.

• The Problem: The Directors have to explain their private, partial view to the Builder. They have to guess what the others know and what they don't. This is called "Epistemic Asymmetry" (a fancy way of saying "we all know different things").
• The Goal: They need to build "Common Ground." Think of this as a shared mental whiteboard where everyone agrees on what the castle looks like. If the whiteboard is messy or wrong, the castle collapses.

2. The Data: A "Multimodal" Treasure Map

The researchers didn't just listen to what people said. They recorded:

• Speech: What they said.
• Gestures: Pointing fingers, waving hands, nodding.
• Actions: Actually moving the Lego blocks.

They created a massive dataset (like a treasure map) where every word, hand wave, and block movement is perfectly synchronized. This allows them to study how humans combine these different clues to solve the puzzle.

3. The AI Test: Can Robots "Get It"?

The researchers asked two types of "brains" to watch the video recordings and figure out what the group was building and what they agreed on:

1. Modern AI (LLMs): These are the big, fancy chatbots (like the ones you might use daily) that are great at writing stories and answering questions.
2. A Logic Robot (Axiomatic Pipeline): This is a strict, rule-based computer program that follows a set of logical laws (like "If I see it, I believe it" or "If you move a block, you intend to build it").

The Results:

• The Logic Robot was surprisingly good. It followed the rules of the game perfectly. It could look at the actions and gestures and deduce the structure almost as well as a human could.
• The Fancy AI (LLMs) struggled. Even though these models are incredibly smart at writing poems or coding, they got confused by the puzzle.
  • Why? They are like a student who is great at reading a textbook but terrible at a live sports game. They can understand the words, but they missed the context. They didn't fully grasp that when a Director points to a block, it means something specific relative to their own hidden view.
  • When the AI tried to guess the final castle, it often got the shape wrong. When it tried to guess what the group "agreed on," it often hallucinated (made up) beliefs that didn't exist.

4. The "Failed Team" Surprise

There was one group of people who failed to build the castle correctly. They were confused and argued a lot.

• The Logic Robot correctly identified that this group had no shared agreement (no common ground).
• The Fancy AI also correctly identified that the group was confused and had no shared plan.
• The Twist: The AI was actually better at spotting a failed team than a successful one! When the team was working well, the AI got confused by the complexity. When the team was failing, the lack of agreement was so obvious that the AI could easily spot it.

The Big Takeaway

This paper is a wake-up call for AI developers.

• Current AI is like a brilliant librarian: It knows every book in the library (text) but doesn't know how to navigate a busy, noisy construction site where people are pointing, shouting, and moving things around.
• Human collaboration is messy and multimodal: We don't just talk; we point, we move things, and we read the room.
• The Future: To build AI that can truly work with humans (like a co-pilot or a teammate), we need to teach it to understand not just words, but actions, gestures, and hidden perspectives.

In short: AI is great at reading the script, but it's still learning how to improvise on stage with a team. This new puzzle (DPIP) is the training ground to help it learn that skill.

Technical Summary

Here is a detailed technical summary of the paper "Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry."

1. Problem Statement

Effective human collaboration relies on establishing common ground (CG)—a shared set of beliefs and mutually recognized facts. However, current AI systems struggle to model CG in multiparty, co-situated, multimodal settings where participants possess epistemic asymmetry (different, partial information).

Existing datasets often fail to capture the simultaneous complexity of:

• Multiparty interaction: More than two agents coordinating.
• Co-situatedness: Agents interacting in a shared physical environment.
• Epistemic Asymmetry: Agents holding unique, private information that must be integrated to solve a task.
• Multimodality: The reliance on speech, gesture, and physical action to convey meaning.

The paper argues that without a benchmark addressing these combined factors, it is impossible to rigorously evaluate how well AI systems (specifically Large Language Models) can track belief dynamics and infer shared understanding in realistic collaborative scenarios.

2. Methodology

A. The Task: Distributed Partial Information Puzzle (DPIP)

The authors introduce the DPIP, a collaborative construction task involving four participants:

• Three Directors: Each is given a unique 2D side view (front, left, or right) of a target 3D Lego structure. They cannot see the physical blocks or each other's views.
• One Builder: Can manipulate the physical Lego blocks but has no visual reference to the target structure.
• Goal: The directors must collaboratively instruct the builder to construct a single contiguous structure that matches all three directors' partial views.
• Constraints: The task requires resolving discrepancies between private perspectives through verbal communication, gestures, and physical actions.

B. Dataset Collection

• Participants: 10 groups of 4 people (33 total groups collected, 10 fully annotated).
• Setup: Participants sit around a table with a Lego board. Directors view tablets showing their specific side views.
• Sensors: 3 Microsoft Kinect Azure cameras capture the scene from different angles; a tabletop microphone records audio.
• Scale: The dataset includes 10 audiovisual recordings with an average duration of ~12 minutes per group.

C. Annotation Pipeline

The authors developed a rigorous, multi-modal annotation pipeline:

1. Speech: Transcribed via Whisper ASR, then human-verified. Annotated with propositional content (relational information between blocks) and speaker identity.
2. Action: The builder's physical manipulations (put, remove, move) were reconstructed in a 3D Structure Annotation Tool (SAT) to create a ground-truth timeline of the board state.
3. Gesture: Annotated using Gesture Abstract Meaning Representation (GAMR), capturing deictic (pointing), iconic (shape), and emblematic (agreement) gestures.
4. Alignment: All modalities were temporally aligned. Propositions from speech and gestures were symbolically grounded to specific block identifiers (e.g., rs1 for a red square block) based on subsequent actions.

D. Evaluation Framework

The authors evaluated two paradigms for modeling Common Ground:

1. Axiomatic Pipeline (CGC): A rule-based system grounded in Dynamic Epistemic Logic (DEL). It uses three axioms to update belief states:
   • Seeing is Believing: Perceptual context updates belief.
   • Acting is Believing: Embodied action reveals intention.
   • Saying is Believing: Language communicates epistemic state.
   • This pipeline incrementally constructs a set of shared beliefs (CG) based on the annotated multimodal inputs.
2. Large Language Models (LLMs): State-of-the-art models (Qwen3-4B, Llama-3.2-3B, GPT-5-mini/GPT-5) were prompted to:
   • Predict the structure state from actions alone.
   • Predict the structure state from aligned multimodal annotations.
   • Predict the group's common ground (shared belief set) directly.

Metrics: The primary metric was the Dice Similarity Coefficient (DSC) to measure the overlap between predicted sets (structure or beliefs) and the ground truth.

3. Key Contributions

1. The DPIP Task & Dataset: A novel benchmark specifically designed to test multimodal, multiparty collaboration under epistemic asymmetry. It bridges the gap between theoretical dialogue research and embodied, physical collaboration.
2. Rich Multimodal Annotation: A high-quality dataset with temporally aligned speech, gesture, and action annotations, including propositional content and belief dynamics (accepted, doubted, negated).
3. Axiomatic vs. Neural Comparison: A systematic evaluation comparing a logic-driven, axiomatic approach (DEL) against modern LLMs for tracking common ground.
4. Empirical Findings on LLM Limitations: Evidence that even advanced LLMs struggle to maintain coherent belief states and track task progression in complex, partially observable, multimodal environments.

4. Results and Analysis

• LLM Performance on Structure Prediction:

  • When given only actions, GPT-5 performed best, but all models struggled with global dialogue-level prediction compared to turn-level prediction.
  • When given aligned multimodal annotations (speech + gesture + action), Qwen3-4B outperformed others. Interestingly, adding multimodal context improved global DSC scores but decreased turn-level (step-wise) scores, suggesting that while context helps overall, it introduces noise for immediate state tracking.
  • GPT-5-mini/GPT-5 surprisingly performed poorly on the multimodal task, failing to predict correct structures for several groups, indicating a potential weakness in processing this specific type of structured, spatial reasoning data despite its general capabilities.

• Axiomatic Pipeline (CGC) vs. LLMs:

  • The deterministic axiomatic approach (CGC) achieved structure prediction scores comparable to or better than GPT-5-mini when given full multimodal input.
  • Common Ground Prediction: When asked to predict the group's shared beliefs (CG), LLMs showed very low overlap (DSC often near 0) with the axiomatic CG calculations. This indicates LLMs infer fundamentally different belief states than the logical ground truth derived from the interaction.

• The "Failure" Case (Group 7):

  • In a group that failed to complete the task (due to confusion), the LLMs surprisingly achieved perfect alignment with the axiomatic CG calculation.
  • Interpretation: Because the group had no shared understanding, the "common ground" was effectively empty or minimal. The LLMs correctly inferred this lack of consensus, whereas in successful groups with complex, evolving consensus, LLMs struggled to track the nuances.

5. Significance

• Benchmark for Multimodal AI: The DPIP establishes a rigorous standard for evaluating AI in scenarios requiring the integration of speech, gesture, and physical action under conditions of incomplete information.
• Challenging the "Black Box" of LLMs: The results suggest that while LLMs are powerful for general reasoning, they lack the robust mechanisms required to track belief dynamics and epistemic states in real-time, multiparty physical collaborations. They often fail to distinguish between private knowledge and shared ground.
• Theoretical Insight: The study highlights that "common ground" is not merely a static set of facts but a dynamic, negotiated state. The failure of LLMs to match axiomatic logic suggests a gap in how current models handle Theory of Mind and epistemic reasoning in collaborative settings.
• Future Directions: The paper calls for new architectures that can explicitly model belief updates and handle the "friction" of collaborative problem-solving, moving beyond simple next-token prediction to true collaborative reasoning.

In summary, the paper demonstrates that current SOTA AI systems are not yet ready to act as effective collaborators in complex, partially observable, multimodal environments, and provides the data and methodology necessary to drive progress in this critical area.