Are you with me? A Framework for Detecting Mental Model Discrepancies in Task-Based Team Dialogues

Technical Summary: "Are you with me? A Framework for Detecting Mental Model Discrepancies in Task-Based Team Dialogues"

Problem Statement

Effective teamwork relies on Shared Mental Models (SMMs), where team members maintain overlapping representations of task states, goals, and teammate capabilities. While traditional SMM assessment methods (e.g., retrospective expert coding, questionnaires) provide global similarity scores, they fail to capture the dynamic, real-time nature of coordination breakdowns. Specifically, existing literature lacks a systematic categorization of how mental models diverge during naturalistic team dialogues and whether specific patterns of misalignment can predict future coordination failures. This paper addresses the gap in understanding the specific types of discrepancies that emerge when updates are not communicated and how these discrepancies evolve over sequential task episodes.

Methodology

Experimental Setup:
The study utilized a collaborative object identification task within a video game environment involving 20 dyad teams. The task featured asymmetric information: a "Spotter" had detailed written instructions but limited visual range, while a "Photographer" had enhanced binoculars but relied entirely on verbal cues from the Spotter. The experiment consisted of four sequential levels (Local Law Enforcement, Mountain Search and Rescue, Beach Patrol, and Poaching), each lasting eight minutes.

Data Processing:
All team communications were transcribed using Large Language Models (LLMs) and verified by human annotators. A previously developed LLM model (currently under review) was employed to detect mental model updates and categorize discrepancies at the level of human annotation.

Discrepancy Categorization Framework:
The authors propose a framework identifying four distinct types of mental model discrepancies observable in dialogue:

1. Belief Contradiction: One member believes $X$, while another believes $\neg X$ (e.g., conflicting views on object stability).
2. Omission: A member's mental model lacks a belief that another member holds, inferred from role-based expectations of knowledge versus communicated content.
3. Unsupported Beliefs: Beliefs identified by a member that are neither contradicted nor supported by ground truth or other members (unverifiable).
4. False Beliefs: Beliefs explicitly contradicted by established ground truth (e.g., incorrect location data).

Prediction Model:
To test if historical discrepancy patterns predict future performance, the authors employed a baseline weighted sum model. The predicted discrepancy count for a target level ($L_{target}$) was calculated as the sum of discrepancy counts from previous levels ($L_i$) multiplied by weights ($w_i$).
$$L_{target}(d) = \sum_{i \neq target} w_i L_i(d), \quad \sum w_i = 1$$
The study utilized a uniform weighting baseline ($w_i = 1/n$), assuming all prior levels contribute equally. More complex schemes (e.g., recency-weighted, type-specific) were proposed as theoretical extensions but not empirically tested in this work.

Key Results

Discrepancy Distribution:
Across 20 teams and four levels, 1,447 total discrepancies were identified.

• Omissions were the dominant category, constituting the largest source of both total discrepancies and prediction errors in absolute terms.
• Belief Contradictions showed high volatility across levels and teams.
• False Beliefs and Unsupported Beliefs were rare, appearing in fewer than 20% of team-level observations and contributing negligibly to overall prediction error.

Predictive Accuracy:
Using the uniform averaging baseline, the model achieved a pairwise correlation of 0.56 ($p=0.01$) between predicted and actual discrepancy counts for Level 4 based on Levels 1–3.

• The authors caution that this correlation may reflect stability in team-specific baseline rates rather than a mechanistic causal link, as Level 4 counts are likely autocorrelated with earlier levels.
• Some teams (e.g., Team 6) showed predictable trajectories where baseline challenges persisted despite temporary improvements, while others (e.g., Teams 13, 16) exhibited substantial deviations in the final level.

Validation:
To validate the LLM-based detection, the authors correlated discrepancy counts with an objective scoring metric based on target identification performance (visual confirmation of specific objects). Team 16, which exhibited significantly more SMM discrepancies (particularly omissions and contradictions), scored lower (26.3%) than Team 8 (42.1%), supporting the claim that communication-based misalignment manifests as measurable performance decrements.

Significance and Claims

The paper claims to provide the first systematic categorization of SMM discrepancy types in naturalistic team coordination. Its primary contributions are:

1. Theoretical Framework: Distinguishing between four specific types of misalignment (omissions, contradictions, false beliefs, unsupported beliefs) rather than treating SMM quality as a monolithic metric.
2. Temporal Dynamics: Demonstrating that different discrepancy types exhibit differential temporal stability. Omissions showed stable patterns across levels, suggesting information-sharing behaviors are resistant to change. In contrast, belief contradictions showed greater volatility, with some teams resolving them through dialogue while others did not.
3. Predictive Potential: Showing that even a simple uniform weighting of historical discrepancy counts can yield meaningful prediction accuracy, suggesting early-stage coordination patterns establish trajectories that persist.

The authors modestly conclude that while the framework enables targeted intervention strategies (e.g., addressing information gaps vs. interpretive disagreements), the current additive model assumes independence between discrepancy types, which may not reflect reality (e.g., omissions cascading into contradictions). Future work is needed to incorporate team-specific learning curves and type-specific dynamics.