💬 NLP

LLMs and their Limited Theory of Mind: Evaluating Mental State Annotations in Situated Dialogue

This paper introduces a two-step framework using large language models to annotate shared mental models in team dialogues and detect discrepancies, revealing that while LLMs perform well on straightforward tasks, they systematically fail in scenarios requiring spatial reasoning or prosodic disambiguation.

Original authors: Katharine Kowalyshyn, Matthias Scheutz

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Katharine Kowalyshyn, Matthias Scheutz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-stakes game of "Blindfolded Treasure Hunt" with a friend. You are the Searcher, walking through a real building with your eyes open. Your friend is the Director, sitting in a different room with a map that is slightly wrong. They can't see you; they can only talk to you over the phone.

To succeed, you both need to build a Shared Mental Model (SMM). This is like a invisible, mental whiteboard where you both write down what you know: "I'm in the hallway," "The box is blue," "You are looking for the green box." If your mental whiteboards don't match, you get lost, and the team fails.

This paper asks a simple but deep question: Can a super-smart AI (a Large Language Model or LLM) act like a referee and accurately track what's on your mental whiteboards just by listening to your phone conversation?

The Experiment: The "Blind" vs. The "All-Seeing"

The researchers set up a clever test using old recordings of humans playing this game. They created three groups of "referees" to analyze the same conversations:

The Naive Humans: Regular people who only heard the audio. They had to guess what the Searcher was seeing based only on the words spoken.
The AI Models: Three different AIs (o3-mini, Claude, and Gemma) that also only heard the audio.
The "God's Eye" View (Ground Truth): A team of humans who watched the video of the game. They knew exactly where the Searcher was standing and what they were looking at at every second. This was the "correct answer key."

The Two-Step Process

The researchers used the AI in two different roles, like a writer and an editor:

Step 1: The Writer (Trace Generation): The AI listened to the conversation and tried to write down a "mental model trace." It had to guess: What does the Searcher believe right now? What is their goal? What have they promised to do?
Step 2: The Editor (Discrepancy Detection): A separate AI (acting as a judge) compared the "Writer's" guess against the "God's Eye" video answer key. It counted the mistakes.

The Mistakes: Where the AI Got Lost

The paper found that while the AIs were good at sounding smart, they struggled with the messy reality of human communication. Here are the main types of errors, explained with analogies:

The "Hallucinating" Director (Unsupported Beliefs):
- What happened: The AI invented facts that weren't there.
- Analogy: Imagine the Director says, "I think there's a cat in the room." The AI, hearing this, writes on the mental whiteboard, "The Searcher believes there is a cat." But the video shows there is no cat. The AI made up a belief just because the words sounded plausible.
- Result: The AI (especially Claude) did this a lot, filling the whiteboard with imaginary details.
The "Missing" Details (Omissions):
- What happened: The AI missed things that were actually said.
- Analogy: The Searcher says, "I'm turning left." The AI's whiteboard stays blank, forgetting to update the location.
- Result: The AI (especially o3-mini) was often too quiet, leaving out important updates.
The "Spatial" Confusion:
- What happened: The AI got confused about where things were in space.
- Analogy: If the Director says, "The box is to your left," the AI might write, "The box is to the right." It struggles to translate words into a 3D map in its head.
The "Stutter" Trap:
- What happened: Humans say "um," "uh," and stumble over words. The AI often treated these stutters as new information.
- Analogy: If the Director stutters, "There's a... uh... door," the AI might think, "Aha! The 'uh' means they are changing their mind about the door!" and update the whiteboard unnecessarily.

The Verdict: Smart but Groundless

The paper concludes that current AIs are like actors who are very good at reading a script but terrible at improvising in a real room.

They can mimic the language of thinking: They can use words like "I believe" or "My goal is."
They cannot ground the thinking: They can't connect those words to the physical reality (the video) or handle the messy, stuttering nature of real human speech.

Interestingly, the "Naive Humans" (who also didn't see the video) made similar mistakes to the AIs. This proves that the task is genuinely hard without seeing the video. However, the AIs were worse at avoiding "hallucinations" (making things up) than the humans were.

The Bottom Line

This study doesn't say AIs can't be used in teams. It says that if you want an AI to understand a team's shared mental state just by listening to them talk, it currently lacks the "common sense" to know what is real and what is just a guess. It needs the "video" (the real-world context) to truly understand what's happening, rather than just guessing based on the words it hears.

Technical Summary: LLMs and their Limited Theory of Mind

Problem Statement

Effective human team performance relies on a Shared Mental Model (SMM), a consistent understanding among team members regarding tasks, goals, and latent belief states. While humans implicitly grasp the subtext of situated dialogue (assumptions, intentions, and knowledge), it remains unclear whether Large Language Models (LLMs) can accurately infer and annotate these internal belief states based solely on dialogue interactions. Current evaluations of Theory of Mind (ToM) in LLMs often rely on static, decontextualized tasks (e.g., multiple-choice questions) that fail to capture the dynamic, real-time reasoning required for situated coordination. This paper addresses the gap in evaluating LLMs' ability to trace belief states in interactive, grounded team contexts and detect discrepancies between inferred mental models and ground truth.

Methodology

The authors propose a novel, two-step framework to evaluate LLMs as generators of mental model traces and as automated detectors of discrepancies.

1. Dataset and Setup

Corpus: The study utilizes the Cooperative Remote Search Task (CReST) corpus, comprising situated task-based interactions between a "searcher" (physically present in an environment) and a "director" (remote, viewing an incomplete map).
Ground Truth: Six dialogues with associated video recordings (searcher's headcam and director's screen) were selected to establish ground truth mental model traces.
Trace Generators: Three independent LLMs (OpenAI o3-mini, Anthropic Claude Sonnet 4, Google Gemma 8.5B) and two human annotator groups were tasked with generating mental model traces for the six dialogues.
- Naive Condition: LLMs and one pair of humans received only the audio transcript (no video).
- Ground Truth Condition: A second pair of humans received the transcripts plus video evidence to establish the ground truth traces.
Annotation Schema: Traces were generated as JSON objects updating belief, goal, and commitment states for each utterance, using a specific verb vocabulary to ensure comparability.

2. Discrepancy Analysis Framework

A separate "LLM judge" compared the audio-only traces (from LLMs and naive humans) against the video-grounded truth traces. The framework defines four specific discrepancy types:

Belief Contradictions: Mutually exclusive propositions held by agents in the trace vs. ground truth.
False Beliefs: Propositions in the trace that disagree with ground truth (even if not direct negations).
Unsupported Beliefs: Propositions in the trace that lack evidential basis in the ground truth (hallucinations).
Omissions: Missing belief/goal updates present in the ground truth but absent in the trace.

A weighted discrepancy score ( $S_{m,d}$ ) was calculated, normalized to [0, 1], to quantify alignment with ground truth, accounting for dialogue length and error severity.

Key Contributions

First Study of its Kind: To the authors' knowledge, this is the first study to use real-world, task-based dialogue belief-state traces as a lens for evaluating ToM in LLMs.
Novel Framework: Introduction of a reproducible, two-step evaluation pipeline for SMM coherence that distinguishes between missing perceptual information and inferential errors.
Dataset Release: Creation of a dataset containing human and LLM mental model traces, along with ground truth traces derived from video, enabling future research on situated ToM.
Discrepancy Taxonomy: Definition and operationalization of four distinct error types (Contradictions, False Beliefs, Unsupported Beliefs, Omissions) with a severity ranking.

Results

The evaluation across six dialogues (1,142 utterances) revealed significant performance gaps between models and humans:

Model Performance:
- Claude Sonnet 4 exhibited the highest overall discrepancy counts (surpassing 700 in Dialogue 3) and the lowest normalized scores. It frequently generated unsupported beliefs and belief contradictions, suggesting a tendency toward speculative reasoning and overgeneralization.
- o3-mini and Gemma 8.5B produced markedly fewer discrepancies. However, they displayed distinct profiles: o3-mini had high omission rates, while Gemma had near-zero omissions but higher rates of belief contradictions.
- False Beliefs were notably sparse across all models, indicating that errors often stem from ungrounded extrapolation rather than confident misidentification.
Human vs. LLM:
- Naive human annotators outperformed LLMs in limiting unsupported beliefs (hallucinations) but showed comparable or higher rates of belief contradictions and omissions.
- Statistical analysis (Two-way ANOVA) confirmed a strong main effect of the model ( $p < 2e-16$ ), with Claude differing significantly from both Gemma and o3-mini.
Error Patterns: Qualitative analysis indicated that LLMs systematically err in scenarios requiring spatial reasoning or the disambiguation of transcription-level disfluencies. Errors clustered in turns where environmental grounding was critical but unavailable in the text alone.

Significance and Claims

The paper claims that while LLMs can simulate limited, measurable ToM-like behaviors (maintaining internally consistent belief structures), they lack grounded situational reasoning.

Limited ToM: LLMs appear to infer mental states from linguistic regularity rather than integrated spatial, temporal, and epistemic understanding. They produce traces that are coherent on straightforward natural language tasks but fail when spatial constraints or environmental grounding are required.
Proxy Task Validation: The study proposes situated dialogue trace generation as a viable proxy task for probing ToM capabilities in AI systems, moving beyond static benchmarks.
Implications: The findings underscore that current LLMs are driven more by linguistic patterns than by the dynamic, real-time reasoning underlying human ToM. This limitation is critical for applications involving human-AI teaming where accurate shared mental models are essential for coordination.

The authors remain modest, acknowledging limitations such as the non-deterministic nature of LLM outputs, the specific scale of the CReST subset, and the lack of a condition where LLMs have access to video (multimodal input), which they identify as a direction for future work.