Dynamic Theory of Mind as a Temporal Memory Problem: Evidence from Large Language Models

This paper introduces the DToM-Track framework to demonstrate that while Large Language Models can reliably infer an agent's current belief, they consistently struggle to maintain and retrieve prior belief states over time due to recency bias and interference, revealing that dynamic Theory of Mind is fundamentally a temporal memory challenge distinct from static false-belief reasoning.

The Gist

Imagine you are playing a game of "Guess What I'm Thinking" with a friend. In a normal conversation, your friend might say, "I think the movie starts at 7 PM." Then, five minutes later, they get a text and say, "Oh wait, I just saw an update; it actually starts at 8 PM."

If you have a good Theory of Mind (the ability to understand what others are thinking), you should be able to do two things:

1. Know what they believe right now (8 PM).
2. Remember what they believed just a moment ago (7 PM), even though they changed their mind.

This paper is about testing Large Language Models (AI chatbots) to see if they can do that second part.

The Big Problem: The "Amnesia" Effect

The researchers found that current AI models are great at knowing what someone believes right now, but they have a terrible case of "amnesia" when it comes to what that person believed before they changed their mind.

It's like watching a movie where the main character keeps changing their mind about the plot, but the AI narrator only remembers the latest version of the story and completely forgets the previous chapters.

The Experiment: DToM-Track

To test this, the researchers created a new game called DToM-Track. Here is how it works, using a simple analogy:

The Setup:
Imagine two AI characters, Alex and Sam, having a conversation.

• The Secret: Before Alex speaks, the AI makes Alex "think out loud" (like a secret inner monologue) about what they believe. For example, Alex thinks, "I believe the restaurant requires a 24-hour notice."
• The Change: Later in the conversation, Alex gets new information and changes their mind. They think, "Wait, I just realized, no reservations are needed."
• The Test: The researchers ask the AI (acting as a judge) questions like:
  • "What did Alex believe before the update?" (The hard question)
  • "What does Alex believe now?" (The easy question)
  • "When did Alex change their mind?"

The Results: The "Recency Bias"

The results were very clear and consistent across different AI models (from small ones to the huge, powerful ones):

1. The "Now" is Easy: When asked what Alex believes currently, the AI got it right almost every time. It's like asking, "What is the weather right now?" and getting a perfect answer.
2. The "Before" is Hard: When asked what Alex believed before the change, the AI failed miserably. It often forgot the old belief entirely and just repeated the new one.

The Analogy:
Think of the AI's memory like a whiteboard.

• When new information comes in, the AI grabs an eraser and wipes the board clean, then writes the new belief.
• It is excellent at reading what is currently written on the board.
• But if you ask, "What was written on the board five minutes ago?", the AI says, "I don't know, I erased it!"

The researchers call this "Recency Bias." It's the same thing that happens to humans when we try to remember a phone number we just heard versus one we heard an hour ago. The most recent information "drowns out" the older information.

Why Does This Matter?

Most tests for AI intelligence only ask, "What does the character believe right now?" (like a snapshot photo). This paper argues that real human conversation is more like a movie. We need to track how beliefs change over time.

If an AI is going to be a helpful assistant, a therapist, or a friend, it needs to remember not just what you think now, but what you thought yesterday or even five minutes ago before you corrected yourself. If it forgets the old belief, it might get confused, repeat itself, or misunderstand your history.

The Takeaway

The paper concludes that while AI is getting smarter at understanding people, it still struggles with the memory part of social interaction. It's not that the AI is "stupid"; it's that its memory works differently than ours. It prioritizes the "newest" information so heavily that it loses the "old" information, making it hard for it to track the full story of a conversation.

In short: AI is great at knowing what you think today, but it's currently terrible at remembering what you thought yesterday before you changed your mind.

Technical Summary

1. Problem Statement

Theory of Mind (ToM) is the capacity to attribute mental states (beliefs, intentions, desires) to others. While essential for social cognition and human-AI interaction, current evaluations of Large Language Models (LLMs) largely treat ToM as a static, single-moment judgment, primarily relying on "false belief" paradigms (e.g., "What does X believe right now?").

This static approach overlooks a critical dynamic dimension: Temporal Belief Tracking. In real-world interactions, agents must not only infer current beliefs but also:

• Maintain representations of prior beliefs.
• Update these representations when new information arrives.
• Retrieve earlier belief states after they have been superseded.

The authors hypothesize that LLMs may struggle with this temporal aspect due to recency bias (recent information dominating judgment) and interference effects (new representations disrupting access to old ones), phenomena well-documented in human cognitive science but under-explored in LLMs.

2. Methodology: The DToM-Track Framework

To address this gap, the authors introduce DToM-Track, a controlled evaluation framework designed to probe temporal belief reasoning in multi-turn conversations.

A. Data Generation (Controlled Conversation)

• Inner-Speech Prompting: The framework uses LLMs (specifically LLaMA-3-70B-Instruct) to generate multi-turn dialogues between two agents. Crucially, agents verbalize their internal mental states (thoughts) via "inner speech" before speaking, but these thoughts are hidden from the conversational partner. This creates information asymmetry, mirroring real-world false-belief scenarios.
• Planned Belief Updates: Unlike unconstrained generation, belief updates are explicitly scheduled at specific turns (e.g., Turn 4–8). Agents are instructed to change their beliefs based on triggers like corrections, new information, or goal shifts.
• Linguistic Markers: Updates are signaled using naturalistic linguistic cues (e.g., "Actually...", "I just realized...") to ensure the dialogue flows naturally while maintaining a ground truth for evaluation.

B. Belief State Tracking

A dedicated tracker maintains a structured representation of mental states across turns, recording:

1. The source turn of the belief.
2. The belief type (Belief, Intention, Desire, Emotion, Knowledge).
3. An update flag (indicating if the belief is superseded).
4. The prior content before the update.

C. Question Types & Evaluation

The framework generates multiple-choice questions to test three specific temporal dimensions:

1. Pre-Update: "What did X believe before the change at Turn T?" (Tests memory retrieval of prior states).
2. Post-Update: "What does X believe after the change?" (Tests current state inference).
3. Update Detection: "At which turn did X's belief change?" (Tests tracking of the transition).

These are compared against standard ToM tasks: Temporal (state at a specific turn), Second-Order (X's belief about Y's belief), and False Belief (X's belief vs. reality).

D. Quality Control

A multi-stage LLM-based filtering pipeline ensures:

• Update Verification: Confirming planned belief changes actually occurred in the text.
• QA Verification: Ensuring questions are unambiguous, answers are derivable from context, and distractors are plausible.
• Sample Validation: Verifying models can parse the output format.

3. Experimental Setup

• Models Evaluated: Six LLMs ranging from 3B to 70B parameters, including open-source (LLaMA 3.1/3.2/3.3, Mistral) and proprietary (GPT-4o-mini) models.
• Dataset: 5,794 validated questions across six question types and five mental state categories.
• Protocol: Zero-shot evaluation where models receive the conversation context and a multiple-choice question.

4. Key Results

The experiments reveal a consistent asymmetry in LLM performance across model families and scales:

• Strong Current Inference: Models perform well on Post-Update questions (63.9% average accuracy) and Update Detection (67.5%), indicating they can successfully infer an agent's current belief state.
• Weak Prior Recall: Models struggle significantly with Pre-Update questions (27.7% average accuracy). This is a stark drop compared to Post-Update performance.
• Comparison to Static Tasks: Pre-update performance (27.7%) is substantially lower than standard False Belief tasks (44.7%) and Second-Order reasoning (47.2%).
• Recency Bias: The gap between current and prior belief tracking persists even in the largest models (LLaMA 3.3-70B), suggesting the issue is not merely a lack of capacity but a fundamental limitation in temporal representation and retrieval.
• Interference: The results align with cognitive science theories where recent updates interfere with access to earlier mental states.

5. Key Contributions

1. Reframing ToM: The paper shifts the definition of ToM evaluation from a static snapshot to a temporal memory problem, highlighting that social reasoning requires maintaining and retrieving evolving mental states.
2. DToM-Track Framework: Introduces a novel, controlled methodology using inner-speech prompting and planned belief updates to isolate temporal components of ToM that are invisible in static benchmarks.
3. Empirical Evidence of Asymmetry: Provides robust evidence that LLMs suffer from a specific "recency bias" in social reasoning—they are good at "what is happening now" but poor at "what was believed before," a limitation that scales with model size.
4. Connection to Cognitive Science: Bridges the gap between LLM evaluation and human cognitive mechanisms (recency bias, interference), suggesting LLMs mimic human memory constraints in social contexts.

6. Significance and Implications

• For Human-AI Interaction: As AI systems are deployed in long-term, multi-turn interactions (e.g., personal assistants, therapy bots), their inability to track belief trajectories over time could lead to misunderstandings, failure to adapt to user corrections, or "hallucinated" consistency with past user preferences.
• For AI Safety & Alignment: Understanding that LLMs struggle to retrieve prior states is crucial for debugging systems where consistency over time is required.
• Future Research: The paper suggests that improving ToM in LLMs requires architectural or training changes specifically targeting long-term memory retrieval and interference mitigation, rather than just scaling model size. It also calls for future work to test if these patterns hold in human judgments and naturalistic (non-synthetic) dialogues.

In conclusion, the paper argues that dynamic belief tracking is a distinct and challenging component of Theory of Mind that current LLMs fail to master, primarily due to memory interference and recency bias, necessitating new evaluation frameworks like DToM-Track.