Language Models Might Not Understand You: Evaluating Theory of Mind via Story Prompting

1. Problem Statement

Large Language Models (LLMs) have shown significant improvements in reasoning tasks (math, coding, logic), but their ability to reason about Theory of Mind (ToM)—the capacity to attribute mental states, beliefs, and perspectives to others—remains uncertain.

• Data Contamination: Existing ToM benchmarks (e.g., Sally-Anne test, TOMI) are likely present in the pretraining data of modern LLMs. Consequently, models may be "memorizing" answers rather than genuinely reasoning, leading to inflated performance metrics.
• Lack of Control: Prior synthetic benchmarks often rely on LLMs to generate the stories themselves, which introduces inconsistency and makes it difficult to isolate specific variables (e.g., character perspective vs. event timing).
• The Gap: There is a need for a robust, contamination-free framework to evaluate whether LLMs can truly model mental states or if they rely on heuristics (like recency bias) and memorization.

2. Methodology: The StorySim Framework

The authors introduce StorySim, a programmable, synthetic framework designed to generate novel stories for evaluating ToM and World Modeling (WM) capabilities.

Core Components

• The STORYBOARD: A high-level, symbolic specification defined as $D = (C, \phi, G, E, n)$, where:
  • $C$: Set of characters.
  • $G$: A directed graph representing locations and connectivity.
  • $E$: A set of deterministic, timestamped key events (e.g., "Character A moves to Room 2 at $t=5$").
  • $n$: Total number of events.
  • $\phi$: Action types (e.g., enter, move).
• Generation Process:
  1. Symbolic Generation: StorySim generates a sequence of events that strictly adheres to the STORYBOARD constraints. It fills in non-deterministic details randomly while ensuring logical consistency (e.g., characters can only move to adjacent locations).
  2. Translation: Events are converted into natural language using templates.
  3. Paraphrasing (Optional): Stories can be further paraphrased by an LLM (GPT-4o-mini) to increase linguistic naturalism, though the core logic remains anchored to the symbolic storyboard.
• Novelty: Because the stories are algorithmically generated with specific constraints and random variations, the probability of any generated story appearing in pretraining data is "astronomically small."

Experimental Design

The authors designed three specific prompt types to isolate different cognitive abilities:

1. ToM Prompt: Asks about a character's belief regarding another's location (e.g., "Where does Bob think Alice is?"). This tests the ability to model a perspective that differs from reality.
2. WM-Human Prompt: Asks about the objective location of a character based on observed events (e.g., "Where did Alice go when Bob was in the room?"). This tests World Modeling without requiring perspective-taking.
3. WM-Inanimate Prompt: Similar to WM-Human but uses inanimate objects moved by external forces. This tests if the model's reasoning is sensitive to the "animate" nature of agents.

Key Variables Manipulated:

• Mislead Distance: The number of time steps between the last time a character ($S1$) saw another ($T$) move and the final location of $T$. This tests for recency bias (predicting the most recent location rather than the last observed location).
• Order of ToM: First-order (A thinks B is at X) vs. Second-order (A thinks B thinks C is at X).
• Character Count: Varying the number of characters to test generalization beyond simple, familiar story structures.

3. Key Contributions

1. StorySim Framework: A fully controllable, synthetic benchmark that eliminates data contamination risks and allows precise manipulation of narrative variables (perspective, timing, graph structure).
2. Decoupling ToM and WM: The study provides a method to evaluate ToM independently of general World Modeling, revealing that models often excel at tracking physical events (WM) but fail at tracking mental states (ToM).
3. Heuristic Identification: The framework successfully identified specific failure modes, such as the "First Common Location Heuristic" (predicting where characters first met rather than where they currently are) and Recency Bias.
4. Comprehensive Benchmarking: The paper evaluates a wide suite of models (GPT-4, GPT-5.4, Llama 3.1/3.2/3.3, R1, Qwen) across zero-shot and few-shot settings.

4. Key Results

• ToM vs. WM Discrepancy: Most models achieved higher accuracy on World Modeling (WM) tasks than on Theory of Mind (ToM) tasks. This suggests LLMs are better at tracking physical states than simulating the subjective perspectives of others.
• The "Human" Effect: Models performed significantly better on WM tasks involving human subjects compared to inanimate objects, suggesting that the presence of "agents" triggers different (and more accurate) reasoning pathways.
• Model Size Correlation:
  • In the Llama 3.1 family, there was a clear positive correlation between model size (parameters) and ToM performance.
  • DeepSeek R1 showed the highest overall ToM accuracy and the smallest gap between ToM and WM performance.
  • GPT-5.4 excelled at WM but struggled with ToM, highlighting that raw reasoning power does not automatically equate to social reasoning.
• Sally-Anne Test Failure: When tested on a standard Sally-Anne setup, models achieved 100% accuracy. However, when tested on randomly generated stories of the same length, accuracy dropped significantly (e.g., to ~75-87%), confirming that the classic test is a poor measure of genuine ToM due to data contamination.
• Heuristic Reliance:
  • Recency Bias: The study found no significant evidence that models rely on recency bias (predicting the last known location) as a primary failure mode; performance remained stable across varying "mislead distances."
  • First Common Location Heuristic: A dominant error pattern was predicting the location where characters first crossed paths, rather than tracking their actual movements.
  • Context Length Issues: Reasoning-heavy models (like R1) frequently failed due to insufficient context length when generating long chains of thought, leading to "prompt refusal" or hallucinated errors.
• Complexity: Increasing the number of characters in a story did not significantly degrade ToM performance for top-tier models, suggesting robustness in handling complex social graphs.

5. Significance and Future Work

• Redefining Benchmarks: The paper argues that the field must move away from static, potentially contaminated datasets (like Sally-Anne) toward dynamic, synthetic benchmarks like StorySim to truly assess LLM capabilities.
• Safety Implications: A lack of robust ToM is a critical safety concern. If an LLM cannot understand that a user has a different perspective or hidden intent, it may fail to detect harmful scenarios or manipulate users.
• Future Directions: The authors suggest using StorySim to explore more complex scenarios, including counterfactual reasoning, goal attribution, and preference modeling. They also highlight the need to improve model context handling to support deeper chains of reasoning required for complex ToM tasks.

In conclusion, the paper demonstrates that while LLMs are improving, they still struggle with the specific cognitive leap required for Theory of Mind, often relying on heuristics or failing to distinguish between objective world states and subjective beliefs. StorySim provides the necessary tool to rigorously measure and improve these capabilities.