💬 NLP

Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs

By evaluating 41 open-weight language models on false belief tasks, this study demonstrates that while larger models show improved sensitivity to knowledge states, their ability to replicate human reasoning is limited, though they successfully model specific linguistic biases in belief attribution, thereby validating the use of diverse open-weight models to both assess AI capacities and test theories of human social cognition.

Original authors: Sean Trott, Samuel Taylor, Cameron Jones, James A. Michaelov, Pamela D. Rivière

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sean Trott, Samuel Taylor, Cameron Jones, James A. Michaelov, Pamela D. Rivière

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand human feelings, specifically the tricky concept that other people can believe things that aren't true. This is called "False Belief Reasoning," or in psychology terms, a basic part of "Theory of Mind."

For example: If you hide a cookie in a jar, but your friend moves it to a box while you aren't looking, you know the cookie is in the box. But your friend thinks it's still in the jar. Can a computer figure out that your friend is wrong?

This paper is a massive experiment to find out. Instead of testing just one or two "black box" robots (which are like sealed vending machines where you can't see how they work), the researchers tested 41 different open-weight AI models. Think of these as 41 different students in a classroom, ranging from small, simple students to giant, super-smart ones, all with their textbooks (training data) open for everyone to see.

Here is the breakdown of what they found, using some everyday analogies:

1. The "Mind-Reading" Test

The researchers gave these AI models a story about a character named John.

Scenario A (True Belief): John sees a ball move from a box to a basket. He knows where it is.
Scenario B (False Belief): John doesn't see the ball move. He thinks it's still in the box, but it's actually in the basket.

The AI had to guess where John would look for the ball.

The Result: About 34% of the AI models got it right. They showed they could "guess" what John was thinking based on the story.
The Catch: Even the smartest AI models weren't perfect. They got it right about 74% of the time, while humans got it right about 83% of the time. The AIs are getting better, but they still aren't quite as good at this specific human trick as we are.

2. The "Size Matters" Rule

The researchers noticed a clear pattern: Bigger models performed better.

Analogy: Imagine a library. A small library (a small AI) might only have a few books about "John and the ball." A giant library (a huge AI) has millions of stories. The more stories the AI has read, the better it gets at guessing what John is thinking.
The Finding: As the AI models got larger (more "parameters," which is like having more brain cells), their ability to understand these mental states improved. However, even the biggest models couldn't fully explain why humans are so good at it.

3. The "Magic Word" Surprise (The Big Discovery)

This is the most fascinating part. The researchers found a weird quirk in how both humans and AIs think, which they call the "Magic Word" effect.

The Setup: They tested two ways of asking the question:
1. Indirect: "John looks for the ball in the..." (No specific verb about thinking).
2. Direct: "John thinks the ball is in the..." (Using the word "thinks").
The Quirk: When the sentence used the word "thinks" (a non-factive verb), both humans and AIs were more likely to make a mistake. They were more likely to say the ball was in the wrong place (the box) just because the word "thinks" was used.
Why? It's like a linguistic trap. When we say "John thinks X," our brains sometimes subconsciously prepare for the possibility that X might be false.
The Surprise: For this specific "Magic Word" trick, humans and AIs were almost identical. The human reaction fell right in the middle of the AI reactions.
What this means: This suggests that for this specific type of error, we don't need a special "human soul" or complex social brain. We might just be reacting to the statistical patterns of how words like "think" are used in language. The AI learned this from reading the internet, and humans learned it from hearing us talk.

4. The "Black Box" Problem

The paper argues that we need to stop testing only "closed-source" models (like the ones from big tech companies that keep their secrets).

Analogy: If you want to understand how a car engine works, you can't just drive a car with the hood welded shut. You need to open the hood (open-weight models) to see the gears, the pistons, and the fuel lines.
By testing 41 different models, the researchers could see that performance varies wildly. Some small models failed completely, while others did surprisingly well. This helps us understand that "AI" isn't a single thing; it's a family of tools with different strengths.

The Bottom Line

This paper tells us two main things:

AI is getting good at "mind-reading," but it's not quite human-level yet. The bigger the AI, the better it gets, but it still misses some of the nuance that humans pick up naturally.
Language statistics are powerful. For certain types of reasoning (like the "Magic Word" trap), AIs can mimic human behavior perfectly just by analyzing how words are used in sentences. This suggests that a lot of what we consider "social intelligence" might actually just be a very sophisticated understanding of language patterns.

In short: AIs are like students who have read every book in the library. They are starting to understand how people think, but they are still learning the difference between "reading the words" and "understanding the heart."

1. Problem Statement

Research into whether Large Language Models (LMs) possess "Theory of Mind" (specifically, the ability to reason about false beliefs) is critical for understanding both human social cognition and the limits of AI. However, existing literature suffers from significant methodological flaws:

Small, Closed-Source Samples: Most studies rely on a handful of closed-source models (e.g., deprecated versions of GPT-3), making it impossible to verify training data, detect data contamination, or reproduce results.
Lack of Generalizability: Findings from single model instances cannot be generalized to the broader class of LMs.
Epistemological Gaps: Without knowing model properties (size, training tokens, architecture), it is difficult to determine if specific behaviors emerge from language statistics or other factors.

The authors aim to address these issues by replicating and extending previous work (Trott et al., 2023) using a large, diverse sample of 41 open-weight LMs to rigorously test the hypothesis that human mental state reasoning emerges from exposure to distributional language statistics.

2. Methodology

2.1 Materials (The False Belief Task)

The study utilized the stimuli from Trott et al. (2023), based on the canonical False Belief (FB) task:

Scenario: A character places an object in a "Start" location. A second character moves it to an "End" location.
Manipulations:
- Knowledge State: The first character is either present (True Belief) or absent (False Belief) during the move.
- Knowledge Cue: The probe sentence uses either an Explicit non-factive verb ("John thinks the book is in...") or an Implicit indirect cue ("John looks for the book in...").
- Control: Variations in word order (First Mention vs. Recent Mention).
Total Stimuli: 192 passages (12 scenarios $\times$ 16 versions).

2.2 Models

The authors tested 41 models across 5 distinct model families:

Families: Gemma 2, Llama 3, OLMo-2, Pythia, and Qwen 2.5.
Variety: Included models ranging from ~14M to 32B parameters, covering both "Base" (pre-trained) and "Instruct" (instruction-tuned) variants.
Hardware: Models were run locally on an NVIDIA DGX-H200 using the HuggingFace transformers package.

2.3 Procedure & Metrics

LM Evaluation: Instead of asking for a direct answer, the models were prompted to complete a sentence. The authors calculated the Log Odds (difference in log-probability) between the token representing the "Start" location (incorrect for FB) and the "End" location (correct).
- Positive Log Odds: Bias toward the incorrect Start location.
- Negative Log Odds: Bias toward the correct End location.
Human Data: Reused data from Trott et al. (2023) ( $N=613$ participants).
Analysis:
- Sensitivity: Linear mixed-effects models (LMM) to test if Log Odds varied significantly by Knowledge State.
- Psychometric Predictive Power (PPP): Measured how well an LM's Log Odds predicted human responses ( $R^2$ ) and whether the LM could "explain away" the human effect of Knowledge State (i.e., does adding LM Log Odds to a model predicting human behavior render the Knowledge State manipulation insignificant?).
- Correlates: Regression analyses linking model performance to parameter count, training tokens, and instruction tuning.

3. Key Contributions

Large-Scale Open-Weight Benchmark: The first study to evaluate false belief reasoning across 41 open-weight models, providing a robust distribution of effect sizes rather than relying on single point estimates.
Replication and Extension: Successfully replicated the finding that LMs show some sensitivity to belief states but fail to match human performance, while extending the analysis to include novel hypotheses about linguistic cues.
Novel Hypothesis Generation: Used LM behavior to generate and test a new hypothesis regarding non-factive verbs (e.g., "thinks") and their role in belief attribution, revealing a shared bias between humans and LMs.
Methodological Framework: Demonstrated the utility of open-weight models for "taxonomizing" AI behaviors and testing theories of human cognition by comparing human effect sizes against the distribution of LM effect sizes.

4. Key Results

4.1 Sensitivity to Knowledge State (RQ1)

34% of models (14/41) showed statistically significant sensitivity to the character's knowledge state (True vs. False Belief).
The direction of the effect was correct: models were more likely to predict the "End" location when the character was absent (False Belief condition).
Accuracy: The top-performing models achieved 74.5% accuracy, comparable to GPT-3 in prior work, but still below the human baseline of ~82.7%.

4.2 Explaining Away Human Effects (RQ2)

No model "explained away" the human effect. Even the best-performing LMs failed to account for the variance in human responses caused by the Knowledge State manipulation.
Conclusion: Distributional statistics alone (as captured by these LMs) are insufficient to fully account for human-level false belief reasoning. Humans are more "sample-efficient" at this task than current LMs, despite LMs being trained on orders of magnitude more data.

4.3 Predictors of Performance (RQ3 & RQ4)

Parameter Count: The number of parameters was the only significant predictor of both accuracy and Psychometric Predictive Power (PPP).
- Each order of magnitude increase in parameters correlated with an ~8% increase in accuracy.
- Larger models also showed higher $R^2$ in predicting human behavior.
Instruction Tuning: While instruction-tuned variants often performed well, parameter count remained the dominant factor.

4.4 The Non-Factive Verb Effect (RQ5)

Shared Bias: Both humans and LMs exhibited a bias toward attributing false beliefs when the probe used a non-factive verb ("John thinks...") compared to an indirect cue ("John looks...").
Distributional Alignment: Unlike the Knowledge State effect (where humans vastly outperformed LMs), the magnitude of the Knowledge Cue effect in humans fell squarely within the distribution of LM effect sizes.
Implication: The bias toward "incorrect" beliefs when using "thinks" can likely be explained by distributional statistics (the statistical association between "thinks" and uncertainty/falsehood in training data), whereas the core ability to reason about belief states cannot.

5. Significance and Implications

Theoretical Insight: The results support a nuanced view of human cognition: while distributional language statistics can explain certain linguistic biases (like the "thinks" effect), they are insufficient to explain the full capacity for Theory of Mind. Human mental state reasoning likely relies on additional mechanisms (e.g., grounding, social interaction, or innate constraints).
Scientific Rigor: The study highlights the necessity of using open-weight, open-data models to avoid data contamination and ensure reproducibility. It establishes a baseline distribution of "statistical" performance against which human cognition can be compared.
Future Directions: The authors suggest that future research should focus on mechanistic approaches (identifying neural circuits for reasoning) and expanding stimulus diversity to rule out "shortcuts" or overfitting to specific prompt structures.

In summary, the paper demonstrates that while LMs trained on language statistics can mimic certain aspects of human social cognition, they fundamentally lack the robust, human-level capacity for false belief reasoning, suggesting that language exposure is a necessary but not sufficient condition for the emergence of full Theory of Mind.