Traces of Social Competence in Large Language Models

This paper investigates the social competence of 17 large language models using an extensive set of False Belief Test variants, revealing that while scaling improves performance, the explicit mention of mental states triggers a cross-over effect rooted in pre-training stereotypes that can be causally isolated via vector steering.

The Gist

Imagine you are trying to teach a robot how to understand human feelings and thoughts. You want to know: Does this robot actually "get" that other people might believe things that are different from reality?

In psychology, there's a famous test for this called the "False Belief Test." It's like a riddle:

Maxi puts his chocolate in a blue box. He leaves. His mom moves the chocolate to a green box. When Maxi comes back, where will he look for the chocolate?

A human who understands "False Belief" knows Maxi will look in the blue box (because he doesn't know it moved). A human who doesn't understand it will say the green box (because that's where the chocolate actually is).

This paper is a deep dive into whether Large Language Models (LLMs)—the brains behind AI chatbots—can pass this test. The researchers didn't just ask the AI the question; they treated the AI like a lab rat in a very complex maze to see how it thinks.

Here is the story of their findings, explained with some everyday analogies.

1. The "Size Doesn't Always Matter" Rule

You might think, "If I make the AI bigger and give it more data, it will get smarter."

• The Finding: Yes, making the AI bigger helps it get the "False Belief" answer right. But there's a catch.
• The Analogy: Imagine a student studying for a test. If you give them a bigger library (more data), they get better at spotting the trick questions. However, if you ask them a simple, straightforward question ("Where is the chocolate?"), the bigger student sometimes gets more confused than the smaller one.
• Why? The bigger models have read so many stories where someone thinks something wrong, that they start expecting a "trick" even when there isn't one. They overthink the simple stuff.

2. The "Magic Word" Trap

The researchers changed the wording of the question slightly.

• Version A (Explicit): "What does Maxi think?"

• Version B (Implicit): "Where does Maxi go to get the chocolate?"

• The Finding: When the AI sees the word "think," it gets very good at the trick questions (False Belief). But when the word is just an action like "go," it struggles.

• The Analogy: It's like a dog trained to sit only when you say the specific command "Sit." If you say "Please sit down," the dog might not listen. The AI has learned a stereotype: "When I see the word 'think,' I must look for a trick." It's not reasoning about the story; it's reacting to a keyword.

3. The "Over-Training" Problem

The researchers looked at how the AI changes as it goes through different training stages:

1. Base Model: Just read a lot of books.
2. Instruction Tuned: Taught to follow orders and be helpful.
3. Reasoning Tuned: Taught to "think step-by-step."

• The Finding:
  • Instruction Tuning helped a little. It made the AI slightly less rigid.
  • Reasoning Tuning actually made things worse for the tricky questions.
• The Analogy: Imagine a chef.
  • The Base Chef just knows recipes.
  • The Instruction Chef learns to follow the customer's order perfectly.
  • The Reasoning Chef tries to be a "food critic" and over-analyze every ingredient.
  • The problem? The "Reasoning Chef" gets so obsessed with the word "think" that they forget the actual flavor of the dish. They become so good at spotting the "trick" that they fail the simple dishes.

4. The "Magic Wand" (Vector Steering)

This is the coolest part. The researchers used a technique called Vector Steering.

• The Analogy: Imagine the AI's brain is a giant radio. The researchers found a specific "knob" (a mathematical vector) that controls the AI's understanding of the word "think."
• The Experiment: They turned the knob up and down while the AI answered questions.
  • When they turned the "think" knob up, the AI suddenly got better at the trick questions.
  • When they turned it down, the AI got worse.
• The Conclusion: This proves that the AI isn't necessarily "understanding" the story in a human way. It's just reacting to a specific signal in its brain that says, "Oh, the word 'think' is here! Time to guess the opposite of reality!"

The Big Takeaway

The paper concludes that while AI is getting very good at passing these tests, it might be cheating.

It's not that the AI has a "mind" that understands other people's secrets. Instead, it has learned a pattern:

"If the story mentions 'thinking' or 'beliefs,' the answer is usually the opposite of what actually happened."

The Human Lesson:
Just because an AI gives the right answer doesn't mean it understands the why. It's like a student who memorized the answer key for a specific type of math problem. If you change the numbers slightly, they might fail, even if they got the previous 100 questions right.

In short: AI is getting smarter, but it's still mostly a master of patterns, not a master of human empathy. We need to be careful not to mistake a clever trickster for a truly understanding friend.

Technical Summary

Here is a detailed technical summary of the paper "Traces of Social Competence in Large Language Models".

1. Problem Statement

The False Belief Test (FBT) is the gold standard for assessing Theory of Mind (ToM)—the ability to attribute mental states to others—in both humans and AI. However, evaluating Large Language Models (LLMs) on FBT has been plagued by three critical issues:

• Data Contamination: Uncertainty about whether test scenarios exist in training data, leading to "memorization" rather than reasoning.
• Insufficient Model Details: Proprietary models often lack transparency regarding architecture, size, and training regimes, making it hard to isolate variables.
• Inconsistent Controls: Previous studies often fail to control for superficial heuristics (e.g., recency bias or surface-level pattern matching) that allow models to pass tests without genuine socio-cognitive understanding.

The authors argue that current evaluations cannot distinguish between genuine social reasoning and the exploitation of statistical patterns in training data.

2. Methodology

The study employs a rigorous, systematic approach to evaluate 17 open-weight LLMs across 6 model families (including OLMo 2/3, Llama 3, Gemma 3, Qwen 3, and K2-V2).

• Dataset: The authors use a balanced set of 192 FBT variants derived from Trott et al. (2023) and the EPITOME dataset. The scenarios are verified to be absent from the training data of the tested models (specifically OLMo 2/3) through extensive data leakage analysis.
• Experimental Design (4 Dimensions):
  1. Knowledge State: True Belief (character sees the object move) vs. False Belief (character does not see the move).
  2. Knowledge Cue: Explicit (e.g., "X thinks the object is in...") vs. Implicit (e.g., "X goes to get the object from...").
  3. Location Mentions: Varying the order of mentioning start/end locations to control for primacy/recency biases.
  4. Model Variants: Base, Instruction-tuned (Instruct), and Reasoning-tuned models.
• Evaluation Metric: Instead of generating free text, the authors compute the conditional probability of specific location tokens (Start vs. End) given the context. They apply a normalization factor to handle subword tokenization issues (following Pimentel and Meister, 2024) to ensure the probability reflects the complete word.
• Statistical Analysis:
  • Bayesian Multilevel Logistic Regression: To model the probability of a correct answer based on model size, variant, knowledge state, and cue, including interactions.
  • Vector Steering: Used to isolate causal drivers. The authors extract a "think vector" (the difference in hidden states between explicit and implicit cues) and inject it during inference to observe causal effects on performance.
• Case Study: A granular analysis of OLMo 2 checkpoints throughout pre-training, mid-training, and post-training (SFT, DPO, RLVR) to trace the emergence of these behaviors.

3. Key Contributions

• Systematic Benchmarking: The largest and most controlled evaluation of ToM in LLMs to date, covering 17 models and 192 task variations.
• Identification of the "Crossover Effect": The discovery that the presence of explicit propositional attitude verbs (like "thinks") fundamentally alters model behavior in a way that is opposite to human intuition.
• Causal Isolation via Steering: Demonstrating that the verb "think" acts as a causal driver for specific response patterns, which can be manipulated via vector steering.
• Training Dynamics Analysis: Showing that ToM capabilities do not emerge linearly with scale or training tokens; rather, they are shaped by specific post-training objectives that can degrade robustness.

4. Key Results

A. Model Size and Scaling

• Non-Linear Impact: Increasing model size improves performance on False Belief tasks but harms performance on True Belief tasks.
• Conclusion: Scaling does not strictly improve ToM reasoning. The divergence suggests models are learning specific patterns associated with "False Belief" scenarios that interfere with "True Belief" logic as they get larger.

B. The Crossover Effect (Knowledge State × Cue)

• Explicit Cues ("thinks"): These facilitate False Belief performance but significantly impair True Belief performance.
• Implicit Cues ("goes to"): These impair False Belief performance but facilitate True Belief performance.
• Mechanism: The authors argue that the verb "think" is learned in training data as a marker of epistemic divergence (uncertainty or contradiction). When a model sees "X thinks," it activates a pattern of assuming the character's belief differs from reality. This works for False Belief (where reality differs) but fails for True Belief (where they align).

C. Impact of Post-Training

• Instruction Tuning (SFT/DPO): Partially mitigates the crossover effect, improving overall alignment with human cooperative communication.
• Reasoning Tuning: Amplifies the crossover effect. Reasoning-tuned models become highly sensitive to the explicit "thinks" cue, performing well on explicit False Belief but failing catastrophically on True Belief. This suggests reasoning training specializes models for specific heuristics rather than robust social reasoning.

D. OLMo 2 Case Study

• Emergence: Formal linguistic capabilities (measured by BLIMP) emerge early (0.05% of tokens), while ToM capabilities emerge much later and are unstable.
• Pre-training: The crossover pattern emerges during pre-training, indicating that models acquire stereotypical response patterns tied to mental-state vocabulary before any instruction tuning occurs.
• Post-training: While SFT and DPO help False Belief performance, they degrade True Belief performance, particularly in larger models (32B), likely due to over-reliance on learned heuristics.

E. Vector Steering

• Injecting the "think vector" (derived from the difference between explicit and implicit contexts) causally shifts model predictions.
• Adding the vector to implicit scenarios makes them behave like explicit ones (improving False Belief, hurting True Belief), confirming that the verb "think" is the primary causal driver of the observed behavior.

5. Significance and Implications

• Redefining LLM "Social Intelligence": The paper challenges the notion that high FBT scores equate to genuine Theory of Mind. LLMs appear to rely on statistical associations between specific linguistic markers (like "thinks") and scenario outcomes, rather than simulating a character's mental state.
• Evaluation Rigor: The study highlights the danger of surface-level heuristics in LLM evaluation. Tests must be robust to variations in phrasing (explicit vs. implicit) to avoid false positives.
• Training Trade-offs: Post-training techniques designed to improve reasoning (like Chain-of-Thought or specific RLVR) may inadvertently degrade the model's ability to handle nuanced social contexts by over-specializing it on specific linguistic triggers.
• Future Directions: The authors call for evaluation frameworks that test robustness across conditions (True/False, Explicit/Implicit) rather than aggregate scores, and suggest that true ToM in AI requires moving beyond pattern matching to genuine causal reasoning about mental states.

In summary, the paper demonstrates that while LLMs can pass False Belief tests, they do so through a fragile mechanism driven by the statistical weight of the word "think," which breaks down when the scenario requires understanding that a character's belief aligns with reality.