💬 NLP

Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility

This paper demonstrates that language models possess reliable, linearly separable representations of event plausibility that emerge consistently during training and can effectively model fine-grained human judgments of modal categories, offering new insights into both AI capabilities and human cognition.

Original authors: Michael A. Lepori, Jennifer Hu, Ishita Dasgupta, Roma Patel, Thomas Serre, Ellie Pavlick

Published 2026-02-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Michael A. Lepori, Jennifer Hu, Ishita Dasgupta, Roma Patel, Thomas Serre, Ellie Pavlick

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that has read almost everything on the internet. It can write poetry, answer trivia, and even tell you how to bake a cake. But here's the tricky part: the internet is a messy place. It's full of facts, wild fantasies, impossible magic, and sentences that make absolutely no sense (like "The colorless green ideas sleep furiously").

The big question this paper asks is: Does this robot actually know the difference between reality and nonsense, or is it just guessing based on how words usually sound together?

Here is a simple breakdown of what the researchers found, using some everyday analogies.

1. The Robot's "Internal Compass" (The Discovery)

Previous studies suggested that these robots (Language Models) are terrible at telling the difference between what's possible and what's impossible. They thought the robot was just looking at the "odds" of a sentence appearing in a book. If a sentence sounds weird, the robot thinks, "That's probably nonsense."

But this paper found something cooler. The researchers discovered that inside the robot's "brain" (its hidden layers of code), there are specific directions or compass needles that point specifically toward different types of reality.

The Analogy: Imagine the robot's brain is a giant room filled with invisible strings. Most strings just wiggle when you talk. But the researchers found special strings that vibrate only when you talk about things that are Impossible (like a fish flying) versus things that are Unlikely (like a fish wearing a hat). They call these "Modal Difference Vectors." It's like the robot has a built-in GPS that knows exactly where "Reality," "Fantasy," and "Nonsense" are located, even if it doesn't say it out loud.

2. Growing Up Like a Human Child (The Development)

The researchers watched how these "compass needles" appeared as the robot got bigger and smarter (through more training). They found a pattern that looks exactly like how human children learn:

First: The robot learns to spot the "gibberish." It quickly realizes that "The laptop bought the teacher" makes no sense. (This is like a toddler knowing that a dog can't talk).
Second: It learns to spot the "impossible." It realizes that "Freezing a drink with fire" breaks the laws of physics.
Third: It learns the "unlikely." It understands that "Freezing a drink with snow" is possible but rare.
Finally: It learns the "probable." It knows that "Freezing a drink with ice" is the normal way to do it.

The Takeaway: Just like a human child, the robot learns the big, obvious rules of the world first, and only later learns the subtle, fine-grained details.

3. Reading the Robot's Mind to Understand Ourselves (The Human Connection)

Here is the most fascinating part. The researchers used these "compass needles" to predict how humans would judge these sentences.

They found that the robot's internal map of "Reality vs. Nonsense" matches how real people think.

The Analogy: Imagine you and the robot are both looking at a picture of a cat flying. You might say, "That's impossible, but I can imagine it." The robot's internal compass points to a spot that matches your feeling.
The Surprise: The robot helped the researchers figure out why humans distinguish between "Impossible" and "Inconceivable."
- Impossible: Things that break physics (flying cats). We can still imagine them.
- Inconceivable: Things that break logic (a cat eating a Tuesday). We can't even imagine them.
- The robot's "compass" showed that the ability to imagine a scenario is the key switch that humans use to tell these two apart.

4. Why This Matters

Why should we care if a robot can tell the difference between a fantasy story and a news report?

Safety: If we want robots to be doctors or lawyers, they need to know the difference between a real medical fact and a made-up story. This paper shows they are better at this than we thought, but we need to look at their "internal compass" to be sure.
Understanding Humans: By seeing how the robot organizes these ideas, we are actually learning how our own brains organize reality. It turns out the robot and the human brain might be using similar "maps" to navigate the world.

The Bottom Line

The paper argues that these AI models aren't just parrots repeating what they've heard. They have built a sophisticated, internal understanding of what is real, what is fake, and what is nonsense. They learn this in a very human-like order, and their internal "thoughts" about reality actually mirror how we, as humans, judge the world.

In short: The robot isn't just guessing; it has a map of reality, and it's getting better at reading it every day.

1. Problem Statement

Language Models (LMs) are increasingly used for tasks ranging from factual question-answering to creative writing. A critical capability for these models is the ability to distinguish between different modal categories of sentences:

Probable: Common, real-world events.
Improbable: Possible but rare events.
Impossible: Events violating known laws of nature (physics, biology).
Inconceivable: Events that are semantically nonsensical (violating selectional restrictions, e.g., "The laptop bought the teacher").

Recent studies suggested that LMs struggle to categorize these sentences, relying instead on surface-level features or next-token probabilities, which are unreliable indicators of modality. The authors investigate whether LMs possess internal, linear representations of these modal categories that go beyond simple probability estimates, and whether these representations align with human intuition and development.

2. Methodology

The authors employ techniques from mechanistic interpretability to analyze the hidden states of various LMs (including GPT-2, Llama-3.2, OLMo-2, and Gemma-2 across different scales).

A. Modal Difference Vectors (MDVs)

The core method involves creating Contrastive Activation Additions (CAA) vectors.

Construction: For a pair of minimal sentences $(x_+, x_-)$ representing two different modal categories (e.g., Probable vs. Impossible), the authors extract hidden state representations at a specific layer $l$ . The difference vector is computed as $v = r_+ - r_-$ .
Aggregation: These pairwise difference vectors are averaged across many examples to form a robust Modal Difference Vector for specific category pairs (e.g., Probable-Improbable, Impossible-Inconceivable).
Classification: To classify a new sentence, its hidden representation is projected onto the MDV. The magnitude of the projection determines the predicted category.

B. Experimental Design

The study is divided into four distinct investigations (Studies 1–4):

Study 1 (Existence & Generalization): Train MDVs on the Hu et al. (2025b) dataset and test generalization on Goulding et al. (2024), Vega-Mendoza et al. (2021), and Kauf et al. (2023) datasets. Compare performance against baselines: Sentence Probability, Principal Component Analysis (PCA), and Random Vectors.
Study 2 (Development): Analyze the emergence of MDVs across three dimensions:
- Scale: Parameter count (1B to 13B+).
- Depth: Layer depth within the model.
- Training Steps: Checkpoints during the training of OLMo-2-7B.
Study 3 (Human Behavior Modeling): Use MDVs as features in a logistic regression model to predict human categorization distributions (not just expert labels). The goal is to see if the vector space captures the "fuzzy" boundaries of human intuition.
Study 4 (Interpretability): Correlate projections along MDVs with human ratings of interpretable features (e.g., event likelihood, imageability, grammaticality, emotional valence) to understand what semantic concepts the vectors encode.

3. Key Contributions

Discovery of Linear Modal Representations: The paper provides strong evidence that LMs contain coherent, linear representations (MDVs) that distinguish between modal categories, contradicting prior claims that LMs only rely on unreliable probability estimates.
Superiority over Probability: MDVs consistently outperform sentence probability and other projection-based baselines in classifying modal categories, even on adversarial datasets.
Developmental Hierarchy: The authors map a specific order of emergence for modal distinctions, mirroring human cognitive development:
1. Inconceivable vs. Others (emerges first).
2. Probable vs. Impossible.
3. Probable vs. Improbable.
4. Improbable vs. Impossible (emerges last, requiring the most scale/depth).
Scale Threshold: A qualitative performance gap exists between models with <2B parameters and those with $\ge$ 2B parameters; smaller models fail to form reliable MDVs for fine-grained distinctions.
Human Alignment: MDVs create a feature space that accurately models human categorization behavior, including graded judgments and disagreement on borderline cases.
Semantic Decoding: The vectors map to interpretable human concepts. Specifically, the Impossible-Inconceivable vector correlates strongly with imageability (the ease of imagining a scenario), suggesting imagination is a key differentiator for these categories.

4. Key Results

Classification Accuracy: For models $\ge$ 2B parameters, MDVs achieve high accuracy (often >0.9) in distinguishing modal categories across diverse datasets, significantly outperforming probability-based classifiers.
Emergence Order:
- Training Steps: Models first learn to reject "nonsense" (inconceivable), then learn physical impossibility, and finally learn to distinguish between probable and improbable events.
- Layers: Coarse distinctions appear in earlier layers; fine-grained distinctions require deeper layers.
Human Modeling: Logistic regression models using MDV projections as features achieved higher correlation with human response distributions and lower Mean Squared Error (MSE) compared to baselines. This indicates MDVs capture the nuance of human intuition, not just expert labels.
Feature Correlation:
- Probable-Improbable: Strongly correlates with Subjective Event Likelihood.
- Impossible-Inconceivable: Selectively correlates with Imageability and the presence of physical objects/places. This supports the hypothesis that the ability to imagine a scenario is the cognitive mechanism humans (and LMs) use to distinguish "impossible" (violates physics but imaginable) from "inconceivable" (semantically broken/unimaginable).

5. Significance

World Models: The findings suggest that LMs encode a "world model" that includes causal principles and constraints of reality, rather than just statistical patterns. The ability to distinguish modal categories implies an internal understanding of what is possible, impossible, or nonsensical.
Safety and Reliability: Understanding how LMs represent plausibility is crucial for deploying them in high-stakes environments (e.g., medical or legal advice) where distinguishing fact from fantasy is vital.
Cognitive Science: The study bridges AI and cognitive science by showing that the developmental trajectory of modal understanding in LMs (Inconceivable $\to$ Impossible $\to$ Improbable) mirrors the developmental trajectory observed in human children.
New Hypothesis: The paper proposes a novel hypothesis for human cognition: that imagination (imageability) is the specific cognitive tool used to distinguish between events that are physically impossible but imaginable, and events that are semantically inconceivable.

In summary, this paper demonstrates that large language models do not merely hallucinate or rely on surface statistics; they develop structured, linear internal representations of event plausibility that align with human cognitive structures and intuitive theories of the world.