💬 NLP

Do BERT Embeddings Encode Narrative Dimensions? A Token-Level Probing Analysis of Time, Space, Causality, and Character in Fiction

This study demonstrates that BERT embeddings encode meaningful token-level information regarding time, space, causality, and character in fiction, as evidenced by a high-accuracy linear probe that significantly outperforms random baselines, though the dimensions are not discretely separable and suffer from "boundary leakage" where rare categories are often misclassified as "others."

Original authors: Beicheng Bei, Hannah Hyesun Chun, Chen Guo, Arwa Saghiri

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Beicheng Bei, Hannah Hyesun Chun, Chen Guo, Arwa Saghiri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to read a novel, like Pride and Prejudice. You want to know if the robot is just memorizing words, or if it actually understands the story behind them. Does it know who the characters are? Does it understand when things happen (time), where they happen (space), why things happen (causality), and who is doing what?

This paper is like a "pop quiz" for a famous AI brain called BERT. The researchers wanted to see if BERT has secretly learned the "skeleton" of a story just by reading millions of books, even though it was never explicitly taught to do so.

Here is the story of their experiment, broken down into simple parts:

1. The Setup: Building a Story Map

The researchers took the first five chapters of Pride and Prejudice and broke them down word-by-word (or "token-by-token"). They asked a smart AI assistant to help them label every single word with one of four story categories:

Character: Who is involved? (e.g., "Mr. Darcy," "she")
Time: When is this happening? (e.g., "yesterday," "now")
Space: Where is this happening? (e.g., "in the garden," "London")
Causality: Why is this happening? (e.g., "because," "therefore")
Others: Everything else (verbs, adjectives, random words).

The Challenge: The dataset was very unbalanced. It was like a jar of M&Ms where 70% were yellow ("Others"), and only a tiny, tiny handful were the rare colors like "Causality."

2. The Test: The "Linear Probe"

To see if BERT understood these categories, the researchers didn't retrain the AI. Instead, they used a Linear Probe.

The Analogy: Imagine BERT is a giant library where every book is stored in a specific spot based on its meaning. The researchers didn't want to rearrange the library; they just wanted to see if they could draw a simple line on a map to separate the "Time" books from the "Space" books.

They fed BERT's internal "thoughts" (embeddings) into a simple classifier.
The Result: The classifier got 94% accuracy.
The Control: They tried the same test with random, meaningless numbers (like static on a TV). That classifier only got 47% (basically guessing).

The Takeaway: This proves that BERT does have a hidden map of story elements. It's not just guessing; it actually "knows" the difference between a character and a time marker.

3. The Glitch: "Boundary Leakage"

While the test was mostly successful, the researchers found a funny flaw. When the AI got confused, it didn't mix up "Time" with "Space." Instead, it almost always dumped the confusing words into the "Others" pile.

The Analogy: Imagine a bouncer at a club. If someone looks a little bit like a VIP (Character) or a little bit like a guest with a ticket (Time), the bouncer usually lets them in. But if someone is wearing a weird hat and holding a sandwich (a rare "Causality" word), the bouncer just says, "I don't know what you are, go stand in the 'Other' line."

Rare story elements (like "because" or "therefore") were often mislabeled as "just regular words" because the AI wasn't 100% sure.

4. The Shape of the Data: A Messy Cloud

The researchers tried to visualize how BERT organizes these story elements. They used a technique called Clustering (grouping similar things together).

The Analogy: If you threw all the "Character" words and "Time" words into a giant 3D fog, you might expect them to form two distinct, separate islands.

What they found: Instead of islands, it was more like a swirly, mixed-up smoothie. The "Character" words were everywhere, and the "Time" words were mixed right in with them.
The AI didn't keep these categories in neat, separate boxes. They overlapped heavily. This suggests that understanding a story isn't about having separate drawers for "Time" and "Space," but rather a complex, blended understanding where everything connects.

5. Why Was It Hard?

The researchers realized two main things made the job tricky:

One Word vs. Many: Some story ideas are just one word (like "he"), but others are whole phrases (like "in the middle of the night"). The AI struggled more with the long phrases because it had to decide which part of the phrase was the "Time" and which was just "space filler."
Old English: The book is from 1813. The way people spoke then was different from today, which sometimes confused the AI's patterns.

The Bottom Line

Did BERT learn the story? Yes.
The study proves that when you train an AI on enough text, it naturally picks up the "skeleton" of a narrative. It knows who the characters are, when things happen, and why they happen.

However, it doesn't store this knowledge in neat, separate folders. It stores it in a complex, overlapping web. The AI is great at spotting the obvious stuff (like names), but it still gets a little fuzzy on the rare, tricky connections (like "because") unless you give it a lot of help.

Future Plans: The researchers want to test this on more modern books, look deeper into the AI's "layers" (like peeling an onion to see how deep the understanding goes), and try to teach the AI to handle those tricky, multi-word phrases better.

1. Problem Statement

Narrative comprehension relies on multidimensional semantic structures (time, space, causality, and character) that organize events across sentences and entire texts. While previous research has established that models like BERT encode rich syntactic information, it remains unclear whether their internal representations capture high-level discourse-level narrative structures.

The core research question is: Do BERT embeddings encode distinct, separable representations for fundamental narrative dimensions (time, space, causality, and character) at the token level?

2. Methodology

A. Dataset Construction

Source Material: The first five chapters of Jane Austen's Pride and Prejudice (1813).
Annotation Strategy: A hybrid Human-in-the-Loop approach.
- LLM Assistance: ChatGPT was used for initial token-level annotation based on an iteratively refined codebook.
- Human Verification: All 5,088 annotated tokens were manually reviewed and corrected to ensure accuracy.
Label Categories:
1. Character: Agents/actors (e.g., "Mr. Bennet," "he").
2. Time: Temporal markers (e.g., "yesterday," "now").
3. Space: Setting/locations (e.g., "in the house").
4. Causality: Logical/eventual relations (e.g., "because," "therefore").
5. Others: Tokens not fitting the above categories.
Class Imbalance: The dataset is severely imbalanced. "Others" constitutes 70.1% of tokens, while "Causality" is the rarest at 0.5% (only 25 tokens in the test set).

B. Embedding Extraction & Alignment

Model: bert-base-uncased (12 layers, 768-dimensional hidden states).
Alignment: Since BERT uses WordPiece tokenization (splitting words into subwords), the authors aligned annotated tokens to BERT subwords. For multi-subword tokens, the embeddings were averaged to produce a single 768-dimensional vector per annotated token.

C. Probing Experiments

Linear Probe: A Logistic Regression classifier (scikit-learn) trained on frozen BERT embeddings to predict the narrative label.
Handling Imbalance: The authors applied balanced class weighting (class_weight='balanced') to the loss function. This assigns higher penalties to misclassifying rare classes (e.g., Causality weight $\approx$ 41.89 vs. Others weight $\approx$ 0.28).
Control Experiment: To rule out artifacts of high-dimensional spaces, a Control Probe was trained on random embeddings sampled from a normal distribution matching the variance of real BERT embeddings.
Unsupervised Analysis: K-Means clustering (k=12) and dimensionality reduction (UMAP, PCA, Isomap) were used to visualize the intrinsic structure of the embeddings.

3. Key Results

A. Classification Performance

Real Probe: Achieved 94% accuracy on the test set.
- Majority Classes: "Character" (F1=0.97) and "Others" (F1=0.96) were classified with high reliability.
- Rare Classes: Despite severe scarcity, the probe achieved meaningful performance on rare categories with balanced weighting:
  - Time: F1 = 0.77 (Recall = 0.80)
  - Causality: F1 = 0.60 (Recall = 0.75)
  - Space: F1 = 0.59 (Recall = 0.66)
- Macro-Average Recall: 0.83, indicating the model successfully identifies positive instances across all categories.
Control Probe: Achieved 47% accuracy (near random chance for a 5-class problem), confirming that the real probe's success is due to meaningful information in BERT's representations, not data artifacts.

B. Error Analysis: "Boundary Leakage"

Confusion matrix analysis revealed a specific failure mode termed "Boundary Leakage."
Rare dimensions (Time, Space, Causality) were not confused with each other (e.g., Time was not mistaken for Space). Instead, they were systematically misclassified as the majority class "Others."
This suggests that when contextual cues are weak or ambiguous, the model defaults to the majority class, even with class weighting.

C. Clustering and Structure

Unsupervised Clustering: K-Means clustering in the 768-dimensional space yielded a very low Adjusted Rand Index (ARI = 0.081) when compared to ground truth labels.
Interpretation: Narrative dimensions do not form discrete, separable clusters in BERT's latent space. Instead, they occupy highly overlapping, non-spherical regions. The "Character" class dominated the cluster centroids due to its frequency, fragmenting the semantic space.

D. Syntactic Correlations

POS Distribution: Narrative dimensions correlate strongly with Part-of-Speech (POS) tags (e.g., Character $\to$ Pronouns/Nouns; Space $\to$ Adpositions).
Token Span: Space and Time often involve multi-word spans (e.g., "in the house"), whereas Causality and Character are predominantly single-word. This complexity contributes to higher misclassification rates for Space and Time due to alignment challenges.

4. Key Contributions

Empirical Evidence: Demonstrates that BERT embeddings encode meaningful narrative information (Time, Space, Causality, Character) at the token level, significantly outperforming random baselines.
Methodological Rigor: Implements a robust control probe (variance-matched random embeddings) to validate that narrative signals are not artifacts of the embedding space geometry.
Dataset Creation: Provides a token-level annotated dataset for narrative dimensions using a hybrid LLM-human workflow, addressing the scarcity of such resources.
Insight into Narrative Representation: Reveals that while narrative dimensions are encoded, they are fluid and intermixed rather than forming discrete clusters, challenging the assumption that semantic categories are linearly separable in transformer embeddings.

5. Significance and Future Directions

Significance: This study bridges the gap between low-level linguistic probing and high-level narrative understanding. It confirms that pre-trained models possess the latent capacity to track narrative structure, even without explicit training on narrative tasks.
Limitations:
- Annotation Noise: Polysemy and tokenization inconsistencies (e.g., multi-word phrases) introduce label ambiguity.
- Domain Shift: The 1813 text style differs from modern usage.
- Confounding Factors: The strong correlation between POS tags and narrative labels suggests the probe might be leveraging syntactic cues rather than pure semantic understanding.
Future Work:
- Layer-wise Probing: Investigating which specific BERT layers encode narrative features.
- POS Baselines: Training a classifier on POS tags alone to disentangle syntactic patterns from semantic encoding.
- Expanded Datasets: Incorporating modern texts and complete story arcs to better capture causality and character development.
- Non-linear Classifiers: Moving beyond linear probes to capture the anisotropic and overlapping nature of narrative semantics.