World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings

This paper demonstrates that static word embeddings derived solely from co-occurrence statistics retain significant spatial and temporal structure, indicating that linear probe recoverability in LLMs does not necessarily prove the existence of internal world models beyond what is already latent in raw text.

The Gist

Here is an explanation of the paper, translated into everyday language with some creative analogies.

The Big Question: Do AI Models "Know" the World?

Imagine you have a super-smart AI (a Large Language Model, or LLM) that has read almost everything on the internet. Recently, researchers found that if you poke around inside the AI's "brain," you can pull out a map. You can ask the AI, "Where is Paris?" and it gives you the exact latitude and longitude. You can ask, "When was Napoleon born?" and it gives you the year.

Because of this, many scientists started saying: "Aha! This AI has built an internal 'World Model.' It understands space and time just like a human does, not just because it memorized words, but because it learned how the world works."

The Plot Twist: The "Magic" Might Be in the Words, Not the Brain

Elan Barenholtz, the author of this paper, says: "Hold on a second. Let's not get too excited yet."

He proposes a simpler explanation: Maybe the AI isn't building a 3D map of the world inside its head. Maybe the "map" was already hidden inside the text it read, and the AI just found it.

To prove this, he didn't use a fancy, modern AI. He used GloVe and Word2Vec. Think of these as the "iPads" of the AI world compared to the "supercomputers" of today. They are simple, old-school models that just count how often words appear next to each other. They don't have "layers" of deep thinking; they are just giant statistical calculators.

The Experiment:
He took these simple, dumb models and tried to pull out the same geographic and temporal data (city locations, birth years) that the fancy AI could.

The Result:
It worked.

• The simple models could predict city locations with about 70–80% accuracy.
• They could guess historical eras with about 50% accuracy.

The Analogy: The "Library of Babel" vs. The "Librarian"

Imagine the internet is a massive library containing every book ever written.

• The Fancy AI (LLM) is like a super-intelligent librarian who has read every book, understands the plot of every story, and can visualize the geography of every fictional world.
• The Simple Model (GloVe/Word2Vec) is like a robot that just counts how many times the word "Paris" appears next to the word "France" or "Eiffel Tower." It doesn't "know" what France is; it just knows the words hang out together.

The paper shows that even the robot can draw a pretty good map. Why? Because in the library, the word "Paris" is surrounded by words like "France," "Europe," "croissants," and "cold winters." The word "Miami" is surrounded by "Florida," "beach," "hot," and "hurricanes."

The robot doesn't need to "understand" geography to know that "Miami" is hot and "Paris" is cold. It just needs to notice that the words describing Miami are different from the words describing Paris. The "map" is hidden in the vocabulary itself.

The Detective Work: Where is the Signal?

To prove this, the researcher played a game of "Whac-A-Mole" with the data.

1. The Temperature Test: He looked at which words made a city seem "hot" or "cold" in the AI's math.

   • Hot cities were linked to words like dengue, cyclone, coconut, tropical, plantation.
   • Cold cities were linked to words like chemist, physicist, violinist, skiing, polar.
   • The AI didn't need a thermometer; it just saw that "tropical" words hang out with "hot" places.

2. The Surgery (Ablation): He took the "brain" of the simple model and surgically removed the parts that dealt with country names and weather words.

   • Result: The model's ability to guess locations crashed. It went from being 70% accurate to barely better than guessing.
   • Conclusion: The "world knowledge" wasn't a magical internal map; it was just a collection of specific words (like "Germany" or "snow") that the model used as clues.

The Takeaway: Don't Confuse "Reading" with "Knowing"

The paper has two main messages:

1. For AI Researchers: Just because you can pull a map out of an AI's brain using a simple math trick (a "linear probe"), it doesn't mean the AI has built a complex, human-like understanding of the world. It might just be very good at spotting patterns in the text. The bar for proving an AI has a "World Model" needs to be much higher.
2. For Everyone Else: This reveals something amazing about language itself. Even without any human teaching, the way we write and speak naturally encodes a compressed map of the world. If you write enough about "tropical places," the words you use will naturally cluster together in a way that creates a map.

In short: The "world" wasn't created by the AI. The world was already written into the text, and even the simplest AI can find it if it knows how to look. The AI didn't learn the world; it just learned the vocabulary of the world.

Technical Summary

Here is a detailed technical summary of the paper "World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings" by Elan Barenholtz.

1. Problem Statement

Recent research (e.g., Gurnee & Tegmark, 2024) has claimed that Large Language Models (LLMs) possess "world-like" internal representations because linear probes can recover spatial (geographic coordinates) and temporal (historical dates) variables from their hidden states. These findings are often interpreted as evidence that LLMs have moved beyond mere linguistic statistics to construct structured "world models."

The Core Question: Does the linear recoverability of world properties from LLMs necessarily imply the emergence of structured internal representations, or is this structure already latent in the raw statistics of text itself?

Hypothesis: The authors propose that static word embeddings, which are direct functions of corpus co-occurrence statistics without contextual processing or layered abstraction, already contain substantial recoverable spatial and temporal structure. If so, linear probe recoverability alone is insufficient evidence to claim that LLMs have achieved a representational leap beyond text.

2. Methodology

A. Models and Data

The study utilizes static word embeddings, which lack the contextual dynamics of LLMs:

• GloVe (6B, 300d): Trained on 6 billion tokens (Wikipedia + Gigaword).
• Word2Vec (Google News, 300d): Trained on ~100 billion tokens using CBOW with negative sampling.
• Rationale: Both models are mathematically derived from co-occurrence matrices (GloVe factorizes log-co-occurrence; Word2Vec implicitly factorizes shifted Pointwise Mutual Information). Any structure found here must derive from text statistics, not explicit grounding.

B. Datasets

1. World Cities ($N=100$): A globally distributed set of cities. Targets included:
   • Recoverable: Latitude, Longitude, Mean Annual Temperature.
   • Negative Controls: Elevation, GDP per capita, Population, Year Founded.
2. Historical Figures ($N=194$): Spanning antiquity to the 20th century. Targets included:
   • Birth Year, Death Year, Midlife Year.

C. Probing Architecture

• Model: Ridge Regression ($\hat{y} = w^\top x + b$) with L2 regularization.
• Protocol: 80/20 train/test split. Hyperparameters ($\lambda$) selected via 5-fold cross-validation.
• Metric: $R^2$ on the held-out test set.
• Design Choice: Linear probes were used deliberately to match the methodology of prior LLM studies, ensuring a fair comparison of "linear decodability."

D. Analysis Techniques

1. Semantic Similarity Analysis: Correlating cosine similarity between city embeddings and all vocabulary words (excluding proper nouns) with actual city temperatures/latitudes to identify data-driven lexical gradients.
2. Subspace Ablation:
   • Identified specific semantic categories (e.g., country names, climate terms).
   • Used PCA to define the subspace spanned by these words.
   • Projected city embeddings onto this subspace and subtracted it (zeroing out those dimensions).
   • Measured the drop in $R^2$ compared to random subspace removal (controls for generic dimensionality loss).

3. Key Results

A. Recovery of Spatial and Temporal Structure

Static embeddings successfully recover significant world properties, challenging the assumption that such structure requires complex modeling:

• Geographic Coordinates: High $R^2$ for Latitude (0.71–0.87) and Longitude (0.66–0.87) in both GloVe and Word2Vec.
• Climate: Moderate-to-strong recovery for Temperature ($R^2 \approx 0.47–0.62$).
• Temporal: Coarse recovery for historical birth years ($R^2 \approx 0.48–0.52$), capturing broad eras (ancient vs. modern) rather than precise dates.
• Selectivity (Negative Controls): Crucially, properties that do not systematically structure co-occurrence patterns (Elevation, GDP, Population) yielded negative or near-zero $R^2$. This proves the probes are not extracting arbitrary world attributes but are sensitive to specific distributional gradients present in the text.

B. Semantic Interpretability

The recovered signals are not opaque; they are tied to interpretable lexical gradients:

• Temperature: Words associated with warmer cities (e.g., "dengue," "cyclone," "coconut") correlate positively with temperature, while words associated with colder cities (e.g., "chemist," "violinist," "skiing") correlate negatively.
• Time: Words like "ancient" and "mythology" correlate with earlier birth years, while "industrial" and "revolution" correlate with later eras.
• Composite Scores: Simple semantic contrasts (e.g., similarity to "cold" minus "warm") alone can predict latitude ($r=0.61$) and temperature ($r=-0.79$).

C. Subspace Ablation Findings

Removing specific semantic subspaces significantly degraded performance, confirming the source of the signal:

• Country Names: Removing the 20-dimensional subspace of country names caused a massive drop in Latitude $R^2$ (from 0.71 to 0.27, $\Delta R^2 = 0.41$) and Temperature $R^2$.
• Climate Terms: Removing climate/weather vocabulary specifically destroyed the temperature signal (dropping $R^2$ from 0.47 to -0.17, worse than a constant predictor).
• Specificity: Random removal of the same number of dimensions had negligible effects, proving the signal is concentrated in specific semantic directions, not uniformly distributed.

4. Key Contributions

1. Demonstration of Latent Structure: Proved that static, co-occurrence-based embeddings preserve substantial, recoverable spatial and temporal structure, revealing an underappreciated capacity of simple distributional models.
2. Semantic Interpretability: Identified that this structure is driven by graded co-occurrence with specific, interpretable vocabularies (country names, climate terms, historical eras).
3. Causal Evidence via Ablation: Showed through targeted subspace ablation that the signal depends on identifiable distributional subspaces, far exceeding random controls.
4. Methodological Critique: Established that linear probe recoverability alone cannot distinguish between "emergent world models" in LLMs and structure already latent in text statistics.

5. Significance and Implications

• Re-evaluating "World Models": The paper argues that the linear recoverability of space and time in LLMs is not definitive proof of structured internal representations. Since the same structure exists in static embeddings (which have no "world model"), the phenomenon may simply be a reflection of the rich relational structure inherent in natural language co-occurrence.
• The Power of Text: The findings suggest that language itself acts as a "dense residue" of the physical and historical world. The "company a word keeps" (Firth, 1957) in text is sufficient to encode a compressed relational map of geography, climate, and history.
• Benchmark for Future Claims: To claim that LLMs have moved beyond text, researchers must demonstrate capabilities (e.g., fine-grained resolution, compositional generalization, or nonlinear structure) that exceed what is recoverable from distributional baselines.
• Limitations Acknowledged: The authors note that LLMs still outperform static embeddings (likely due to context, larger corpora, and higher dimensionality), and that nonlinear structures in LLMs might exist beyond what linear probes on static embeddings can detect. However, the baseline established here raises the bar for proving "emergent" world knowledge.

Conclusion: The paper concludes that while LLMs may indeed build complex representations, the mere ability to linearly decode world properties is insufficient evidence for this. A significant portion of "world knowledge" in AI is likely inherited directly from the statistical regularities of the training text.