💬 NLP

Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence

This paper proposes and validates a distributional theory demonstrating that the hierarchical geometric structure of concepts in language models emerges naturally from the spectral properties of word co-occurrence statistics, rather than requiring specialized functional mechanisms.

Original authors: Andres Nava, Matthieu Wyart

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andres Nava, Matthieu Wyart

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, messy library where every book is a word. In this library, books that talk about similar things tend to sit on the same shelves or appear in the same conversations. This paper argues that the "geometry" (the shape and arrangement) of how computers understand language isn't built by a complex, pre-programmed rulebook telling them how to organize concepts. Instead, the structure emerges naturally, like a tree growing from the soil, simply because words that are related appear together more often.

Here is the breakdown of the paper's findings using simple analogies:

1. The Core Idea: The "Co-occurrence" Garden

Think of language learning like planting a garden.

The Seeds: The seeds are words (like "dog," "cat," "animal").
The Soil: The soil is the text the computer reads (like Wikipedia).
The Rule: If two seeds are planted close together in the soil (meaning the words appear near each other in sentences), they grow roots that tangle together.

The authors propose a simple rule: Words that are "close" in meaning (like "dog" and "puppy") appear together in text much more often than words that are "far" apart (like "dog" and "rock").

2. The Discovery: The "Spectral Splitting" Tree

When you take these tangled roots (the statistical data of how often words appear together) and try to flatten them out to see their shape, something magical happens. The computer doesn't just make a random pile; it builds a hierarchical tree.

Imagine a large, multi-story apartment building:

The First Floor (Broadest Split): The computer's first "thought" separates the entire building into two massive wings: Animals vs. Plants. This is the biggest, most obvious difference.
The Second Floor (Medium Split): Inside the "Animal" wing, the next level of thought splits it into Birds vs. Fish.
The Third Floor (Fine Split): Inside the "Bird" wing, it splits Owls vs. Eagles.
The Attic (Tiny Split): Finally, it separates specific items like Daisies vs. Poppies.

The paper calls this "Hierarchical Splitting Geometry." It's like a set of Russian nesting dolls, where the computer peels back layers from the biggest categories down to the tiniest details, purely based on how often the words hang out together.

3. The "Magic" of Word2vec and LLMs

The researchers tested this on two types of AI:

Word2vec: An older, simpler type of AI that just looks at word statistics.
Gemma: A modern, large language model (LLM) that is much more complex.

The Surprise: They found that both types of AI built this exact same "tree-like" structure.

In the simple AI, this makes sense because it was only looking at word statistics.
In the complex AI (Gemma), people often think the AI is using some special, secret "hierarchy module" to understand that a "dog" is an "animal."
The Paper's Claim: No, it doesn't need a secret module. The complex AI built this tree structure automatically just because it learned from the same word statistics. The "is-a" relationship (hypernymy) is just a shadow cast by the fact that related words appear together frequently.

4. The "Spectral" Map

The paper uses math (eigenvectors and eigenvalues) to prove this. Think of it like shining a flashlight through a stained-glass window:

The light is the word statistics.
The window is the mathematical structure of the AI.
The pattern on the wall is the hierarchy.

The math shows that the "light" (statistics) naturally creates a pattern where the first "beam" of light separates the biggest groups, the next beam separates the medium groups, and so on. The computer doesn't have to be told to do this; the math of the statistics forces it to happen.

Summary

The paper concludes that the beautiful, organized way AI understands the world (knowing that a "poodle" is a "dog," which is an "animal") isn't necessarily because the AI was programmed with a family tree of concepts. Instead, it's a natural side effect of the fact that words that belong to the same family tend to appear in the same sentences.

Just as a river naturally carves a path through a landscape based on the terrain, the AI naturally carves a "concept tree" based on the terrain of word usage. The hierarchy is a statistical echo, not a functional instruction.

Technical Summary: Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence

Problem Statement

Large Language Models (LLMs) and word embeddings exhibit geometric structures where semantic relationships, such as hypernymy (the "is-a" relation), appear to be encoded via specific directions or subspaces. Previous work, notably Park et al. [13], proposed a "linear representation hypothesis" suggesting that hierarchical geometry arises from a functional mechanism where concept vectors are orthogonal to their parents, allowing for clean semantic manipulation (e.g., steering from "bird" to "fish" without losing "animalhood"). However, the origin of this geometry remains debated: is it a result of explicit functional design or an emergent property of the data distribution? This paper investigates whether the hierarchical splitting geometry observed in LLMs and word embeddings is a necessary consequence of word co-occurrence statistics rather than a hierarchy-specific functional mechanism.

Methodology

Theoretical Framework

The authors propose a distributional theory based on the assumption that words closer in the WordNet hypernym graph co-occur more frequently in text.

Co-occurrence Model: They define a normalized co-occurrence matrix $M^\star$ (related to Pointwise Mutual Information) where entries depend on the semantic distance between words. They assume a kernel function $f(\text{dist}(i, j))$ that is positive and strictly decreasing with distance.
Spectral Decomposition: The paper analyzes the eigendecomposition of $M^\star$ $M^{⋆}$ for words forming a perfect binary subtree of WordNet. Using a hierarchy-adapted Haar wavelet basis, they prove that the eigenvectors of $M^\star$ $M^{⋆}$ organize hierarchically:
- The leading eigenvector (scaling mode) varies only with depth.
- Subsequent eigenvectors (wavelet modes) split subtrees from coarse to fine levels.
- The spectrum exhibits a "coarse-to-fine" ordering, where dominant non-scaling modes correspond to higher-level taxonomic splits before finer sub-branches.
Parametric Model: To make quantitative predictions, the authors instantiate the kernel with an exponential decay function $f(d) = \alpha e^{-\beta d}$ , fitting parameters to empirical co-occurrence data.

Empirical Validation

The theory is tested against two types of representations:

Word2vec Embeddings: The authors construct word2vec embeddings directly from the top eigenvectors of the empirical co-occurrence matrix derived from the Wikipedia corpus. They sample perfect binary subtrees of depth $L=3$ from the WordNet hierarchy.
LLM Representations: They analyze the unembedding vectors of the Gemma 2B model (and residual stream activations of Llama 3.2-1B for robustness). The vectors are centered and whitened to reduce global anisotropy.

Metrics:

Top-k Eigenspace Alignment: Since theoretical eigenvectors for degenerate subspaces are defined only up to rotation, the authors compare the alignment of the top- $k$ eigenspaces between the empirical Gram matrices and the theoretical Gram matrices derived from the fitted kernel.
Concept-Vector Orthogonality: They replicate the diagnostic from Park et al. [13], measuring the cosine similarity between parent vectors and the "innovation" vectors (child minus parent) to test for approximate orthogonality.

Key Contributions

Distributional Origin of Hierarchy: The paper provides a simple statistical mechanism demonstrating that hierarchical splitting geometry emerges naturally from pairwise word co-occurrence statistics, without requiring explicit functional constraints or hierarchy-specific training objectives.
Coarse-to-Fine Spectral Organization: The authors prove and verify that the leading eigenvectors of the co-occurrence Gram matrix organize hierarchically, first separating broad taxonomic branches (e.g., plants vs. animals) and then progressively finer sub-branches (e.g., birds vs. fish).
Extension to LLMs: The study demonstrates that this specific spectral signature extends strikingly well to LLM unembeddings (Gemma 2B), suggesting that the hierarchical geometry in LLMs may be a byproduct of the same co-occurrence statistics that drive word2vec.
Reinterpretation of Concept Vectors: By showing that a purely co-occurrence-driven model reproduces the "orthogonal innovation" pattern observed by Park et al., the paper argues that this geometric feature does not necessarily imply a functional mechanism for hierarchical reasoning but can arise from the spectral structure of pairwise statistics.

Results

Theoretical vs. Empirical Alignment: In both word2vec and Gemma 2B unembeddings, the top- $k$ eigenspaces of the empirical Gram matrices align significantly better with the theoretical predictions (based on the fitted exponential kernel) than with shuffled-label baselines. This alignment holds across various semantic hierarchies (e.g., organisms, cognition).
Visual Confirmation: Projections of taxonomy nodes onto principal components (PCs) reveal the predicted hierarchical splitting. For instance, in the organism taxonomy, PC2 separates plants from animals, while degenerate PCs 3–4 split birds from fish and flowers from trees.
Robustness: The results persist across different kernel parameterizations (exponential vs. power-law), different tree depths ( $L=2$ and $L=3$ ), and different representation types (whitened unembeddings, centered unwhitened embeddings, and internal residual stream activations).
Concept Vector Diagnostics: The co-occurrence model reproduces the near-orthogonality of parent-child innovations found in LLMs, confirming that this pattern is consistent with a purely statistical origin.

Significance and Claims

The paper claims to refine the functional picture of hierarchical geometry in representation spaces. While the geometry is real and robust, the authors argue it need not arise from exact orthogonal concept directions driven by a hierarchy-specific functional mechanism. Instead, the approximate orthogonality observed in practice is a consequence of the spectral structure induced by pairwise word statistics.

The work supports a broader view where word embeddings are characterized by discrete, continuous, and hierarchical attributes. Words with similar attributes co-occur more often, and this statistical regularity alone gives rise to the elegant geometric organization of embeddings. While such organization may be useful for function, the paper posits that it is not driven by it. The findings suggest that the hierarchical structure in LLMs is an emergent property of the data distribution rather than a learned functional necessity.