The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers

Inside the digital minds of modern language models, words do not sit still. When a computer reads a sentence, it transforms each word into a long list of numbers, a point floating in a vast, invisible space with hundreds of dimensions. These points are not scattered randomly; they cluster together, forming thin, curved sheets within that high-dimensional void. Scientists call these sheets manifolds, and the thickness of the sheet tells a story about how much information the word is carrying. If a word is free to change in many directions, the sheet is thick and complex. If the word is tightly constrained by its neighbors, the sheet becomes thin and simple. This compression of information is a fundamental way these models understand the world, but until now, researchers have struggled to see how the specific job a word performs in a sentence—whether it is a heavy noun like "cat" or a light connector like "the"—shapes this geometry.

A team of researchers set out to map this hidden landscape, asking whether the grammatical role of a word dictates the path it takes through the layers of a neural network. They focused on the distinction between open-class words, which carry rich meaning like nouns and verbs, and closed-class words, which serve a structural purpose like prepositions and pronouns. By feeding thousands of sentences into four different types of language models, they tracked how the shape of the data changed layer by layer. They found that these two types of words do not travel the same route. Closed-class words expand into a wider, more complex space early in the process, only to collapse back down into a tight, simple shape much sooner than their content-heavy counterparts. This suggests that the model first stretches out these structural words to figure out how they connect to other parts of the sentence, and once that connection is made, it folds them back up, having extracted all the necessary relational information.

The researchers discovered that this geometric folding is not just a random fluctuation but a precise reorganization of the neighborhood around each word. As a word moves deeper into the network, its closest neighbors in the mathematical space change. For structural words, this change happens quickly and decisively. A word like "the" might start its journey surrounded by other determiners, but as the network processes the sentence, it detaches from that group and moves closer to the specific noun it modifies. This shift in neighbors corresponds exactly to the moment the word's internal representation becomes simpler and more compressed. In contrast, content words like "elephant" or "run" maintain a more complex, open geometry for longer, reflecting the ongoing need to integrate their specific meanings with the evolving context of the sentence.

The study also revealed that the architecture of the model matters deeply. Models that read sentences in both directions at once, like encoders, resolve these structural connections early, causing the geometric changes to happen in the middle layers. Models that read strictly from left to right, like decoders, take a different path, with these geometric shifts occurring later, as the context accumulates. Despite these architectural differences, the core finding held true: the path a word takes through the network is a direct reflection of its grammatical function. The researchers proved this by showing that a computer could look only at the shape of the data points—their thickness and how their neighbors shifted—and correctly guess whether a word was a function word or a content word, or even identify its specific part of speech, without ever seeing the word itself.

This work suggests that the grammar of a sentence is not just a set of rules the model follows, but a physical reality within its mathematical structure. The model does not simply store the meaning of a word; it physically reshapes the space around that word to solve the puzzle of how it fits into the sentence. The moment a structural word finds its partner in the sentence, the model collapses the extra dimensions it needed to explore, leaving behind a compact, efficient representation. It is as if the model builds a temporary scaffold to hold the sentence together, and once the structure is secure, it removes the scaffolding, leaving only the essential shape of the meaning. This geometric dance of expansion and contraction provides a new window into how artificial intelligence understands the delicate balance between the words that carry meaning and the words that hold it all together.