A Hyperbolicity Atlas of Large Language Model Hidden States
This paper presents the first systematic study demonstrating that Gromov hyperbolicity in LLM hidden states is primarily driven by layer depth—peaking in middle layers and becoming more tree-like in final layers—rather than model scale, offering a practical diagnostic for understanding hierarchical distance structures across different model families and input domains.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Inside the computers that power modern artificial intelligence, there is a hidden world of numbers. When these machines read a sentence or process a question, they do not store the words as text. Instead, they translate every word into a long list of numbers, a vector, that exists in a vast, multi-dimensional space. For years, scientists have known that the things these machines process—like the structure of a family tree, the steps in a code program, or the logic of a story—are often hierarchical. They have a top, a bottom, and many branches in between. A natural question arose: do the internal number-lists that the machine creates to understand these things also arrange themselves in a tree-like shape? If the machine's internal map looks like a tree, it might mean the computer is organizing information in a way that mirrors the complex structures of human language and logic. To answer this, researchers turned to a mathematical concept called hyperbolicity, which acts like a ruler for measuring how "tree-like" a collection of points is. A low score means the points are arranged in a neat, branching hierarchy, while a high score means they are scattered in a more chaotic, flat way.
A team of researchers from Beijing and Guangzhou set out to map this hidden geometry across the most advanced language models available today. They did not build a new machine or change how the existing ones worked. Instead, they acted as cartographers, taking ten different open-source models and sending them through four very different types of tasks: solving math problems, writing computer code, answering questions about common sense, and judging the truthfulness of statements. For every single word in the input prompts, they measured the distance between the number-lists the machine created. They did this for over 800,000 specific measurements, covering ten different models of varying sizes and every single layer of processing inside each machine. Their goal was to create an atlas, a detailed map showing exactly where the internal structure of these machines looks like a tree and where it does not.
The most striking discovery was that the size of the machine mattered far less than where you looked inside it. Many people assume that a larger model with more parameters is automatically "smarter" or more structured. The researchers found this to be largely untrue. Simply making a model bigger did not consistently make its internal structure more tree-like. In some cases, a larger model was actually less organized than a smaller one, and in others, it was more organized. The true pattern emerged when they looked at the depth of the machine's processing. As the information moved from the first layer of the network to the middle, the structure became messy and flat, losing its tree-like shape. The numbers in the middle layers were crowded and complex, reflecting a mix of many different types of information. However, as the data reached the final layers, just before the machine produced an answer, the structure changed dramatically. The numbers snapped back into a much clearer, more tree-like arrangement. This suggests that the machine starts with a chaotic mix of possibilities and then compresses them into a structured form right at the end.
The researchers also found that the specific type of model and the task it was trained for changed the map significantly. Two models of the exact same size could have completely different internal geometries depending on their training. For instance, a model specialized in writing code showed a very strong tree-like structure when processing programming tasks, but a model trained for general math did not show the same pattern, even when asked to solve math problems. In fact, the specialized code model made the structure of code prompts look more like a tree than the general model did, while the math-specialized model kept its structure more complex. This means that the "shape" of the machine's thinking is not a fixed property of its size, but a flexible result of how it was trained and what it is currently doing.
Ultimately, this study provides a new way to understand how artificial intelligence organizes information. It shows that the internal world of these models is not a static, uniform landscape. Instead, it is a dynamic journey where information starts in a complex, flat state, wanders through a crowded middle section, and finally settles into a structured, tree-like form at the very end. The researchers conclude that looking only at the final answer or the total size of a model is not enough to understand its inner workings. To truly see how these machines think, one must look at the path the information takes, noting that the most important structural changes happen in the final moments before a decision is made. This map offers a practical tool for scientists to see where these models are organizing their thoughts and where they are still struggling, revealing that the geometry of intelligence is far more nuanced than a simple measure of size could ever show.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.