🤖 machine learning

Sky sphere representation in language models

This paper demonstrates that large language models (~100B parameters) possess a decodable, high-dimensional curved manifold representation of the night sky within their residual streams, capable of predicting celestial proximities with significant accuracy and low angular error.

Original authors: Aleksandr Berdnikov, Yevgeny Liokumovich

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aleksandr Berdnikov, Yevgeny Liokumovich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, invisible library where every book is a different thought, and the shelves are arranged not by author or title, but by how the ideas feel to each other. This is the world of mechanistic interpretability, a field where scientists try to peek inside the "brain" of artificial intelligence to see how it organizes information. We know these AI models, called Large Language Models (LLMs), are incredibly good at writing and answering questions, but they are often like black boxes: we see the input and the output, but the middle part is a mystery.

To understand this mystery, researchers use tools like linear probes, which act like metal detectors, scanning the AI's internal signals to see if they are hiding simple, straight-line patterns (like a list of numbers). But sometimes, the patterns aren't straight lines; they are curves, loops, or shapes. Think of it like trying to map the Earth. If you try to draw the whole world on a flat piece of paper, you get a map that stretches and squishes things. But if you draw it on a globe, the distances and relationships are perfect. Scientists have found that AIs sometimes use these "globe-like" shapes to organize concepts, such as arranging the days of the week in a circle so the model can easily calculate that "Friday plus two days" is "Sunday."

The big question is: How deep does this geometric thinking go? Does the AI just memorize facts, or does it build a mental map of the world that looks like a real sphere? This is where a new study steps in to explore the night sky.

The AI's Secret Star Map

In a recent paper titled "Sky sphere representation in language models," researchers Aleksandr Berdnikov and Yevgeny Liokumovich decided to test if massive AI models (specifically those with around 100 billion parameters) have secretly built a 3D map of the night sky inside their brains. They didn't ask the AI to recite coordinates or draw a picture. Instead, they asked it simple, conversational questions like, "What is close to this object in the night sky?" or "Stargazers spot X right beside..."

They fed these questions to seven different open-source AI models, including giants like Mistral Large 2 and Qwen3. As the AI processed these words, the researchers watched the "residual stream"—a high-speed data highway inside the model where information flows from one layer to the next. They were looking for a specific signal: a pattern that looked like a sphere, where stars and constellations were arranged exactly as they are in the real sky.

The Discovery: A Hidden Globe
The results were surprising. In almost every model they tested, the AI did have a representation of the night sky. When the researchers looked at the top layers of the AI's processing, they found that the positions of stars and constellations were arranged on a curved, spherical surface, just like the real celestial sphere.

It wasn't just a faint echo. The researchers could decode this hidden map with impressive accuracy. In most models, they could predict the location of a star or constellation with an error of only 12° to 21° (about the width of your fist held at arm's length). The model explained between 65% and 85% of the variance in the data, meaning the spherical shape was a very strong, dominant feature in the AI's thinking process.

Why This Is Special
Before this, scientists had found that AIs use flat maps (like a 2D chart of the US states) or simple circles (like the days of the week). This is the first time anyone has found a curved, high-dimensional sphere that isn't just a combination of simpler lines. It's as if the AI didn't just memorize a list of star names; it built a mental "globe" to understand how close things are to each other in the sky.

What It Is NOT
The researchers were very careful to rule out other possibilities. They asked: "Is the AI just remembering the text strings 'Right Ascension' and 'Declination' (the coordinates used by astronomers) and arranging them in a flat 2D chart?"

The Flat Map Ruled Out: They tested this by looking for a "discontinuous" line that would exist on a flat map (like the edge of the world on a paper map). They found no evidence of this. The AI wasn't using a flat 2D chart; it was truly using a sphere.
The 3D Distance Ruled Out: They also checked if the AI was mapping the actual distance of stars from Earth (some are light-years away, others are closer). They found that the AI's map didn't care about real-world distance. Instead, the "depth" of the sphere seemed to be influenced by how often the object is mentioned in text or what type of object it is (stars were placed "closer" in the map, while constellations were placed "farther" away).

The "Gpt-Oss" Mystery
There was one oddball in the group: a model called gpt-oss-120b. In this specific model, the star map was much harder to find. The researchers suspect this is because this model was trained on a special "harmony" format (a specific way of formatting conversations) rather than raw text, which might have scrambled the signal. However, when asked directly for star coordinates, this model was still very good at answering, suggesting it knows the facts but just doesn't organize them in a visible sphere in the same way the others do.

Why It Matters
This discovery suggests that when we ask an AI about the night sky, it isn't just searching a database. It is accessing a structured, geometric understanding of proximity. The researchers found that this "star map" becomes most visible when the AI is asked about closeness (e.g., "what is next to X?"). If you ask generic questions like "What is the history of X?", the map fades away and sinks deeper into the noise.

In short, these massive AI models have spontaneously learned to arrange the night sky on a sphere inside their neural networks. It's a beautiful example of how AI can develop its own internal geometry to make sense of the world, creating a curved, 3D map of the stars that is surprisingly accurate, even though it was never explicitly taught to do so.

Technical Summary: Sky Sphere Representation in Language Models

Problem Statement
Mechanistic interpretability research seeks to understand how Large Language Models (LLMs) represent concepts within their high-dimensional activation spaces. While linear methods (e.g., probes, sparse autoencoders) and deep learning-based methods (e.g., natural language autoencoders) have identified various structures, a gap remains in understanding non-linear, curved feature manifolds. Previous discoveries of geometric representations in LLMs have largely been limited to flat structures (e.g., continents, US states, color space) or one-dimensional manifolds (e.g., days of the week, months, sentence length). This paper investigates whether LLMs possess a representation of the night sky that corresponds to the spherical geometry of celestial objects, specifically testing if the positions of stars and constellations are encoded as a curved, high-dimensional manifold in the residual stream.

Methodology
The authors analyzed seven open-source LLMs ranging from 32B to 235B parameters (including Qwen3, Llama-3.3, Mixtral, Mistral Large 2, GLM-4.5-Air, and gpt-oss-120b). The experimental pipeline involved the following steps:

Data Construction: A dataset of 188 astronomical objects was created, comprising 85 bright stars, 88 official constellations, and 15 miscellaneous objects (galaxies, nebulae, clusters).
Prompt Engineering: The objects were fed into the models using 25 distinct prompt templates focused on proximity in the night sky (e.g., "the closest constellation to X is," "stargazers spot X right beside"). Crucially, these prompts avoided explicit coordinate strings (Right Ascension/Declination) to ensure the model was not merely echoing training data coordinates but rather retrieving a conceptual notion of proximity.
Feature Extraction: For each prompt, the residual stream vectors at the final token were extracted across all layers. The top 128 Principal Components (PCs) were computed for each layer.
Decodability Analysis: The authors employed a Leave-One-Out (LOO) validation strategy. They trained a linear regression model to map the top $n$ PCA components (where $n \in \{4, 8, 128\}$ ) of the residual stream to the true 3D unit sphere coordinates of the objects. The held-out object was then predicted, and performance was measured using the $R^2$ score (variance explained) and median angular error.
Alternative Hypothesis Testing: To verify the representation was not a "leak" from a flat 2D map (RA/Dec) or a 3D Euclidean map, the authors performed RDM (Representational Dissimilarity Matrix) analysis. They tested whether the discontinuous dimension inherent in flat maps or the radial distance component of 3D maps were present in the top PCA components.

Key Results

Existence of Spherical Manifold: In six of the seven models tested, a spherical representation of the night sky was successfully decodable from the residual stream. The representation typically emerged in the top 8 principal components.
Performance Metrics: For the successful models, LOO validation yielded $R^2$ scores ranging from 65% to 85%. The median angular error for predicting the position of unseen objects was between $12^\circ$ and $21^\circ$ .
Layer Depth: The representation was weak and dispersed across many components in early layers but strengthened and consolidated into the top $\sim8$ PCs in deeper layers (e.g., layer 66 in Mistral Large 2).
Prompt Dependency: The signal was strongly dependent on prompt phrasing. When using generic prompts (e.g., "the age of X is..."), the representation weakened significantly, failing to appear in the top 8 PCs and decaying in deeper layers.
Exclusion of Alternatives:
- Flat Maps: RDM analysis showed that the discontinuous dimension required for a flat 2D map (Right Ascension) had coefficients near zero or negative in the top-8 PCA space, ruling out a flat map representation.
- 3D Euclidean Maps: No significant correlation was found between the radial component of the representation and the actual physical distance to the stars. Instead, the radial component correlated with object type (constellations placed further, stars closer) and textual frequency.
Model Anomalies: The gpt-oss-120b model showed poor decodability of the spherical manifold. However, the authors noted this model performs well when explicitly asked for coordinates, suggesting the issue lies in the specific representation structure rather than a lack of knowledge.

Significance and Claims
The paper claims to present the first known example of a curved, high-dimensional, irreducible feature manifold in language models. Unlike previous findings of flat or 1D manifolds, this representation captures the intrinsic spherical geometry of the night sky without relying on explicit coordinate strings in the input.

The authors emphasize that this representation is not a trivial echo of training data coordinates but likely reflects a learned concept of "proximity" for night sky objects. They position this work as a middle ground in interpretability tools: capturing non-linear structures while remaining explicit and human-inspectable via linear probes on principal components. The study leaves open questions regarding the mechanism of construction (memorization vs. reasoning) and the functional role of this representation in the model's reasoning process, noting that the models' ability to answer coordinate questions directly far exceeds the accuracy of the decodable manifold.

The AI's Secret Star Map

Technical Summary: Sky Sphere Representation in Language Models

More like this