💻 computer science

Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory

This paper introduces the Transformer Geometry Observatory (TGO) framework, specifically TGO-I, to analyze the spectral geometry of Vision Transformers and reveals that training progressively redistributes variance across representational dimensions, leading to increased effective dimensionality and reduced anisotropy contrary to the intuition of information concentration.

Original authors: Kaustubh Kapil, Kishor P. Upla

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaustubh Kapil, Kishor P. Upla

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, complex machine called a Vision Transformer (ViT). This machine is designed to look at pictures and understand what's in them. For a long time, scientists have been obsessed with what the machine decides (e.g., "That's a cat!") or how it pays attention (e.g., "It's looking at the ears").

But this paper, TGO-I, asks a different question: "What does the machine's internal 'brain' actually look like as it learns?"

Think of the machine's brain not as a list of facts, but as a giant, multi-dimensional room where information lives. The authors built a special "observatory" (a set of tools) to watch how the shape of this room changes as the machine trains.

Here is the story of what they found, explained simply:

1. The Setup: Watching the Room Evolve

The researchers trained a Vision Transformer on a dataset of 100 types of images (like a mini version of the whole internet). They didn't just watch the final answer; they peeked inside the machine at every step of the training process (from day 1 to day 100).

They measured the "geometry" of the data. Imagine the data as a cloud of points in a room.

Early on: The cloud might be squashed flat against one wall (very narrow).
Later on: Does it stay flat? Does it spread out? Does it fill the whole room?

2. The Big Surprise: The "Spreading Out" Effect

Common sense might suggest that as a machine gets smarter, it should get more focused. You might think, "It should learn the most important things and ignore the rest, squeezing all its knowledge into a few tight, powerful directions."

The paper found the exact opposite.

Instead of squeezing into a tight corner, the machine's internal representation spread out to fill the entire room.

The Analogy: Imagine a group of people in a dark room. At the start, everyone is huddled in a tight circle, shouting the same thing. As the "training" happens, they don't huddle tighter. Instead, they slowly walk apart, spreading out to fill the whole room, each holding a unique piece of information.
The Result: The "Effective Rank" (a fancy way of saying "how many different directions are being used") went up. The "Anisotropy" (how lopsided or squashed the shape is) went down.

3. The "CLS Token": The Ultimate Organizer

In these machines, there is a special token called the CLS token. Think of this as the "Class President" or the "Team Captain" that gathers all the information from the rest of the group to make the final decision.

The paper found that this "Captain" became the most interesting part of the room.

By the end of training, the CLS token's internal space was the most spread out of all.
It had the least squashed shape (lowest anisotropy).
It was using the most dimensions to hold information.

It's as if the Team Captain didn't just memorize a few key facts; they organized the whole team so that every single member's unique perspective was preserved and utilized.

4. The "Diagonal" Pattern

The researchers also looked at how different parts of the machine's brain talked to each other (correlations).

Early on: The parts were very tangled and dependent on each other (like a knot of headphones).
Later on: The "knot" untangled. The parts became more independent.
The Visual: If you drew a map of these connections, it started looking like a clean diagonal line (where things only talk to themselves) rather than a messy web of cross-connections. This means the machine learned to keep its different features distinct and non-redundant.

5. Why Did This Happen? (The Theories)

The paper doesn't say exactly why this happened, but they offer three guesses (hypotheses):

Token Diversification: The machine learned to make every piece of the image (every "token") unique and distinct, so they all needed their own space.
Semantic Expansion: The machine discovered more and more different "meanings" or features, so it needed more room to store them all.
Redundancy Reduction: The machine stopped repeating itself. Instead of having three features that say the same thing, it rearranged them so every feature said something new and useful.

The Takeaway

The main point of TGO-I is that we now have a new way to "watch" AI learning. We used to think learning meant getting more focused and narrow. This paper shows that, at least for Vision Transformers, learning looks more like expanding and distributing information across a wide, flat, and efficient landscape.

The authors call this the Transformer Geometry Observatory. They are just starting; this is the first "installment" (TGO-I) that looked at the shape of the data. They plan to build more observatories in the future to watch how the "Team Captain" moves, how the "Attention" works, and how the machine optimizes its path.

In short: The machine didn't get smarter by narrowing its focus; it got smarter by learning to use its entire brain, spreading its knowledge out evenly so nothing is wasted.

Technical Summary: TGO-I: Spectral Geometry Observatory

Problem Statement

Despite the widespread adoption of Vision Transformers (ViT) in computer vision, the fundamental understanding of their representational geometry and how it evolves during training remains under-explored compared to application-level adaptations. Existing literature often limits analysis to specific layers, isolated training stages, or individual observables, leaving a gap in understanding how representation geometry jointly evolves across network depth and training time. Specifically, questions regarding how dimensional utilization changes, how variance is distributed across representational directions, and how global representations emerge within transformer architectures lack systematic empirical investigation.

Methodology

To address these gaps, the authors introduce the Transformer Geometry Observatory (TGO), a modular framework designed to systematically instrument and analyze the internal geometry of transformer representations. The first installment, TGO-I, focuses specifically on Spectral Geometry.

The experimental setup involves the following components:

Model & Dataset: A ViT-Small/16 model trained on the ImageNet-100 dataset for 100 epochs.
Observation Protocol: Activations are extracted via forward hooks from the Patch Embedding, Positional Embedding, all 12 Transformer blocks, and the final CLS token representation.
Data Consistency: To ensure observational consistency, all spectral measurements are performed on a fixed analysis subset of 1,000 validation images, rather than dynamic training mini-batches. This results in an aggregated representation matrix of size $197,000 \times 384$ (where $B=1000$ images, $T=197$ tokens, $D=384$ dimensions).
Spectral Analysis: The framework computes the feature covariance matrix ( $C_l$ $C_{l}$ ) for each layer and performs eigendecomposition. It then derives a suite of spectral observables to characterize the geometry:
- Effective Rank ( $r_{eff}$ ): The exponential of the entropy of the normalized eigenvalue distribution.
- Stable Rank ( $r_{stable}$ ): The ratio of total spectral energy to the dominant spectral direction.
- Participation Ratio (PR): An estimate of the effective number of dimensions contributing to the spectrum.
- Spectral Entropy: Quantifies the uniformity of the eigenvalue distribution.
- Spectral Anisotropy: Measures the dominance of the principal spectral direction relative to total variance.
- Visualizations: Direct analysis of covariance matrices, eigenspectra, singular value spectra, and spectral decay curves.

Key Results

The study reveals a consistent pattern of progressive spectral redistribution across the network as training progresses:

Increased Dimensional Utilization: There is a consistent increase in Effective Rank, Stable Rank, and Participation Ratio across nearly all monitored layers. This indicates that variance is being distributed across a larger number of spectral directions rather than being concentrated in a few.
Decreased Anisotropy: Spectral Anisotropy decreases significantly, particularly in deeper layers. This suggests a reduction in the dominance of any single spectral direction.
Increased Spectral Entropy: Spectral Entropy rises, indicating that the eigenvalue distribution becomes more uniform and less concentrated over time.
Covariance Structure Evolution: Covariance matrices become progressively more diagonal. Off-diagonal correlations weaken, and the final CLS representation exhibits largely decorrelated feature dimensions with variance concentrated along the diagonal.
Layer-Specific Dynamics:
- Patch and Positional Embeddings: Show relative stability in rank and geometry throughout training.
- Transformer Blocks: Exhibit progressive increases in effective dimensionality and decreases in anisotropy.
- CLS Token: The final CLS representation displays the most pronounced effects, achieving the highest effective dimensionality, highest spectral entropy, and lowest anisotropy in the network.
Spectral Flattening: Eigenspectra and singular value spectra become progressively flatter, with the dominance of leading eigenvalues decreasing relative to the rest of the spectrum.

Significance and Claims

The paper positions TGO-I not as a causal study explaining why these geometric shifts occur, but as a foundational observational framework that establishes reproducible measurements of transformer learning dynamics.

Empirical Contribution: The work provides empirical evidence contradicting the common intuition that training concentrates information into a small number of dominant directions. Instead, ViT representations become progressively less spectrally concentrated and increasingly distributed.
Framework Utility: TGO-I establishes a systematic pipeline for tracking geometric evolution, serving as a baseline for future investigations.
Hypothesis Generation: While the paper does not definitively prove the underlying mechanisms, the observed data motivates three plausible hypotheses for future investigation:
1. Token Diversification: Tokens become increasingly distinguishable, occupying more spectral directions.
2. Semantic Expansion: The model discovers and encodes a larger number of semantically meaningful feature directions.
3. Redundancy Reduction: Training reorganizes existing information into a more efficient, less redundant basis, reducing statistical dependence between features.

The authors conclude that distinguishing between these hypotheses requires future observatories (TGO-II through TGO-V) capable of analyzing token dynamics, representation similarity, attention geometry, and optimization landscapes. The primary value of TGO-I lies in its ability to generate systematic, reproducible measurements that identify unexpected phenomena and motivate mechanistic inquiries into transformer learning.