💬 NLP

A Gauge Theory of Superposition: Toward a Sheaf-Theoretic Atlas of Neural Representations

This paper proposes a sheaf-theoretic gauge framework for interpreting superposition in large language models by replacing global dictionaries with local semantic charts, thereby quantifying three specific obstructions to interpretability—local jamming, proxy shearing, and nontrivial holonomy—and validating these theoretical bounds through empirical experiments on Llama-3.2-3B.

Original authors: Hossein Javidnia

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hossein Javidnia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a massive, super-smart AI (like the Llama model mentioned in the paper) thinks.

For a long time, researchers believed the AI had a single, giant dictionary in its brain. They thought that if you looked at a specific word or concept, it would always be represented by the same set of "neurons" firing in the same way, no matter what the AI was talking about. It was like having one universal translator that worked perfectly for every language and every context.

The Big Problem:
This paper argues that this "single dictionary" idea is wrong. The AI's brain is actually more like a patchwork quilt or a travel guide with many local maps.

Here is the breakdown of the paper's ideas using simple analogies:

1. The "Atlas" vs. The "Single Map"

Imagine you are a tourist in a huge, complex city.

The Old View: You have one giant map of the whole city. You assume "Main Street" looks the same whether you are in the north or the south.
The New View (This Paper): The city is too big for one map. Instead, you have a collection of local maps (called "charts").
- In the business district, "Main Street" means skyscrapers and suits.
- In the art district, "Main Street" means galleries and coffee shops.
- In the industrial zone, "Main Street" means factories and trucks.

The AI doesn't have one global definition for a concept; it has local definitions that change depending on the context (the "district" it is currently in).

2. The Three Ways the "Maps" Clash

The paper introduces a fancy mathematical framework (Gauge Theory and Sheaves) to measure exactly how these local maps fail to fit together into one perfect global picture. They found three specific ways the AI gets confused when trying to stitch these local views together:

Obstacle 1: Local Jamming (The Traffic Jam)

The Analogy: Imagine a small parking lot (the AI's local memory) that is trying to park 100 cars (features) but only has space for 50.
What happens: The cars have to squeeze in. They start bumping into each other. The AI is trying to pack too many ideas into too little space.
The Result: The features get "jammed." The AI can't clearly distinguish between them anymore because they are overlapping too much. This is Local Jamming.

Obstacle 2: Proxy Shearing (The Mismatched Puzzle Pieces)

The Analogy: Imagine you are moving from the Business District to the Art District. You have a "local map" for each.
- On the Business map, "Main Street" points North.
- On the Art map, "Main Street" points East.
- You try to glue the two maps together, but the streets don't line up. The "North" of one map is the "East" of the other.
What happens: The AI tries to translate a concept from one context to another, but the translation is "sheared" (twisted). The meaning gets distorted because the local rules don't match the global rules. This is Proxy Shearing.

Obstacle 3: Nontrivial Holonomy (The Lost Compass)

The Analogy: Imagine you take a walk in a loop. You start at a park, walk through the Business District, then the Art District, then the Industrial zone, and finally back to the park.
- In a perfect world, when you return to the park, your compass should point the exact same way it did when you left.
- In this AI, when you walk that loop, your compass ends up pointing in a different direction.
What happens: The meaning of a concept changes depending on the path you took to get there. If you approach an idea via a coding context, it means one thing. If you approach it via a poetry context, it means something else. When you try to bring them back to the same spot, they don't match. This path-dependency is called Holonomy.

3. Why Does This Matter?

The authors built a toolkit to measure these three problems. They didn't just guess; they tested it on a real AI model (Llama 3.2).

They proved the "Single Dictionary" is broken: They showed mathematically that you cannot force the AI to have one consistent global dictionary without it breaking down.
They found the "Glue" is weak: They measured exactly how much the local maps clash (Shearing) and how much the AI gets confused by its own loops (Holonomy).
They made it reproducible: They showed that these aren't just random glitches; they are structural features of how the AI learns.

The Takeaway

Think of the AI not as a library with one perfect catalog, but as a team of local guides who all know their own neighborhood perfectly but don't always agree on how to describe the city as a whole.

This paper gives us a new way to look at AI: instead of trying to force it to have one single "truth," we should accept that its understanding is local, patchy, and context-dependent. By measuring the "glitches" where the patches don't fit (the jamming, the shearing, and the compass spinning), we can finally understand why the AI sometimes makes mistakes or behaves strangely in different situations.

In short: The AI's brain is a patchwork of local realities, and this paper provides the ruler to measure the seams where they don't quite line up.

1. Problem Statement

Current mechanistic interpretability in Large Language Models (LLMs) often relies on the assumption that a single, globally consistent feature dictionary can explain a model's behavior across all contexts. However, evidence from superposition (where models encode more features than their ambient dimension) suggests this premise fails. Representations are often context-dependent, leading to interference and feature identities that shift based on the input.

The paper argues that interpretability should not be viewed as finding a single global dictionary, but rather as a local-to-global construction problem. The core challenge is quantifying the obstructions that prevent local feature descriptions (valid in specific contexts) from being "glued" together into a coherent global account.

2. Methodology: The Sheaf-Theoretic Atlas Framework

The authors propose a discrete gauge-theoretic framework modeled as a sheaf-theoretic atlas. This approach treats the space of contexts as a stratified simplicial complex (a graph of context clusters) where each cluster is a "local chart" with its own feature space.

Key Components:

Context Complex & Clustering: Token activations from a frozen LLM are clustered into $C$ local charts.
Local Feature Spaces:
- Transports: For overlapping contexts, a linear transport map ( $T_{vu}$ ) is estimated to map features from one chart to another.
- Local Metrics: A local Fisher Information Matrix (or Gauss-Newton metric) is computed for each chart to identify which feature interactions are predictively consequential.
Three Geometric Obstructions: The framework decomposes the failure of global interpretability into three measurable obstructions:
1. Local Jamming (O1): Occurs when the active load of features exceeds the local information-theoretic bandwidth (Fisher bandwidth).
2. Proxy Shearing (O2): A mismatch between the geometrically derived transport (how features actually move) and a fixed "correspondence proxy" (how we assume they should map).
3. Nontrivial Holonomy (O3): Path-dependence where transporting features around a loop of contexts results in a different state than the starting state, indicating a topological obstruction to global trivialization.

Theoretical Tools:

Gauge Theory: The framework uses gauge transformations (rotations of local bases) to define "defects" (errors) in the connections between charts.
Spanning-Tree Gauge Fixing: A constructive theorem allows the authors to fix a gauge on a spanning tree of the context graph, reducing the computation of complex loop holonomies to simple chord residuals.

3. Key Contributions & Theoretical Results

The paper establishes four main results (A–D) that connect the theoretical framework to empirical reality:

Result A: Spanning-Tree Gauge Identity (Theorem 5.1):
- Proves that after fixing a gauge on a spanning tree, the residual error on any "chord" (an edge not in the tree) is exactly equal to the holonomy of the fundamental cycle formed by that chord.
- Significance: This makes holonomy computable and gauge-invariant, transforming it from an abstract loop statistic into a directly measurable quantity.
Result B: Shearing as an Unavoidable Mismatch (Theorem 4.2):
- Proves that the "shearing score" (disagreement between geometric transport and the proxy) lower-bounds a concrete transfer mismatch energy.
- Significance: High shearing is not just a geometric curiosity; it mathematically guarantees a failure in transferring feature representations between contexts.
Result C: Certified Jamming/Interference (Theorem 6.2):
- Provides a non-vacuous, certified lower bound on interference energy for a "consequential subset" of atoms (features) where the Fisher weights are high.
- Significance: This moves interference from a qualitative observation to a certified, quantitative bound that holds with zero violations across random seeds.
Result D: Estimator Stability (Section 7):
- Demonstrates via bootstrap resampling that the diagnostics (shearing and holonomy) are stable under finite sampling, particularly when filtering for "persistent" (well-conditioned) subsystems.
- Significance: Confirms that the observed obstructions are structural properties of the learned representation, not artifacts of sampling noise.

4. Experimental Results

The framework was evaluated on a frozen Llama 3.2 3B Instruct model using diverse corpora (WikiText, C4, Code). Extensive robustness checks were performed on Qwen 2.5 and Gemma 2, across different layers, seeds, and hyperparameters.

Validation of Gauge Identity: Tree-edge residuals were numerically near zero ( $\sim 10^{-5}$ ), while chord residuals matched fundamental-cycle holonomy to numerical precision, confirming Result A.
Holonomy Magnitudes: Nontrivial holonomy was observed across all models, with Qwen showing the highest cycle inconsistency and Gemma the lowest. The signal was robust to random-bases null controls (where learned bases were replaced by random ones, causing holonomy to spike to $\approx 1.0$ , proving the baseline signal relies on learned geometry).
Shearing: Significant shearing was detected, with a median score of $\approx 0.2$ . Theoretical lower bounds held true, though often with conservative slack due to anisotropic data.
Jamming: Certification of interference was achieved on 85–90% of analyzed charts across different seeds and hyperparameters, with zero violations of the lower bound.
Depth Dependence: The obstructions (holonomy and shearing) were not uniform; they exhibited a non-monotonic profile across network layers (strongest in early layers, weakest in the middle, rising again in deeper layers).

5. Significance and Impact

Paradigm Shift: The paper moves interpretability from a "global dictionary" search to an "atlas construction" problem, acknowledging that global consistency may be mathematically impossible due to intrinsic obstructions.
Diagnostic Toolkit: It provides the first set of reproducible, falsifiable diagnostics to determine when and why global interpretability fails:
- Is it Local Overload? (Check Jamming Index)
- Is it Cross-Chart Mismatch? (Check Shearing)
- Is it Path-Dependent Transport? (Check Holonomy)
Robustness: The framework is shown to be robust across model families, layers, and hyperparameters, suggesting these obstructions are fundamental structural properties of superposition in LLMs rather than artifacts of specific training runs.
Safety & Debugging: By identifying specific geometric obstructions, this method offers a rigorous way to audit models for representation instability, potentially aiding in debugging and safety evaluations.

In conclusion, the paper successfully formalizes the intuition that LLM representations are locally coherent but globally obstructed, providing a mathematical and empirical toolkit to measure these obstructions with high precision.