🤖 machine learning

Geometric Properties of the Voronoi Tessellation in Latent Semantic Manifolds of Large Language Models

This paper empirically validates a linear scaling law for the expressibility gap in Qwen3.5-4B-Base's Voronoi tessellation and demonstrates that Fisher information-based margin refinement can geometrically reorganize the model to improve decision margins by 28% with invariant downstream performance, though its practical utility is limited by a concentration of benefits on high-frequency structural tokens.

Original authors: Marshall Brett

Published 2026-04-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marshall Brett

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) like Qwen3.5 as a massive, high-tech library. Inside this library, every possible word or phrase (a "token") has its own specific shelf. When the model reads a sentence, it tries to figure out which shelf the next word belongs to.

Usually, the shelves are arranged neatly. But sometimes, two shelves are right next to each other, making it hard for the model to decide which one to pick. This confusion creates a "fuzzy zone" where the model might guess wrong.

This paper is about a team of librarians who discovered a way to rearrange the shelves to make the library clearer, without having to rebuild the whole building.

Here is the breakdown of their discovery using simple analogies:

1. The Problem: The "Fuzzy Zone" (The Expressibility Gap)

The researchers found that in the model's "mind," there is a specific area where the shelves are too close together. If a sentence lands in this fuzzy zone, the model gets confused.

The Discovery: They proved mathematically that this fuzzy zone follows a predictable pattern (like how water flows down a hill). They also fixed a measurement error: previously, they were using a blurry ruler (low-precision math) to measure the shelves, which made the fuzzy zones look worse than they actually were. When they used a super-precise ruler, the pattern became crystal clear.

2. The Solution: Two Ways to Rearrange the Shelves

The team tried two different methods to push the confusing shelves apart. Think of this as trying to organize a messy closet.

Method A: The "Brute Force" Push (Margin Maximization)

How it works: Imagine you see two shirts hanging too close together. You just grab them and shove them apart as hard as you can.
The Result: It works! The shirts are further apart. But because you shoved them so hard, you accidentally knocked over other clothes nearby.
The Catch: If you push too hard, you break the closet. The more you try to fix the confusion, the more you ruin the rest of the library. It's a "high risk, high reward" approach that quickly becomes destructive.

Method B: The "Smart Navigator" (Fisher Information Distance)

How it works: Instead of just shoving things apart, this method asks: "What is the most natural way to separate these items without disturbing the rest of the room?" It looks at the "shape" of the library and gently rotates the shelves to find a better spot.
The Result: It successfully separates the confusing shelves, but it doesn't knock over anything else.
The Magic: Even when they pushed the "Smart Navigator" to its maximum strength, the rest of the library stayed perfectly organized. The model's ability to answer general questions (like math or logic puzzles) didn't change at all.

3. The Surprise: What Actually Got Better?

The researchers expected the model to get better at everything. But when they looked closely, they found a surprise:

The "Head" Tokens: The improvements were almost entirely focused on common, structural words like commas, periods, spaces, and "the" or "and."
The Analogy: Imagine you hired a professional organizer to fix your kitchen. They did a fantastic job organizing the salt, pepper, and spoons (the common stuff). But they didn't really improve the placement of the exotic spices or the fancy knives (the rare, complex words).
The Trade-off: At moderate settings, the model got better at everything. But if they pushed the "Smart Navigator" too hard, the model became too obsessed with commas and periods, and actually started getting slightly worse at handling complex names or specific content words.

4. The "Mid-Layer" Mystery

The team also looked inside the model's "brain" layer by layer.

Early Layers: The model is just gathering context (like reading the first few words of a sentence).
Middle Layers (The "Ambiguity Zone"): They found a strange middle section where the model seems confused, even if it eventually gets the right answer. It's like a detective who has all the clues but hasn't connected them yet.
Final Layers: The model suddenly "crystallizes" the answer and makes the right choice.

5. The Bottom Line

The paper proves that you can take a finished, trained AI model and give it a "geometric polish" to make it more confident and accurate.

The Good News: You can fix the model's confusion without breaking its general intelligence.
The Warning: You have to be careful not to over-tune it. If you push too hard, the model becomes a master of punctuation but a bit clumsy with complex ideas.
The Future: The researchers suggest that in the future, we should teach the model to be "smart" about which words matter most, so we can fix the common stuff without hurting the rare stuff.

In short: They found a way to gently reorganize the AI's internal map to reduce confusion. It works beautifully, but like any good renovation, you have to know when to stop so you don't accidentally knock down the walls.

1. Problem Statement

Large Language Models (LLMs) operate by mapping continuous hidden states to discrete tokens via a final linear layer (unembedding). This process induces a Voronoi tessellation in the latent semantic manifold, where each region corresponds to a specific token.

The Expressibility Gap: Defined as the fraction of the semantic space where the model cannot confidently assign a token (i.e., where the margin between the top two logits is small).
The Challenge: Previous theoretical work (Mabrok, 2026) suggested this gap follows a linear scaling law, but empirical validation was limited to smaller models. Furthermore, it was unclear if the Voronoi geometry of a converged model could be reshaped via post-hoc intervention without catastrophic damage to the model's capabilities.
Technical Artifacts: Initial attempts to measure these margins were plagued by bfloat16 quantization artifacts, which obscured the fine-grained geometric structure of the manifold.

2. Methodology

The study focuses on Qwen3.5-4B-Base, a 4.2B parameter dense transformer with a 248k vocabulary. The authors employ Margin Refinement Procedures (MRP): short, targeted optimization runs applied to a converged model to reshape local Voronoi geometry without full retraining.

A. Precision Correction

To resolve quantization issues, the authors implemented a float32 margin recomputation technique. Instead of relying on bfloat16 logits, they extract the final hidden state, upcast to float32, and recompute the linear projection. This eliminated quantization noise, revealing the true continuous geometric structure.

B. Intervention Strategies

Two distinct loss functions were compared to maximize margins:

Direct Margin Maximization (MM): Optimizes the difference between the top-1 and top-2 logits ( $L = -E[m(h)]$ ). This pushes representations away from the Voronoi boundary along a fixed axis.
Fisher Information Distance Maximization (Fisher): Optimizes the pairwise Fisher distances between the top- $k$ tokens weighted by their probabilities. This moves representations along the direction of maximum information divergence in the model's own metric tensor, effectively rotating the "runner-up" token to a more naturally separable alternative.

C. Experimental Protocol

Dose-Response: The weight $\lambda_{MRP}$ was varied across a range (0.1 to 2.0) to observe the trade-off between margin improvement and collateral damage.
Evaluation: Metrics included expressibility gap scaling ( $R^2$ ), prediction churn (Wrong $\to$ Right vs. Right $\to$ Wrong), downstream benchmark performance (ARC, HellaSwag, etc.), and granular audits by token frequency and class.

3. Key Contributions

Validation of Theoretical Scaling: Achieved an unprecedented $R^2 = 0.9997$ in the linear scaling law of the expressibility gap, confirming Mabrok's Theorem 10.5 on a large-scale (4B) Gated DeltaNet architecture.
Resolution of Quantization Artifacts: Demonstrated that bfloat16 inference artificially suppresses the measured expressibility gap and that float32 recomputation is essential for accurate geometric analysis.
Discovery of Mid-Layer Ambiguity: Identified a "geometric ambiguity regime" in layers 24–28 where intermediate representations are geometrically ambiguous (low margin) even when the final prediction is correct, suggesting a late-layer linearization process.
Fisher vs. Margin Damage Profiles:
- Margin Maximization: Shows escalating collateral damage. As $\lambda_{MRP}$ increases, the flip ratio (Right $\to$ Wrong) degrades rapidly (from 2.9x to 0.4x), making the method destructive beyond $\lambda_{MRP} \approx 0.3$ .
- Fisher Maximization: Exhibits constant collateral damage. Across the validated range ( $\lambda_{MRP} = 0.15$ to $0.6$), the number of corrupted positions remains stable at ~5,300, while median margins improve significantly (+28%).
Token-Value Distribution Analysis: Revealed that while aggregate benchmarks remain flat, the gains from Fisher refinement are highly concentrated in high-frequency structural tokens (punctuation, formatting). At higher $\lambda_{MRP}$ , gains in content and entity-like tokens diminish, and regressions in medium-frequency function words emerge.

4. Key Results

Expressibility Gap: The gap coefficient $\alpha$ was established at 0.762. Fisher refinement at $\lambda_{MRP}=0.6$ reduced the probability of low margins ($Pr(m<0.5)$) from 0.300 to 0.250 (-16.7%) and compressed the gap coefficient to 0.612.
Correction Ceiling: Both methods hit a "corrections ceiling" of approximately 16,300 correctable positions out of 256,000. This suggests a fixed reservoir of geometrically fixable errors; errors beyond this are due to missing knowledge, not boundary proximity.
Benchmark Stability: Downstream benchmarks (ARC-Challenge, HellaSwag, etc.) remained flat across the entire Fisher range ( $\lambda_{MRP} = 0.15$ to $2.0$), despite significant changes in the internal Voronoi geometry.
The "Runner-Up" Mechanism: Fisher refinement works not by forcing the top-1 token further away, but by rotating the runner-up token to a different, more separable token. This preserves the top-1 prediction while widening the margin naturally.
Frequency Bias: At $\lambda_{MRP}=0.6$ , the 100+ frequency bucket accounted for 84.0% of net corrections. By $\lambda_{MRP}=2.0$ , this rose to 92.1%, while entity-like tokens became net negative in fine-tuned samples.

5. Significance and Implications

Geometric Plasticity: The study proves that the Voronoi tessellation of a converged LLM is malleable. It can be reshaped via short optimization runs, compressing the expressibility gap without retraining.
Superiority of Fisher Metric: Fisher Information Distance maximization is identified as a low-damage geometric polishing tool. Unlike direct margin maximization, it respects the model's intrinsic geometry, allowing for margin expansion without distorting the representation space.
Nuanced Trade-offs: The paper challenges the assumption that improved aggregate accuracy implies uniform utility. It highlights a critical trade-off: geometric refinement improves structural/formatting tokens disproportionately, potentially at the expense of medium-frequency content words and entity recognition, especially at high intervention strengths.
Future Directions: The authors suggest that the optimal path forward is not just bulk geometric polishing, but token-value-aware refinement (e.g., weighting the Fisher loss by token importance) and training-time geometry shaping to create a more robust manifold substrate before post-hoc intervention.

In conclusion, this work provides the strongest empirical validation to date of latent semantic manifold theory and introduces a robust, low-damage method (Fisher MRP) for post-hoc geometric refinement, while cautioning that such refinements must be carefully managed to avoid skewing token-level utility toward high-frequency structural elements.