🤖 machine learning

Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

This paper introduces Power Law Graph Attention (PLGA), a theoretically grounded generalization of scaled dot-product attention that replaces fixed bilinear forms with learned, positive tensor-based operators, while establishing an inference-collapse theorem that limits its practical advantage to near-identical performance under specific conditions.

Original authors: Burc Gokden

Published 2026-08-12

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Burc Gokden

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers try to learn human language by reading billions of books. To do this, they use a special kind of brain called a "Large Language Model" (LLM). Think of these models as giant, digital detectives. When they read a sentence, they have to guess what word comes next. To make a good guess, they need to pay attention to the other words in the sentence. Some words are more important than others; for example, in the sentence "The cat sat on the mat," the word "cat" is very important for understanding "sat."

The standard way these detectives do their job is called "Scaled Dot-Product Attention" (SDPA). It's like a very fast, rigid calculator that compares every word to every other word using a fixed, unchangeable rule. It's efficient, but it's a bit like using the same ruler to measure a mountain and a grain of sand. It works, but it can't bend or adapt its measuring tool based on what it's looking at.

Now, imagine if that calculator could build its own custom ruler for every single sentence it reads. It could look at the sentence, realize it needs a flexible, stretchy ruler, and then build that ruler on the spot before doing the math. This is the big idea behind the new method described in this paper: Power Law Graph Attention (PLGA). Instead of using a fixed rule, the model learns to generate a unique, dynamic "scoring rule" for every input. It's like the detective doesn't just use a ruler; they craft a custom measuring tape out of the very words they are reading.

The paper introduces a new type of AI architecture called PLDR-LLM (Large Language Model from Power Law Decoder Representations). Its main job is to figure out if this fancy, self-crafting ruler is actually necessary, or if the model eventually stops needing to build a new one every time. The researcher wanted to know: Does the model keep changing its mind about how to measure things, or does it eventually settle on a single, perfect way of measuring that works for almost everything?

Here is what the paper found, explained through the story of a very efficient detective.

The Detective's New Superpower

The author built a model where the "attention" part (the part that decides which words matter) isn't just a simple math formula. Instead, it's a complex machine that takes the input text, crunches it through a deep neural network, and spits out a brand-new, custom "bilinear operator." You can think of this operator as a dynamic lens.

In the old way (SDPA), the detective looks through a fixed, glass lens. In this new way (PLGA), the detective looks through a lens that reshapes itself based on the scene. If the scene is chaotic, the lens becomes a wide-angle. If the scene is focused, it becomes a zoom lens. The paper proves mathematically that this new system contains the old system as a special case. If the detective decides to stop reshaping the lens and just use a flat piece of glass, the new system becomes exactly the old system. So, this new method is a superset; it's strictly more flexible.

The Great Surprise: The Lens Stops Moving

The most exciting part of the paper is what happens after the detective has trained for a long time. You might expect that because the model is so flexible, it would keep changing its lens for every new sentence. But the researcher discovered something strange and wonderful: the lens stops moving.

After the model is trained, the "dynamic lens" (the complex math that generates the scoring rule) becomes almost completely invariant. This means that no matter what sentence you give it, the model generates the exact same lens. It's as if the detective spent months learning how to build custom rulers, only to realize that for the job at hand, they only ever needed one specific, perfect ruler.

The paper calls this "Empirical Collapse."

What they measured: They tested this on a released version of the model. They found that the "lens" the model generated for different sentences was identical up to a tiny, tiny error (about 1 part in a million, or $10^{-6}$ ). For the best models, the lens was identical down to the very last bit of computer memory.
The consequence: Because the lens is the same for everything, the model doesn't need to do the hard work of building a new lens every time. It can just cache (save) the lens once and reuse it forever. This makes the model much faster and cheaper to run.

Why Does This Happen? (The Three-Stage Magic Trick)

The paper doesn't just say "it happens"; it tries to explain how it happens using a three-stage mechanism. Imagine the model is a chef trying to make a soup.

The Twirl (Rotary Embeddings): The model starts by spinning the ingredients (the words) in a specific way. This "twirl" acts like a filter that washes away the specific details of where a word appeared in the sentence, leaving only the general "flavor" of the word.
The Concentration: As the model looks at more and more sentences, the "flavor" of the ingredients starts to look the same. The math shows that the variations between different sentences get smaller and smaller, like a crowd of people all wearing slightly different shades of blue eventually looking like a single block of blue from a distance.
The Squeeze (Contraction): Finally, the part of the model that builds the lens (the "metric learner") acts like a powerful press. Because the inputs are all so similar (thanks to the first two steps), the press squeezes them all into the exact same shape. The result is a single, stable lens that works for almost every sentence.

The paper is careful to say this is a measured phenomenon, not a magic trick. They proved the math behind the "twirl" and the "squeeze," but the fact that the model actually does this in real life is something they observed in experiments. They measured the "order parameter" (a fancy way of saying "how much does the lens wiggle?") and found that for models trained in a specific "critical" regime, the wiggling stops almost completely.

What This Means for the Future

The paper makes a few very specific claims, and it is very honest about what it doesn't claim.

It IS a generalization: The new method definitely includes the old method. If you set the new method to do nothing special, it becomes the old method.
It IS faster: Because the lens stops moving, you can save it and reuse it. The paper shows this gives a speed boost of about 3 times during the "inference" phase (when the model is actually answering questions).
It ISN'T a magic bullet for intelligence: The paper does not claim that this new model is smarter than the old ones. In fact, they say that for the specific models they tested, the new method and the old method (if you just used the cached lens) produced the exact same answers. The "smarts" come from the training, not just the architecture.
It ISN't a proof of "Self-Organized Criticality" (yet): The paper uses the idea of "Self-Organized Criticality" (like a sandpile that naturally finds a balance point) as a framework to understand why the model behaves this way. They call it a "phenomenological framework," meaning it's a helpful story to explain the data, but they haven't proven it's a fundamental law of physics for AI.

The "Collapse" Theorem

The paper has a theorem called the "Inference-Collapse Theorem." It says: If the model's lens becomes perfectly invariant (never changes), then the complex, slow process of generating a new lens every time can be replaced by a simple, fast process of just using the saved lens.

The paper proves this mathematically. But the real punchline is the measurement: they checked a real, trained model and found that the lens did become invariant. The "collapse" isn't just a theory; it's what actually happened in their experiments.

The Caveats and Limits

The author is very careful not to overhype.

They admit that the "invariance" isn't perfect. There is a tiny error (around $10^{-6}$ ), but it's so small that it doesn't change the answers the model gives.
They note that this only happens if the model is trained in a specific way (the "critical" regime). If you train it differently, the lens might keep wobbling, and you wouldn't get the speed boost.
They explicitly state that they haven't proven that this new method is better at learning than the old method. They only proved it's more flexible and, under the right conditions, faster to run.

The Takeaway

This paper tells the story of a model that learns to build its own tools, only to discover that it only needs one tool. It's a bit like a carpenter who learns to forge a different hammer for every nail, only to realize that one perfect hammer works for everything.

The paper provides a rigorous mathematical description of this new "Power Law Graph Attention," proves that it contains the old methods, and shows through careful measurement that trained models naturally "collapse" into a state where they don't need to do the heavy lifting of generating new rules. This allows for a massive speedup in how these models run, making them more efficient without necessarily making them smarter. It's a story of efficiency, stability, and the surprising simplicity that can emerge from complex systems.

Technical Summary: Power Law Graph Attention

Problem Statement
Modern Large Language Models (LLMs) rely on the Transformer architecture with Scaled Dot-Product Attention (SDPA), where attention scores are computed via a fixed Euclidean inner product of linearly projected token representations. This paper addresses the limitations of this static bilinear form by proposing a mechanism where the attention operator itself is learned, input-dependent, and nonlinear. The author aims to define a mathematically rigorous architecture, the Power Law Graph Attention (PLGA), and the associated Large Language Model (PLDR-LLM), while formally characterizing the conditions under which this complex, input-generated operator collapses into a simpler, cached form during inference.

Methodology and Architecture
The core of the methodology is the Power Law Graph Attention (PLGA) operator, which replaces the fixed bilinear form of SDPA with a learned, input-generated operator $G_{LM}$ . The architecture is defined through a specific chain of transformations:

Density Operator: The input query matrix $Q$ is used to compute a Gram matrix $D_Q = Q^\top Q$ , representing the uncentered second moment of the query vectors.
Metric Generator: A shared deep residual network $\Phi_{res}$ (composed of SwiGLU blocks and LayerNorm) processes the row-wise normalized density operator to produce a generator matrix $A$ .
Power Law Transformation: The generator $A$ is transformed into a strictly positive "metric tensor" $A_{LM}$ via an element-wise power law: $A_{LM} = \text{iSwiGLU}(W A + b_W) + \epsilon$ . This tensor is then raised to a learned power matrix $P$ to form a "potential tensor" $A_P = A_{LM}^{\odot P}$ .
Bilinear Score Operator: The final operator $G_{LM}$ is constructed as a linear superposition of the potential tensor: $G_{LM} = a A_P + b_a$ .
Attention Computation: The attention scores are computed as $E = Q G_{LM} K^\top / \sqrt{d_k}$ , followed by a causal masked softmax.

The full PLDR-LLM architecture integrates this operator into a decoder-only structure with Rotary Position Embeddings (RoPE). A key design feature is the separation of deductive outputs (the tensors $A, A_{LM}, A_P, G_{LM}$ ) from the inductive output (next-token probabilities). The model is trained using a blockwise cross-entropy objective augmented by a DAG regularizer (based on the NOTEARS algorithm) applied to the deductive tensors to penalize cyclic dependencies in the learned interaction graphs.

Key Contributions and Theoretical Results

Exact Generalization of SDPA:
The paper proves that SDPA is an exact special case of PLGA. Specifically, when the learned operator $G_{LM}$ is the identity matrix ( $I$ ), the PLGA equations reduce precisely to the standard SDPA equations. This establishes PLGA as a strict superset of the standard attention mechanism in terms of parameterization, though the paper explicitly notes it does not claim strict function-class separation or performance superiority at matched resources.
The Inference-Collapse Theorem:
A central theoretical contribution is the Inference-Collapse Theorem. It states that if the deductive outputs (specifically $G_{LM}$ ) become exactly input-invariant after training, the entire deep nonlinear subnetwork generating these tensors can be replaced at inference time by a single, constant cached tensor.
- Conditional Nature: The theorem is conditional on the hypothesis of exact invariance. The paper distinguishes between the theorem (proven under the assumption of invariance) and the empirical observation (that trained models exhibit near-invariance).
- Consequences: If invariance holds, the model collapses to a "Generalized SDPA" with a learned constant bilinear form. If RoPE is present, the paper further characterizes when this constant form preserves relative-position dependence (requiring $G_{LM}$ to lie in the commutant of the rotation group).
Structural Properties of the Learned Operator:
- Perron–Frobenius Structure: Due to the element-wise positivity enforced by the power law and $\epsilon$ , the interaction tensor $A_{LM}$ possesses a Perron–Frobenius spectral structure (strictly positive eigenvalues and eigenvectors).
- Learned Singularity: Empirical measurements on released checkpoints reveal that at convergence, the generator $A$ becomes numerically rank-one with identical rows across all attention heads and layers. This "learned singularity" is the algebraic mechanism behind the observed invariance.
- DAG Regularizer Limitations: The paper clarifies that while the DAG regularizer (NOTEARS) is used to encourage acyclicity, the strict entry-wise positivity of $A_{LM}$ and $A_P$ makes exact acyclicity mathematically impossible. The regularizer instead acts as a soft penalty on cycle content.
Mechanism of Invariance (Three-Stage Analysis):
The paper proposes a conditional, three-stage mechanism explaining how the metric learner generates an invariant operator:
- Stage 1 (Rotary Twirl): The application of RoPE before the density operator acts as a "twirl," projecting the density operator onto the commutant of the rotation group, suppressing position-dependent components.
- Stage 2 (Statistical Concentration): The normalized density operator concentrates around a dataset-level mean due to the law of large numbers (modeled via matrix Bernstein inequalities).
- Stage 3 (Row-Map Contraction): The learned row map $\phi$ (the composition of the residual network) is hypothesized to be contractive on the visited set of inputs, collapsing the concentrated inputs into a single output profile.

Empirical Results
The paper reports measurements on released checkpoints (referenced as [12, 13, 14]) to validate the theoretical claims:

Invariance: On models trained in a "near-critical" regime, the deductive outputs exhibit minute relative fluctuations ( $10^{-6}$ to $10^{-11}$ ), effectively behaving as constants.
Cache Fidelity: The "cached" inference (reusing the frozen $G_{LM}$ ) produces bit-identical or near-identical results compared to the "uncached" recomputation on tested prompts, with deviations well within decoding margins.
Rank Collapse: Singular value audits confirm that the generator $A$ is numerically rank-one, while the downstream tensor $A_{LM}$ remains near-full rank despite having a float-zero determinant (due to underflow of small singular values).
Criticality: The paper observes that models trained in a critical-like regime (controlled by learning rate and warm-up steps) exhibit the invariance and coherence required for generalization, while sub-critical models do not.

Significance and Claims
The paper positions itself as a formal analytical description of the PLDR-LLM architecture. Its significance lies in:

Formalization: Providing a complete, machine-checked (in Lean 4) mathematical definition of the architecture and its properties.
Epistemic Clarity: Rigorously distinguishing between proven theorems, conditional theorems, empirical observations, and conjectures. It explicitly avoids claiming that PLGA is universally superior to SDPA, instead framing the "collapse" as a phenomenon that allows for efficient inference without loss of expressivity for specific trained models.
Phenomenological Framework: Treating the "Self-Organized Criticality" (SOC) of the training dynamics as a phenomenological framework for understanding the emergence of invariance, rather than an established law.
Diagnostic Utility: Introducing the "order parameter" (based on the RMS fluctuation of deductive outputs) as a measurable diagnostic to distinguish between phases of model behavior (critical vs. sub-critical).

The paper concludes that PLGA offers a larger parameterized family with distinct training dynamics and an inspectable law representation, but it acknowledges the costs: the training objective involves target-exposed auxiliary scores, and the global Gram operator breaks "historical-row prefix consistency" (a property of standard SDPA) unless the model collapses to an invariant state. The work invites further investigation into the spectral form of the order parameter and the transferability of the invariant operators across domains.