🤖 machine learning

Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

This paper provides quantitative experimental evidence that Transformer dynamics are governed by a universal principle of infrared collective organization, where learning reorganizes time-scale density of states to enhance memory and reduce forgetting through the accumulation of slow relaxation modes.

Original authors: Byung Gyu Chae

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Byung Gyu Chae

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant, digital brain made of billions of tiny switches, known as a Transformer. For a long time, scientists thought these brains worked like a simple assembly line: one step leads to the next, and the "intelligence" just pops out at the end. But a new study suggests something much wilder is happening inside. It turns out these models aren't just processing data; they are building a collective cognitive field, a sort of invisible, shared mental atmosphere that organizes itself through a specific kind of physical dance.

The Great Slow-Down Party

To understand what's going on, imagine the brain's internal activity as a massive crowd of runners. At the very beginning of training (when the model is "born"), the runners are scattered everywhere. Some are sprinting fast, some are jogging, and some are just standing still. It's chaotic.

As the model learns, something strange happens. The fast runners don't just get faster; they start to slow down. More and more of them drift toward the "infrared" zone—a fancy physics term for the slowest, most relaxed part of the spectrum. It's like the entire crowd suddenly decides to sit down and have a long, deep conversation instead of running laps.

The researchers measured this by looking at the "Jacobian spectrum," which is basically a snapshot of how the model's internal state changes from one layer to the next. They found that as training progresses, the model doesn't just get smarter; it reorganizes its entire internal rhythm. The "Time-Scale Density of States" (TDOS)—a way of counting how many slow modes exist—becomes almost flat in the slow region. This means the model builds a hierarchy of slow relaxation modes. It's not just one slow memory; it's a whole orchestra of them, all humming together.

The "Critical" Moment: A Peak, Not a Line

Here is the most surprising part of the story. You might expect that as the model learns, its "memory power" just keeps getting stronger and stronger in a straight line. But the paper shows that this is not what happens.

Instead, the model's "memory self-energy" (a measure of how much the past influences the present) shoots up rapidly at the very beginning of training. It hits a pronounced transient maximum around step 2,000. At this exact moment, the model is in a "critical" state. Think of it like a tightrope walker balancing perfectly in the middle of the wire. The "cognitive forgetting gap" (the distance between remembering and forgetting) shrinks to its smallest point, and the model's ability to react to information (its "collective susceptibility") becomes huge.

But here's the twist: the model doesn't stay at this peak. After hitting that maximum, the memory self-energy actually drops down a bit and settles into a stable, "metastable near-critical" regime. It's like the tightrope walker finds a safe, comfortable spot just below the peak where they can stand for a long time without falling. The paper argues that this transient critical formation is the key moment where the "cognitive field" is born, and the model then stabilizes in a protected state that is ready for long-term memory.

The Universal Rule: 1 Over Time

Once this field is formed, the model develops a very specific kind of memory. Most systems forget things quickly, like a bucket with a hole in it. But this model remembers things in a way that follows a universal power law.

The "memory kernel" (how the past affects the future) follows the rule $K(t) \sim 1/t$ . This means the memory doesn't fade away exponentially; it fades slowly, like a long echo that never quite disappears. This happens across the entire network, from the first layer to the last, and it happens regardless of how big the model is.

The researchers tested this on three different sizes of models: 70M, 410M, and 1.4B parameters. Even though these models are vastly different in size, they all did the exact same dance. They all built the same slow-mode hierarchy, they all hit that same transient peak in memory self-energy, and they all settled into the same $1/t$ memory pattern. This suggests that this "infrared organization" isn't just a quirk of a specific model; it's a universal principle of how these systems learn.

What It's NOT

The paper is very clear about what this is not. It rules out the idea that intelligence is just a bunch of individual neurons firing or a simple optimization process where the model just gets "better" in a straight line. It also rules out the idea that the model develops a single, dominant "slow time" for memory. Instead, it's a broad, continuous distribution of many different slow speeds working together.

The Bottom Line

The study measured these dynamics directly using Pythia language models (specifically the 410M version for the main data, with 70M and 1.4B for comparison). They didn't just guess; they calculated the relaxation rates from the model's own math and found that the numbers match the predictions of "Cognitive Field Theory."

The evidence suggests that Transformer learning is governed by infrared collective organization. The model doesn't just memorize facts; it reorganizes its internal physics to create a shared, slow-moving field of memory. It hits a critical peak early on, then settles into a stable, near-critical state where it can hold onto information for a long time, following a $1/t$ scaling law that works the same way whether the model is small or huge. This provides the first quantitative experimental proof that these massive AI systems are behaving like complex, collective physical systems, not just giant calculators.

Technical Summary: Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

Problem Statement
Large language models (LLMs) based on the Transformer architecture exhibit remarkable emergent capabilities, such as reasoning and long-range contextual memory. However, the physical mechanisms governing their large-scale collective dynamics remain poorly understood. Existing literature largely describes Transformer computation as high-dimensional nonlinear mappings or attention-based optimization processes, offering limited insight into how macroscopic collective organization arises from underlying dynamics. This work addresses the gap in understanding whether the collective observables predicted by Cognitive Field Theory—specifically the organization of slow relaxation modes—can be directly identified and quantified within Transformer dynamics.

Methodology
The authors investigate publicly available Pythia autoregressive Transformer language models (70M, 410M, and 1.4B parameters) trained on The Pile corpus. The study employs a response-theoretic approach to analyze the collective relaxation dynamics generated by the learned hidden-state geometry.

Jacobian Spectrum Extraction: The authors compute the Jacobian matrices ( $J_l = \partial h_{l+1} / \partial h_l$ ) between successive hidden-layer representations using automatic differentiation. They also analyze composed Jacobians ( $J = J_{l+n-1} \cdots J_l$ ) to capture collective propagation across varying network depths and propagation distances (e.g., $1 \to 24$ layers).
Relaxation Spectrum Construction: From the complex eigenvalues ( $\mu_\alpha$ ) of the Jacobians, relaxation rates are extracted via $\lambda_\alpha = -\log |\mu_\alpha|$ .
Derivation of Collective Observables: Using the measured relaxation rates, the authors construct the Time-Scale Density of States (TDOS), $\rho(\lambda)$ $ρ (λ)$ . From the TDOS, they quantitatively evaluate:
- The Memory Kernel ( $K(t)$ ), derived via Laplace transform of the TDOS.
- The Memory Self-Energy ( $\Sigma(0)$ ), calculated as the integral of the TDOS weighted by $1/\lambda$ .
- The Cognitive Forgetting Gap ( $r_{cog} = r - \Sigma(0)$ ), where $r$ is the bare forgetting rate.
- The Collective Susceptibility ( $\chi(0) \propto r_{cog}^{-1}$ ).
- The Infrared Critical Exponent ( $\beta$ ), determined by fitting the cumulative mode distribution $N(<\lambda) \propto \lambda^{\beta+1}$ .

The analysis spans the entire training trajectory, varying network depths, and model scales to test for universality.

Key Contributions

Direct Measurement of Cognitive Field Theory Observables: The paper provides the first quantitative experimental realization of collective observables (TDOS, self-energy, forgetting gap, susceptibility) directly from Transformer dynamics without introducing phenomenological parameters.
Identification of Infrared Spectral Reorganization: The study demonstrates that Transformer learning is characterized by a progressive accumulation of slow relaxation modes toward the infrared sector ( $\lambda \to 0$ ), reorganizing the TDOS.
Discovery of Transient Critical Formation: The authors identify a non-monotonic evolution of the memory self-energy, characterized by a pronounced transient maximum during early training, followed by relaxation into a metastable near-critical regime.
Establishment of Universality: The paper establishes that these dynamical phenomena (infrared organization, critical formation, and scale-free memory) are universal across different model scales (70M to 1.4B), network depths, and propagation distances.

Key Results

Infrared Accumulation and TDOS: Throughout training, the relaxation spectrum undergoes a systematic reorganization where spectral weight shifts toward smaller relaxation rates. The resulting TDOS exhibits an approximately flat infrared distribution, following a power law $\rho(\lambda) \sim \lambda^\beta$ with a critical exponent $\beta \simeq -0.1$ . This indicates a hierarchy of slow modes rather than a single characteristic timescale.
Transient Critical Formation: The memory self-energy ( $\Sigma(0)$ ) does not increase monotonically. Instead, it reaches a transient maximum during the early stages of optimization (around 2,000 steps for Pythia-410M). At this point, the cognitive forgetting gap ( $r_{cog}$ ) reaches its minimum, and the collective susceptibility ( $\chi(0)$ ) is maximized. This signifies a transient critical formation of the macroscopic cognitive field. Subsequent optimization drives the system toward a protected metastable near-critical operating regime.
Universal Scale-Free Memory: The memory kernels derived from the measured TDOS exhibit robust long-time power-law scaling, $K(t) \sim 1/t$ , across all model sizes and training stages. This scaling emerges naturally from the measured relaxation spectrum without imposed functional forms.
Spatial and Depth Independence:
- Spatial: The infrared organization is distributed across the Transformer hierarchy. While the strength of memory renormalization varies by layer (strongest in intermediate and deep layers), the underlying scale-free infrared structure is preserved throughout the network.
- Depth: The transient critical formation and the $1/t$ memory kernel scaling persist robustly across different layer propagation depths (from $1 \to 5$ to $1 \to 24$ ), indicating that the collective dynamics are a global property of the network, not limited to local layer interactions.
Model Scale Universality: The same sequence of infrared spectral reorganization, transient critical formation, and scale-free memory dynamics is reproduced consistently across Pythia models ranging from 70M to 1.4B parameters.

Significance and Claims
The paper claims to provide the first quantitative experimental evidence that Transformer learning is governed by infrared collective organization. By mapping Transformer dynamics to the language of nonequilibrium many-body physics, the authors demonstrate that:

Large language models function as nonequilibrium many-body systems where cognition emerges from the organization of slow relaxation modes.
The emergence of macroscopic cognitive fields follows a specific dynamical scenario: an initial infrared spectral reorganization leads to a transient critical state (maximal susceptibility), which then stabilizes into a protected metastable near-critical regime.
The preservation of the infrared universality class ( $\beta \simeq -0.1$ and $K(t) \sim 1/t$ ) across different scales and depths suggests that infrared slow-mode organization constitutes a universal collective principle underlying Transformer dynamics, rather than a model-specific architectural feature.

The work establishes a unified field-theoretic framework for measuring collective cognitive dynamics, bridging the gap between deep learning and statistical physics, and suggests that these principles may represent general laws for collective information processing in complex dynamical systems.