💻 computer science

Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures

This paper demonstrates that despite fundamentally different microscopic mechanisms, both Transformer and Mamba architectures develop a shared, near-marginal infrared collective dynamics characterized by a slow-mode continuum, thereby extending the universality of such organization beyond Transformers and supporting predictions from Cognitive Field Theory.

Original authors: Byung Gyu Chae

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Byung Gyu Chae

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of artificial intelligence, researchers have long sought to understand how machines learn to remember. Two dominant families of models have emerged to handle sequences of information, such as words in a sentence or frames in a video. The first family, known as Transformers, relies on a mechanism called self-attention, which allows the model to weigh the importance of every piece of information against every other piece simultaneously. The second family, represented by newer models like Mamba, uses a different approach based on state-space dynamics, where the model maintains an internal state that evolves over time, updating its memory as it processes each new piece of data. For years, scientists have wondered if these two fundamentally different ways of thinking about time and memory might actually lead to the same underlying behavior deep within the machine. This question matters because if different architectures converge on the same patterns, it suggests that the ability to learn and remember is governed by universal principles rather than just the specific engineering choices of the designers.

A researcher set out to test this idea by looking inside a trained Mamba model to see how their memory actually works. They focused on a specific property called the "time-scale density of states," which is essentially a map showing how many different speeds of memory the model possesses. Imagine a library where books are sorted not by topic, but by how quickly they fade from a reader's mind. Some books are forgotten in seconds, others in minutes, and some linger for years. The researcher wanted to see the distribution of these "fading speeds" in the Mamba model. They knew that Mamba has a built-in mechanism for memory that can be examined in layers: first, the raw, internal memory speeds encoded in its design; second, how those speeds change depending on the specific input it receives; and finally, the collective memory speed of the entire block after all its complex calculations are finished.

The researcher discovered that these three layers are not the same. The raw internal memory of the Mamba model starts with a broad range of speeds, but as the model processes information, it selectively rescales these speeds based on what it is reading. This input-dependent adjustment is a unique feature of Mamba, allowing it to slow down or speed up its internal clock depending on the context. However, the most surprising finding came when the researcher looked at the final, collective output of the entire model block. Despite the complex reorganization of speeds happening inside, the final result settled into a very specific, stable pattern. When they measured the memory speeds across different sequence lengths, they found that the model consistently developed a broad, continuous range of very slow memories. These slow memories did not disappear; instead, they formed a smooth, predictable continuum that became clearer and more defined as the model processed longer sequences of data.

To understand if this was a quirk of the Mamba design or a general rule of intelligent systems, the researcher compared their findings with the behavior of Transformer models. Transformers do not have the same internal state-space machinery as Mamba; they rely on attention mechanisms instead. Yet, when the researcher measured the collective memory speeds of the Transformers, they found a strikingly similar pattern. Both architectures, despite using completely different microscopic methods to process information, developed a nearly flat distribution of memory speeds that was slightly weighted toward the very slowest, longest-lasting memories. In the language of physics, this is a "near-marginal" state, meaning the models are poised right at the edge of having infinite memory, allowing them to retain information for long periods without becoming stuck.

The study confirms that this slow-memory organization is not just a feature of one specific type of neural network. The researcher showed that even though Mamba explicitly builds its memory using a different mathematical structure than Transformers, both systems end up organizing their collective dynamics in the same way. They found that the Mamba model's memory behavior stabilizes around a specific value, indicating a robust, reproducible pattern that emerges regardless of the sequence length, provided the sequence is long enough to reveal the full picture. This suggests that the ability to maintain long-term memory is a fundamental property of learned sequence processing, one that arises naturally in different architectural forms. The findings extend our understanding of how artificial intelligence works, showing that beneath the surface-level differences in design, these powerful models share a common, deep structure for handling time and memory.

Title: Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures

Problem Statement
A fundamental open question in deep learning is whether distinct neural architectures, despite utilizing different microscopic computational mechanisms, develop common collective dynamical behaviors. Recent analyses of Transformer language models revealed a nearly flat, weakly infrared-enhanced time-scale density of states (TDOS) in their relaxation spectra, associated with near-marginal long-memory dynamics ( $K(t) \sim 1/t$ ). However, it remains unclear whether this infrared organization is a specific artifact of the self-attention mechanism or a more general principle of learned sequence processing. To test this, a comparison is required with an architecture that employs a qualitatively different mechanism for temporal computation. State-space models, particularly Mamba, offer such a contrast by utilizing input-dependent selective state-space dynamics rather than self-attention.

Methodology
The study employs a multi-level spectral analysis on a pretrained Mamba-370M model to distinguish between microscopic state-space memory and macroscopic collective organization. The methodology proceeds through three distinct dynamical levels:

Intrinsic State-Space Spectrum: The author analyzes the eigenvalue spectrum of the learned state-space generator ( $A$ ), defining intrinsic relaxation rates $\lambda^{int}_i = -\text{Re}(a_i)$ . This represents the hierarchy of time scales encoded directly in the model parameters before input conditioning.
Input-Conditioned Effective Spectrum: The study examines the effective relaxation rates $\lambda^{eff}_{i,t} = \Delta_t(x) \lambda^{int}_i$ , where the input-dependent selective time step $\Delta_t(x)$ rescales the intrinsic rates during inference. This captures how the selective mechanism dynamically reorganizes relaxation scales based on the input sequence.
Full-Block Collective Spectrum: The author computes the Jacobian of the complete Mamba block transformation ( $J_\ell = \partial \text{vec}(H_{\ell+1}) / \partial \text{vec}(H_\ell)$ ). The collective relaxation rates are defined as $\lambda^{block}_\alpha = -\ln |\zeta_\alpha|$ , where $\zeta_\alpha$ are the eigenvalues of the full-block Jacobian. This measures the collective response of the entire block, including projections, gating, nonlinearities, and residual propagation.

The collective Time-Scale Density of States (TDOS), $\rho(\lambda)$ , is constructed from these rates. The analysis systematically varies the retained sequence length ( $L$ ) from 8 to 512 tokens to test the convergence and stability of the infrared sector. The infrared scaling exponent $\beta$ is determined via cumulative mode statistics ( $N(<\lambda) \propto \lambda^{1+\beta}$ ) over a fixed, well-resolved interval to avoid binning artifacts. These results are directly compared with previous measurements from Transformer models (specifically Pythia-410M) using the same full-block Jacobian construction.

Key Results

Hierarchical Reorganization: The intrinsic, effective, and full-block spectra are not identical. While the learned SSM generator contains a broad intrinsic hierarchy of relaxation rates, the selective mechanism and subsequent block transformations substantially reorganize this hierarchy. The full-block collective spectrum is a distinct emergent property, not a direct copy of the microscopic SSM spectrum.
Emergence of a Collective Infrared Continuum: The full-block Mamba dynamics develops a reproducible slow-mode continuum. As sequence length increases, the infrared sector becomes progressively better resolved and smoother, extending toward smaller relaxation rates.
Infrared Scaling Exponent: Cumulative spectral analysis reveals a weakly negative power-law scaling $\rho(\lambda) \sim \lambda^\beta$ . For long sequences ( $L \ge 64$ ), the exponent stabilizes near $\beta_M \simeq -0.17$ . This corresponds to a memory kernel $K(t) \sim t^{-(1+\beta)} \approx t^{-0.83}$ , which is close to the marginal $1/t$ regime.
Architecture Independence: Despite the fundamentally different microscopic mechanisms (self-attention vs. selective state-space), Transformers and Mamba exhibit closely related collective infrared organizations. Transformers show a representative exponent of $\beta_{Tr} \sim -0.1$ , while Mamba stabilizes near $\beta_M \sim -0.17$ . Both architectures occupy a "near-marginal" regime characterized by a broad, nearly flat, but weakly infrared-enhanced TDOS.

Significance and Claims
The paper claims to separate explicit microscopic state-space memory from collective infrared organization, demonstrating that distinct sequence architectures can develop closely related near-marginal slow-mode dynamics. The primary significance lies in showing that this infrared organization is not unique to the self-attention mechanism of Transformers but emerges in state-space models as well, despite their different computational roots.

The author argues that this suggests near-marginal slow-mode dynamics may represent a general dynamical principle of learned sequence processing, extending the observation of infrared collective organization beyond Transformers. This finding provides an independent test of the dynamical structure described by Cognitive Field Theory, which associates long-lived macroscopic memory with the infrared organization of collective relaxation modes. The results indicate that while microscopic implementations differ, the macroscopic collective dynamics of learned sequence models may converge toward a similar, physically meaningful regime.

More like this