🤖 machine learning

On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

This paper characterizes the diverse long-time dynamical behaviors of deep linear transformers in two dimensions by modeling token interactions as a generalized Kuramoto system, revealing intrinsic low-dimensional dynamics, hidden Hamiltonian structures, and robust phenomena like clustering and oscillations that persist in higher dimensions.

Original authors: Sixu Li, Thomas Jacob Maranzatto, Jan Peszek, Trevor Teolis, Semih Akkoc, Konstantin Riedl, Sennur Ulukus, Nicolás García Trillos

Published 2026-07-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Sixu Li, Thomas Jacob Maranzatto, Jan Peszek, Trevor Teolis, Semih Akkoc, Konstantin Riedl, Sennur Ulukus, Nicolás García Trillos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where data isn't just static numbers on a page, but a bustling crowd of tiny travelers moving through a vast, invisible city. This is the realm of machine learning, specifically the technology behind modern Artificial Intelligence known as Transformers. You've likely heard of them; they are the engines that power chatbots, translation tools, and image generators. At the heart of these machines is a mechanism called self-attention. Think of it as a super-powered social network for data points (called "tokens"). When a Transformer reads a sentence, every word gets to look at every other word to decide how important it is. This process happens layer by layer, like a game of telephone where the message gets refined at every step.

But here's the mystery: while we know these machines work incredibly well, we often don't understand how they work inside. It's like knowing a car drives fast but not understanding the engine. Scientists have been trying to map out the "traffic patterns" of these data tokens as they move through the layers. Do they all march in lockstep? Do they form groups? Do they get stuck in a loop? Understanding this is crucial because if we know how the data moves, we can build better, safer, and more efficient AI. This paper dives deep into that traffic, treating the data tokens not as computer code, but as a swarm of particles dancing to a mathematical rhythm.

The Great Token Dance: A Study in Chaos and Order

In this paper, a team of mathematicians and engineers decided to watch what happens when you strip a Transformer down to its bare bones. They focused on a simplified version called a Linear Transformer. Imagine a dance floor where the music is simple and predictable, and the dancers are "tokens" (the data points). The researchers wanted to see: if we let these dancers interact layer after layer, do they eventually stop and stand still in a perfect line? Do they form two opposing groups? Or do they just keep spinning in circles forever?

To figure this out, the authors used a clever trick. They realized that in a specific, two-dimensional setting (think of the tokens moving on a flat circle), the complex math of the Transformer behaves exactly like a famous physics model called the Kuramoto model. You can think of the Kuramoto model as a way to describe how fireflies flash in sync or how pendulum clocks eventually tick together. The authors discovered that their Transformer tokens were essentially "fireflies" that only interacted through a specific, second-order rhythm (a "double-beat" instead of a simple beat).

Because of this connection, they could use powerful mathematical tools (the Watanabe–Strogatz transformation and the Ott–Antonsen ansatz) to shrink the problem. Instead of tracking thousands of individual dancers, they could describe the entire crowd's behavior with just a single, simple equation. It's like realizing that instead of tracking every single person in a stadium, you only need to track the "wave" moving through the crowd.

The Surprising Results: Not Just a Straight Line

The team found that the behavior of these tokens depends entirely on the "rules of the dance," which are set by three specific matrices (tables of numbers) called Key, Query, and Value. By changing these numbers, they could make the tokens do wildly different things:

The Great Clustering (The "Huddle"): In many cases, the tokens do exactly what we expect. They group together. Sometimes they all huddle into one single point (perfect agreement). But often, they split into two opposing groups (antipodal clusters) that sit on opposite sides of the circle. This is like a debate where everyone eventually agrees on one of two extreme sides.
The Eternal Spin (The "Carousel"): Here is where it gets weird. The researchers found specific settings where the tokens do form two groups, but these groups never stop moving. They rotate around the circle forever, like a carousel that never stops. The tokens are clustered, but they never settle down. This is a behavior the authors hadn't seen predicted before in this context.
The Hidden Hamiltonian (The "Swing"): In another scenario, the system behaves like a pendulum or a swing. If the parameters are just right, the tokens oscillate back and forth in a perfect, repeating loop. If you tweak the parameters even slightly, the system can suddenly switch from swinging forever to settling down. This is called a bifurcation, and it shows that the Transformer's behavior is incredibly sensitive to its settings.

What About Real Life? (The "What If" and "What Is")

The authors were careful to note that their most detailed proofs apply to a simplified, two-dimensional world. However, they didn't just stop there. They ran numerical experiments (computer simulations) with much larger, more realistic models (100 dimensions and 200 tokens).

The results were exciting: the strange behaviors they found in the simple 2D world—like the eternal spinning and the sudden switches between swinging and settling—persisted in the higher-dimensional simulations. This suggests that these complex, non-linear dances aren't just mathematical curiosities; they might be happening inside the massive AI models we use every day.

They also compared their "Linear" model to the more common "Softmax" model (the one used in real-world AI). They found that while the Softmax model tends to force everyone into a single, perfect huddle, the Linear model is more diverse, capable of forming two distinct groups or spinning forever. This suggests that the "linear" simplification isn't just a toy; it reveals hidden dynamics that the more complex models might also be hiding.

The Bottom Line

This paper doesn't claim to have solved the mystery of all AI. Instead, it opens a new window. By treating data tokens as interacting particles and using the language of physics, the authors showed that Transformers are capable of a much richer variety of behaviors than just "converging to a single answer." They can oscillate, bifurcate, and rotate.

The authors proved these behaviors mathematically for the 2D case and suggested through simulations that they hold true for larger, more complex systems. They didn't just find that the tokens move; they found how they move, revealing that the internal world of a Transformer is a dynamic, sometimes chaotic, and beautifully structured dance floor. This insight could help us understand why AI sometimes gets stuck in loops or why it behaves unpredictably, paving the way for more stable and understandable artificial intelligence in the future.

Technical Summary: Diverse Dynamical Behaviors Arising in Deep Linear Transformers

Problem Statement

The paper addresses the lack of mathematical understanding regarding the internal transformation of data within transformer architectures, specifically focusing on the inference-time dynamics of deep encoder-only transformers. While prior work has characterized token synchronization and "rank collapse" (convergence to a single point) under restrictive assumptions (e.g., specific matrix structures and softmax attention), the behavior of these systems under more general parameter choices remains unclear.

The authors investigate the long-time behavior of the token representations $x_k(t)$ evolving through successive linear self-attention layers. The dynamics are modeled as a system of coupled ordinary differential equations (ODEs) on the unit sphere $S^{d-1}$ :
$\dot{x}_k(t) = P^\perp_{x_k(t)} \left( \frac{1}{n} \sum_{j=1}^n \beta \langle x_k(t), A x_j(t) \rangle V x_j(t) \right)$
where $A = Q^\top K$ and $V$ are the key/query and value parameter matrices, respectively. The central question is: For which choices of matrices $A$ and $V$ does the system converge to consensus (clustering), and under which choices do other long-time behaviors (oscillations, bifurcations) emerge?

Methodology

The authors adopt an interacting particle system perspective, modeling tokens as particles. Their methodology diverges from the Wasserstein gradient flow approach used in prior literature by focusing on the linear self-attention (LSA) model ( $h(y)=y$ ) in dimension $d=2$ .

Kuramoto Reformulation: The authors demonstrate that in $d=2$ , the LSA dynamics can be reformulated as a generalized Kuramoto model with pure second-harmonic coupling. By representing tokens as angles $\theta_k$ on the unit circle ( $x_k = e^{i\theta_k}$ ), the dynamics depend only on the second-order parameter $R_2(t) = \frac{1}{n} \sum e^{i2\theta_j}$ .
Dimensionality Reduction: Leveraging the pure second-harmonic structure, the authors apply the Watanabe–Strogatz (WS) transformation. This reveals that the $n$ -particle system is intrinsically low-dimensional, governed by only two macroscopic variables ( $\alpha, \eta$ ) regardless of $n$ .
Ott–Antonsen (OA) Ansatz: For a specific class of initializations where the distribution of double-angles $\Xi = 2\Theta$ lies on the "OA manifold" (specifically, wrapped Cauchy distributions), the WS system further reduces to a single complex-valued ODE for the order parameter $R_2(t)$ .
Stability Analysis: To address the limitation of the OA ansatz (which requires specific initializations), the authors establish structural stability results. They prove that for certain parameter regimes, dynamics initialized near the OA manifold exhibit the same long-time behavior as those initialized on the manifold, extending the theoretical results to a broader class of initial conditions.
Numerical Validation: Theoretical findings in $d=2$ are validated through numerical experiments in higher dimensions ( $d=100$ ) for both linear self-attention and unnormalized softmax attention (USA).

Key Contributions

The paper makes four primary contributions:

Pure Second-Harmonic Kuramoto Formulation: The authors show that linear self-attention in $d=2$ is equivalent to a Kuramoto-Daido model with pure second-harmonic coupling for arbitrary matrices $A$ and $V$ . This establishes an intrinsic low-dimensional structure independent of the specific parameter values, distinct from the Wasserstein gradient flow framework.
Diverse Long-Time Behaviors on the OA Manifold: By analyzing the reduced ODE for $R_2(t)$ $R_{2} (t)$ , the authors identify four distinct dynamical regimes based on the properties of $A$ $A$ and $V$ $V$ :
- Clustering with Convergence: Tokens cluster into two fixed antipodal points (e.g., when $V=I$ and $A$ satisfies specific trace/determinant conditions).
- Clustering without Convergence: Tokens form two antipodal clusters, but the cluster locations rotate indefinitely (non-convergent oscillation).
- Oscillations: The system exhibits persistent periodic oscillations without forming stable clusters.
- Bifurcations: The system undergoes saddle-node bifurcations where small changes in $V$ drastically alter the long-term behavior (e.g., transitioning from oscillation to clustering).
- Note: These behaviors include oscillations and bifurcations, which the authors note have not been previously characterized in transformer dynamics.
Structural Stability Beyond the OA Manifold: The authors prove that for specific parameter regimes (e.g., $V=I$ with positive definite symmetric part of $A$ , or $A=I$ with $\lambda_{\max}(V) > 0$ ), the long-time behavior (convergence to antipodal points) is robust. Dynamics initialized near the OA manifold converge to the same state as those on the manifold, provided the initial distribution is sufficiently close to uniform in the bounded-Lipschitz distance.
High-Dimensional Persistence: Numerical experiments suggest that the qualitative behaviors identified in the 2D theoretical analysis (clustering, non-convergent clustering, and oscillations) persist in higher-dimensional settings ( $d=100$ ) for both linear and softmax attention models.

Results

The paper categorizes the long-time behavior of the order parameter $R_2(t)$ (and consequently the token distribution) into three types:

Type 1 (Convergence): $|R_2| \to 1$ and $\arg(R_2) \to \text{const}$ . Tokens converge to two fixed antipodal points.
Type 2 (Rotating Clusters): $|R_2| \to 1$ but $\arg(R_2)$ does not converge. Tokens cluster into two antipodal points that rotate continuously.
Type 3 (Oscillation): Neither $|R_2|$ nor $\arg(R_2)$ converges. The system exhibits persistent oscillations.

Specific findings include:

Case 1 ( $V=I, A$ arbitrary): Clustering occurs if $\text{tr}(A) > 0$ or $\det(A) \le 0$ . If the symmetric part of $A$ is negative definite, the system converges to an interior point (no clustering).
Case 2 ( $A=I, V$ symmetric): Clustering occurs if $\lambda_{\max}(V) \ge 0$ . If $\lambda_{\max}(V) < 0$ , the system converges without clustering.
Case 3 (Non-commuting $A, V$ ): The authors identify a regime where tokens form two antipodal clusters that rotate indefinitely (clustering without convergence).
Case 4 (Hidden Hamiltonian): For specific skew-symmetric-like $V$ , the system exhibits a hidden Hamiltonian structure leading to periodic orbits (oscillations) when $v_{11} < v_{12}$ , and convergence when $v_{11} \ge v_{12}$ .

Significance and Claims

The paper claims to provide the first theoretical characterization linking diverse dynamical behaviors in transformer models (specifically oscillations and bifurcations) to explicit parameter choices of the weight matrices.

Beyond Clustering: The work challenges the prevailing view that transformer dynamics primarily lead to token uniformity or rank collapse. It demonstrates that linear transformers can sustain complex, non-convergent dynamics (oscillations, rotating clusters) depending on $A$ and $V$ .
Analytical Framework: By connecting transformer dynamics to Kuramoto-type models, the authors provide a new analytical framework that is distinct from and complementary to the Wasserstein gradient flow approach. This framework allows for the analysis of general matrices $A$ and $V$ , which were previously intractable under the gradient flow assumptions.
Robustness: The structural stability results suggest that these diverse behaviors are not artifacts of idealized initializations (the OA manifold) but are robust features of the system for a wide range of initial conditions.
Practical Relevance: The authors note that the "clustering without convergence" behavior (rotating clusters) qualitatively resembles cyclic trajectories observed in "Looped Transformers" (recurrently applied LLMs), suggesting potential implications for understanding the dynamics of deep or recurrent transformer applications. They also highlight the sensitivity of these dynamics to parameter perturbations, which may be relevant for model quantization.

The authors remain modest regarding the extension of these results, acknowledging that the rigorous low-dimensional reduction is currently specific to $d=2$ and linear attention, and that extending these techniques to higher dimensions and nonlinear (softmax) kernels remains an open question.