📊 statistics

RotRNN: Modelling Long Sequences with Rotations

This paper introduces RotRNN, a linear recurrent neural network that leverages the properties of rotation matrices to offer a simple, efficient, and robustly normalised alternative to complex state-of-the-art models for long sequence modelling, achieving competitive performance on relevant benchmarks.

Original authors: Kai Biegun, Rares Dolga, Jake Cunningham, David Barber

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kai Biegun, Rares Dolga, Jake Cunningham, David Barber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to remember a very long story, like a novel or a complex movie plot. Your brain (or a computer model) needs to hold onto the beginning of the story while reading the end, without getting overwhelmed or forgetting the details.

For a long time, computers struggled with this. Standard models either got "tired" and forgot the start (the vanishing gradient problem) or got "scrambled" and exploded into chaos (the exploding gradient problem).

Recently, a new type of computer model called a Linear Recurrent Neural Network (like S4 or LRU) became very popular because it's great at remembering long sequences. However, these models are a bit like high-performance race cars that are incredibly difficult to tune. They require very specific, complicated "initial settings" (initialization) to work, and even then, it's not always clear why they work so well. Sometimes, the math behind them doesn't quite match how they are actually built in code.

Enter RotRNN, the new model proposed in this paper.

The Core Idea: The Spinning Top

The authors of this paper asked: "What if we built our memory system using rotation?"

Think of a rotation matrix as a perfect spinning top.

If you spin a top, it stays upright and stable. It doesn't grow larger or smaller; it just turns.
In math, a rotation matrix has a special property: it preserves the "size" (or norm) of whatever it touches. If you feed a number into a rotation, the output is the same size, just turned in a different direction.

The authors realized that if they built their memory system entirely out of these "spinning tops," the system would naturally stay stable. It wouldn't accidentally grow too big (explode) or shrink to nothing (vanish).

How It Works (The Simple Version)

The Problem with Previous Models:
Imagine trying to keep a stack of papers balanced. Previous models (like LRU) tried to balance the stack by constantly adjusting the weight of the papers based on complex math rules. Sometimes the stack wobbled, and the "weight" (the hidden state) would get too heavy or too light, making the model unstable.
The RotRNN Solution:
Instead of balancing weights, RotRNN just spins the information.
- The Spin: Every time the model reads a new piece of data, it rotates the memory slightly.
- The Stability: Because it's a pure rotation, the "size" of the memory stays exactly the same. It's like a dancer spinning in place; no matter how many times they spin, they don't get taller or shorter.
- The Decay: To make sure the model doesn't remember everything forever (which is also bad), they add a gentle "brake" (a decay factor). This lets the model slowly fade out old memories while keeping the current ones stable.

Why Is This Better?

No Complicated Setup: Old models needed a very specific, math-heavy starting point to work. RotRNN is like a toy that works right out of the box. You don't need to fiddle with complex settings; the math of the rotation keeps it stable automatically.
It Does What It Says on the Tin: Sometimes, computer models are built one way in theory but work differently in practice. RotRNN is "faithful" to its theory. The code does exactly what the math says it should do.
The "Multi-Head" Trick: To handle different types of memories (some short, some long), the model uses multiple "heads" (like having several spinning tops working at once). Each top spins at its own speed, allowing the model to pay attention to both recent events and things that happened a long time ago.

The Results

The authors tested RotRNN on several difficult tasks, like reading long documents, classifying text, and recognizing spoken words.

Performance: It performed just as well as the best existing models (the "state-of-the-art").
Stability: When they looked at the "size" of the memory inside the model during training, RotRNN stayed perfectly steady. In contrast, the previous popular model (LRU) had memory sizes that jumped around wildly, taking a long time to settle down.

The Bottom Line

The paper introduces RotRNN, a new way for computers to remember long sequences. Instead of using complex, fragile balancing acts, it uses the simple, stable physics of rotation. It's easier to build, more stable to run, and just as good at the job as the most advanced models currently available. It's a reminder that sometimes, the most elegant solution is just to keep things spinning.

Technical Summary: RotRNN – Modelling Long Sequences with Rotations

Problem Statement
Long sequence modelling remains a challenging domain in machine learning due to the difficulty of capturing long-range dependencies. While Transformers suffer from quadratic computational scaling and poor inductive biases, and non-linear RNNs struggle with vanishing and exploding gradients, recent State Space Models (SSMs) and Linear Recurrent Units (LRUs) have achieved state-of-the-art performance. However, these models exhibit significant drawbacks:

Complex Initialisation: SSMs like S4 rely on intricate theoretical derivations (e.g., HiPPO initialisation based on optimal polynomial projections) that are not well understood during the training process.
Implementation-Theory Mismatch: The LRU, while conceptually simpler, does not fully reflect its theoretical motivation in practice. Specifically, it fails to enforce conjugate pairs for eigenvalues of real matrices, lacks constraints to ensure real-valued outputs, and relies on a learned normaliser that does not guarantee consistent hidden state magnitudes throughout training.
Stability: Existing methods often require complex normalisation schemes or initialisation procedures that may not be robust across different depths and random seeds.

Methodology
The authors propose RotRNN (Rotational Recurrent Neural Network), a linear recurrent model that parametrises the recurrent state matrix $A$ as a rotation matrix. The methodology is built on three core pillars:

Parametrisation via Rotation Matrices:
Instead of learning a constrained matrix directly, RotRNN learns a general weight matrix $M$ and maps it to the special orthogonal group $SO(N)$ using the matrix exponential of a skew-symmetric matrix ( $A = \exp(M - M^\top)$ ). This ensures $A$ remains a valid rotation matrix throughout training.
To enable efficient computation of matrix powers (required for the convolutional form of linear recurrence), the authors decompose the rotation matrix $A$ into a block-diagonal form: $A = P \Theta P^\top$ . Here, $P$ is an orthogonal matrix and $\Theta$ consists of $2\times2$ rotation blocks defined by angles $\theta_i$ . This structure allows for fast computation of $A^k$ without expensive dense matrix operations.
Principled Normalisation:
Leveraging the orthogonality of rotation matrices (where eigenvalues lie on the unit circle), the authors derive a normalisation procedure that maintains a constant expected hidden state norm ( $E[\|x_t\|^2] = c$ ) throughout training.
The recurrence is defined as $x_t = \gamma A x_{t-1} + \xi B u_t$ , where $\gamma$ is a learnable scalar decay factor and $\xi$ is a rescaling coefficient derived to satisfy the constant norm condition. Unlike the LRU, which normalises based on infinite sequence limits, RotRNN's normalisation holds for all timesteps, preventing the exponential decay of early inputs often seen when normalisation constants are raised to the power of $t$ .
Multi-Head Architecture:
To address the limitation of a single scalar decay factor $\gamma$ (which may not generalize well to tasks requiring different memory horizons), RotRNN employs $H$ independent low-dimensional heads. Each head operates with its own rotation matrix $A^{(h)}$ , input matrix $B^{(h)}$ , and decay factor $\gamma^{(h)}$ . The outputs are concatenated and linearly mixed, allowing the model to capture dependencies at multiple scales.

Key Contributions

Novel Linear RNN Layer: The proposal of a recurrent layer where the state matrix is parametrised as a rotation, ensuring the algorithm remains faithful to its theoretical motivation throughout training.
Efficient Computation: A method for computing efficient matrix powers of parametric rotations by decomposing them into block-diagonal forms, enabling fast linear recurrence.
Robust Normalisation: A principled derivation of a normalisation procedure that retains a constant expected hidden state magnitude, offering stronger stability guarantees than prior works that rely on asymptotic convergence.
Theoretical Insight: A derivation showing the mathematical equivalence between a special case of the LRU (where eigenvalues form conjugate pairs) and the multi-head RotRNN, potentially shedding light on the internal mechanics of the LRU.

Results
The authors evaluated RotRNN on several long sequence modelling benchmarks:

Long Range Arena (LRA): RotRNN demonstrated competitive performance against state-of-the-art linear recurrent models (S4, S5, LRU). It achieved the highest scores among baselines on discrete input tasks such as ListOps, Text (IMDB), and Retrieval. However, it performed slightly below some baselines on pixel-level image tasks (Path-X, CIFAR).
Speech Commands: On the 35-way Speech Commands classification task, RotRNN performed identically to the reproduced LRU (95.2% accuracy) with a similar parameter count, remaining competitive with more complex deep state space models.
Stability Analysis: Empirical analysis of hidden state norms on the ListOps dataset revealed that RotRNN maintains an almost constant magnitude across training, layer depth, and random seeds. In contrast, the LRU exhibited wild variations in hidden state norms and took significantly longer to converge to a stable magnitude.

Significance and Claims
The paper claims that RotRNN offers a conceptually simple, efficient, and mathematically principled alternative to existing linear recurrent models. Its primary significance lies in bridging the gap between theoretical motivation and practical implementation. By using rotation matrices, RotRNN avoids the complex initialisation schemes of SSMs and the implementation inconsistencies of the LRU. The authors assert that their model provides a robust normalisation procedure that ensures stability without relying on "black box" initialisation, while achieving competitive performance on standard benchmarks. Furthermore, the work suggests that viewing the LRU through the lens of rotation-based block structures may help clarify the mechanics of linear recurrence in prior models.