Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks

This paper introduces Jordan-RoPE, a non-semisimple relative positional encoding that leverages complex Jordan blocks to generate oscillatory-polynomial features for modeling distance-modulated phase interactions, demonstrating structural advantages and specific performance gains over standard RoPE and ALiBi baselines in certain contexts.

The Gist

Imagine you are trying to understand a story where the order of events matters. In a computer model called a Transformer, the "attention" mechanism is like a reader deciding which previous words in a sentence are important for understanding the current word.

To do this, the model needs to know how far apart two words are. If the model just looks at the words themselves, it doesn't know if Word A came right before Word B or 100 words before. This is where Positional Encoding comes in—it's the "ruler" the model uses to measure distance.

The Problem: The Old Rulers

The paper looks at two popular ways models currently measure distance:

1. RoPE (Rotary Positional Encoding): Think of this like a spinning top. It rotates the meaning of words based on their position. It's great at handling the rhythm or phase of a sentence (like the beat in a song), but it treats distance as a simple rotation.
2. ALiBi: Think of this like a straight line. It adds a simple penalty for being far away. It's good at saying "closer is better," but it doesn't capture the complex, wavy patterns of language.

Most models use these two separately, like having a ruler for rotation and a separate ruler for distance. They don't mix them together in a single, unified tool.

The New Idea: Jordan-RoPE

The author, Yaobo Zhang, asks: What if we could combine the spinning top and the distance ruler into one single, more complex tool?

In mathematics, there is a concept called a Jordan Block. Usually, math tools are "nice" and separate (like the spinning top and the ruler being distinct). But a "defective" or "non-semisimple" Jordan Block is a tool where the parts are glued together in a way that creates something new.

The Creative Analogy: The Wobbly Spinning Top
Imagine a spinning top (the rotation) that is slightly unbalanced. As it spins, it doesn't just rotate; it also wobbles.

• The spin represents the rhythm of the language (the phase).
• The wobble represents the distance.
• In the new Jordan-RoPE, the wobble gets bigger the further you go. It's not just a simple spin or a simple distance; it's a distance-modulated spin.

Mathematically, this creates a feature that looks like:

Distance × (Spin × Cosine + Spin × Sine)

Instead of just knowing "it's 5 steps away" or "it's at a 90-degree angle," the model now sees "it's 5 steps away and the angle is shifting because of that distance." It captures a specific type of pattern where the rhythm of the sentence changes depending on how far back you look.

How They Tested It

The author didn't just build this tool; they tested if it actually helps in specific situations.

1. The "Synthetic" Test: They created a fake language task where the answer strictly depended on this "distance-modulated spin" pattern (like a secret code where the message changes based on how far back you read).

   • Result: The new tool (Jordan-RoPE) solved this puzzle much better than the old tools (RoPE or ALiBi). It was the only one that naturally understood the "wobbly spin" pattern.

2. The "Real World" Test: They tried it on a small language model trained on Wikipedia text (WikiText-103).

   • Result: It did better than the standard RoPE tool, but it didn't beat the "champion" combination of RoPE + ALiBi.
   • The Catch: The paper is careful to say this isn't a magic bullet for all language. In real human language, the "wobble" might not always be the most important thing. The tool is most useful when the task specifically requires that complex, distance-dependent rhythm.

The "Stabilized" Version

There was a problem: in the pure math version, the "wobble" (the nilpotent part) grows infinitely large as the distance increases, which can break the computer's math.

• The Fix: They created a "Stabilized" version that puts a cap on the wobble. It's like putting a governor on the spinning top so it wobbles a lot, but never spins out of control. This version worked very well in the tests.

The Bottom Line

This paper introduces Jordan-RoPE, a new way to measure distance in AI that combines rotation and distance into a single, "glued-together" mathematical structure.

• What it does: It allows the AI to see patterns where the rhythm of the text changes based on distance.
• When it works best: When the task involves complex, distance-dependent oscillations (like the synthetic test).
• What it doesn't do: It doesn't claim to be the absolute best tool for every single language task. In fact, the standard "RoPE + ALiBi" combo is still stronger for general text.

Think of it as a specialized wrench. If you have a bolt that requires a specific "wobbly spin" to loosen, this wrench is perfect. But if you just need to turn a standard screw, your old tools might still be the best choice. The paper proves that this specialized wrench exists, works as intended, and is useful for specific, complex jobs.

Technical Summary

Technical Summary: Jordan-RoPE

Problem Statement
Relative positional encodings (RPE) define the primitive functions of the query-key lag available to attention mechanisms. While successful mechanisms like RoPE (rotary phase) and ALiBi (additive distance bias) are well-understood through group-theoretic classifications of linear, translation-invariant operators, they typically rely on semisimple (diagonalizable) generators. This leaves the non-semisimple corner of the classification underexplored. Specifically, standard approaches treat phase (rotary) and distance (polynomial/shear) features as separate channels or additive biases. The paper investigates whether coupling a complex rotary eigenvalue with a nilpotent response within a single defective Jordan block yields new primitive relative-position features that are structurally distinct from simple direct sums.

Methodology
The authors propose Jordan-RoPE, a construction that embeds the rotary complex eigenvalue and a nilpotent response into a single order-two complex Jordan block.

1. Algebraic Formulation:
   The generator is defined as $J_{\gamma, \omega, \eta} = (-\gamma + i\omega)I + \eta N$, where $N$ is a nilpotent matrix ($N^2=0$). The resulting relative operator for causal lag $d = i-j \ge 0$ is:
   $$G_{exact}(d) = \exp(d J) = e^{(-\gamma + i\omega)d} (I + \eta d N)$$
   This generates a basis of oscillatory-polynomial features:
   $$e^{-\gamma d} \cos(\omega d), \quad e^{-\gamma d} \sin(\omega d), \quad d e^{-\gamma d} \cos(\omega d), \quad d e^{-\gamma d} \sin(\omega d)$$
   Crucially, the nilpotent channel supplies the frequency-tangent feature $d e^{i\omega d}$, coupling distance and phase directly rather than adding them separately.

2. Contragredient Query Action:
   Since the Jordan block is non-orthogonal, applying the same transform to queries and keys does not yield a pure relative operator ($G(i)^\top G(j) \neq G(j-i)$). To recover the correct relative score, the authors formulate a contragredient query action: queries are transformed by the inverse transpose of the position-dependent matrix, while keys use the primal transform. This ensures the attention score depends strictly on the lag $d$.

3. Stabilization:
   The exact nilpotent term grows linearly with $d$, which is problematic for long contexts. The authors introduce Stabilized Jordan-RoPE, replacing $d$ with a bounded shear function $\tau(d) = d / (1 + d/L)$. While this breaks the exact one-parameter group law, it preserves the local Jordan response and prevents unbounded growth. A Scaled-exact variant is also proposed, which preserves the group law by normalizing the shear magnitude by the context length $L$.

Key Contributions

• Structural Identification: The paper identifies the order-two complex Jordan block as the minimal non-semisimple extension of rotary RPE where phase and nilpotent response are coupled in a single defective representation, rather than separated into subspaces.
• Primitive Basis: It demonstrates that this construction directly provides the primitive logit basis $d e^{i\omega d}$ (and its real components $d \cos(\omega d), d \sin(\omega d)$), realizing a "distance-modulated phase" basis at the pre-softmax level.
• Implementation: It provides the real block implementation and the necessary contragredient query action for non-orthogonal maps.
• Distinction from Baselines: It separates the exact representation from stabilized implementations, clarifying that bounded shear improves numerical behavior but sacrifices the exact group law.

Experimental Results
The evaluation focuses on structural evidence rather than broad performance claims, using three types of tests:

1. Kernel-Level Probes: On a mixed target $y(d) = (d/L)\cos(\omega d)$, the Exact/raw Jordan basis achieves the lowest Mean Squared Error (MSE), significantly outperforming RoPE, ALiBi, and Direct-sum baselines. This confirms the basis directly matches the target's coupled structure.
2. Synthetic Language Model: In a task requiring the model to learn a distance-modulated phase rule ($K(d) = (d/L)\cos(\omega d)$), Stabilized Jordan-RoPE achieves 0.906 accuracy at length 8192, outperforming RoPE (0.781) and Direct-sum (0.500). This suggests Transformers can utilize the coupled mode when the task rewards it.
3. Natural Language (WikiText-103): On a small byte-level language model, Scaled-exact Jordan-RoPE ($c=1$) achieves the lowest mean loss within the Jordan family (1.869) and is competitive with Damped RoPE (1.884). However, RoPE+ALiBi remains the strongest overall (1.796). The authors note that larger forced initial shear ($\eta$) worsens long-length loss in this setting, suggesting natural language tasks primarily reward damping and recency bias rather than strong oscillatory-polynomial shear.

Significance and Claims
The paper makes modest, structural claims rather than asserting a new state-of-the-art positional encoding:

• Structural Extension: Complex Jordan blocks provide a controlled, non-semisimple extension of rotary RPE.
• Conditional Utility: The coupled Jordan basis is useful specifically when the target kernel rewards distance-modulated phase interactions (e.g., $d \cdot \text{phase}$).
• Limitations: The authors explicitly state they do not claim nilpotent mechanisms are new, nor that the Jordan family dominates existing encodings on general natural language modeling. The evidence is that the construction offers a specific primitive basis ($d e^{i\omega d}$) that is distinct from the direct sum of phase and distance channels.

In summary, Jordan-RoPE offers a mathematically rigorous way to couple distance and phase within a single attention mechanism, proving effective in synthetic tasks requiring such coupling, while showing that natural language tasks may still prefer simpler, decoupled or additive biases.