🤖 machine learning

RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways

The paper introduces RoVE, a parameter-free modification that rotates value embeddings alongside keys to make the value pathway position-sensitive, thereby unifying various attention formulations and demonstrating consistent performance improvements over standard RoPE in long-context and in-context learning tasks.

Original authors: Alejandro García-Castellanos, Maurice Weiler, Erik J Bekkers

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alejandro García-Castellanos, Maurice Weiler, Erik J Bekkers

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Blind Messenger"

Imagine a large language model (like the one you are talking to) as a giant team of workers passing notes to each other.

The Query (The Reader): One worker asks a question.
The Key (The Index): Other workers hold up signs to say, "I have the answer you need!"
The Value (The Message): Once the Reader decides who to listen to, those workers pass over their actual notes (the data).

The paper points out a flaw in how modern models (using something called RoPE) handle this.

The Good Part: The model is very smart about who to listen to. It knows that a note from a worker standing 5 steps away is different from a note from someone 50 steps away. It treats distance carefully.
The Bad Part: Once the Reader decides to listen to a worker, the content of the note is treated the same way, no matter how far away that worker is.
- Analogy: Imagine you are listening to a friend tell a story. If they are standing right next to you, you hear their voice clearly. If they are 100 feet away, you still hear the exact same words, just as if they were right next to you. The model fails to realize that the "distance" should change how the message is interpreted, not just who gets to speak.

The Solution: RoVE (Rotary Value Embeddings)

The authors propose a simple, free fix called RoVE.

Instead of just rotating the "signs" (Keys) to show distance, RoVE also rotates the "notes" (Values) before they are read.

The Analogy: Imagine the workers are all holding maps.
- Old Way (RoPE): The Reader knows exactly where the worker is standing. But when the worker hands over their note, the note is written in a standard font, regardless of where the worker is.
- New Way (RoVE): Before the worker hands over the note, they rotate the paper based on how far away they are. If they are far away, the note is tilted slightly. If they are close, it's flat.
- The Result: When the Reader gets the note, the tilt of the paper tells them exactly how to interpret the message relative to their own position. The message itself now "knows" its distance.

Why This Matters: The "Attentive Convolution"

The paper claims this tiny change turns the model's attention mechanism into something called an "Attentive Convolution."

The Metaphor: Think of a standard attention mechanism like a spotlight. It shines on different people, but the light is the same color everywhere.
The RoVE Change: RoVE makes the spotlight change color depending on how far the person is from the center.
The Benefit: This creates a "block-Toeplitz" structure (a fancy math term), which the authors say is the same structure used in convolutions (the math behind how computers see images).
- In simple terms: By doing this, the model starts to process language more like it processes images or physical objects, where position and distance fundamentally change the meaning of the data.

What the Experiments Showed

The authors tested this on two sizes of language models (small and medium) and found:

Better Memory: The models got better at remembering things over long distances (long-context retrieval).
Better Guessing: When asked to solve puzzles with very few examples (few-shot learning), they performed better.
Longer Reach: The models handled text that was much longer than they were trained on without getting confused (out-of-distribution perplexity).

The "Free" Upgrade

The most exciting part of the paper is that RoVE requires no extra training or extra memory.

It doesn't add new "weights" (parameters) to the model.
It doesn't slow the computer down significantly.
It is a "drop-in" replacement. You can swap the old method for RoVE, and the model immediately becomes smarter at handling distance and long stories.

Summary

The paper argues that for a language model to truly understand a story, it shouldn't just know who is speaking; it needs to know how far away the speaker is, and adjust the message accordingly. RoVE is a clever, zero-cost trick that makes the "message" (Value) aware of its distance, turning the model into a more robust listener that excels at long, complex tasks.

Technical Summary: RoVE (Rotary Value Embeddings Attention)

Problem Statement

Standard Rotary Position Embeddings (RoPE) successfully render attention scores position-relative by rotating queries and keys based on their absolute positions. However, the value pathway remains position-blind. In the language of transformer circuits, while the Query-Key (QK) circuit is biased toward relative positions, the Output-Value (OV) circuit treats the value token $x_j$ identically regardless of its distance from the query $x_i$ . Specifically, the channel map $W_V$ applied to a value token carries no information about its relative offset $\delta = j - i$ . This asymmetry limits the model's ability to perform relative-position-sensitive transformations on the values before aggregation, a capability inherent in convolutional operations but absent in standard RoPE attention.

Methodology: RoVE

The authors propose RoVE (Rotary Value Embeddings), a parameter-free modification that extends RoPE to the value pathway. The core mechanism involves rotating the value vectors by the same position-dependent rotation matrices used for keys, but in a manner that aligns them with the query's reference frame before aggregation.

Mathematical Formulation

In standard RoPE, the output at position $i$ is computed as:
$y_i = \sum_{j \in N(i)} A(X)_{ij} W_V x_j$
where $A(X)_{ij}$ are the RoPE attention weights dependent on the relative offset $j-i$ .

RoVE modifies this by rotating the value $x_j$ by $R_j$ (the rotation matrix at position $j$ ) and then inverting the rotation at the query position $i$ by $R_i^{-1}$ :
$\tilde{y}_i = R_i^{-1} \sum_{j \in N(i)} A(X)_{ij} R_j W_V x_j = \sum_{j \in N(i)} A(X)_{ij} \underbrace{R_{j-i} W_V}_{\psi_{j-i}} x_j$

Structural Interpretation

This modification fundamentally changes the nature of the attention operator:

Attentive Convolution: RoVE transforms the attention mechanism into an attentive convolution. The constant value map $W_V$ is replaced by an offset-dependent kernel family $\psi_\delta = R_\delta W_V$ . The relative position $\delta$ now modulates the token transformation (OV circuit) just as it modulates token selection (QK circuit).
Matrix Mixer Structure: In the matrix mixer framework, the standard RoPE mixer has a Kronecker structure ( $A \otimes W_V$ ), decoupling routing from channel projection. RoVE replaces this with a block-Toeplitz structure, where blocks along the same relative-offset diagonal share the same rotated value kernel, scaled by the attention weight.
Frame-Change Perspective: The operation can be viewed as a frame change: values are transformed from their local frames into a shared global frame via $R_j$ , aggregated, and then transformed back into the query's local frame via $R_i^{-1}$ .

Efficiency

RoVE introduces no additional learnable parameters. The rotation operations act independently on $n$ tokens and $d^2$ channel pairs, resulting in linear complexity $O(nd)$. This is negligible compared to the $O(nd^2)$ linear layers and $O(n^2d)$ attention layers. Furthermore, RoVE is compatible with efficient attention kernels like FlashAttention, as the rotations occur before and after the kernel call.

Key Contributions

Structural Characterization: The paper provides a theoretical analysis showing that RoVE unifies the QK and OV circuits by making the value pathway relative-position sensitive. It demonstrates that this turns RoPE attention into an attentive convolution, endowing the model with a block-Toeplitz mixer structure.
Empirical Validation: The authors trained GPT-2 style models (124M and 354M parameters) on the FineWebEdu-10B dataset. They evaluated RoVE as a drop-in replacement for the value pathway in standard (non-shared-KV) language models.
Complementarity with YaRN: The study investigates the interaction between RoVE and YaRN (a frequency interpolation method for long-context extrapolation).

Experimental Results

The evaluation focused on few-shot in-context learning (ICL), out-of-distribution (OOD) perplexity, and long-context retrieval (RULER benchmark).

In-Distribution Performance: RoVE consistently improved Core ICL accuracy and in-context perplexity within the training context (1024 tokens) compared to standard RoPE. This indicates that the relative-position bias is beneficial even without extrapolation.
Out-of-Distribution (OOD) Perplexity: RoVE substantially reduced perplexity at context lengths up to 16k tokens. When combined with YaRN, the improvements were additive, suggesting the two methods address complementary aspects of long-context generalization.
Long-Context Retrieval: On the RULER benchmark (4k/8k tokens), RoVE showed significant gains, particularly in tasks requiring information maintenance and recombination across the context (e.g., Variable Tracking, Needle-in-a-Haystack). The improvements were most pronounced when RoVE was combined with YaRN.

Significance and Claims

The paper claims that RoVE offers a robust structural bias for Large Language Models (LLMs) by correcting a structural asymmetry in RoPE attention. By making the value pathway position-sensitive, RoVE unifies independent formulations of similar operations found in computer vision, robotics, and recent LLM architectures (e.g., DeepSeek-V4) under a single theoretical framework: attentive convolution.

The authors emphasize that RoVE is a parameter-free, efficient modification that improves performance across model scales and tasks, particularly those requiring long-range aggregation. The results suggest that relative-position awareness in the value stream is a critical component for improving long-context robustness and retrieval capabilities in transformer architectures.