💬 NLP

Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling

This technical survey provides a unified framework for understanding position encoding methods in Transformers—from absolute and relative approaches to Rotary Position Embeddings (RoPE)—while analyzing their computational properties, long-context scaling techniques, and the critical distinction between length extrapolation capability and reliable long-context generalization.

Original authors: Jiguo Li

Published 2026-08-12

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiguo Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a story. You have a super-fast brain that can look at all the words in a sentence at the same time, rather than reading them one by one like a human. This is the magic of the Transformer, the engine behind modern AI. But there's a catch: because this brain sees everything at once, it has no idea which word came first, second, or last. If you scrambled the words in a sentence, this brain would think the scrambled version was just as good as the original. To fix this, we have to give the AI a way to know "where" each word sits in the line. We call this Position Encoding. Think of it like giving every word in a sentence a unique name tag or a seat number. Without these tags, the AI is like a guest at a party who can see everyone's face but has no idea who is standing next to whom, or who arrived first.

This paper is a deep dive into the different "name tag" systems we've invented for AI over the years. It traces the history from simple seat numbers to complex, rotating clocks, and finally to a new generation of tricks that let AI read books that are hundreds of thousands of words long without getting confused. The author isn't just listing these methods; they are figuring out which ones actually work when we try to stretch them to handle massive stories, and they warn us that just because a system can accept a long input doesn't mean it actually understands it.

The Evolution of the AI's "Seat Number"

In the beginning, the simplest way to give a word a seat number was to just look it up in a dictionary. This is called Learned Absolute Position. Imagine a teacher handing out a list where "Word 1" gets a specific color, "Word 2" gets another, and so on. It works great for short stories, but if you try to read a novel and the teacher only made a list for the first 500 words, the AI gets lost when it hits word 501.

Then came Sinusoidal Encoding. Instead of a static list, the AI uses a mathematical wave pattern (like a sine wave) to generate seat numbers. Think of this as a set of clocks spinning at different speeds. Some clocks tick fast to tell the difference between word 1 and word 2, while others tick slowly to tell the difference between word 1 and word 1,000. This is clever because the AI can calculate a seat number for any position, even ones it hasn't seen before. However, the paper points out that just because you can calculate the number doesn't mean the AI knows how to use it for very long distances.

Next, researchers realized that in language, what matters most is often the distance between words, not their absolute seat numbers. Is the word "it" close to the noun it refers to? This led to Relative Position Encoding. Instead of saying "I am at seat 50," the AI learns to say, "I am 3 seats away from you." This is like a game of musical chairs where you only care about who is sitting next to you, not the exact number on your chair. Methods like T5 and Transformer-XL use this to handle long texts by focusing on the gap between words.

The Star of the Show: RoPE

The paper highlights a third-generation method called RoPE (Rotary Position Embedding) as the current champion for many large AI models. Instead of adding a number to the word or calculating a distance, RoPE treats the word's data like a spinning arrow. As the word moves down the line, its arrow rotates. When the AI compares two words, it checks the angle between their arrows. If the arrows are close together, the words are close in the story; if they are far apart, the words are far apart.

The beauty of RoPE is that it naturally handles the "distance" between words without needing a separate lookup table. It's like if everyone at the party wore a watch that spun at a speed based on when they arrived. When two people talk, they just look at the difference in their watch hands to know how long they've been apart.

The Long-Context Challenge: Stretching the Story

The biggest problem with these systems is that they are usually trained on short stories (like 4,000 words). When we try to use them for massive documents (like 128,000 words or more), the "clocks" or "spinning arrows" start to spin into patterns the AI has never seen before. It's like trying to drive a car designed for city streets onto a highway; the engine might run, but the handling feels wrong.

The paper surveys several "stretching" techniques to fix this:

Position Interpolation (PI): This squishes the long story back into the short space the AI knows. It's like compressing a 10-hour movie into 2 hours. The AI can watch it, but the details get blurry, and it might miss the subtle differences between words that are close together.
NTK-aware Scaling: This is smarter. It keeps the "fast clocks" (which handle close details) spinning at their normal speed but slows down the "slow clocks" to cover the long distance. It's a compromise that keeps the local details sharp while reaching further out.
Dynamic NTK: This method changes the stretching factor depending on how long the current story is. If the story is short, it doesn't stretch at all. If it's long, it stretches. However, the paper warns this can be tricky to manage in real-time because it might confuse the AI's memory of past words.
LongRoPE and LongRoPE2: These are the most advanced methods. Instead of using a single rule for everyone, they search for the perfect stretching factor for each part of the AI's brain. They even mix short and long stories during training so the AI learns to handle both. The author suggests this is the most promising path, but it requires careful tuning and specific data.

The Big Warning: Just Because It Fits, Doesn't Mean It Works

The most important finding in this paper is a reality check. Just because an AI model can accept a 100,000-word input doesn't mean it can understand it. The author argues that many models fail the "Needle in a Haystack" test (finding a specific fact in a huge text) or struggle with reasoning over long distances, even if they have the fancy "long-context" settings turned on.

They emphasize that effective context length (how much the AI actually uses) is often much shorter than the nominal context length (the maximum number of words it can technically hold). They also point out that simply changing the math to handle longer numbers isn't enough; the AI needs to be retrained or fine-tuned with these new settings to actually learn how to use them.

The Verdict

The paper concludes that while we have made incredible progress from simple seat numbers to complex rotating arrows, we haven't solved the problem of long-context understanding yet. There is no single "magic bullet." The best approach seems to be a mix of smart scaling (like LongRoPE), careful training with both short and long examples, and rigorous testing to see if the AI is actually paying attention to the whole story, not just the beginning and the end.

For anyone building or using these AI models, the takeaway is clear: Don't just trust the marketing numbers. If you want an AI to read a whole novel, you need to check if it can actually find the plot twists in the middle, not just if it can swallow the whole book in one bite. The journey to truly long-context AI is still ongoing, and the road ahead requires more than just bigger numbers; it requires smarter ways to keep the AI's attention focused.

Technical Summary: Position Encoding in Transformers

Problem Statement

Self-attention mechanisms in Transformers enable content-dependent interactions between tokens but lack an inherent mechanism to encode token order. Without explicit positional information, the model is equivariant to input permutations; shuffling the input sequence results in a correspondingly shuffled output, rendering the model unable to distinguish sequence structure. While early solutions injected position information into input representations or attention scores, extending these mechanisms to long contexts (beyond the training length) presents significant challenges. Specifically, simple extrapolation of position encodings often leads to out-of-distribution (OOD) phase shifts, attention score degradation, and a loss of short-context performance. Furthermore, there is a lack of unified understanding regarding how different encoding strategies (absolute, relative, and rotational) interact with KV caching, length extrapolation, and specific architectural choices like Multi-Query Attention (MLA).

Methodology and Technical Evolution

The paper provides a unified technical survey and derivation of position encoding methods, categorizing them into three generations:

1. Absolute Position Encoding

Learned Embeddings: Tokens are assigned learnable vectors from a parameter table ( $P \in \mathbb{R}^{L_{max} \times d}$ ). This is simple but bound to a maximum length and lacks structural constraints on adjacent positions.
Sinusoidal Encoding: Uses sine and cosine functions with varying frequencies to generate position vectors. While the form is absolute, the geometric properties (sum of angles) imply relative displacement. However, the model must learn to exploit this structure, and it does not guarantee reliable long-context generalization.

2. Relative Position Encoding

Shaw et al.: Adds learnable relative distance vectors to Keys and Values.
Transformer-XL: Decomposes attention into content-content, content-position, and global offsets to support segment-level memory.
T5: Compresses relative distances into scalar biases learned per attention head and distance bucket.
Limitations: These methods often increase computational complexity due to $n \times n$ pairwise calculations and complicate operator fusion and KV caching.

3. Rotary Position Embeddings (RoPE)

Mechanism: RoPE rotates Query and Key vectors based on their absolute positions. Crucially, the dot product of rotated vectors depends only on the relative position ( $n-m$ ), effectively encoding relative distance while retaining the standard $QK^\top$ form.
Derivation: Based on 2D rotation matrices or complex number multiplication ( $z' = z e^{i m \theta}$ ).
Properties: It does not guarantee monotonic attention decay with distance (scores oscillate periodically), but it allows for explicit relative distance representation within the attention score.

4. Long-Context Extensions (RoPE Scaling)

The paper details methods to extend RoPE beyond training lengths, addressing the "untrained phases" problem where low-frequency dimensions enter unseen phase intervals:

Position Interpolation (PI): Uniformly compresses position indices to fit the target length into the training range. This reduces local resolution.
NTK-aware: Modifies the RoPE base ( $\beta$ ) to scale frequencies non-uniformly, preserving high-frequency local structure while expanding low-frequency coverage.
Dynamic NTK: Adjusts the scaling factor dynamically based on the current sequence length to avoid compressing short inputs.
NTK-by-parts & YaRN: Applies different interpolation strategies based on frequency intervals (rotations within training length) and includes attention temperature calibration to manage entropy changes.
LongRoPE & LongRoPE2: Uses evolutionary search to find dimension-wise scaling factors and position-dependent retention intervals. LongRoPE2 introduces "effective training periods" and hybrid training (mixing short and long contexts) to optimize both long-context extension and short-context retention.

5. Alternative: ALiBi

ALiBi adds a linear distance penalty ( $-m_h|i-j|$ ) to attention logits. It requires no position table and injects a recency bias, though it lacks the multi-frequency phase structure of RoPE.

Key Contributions

Unified Derivation: The paper derives how RoPE converts absolute position indices into relative phase differences in Query-Key inner products, clarifying the mathematical relationship between absolute and relative encoding.
Systematic Comparison: It compares methods based on injection points (input, logits, or geometric transformation), computational costs, KV cache compatibility, and length extrapolation capabilities.
Scaling Law Analysis: It formalizes the relationship between the RoPE base, training length, and extrapolation range, defining critical dimensions where phase coverage becomes insufficient.
Implementation Guidelines: The report emphasizes that position configuration (e.g., rope_theta, pairing conventions, scaling factors) must be bound to the checkpoint and that "nominal length" does not equal "effective length."
Evaluation Protocols: It advocates for evaluation beyond simple input acceptance, proposing metrics such as short-context retention rates, position-wise perplexity, and retrieval/reasoning tasks (e.g., needle-in-a-haystack, code definition-use spans).

Results and Findings

Extrapolation vs. Generalization: The ability to compute positional features beyond training length does not imply reliable long-context generalization. Successful extension requires adaptation of model weights to new phase distributions.
Trade-offs: Uniform scaling (PI) preserves stability but degrades local resolution. Non-uniform scaling (NTK, LongRoPE) better preserves local structure but requires careful tuning or search.
KV Cache Consistency: Dynamic scaling methods (like Dynamic NTK) conflict with standard KV caching if the rotation angle of historical tokens changes; strict implementations require re-rotating cached keys or fixing magnification per session.
Model Diversity: While RoPE has become dominant in decoder-only LLMs (e.g., LLaMA, PaLM, Gemma), significant variations exist. Some models (BLOOM, Falcon) use ALiBi, T5 uses relative biases, and newer architectures (DeepSeek, GLM-5, Kimi K3) employ decoupled RoPE/NoPE subspaces or implicit position mechanisms (KDA) to support MLA or sparse attention.

Significance and Conclusion

The paper concludes that the field has moved from simple coordinate injection to sophisticated frequency scaling and hybrid training strategies. However, several challenges remain unresolved:

Effective Context: Accepting long inputs does not guarantee reliable utilization of long-range dependencies.
Phase Aliasing: Periodic functions can cause different distances to produce similar phases, a problem not fully eliminated by multi-frequency combinations.
Short-Context Conflict: Interpolation and scaling often degrade short-range capabilities, and stable rules for cross-model transfer are lacking.
System Semantics: Engineering implementations (packing, sliding windows, speculative decoding) can inadvertently break position semantics if not carefully aligned.

The author asserts that future research must move beyond merely amplifying nominal context lengths. Instead, the goal should be to establish position mechanisms that are predictable, trainable, cacheable, and capable of expressing complex, multi-dimensional structures (such as code call graphs or multimodal spatial relationships), evaluated through real long-dependency tasks rather than synthetic retrieval benchmarks.