🤖 AI

Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

This paper proposes an encoder architecture that explicitly disentangles semantic, absolute, and relative positional information into separate streams, revealing that isolated positional subspaces naturally form low-dimensional manifolds capturing document structure and demonstrating that this approach significantly improves linguistic representation compared to standard entangled positional encodings.

Original authors: Pierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Transformer (the brain behind modern AI text models) as a busy, high-speed train station. In this station, every word (token) on a train needs two things to do its job correctly:

What it is: The meaning of the word (e.g., "dog," "run," "blue").
Where it is: Its position in the sentence and the document (e.g., "first word," "middle of paragraph 2").

The Problem: The "Messy Backpack"

In most current AI models, these two pieces of information are stuffed into the same "backpack" (the same set of numbers) for every word. The paper argues that this is messy.

When the model tries to predict the next word (a task called "Masked Language Modeling"), it gets so focused on the meaning that it accidentally kicks the position information out of the backpack, especially in the final layers of the network. It's like a student trying to study for a history test while carrying a heavy backpack full of maps; eventually, they drop the maps to make room for the history notes.

Furthermore, newer models that try to calculate position only when looking at other words (Relative Positional Encoding) often forget the "big picture" structure of the document, like where paragraphs begin and end.

The Solution: "Give It Space!"

The authors built a new type of model called DSTG-NeoBERT. Instead of one messy backpack, they gave the model three separate streams of information:

The Semantic Stream (Blue): Carries the meaning of the words.
The Absolute Position Stream (Red): Carries the "where am I in the whole document" info.
The Relative Position Stream (Green): Carries the "where am I relative to my neighbor" info.

Crucially, they told the model: "When you are trying to guess the next word (the test), only look at the Blue (meaning) stream. Leave the Red (position) stream alone." This prevents the model from accidentally deleting the position info to make room for the answer.

What They Discovered (The "Aha!" Moments)

1. The Position Map Collapses into a Simple Line
When they looked at the "Red" stream (Absolute Position), they expected it to be a complex, 48-dimensional mess. Instead, it spontaneously organized itself into a simple, 2-dimensional map.

Analogy: Imagine a chaotic crowd of people. Suddenly, they all line up in a perfect, low-frequency wave. The model realized it didn't need a complex 3D map to know where it is in a document; a simple, smooth wave was enough to capture the entire structure of the text.

2. The Train Station Staff Specialized
In the old models, every "attention head" (a worker in the station) tried to do everything. In the new model, the workers split into two distinct teams:

The Structure Team: These workers only look at the "Red" stream. They know where sentences and paragraphs start and end.
The Meaning Team: These workers only look at the "Blue" stream. They understand the words.
The Result: They stopped stepping on each other's toes. The "Relative Position" info (Green) became a helpful tool only for the Meaning Team, helping them focus on the right words nearby.

3. The "Big Picture" Was Saved
In standard models, the "big picture" structure (like knowing you are in the second paragraph of a long story) gets lost by the time the model reaches the final layer because the pressure to predict words overwrites it.

The Fix: Because the new model keeps the position info in a separate, protected stream that isn't used for guessing words, the "big picture" structure remains intact all the way to the end.

The Results

When they tested this new model against standard ones:

Standard Tests: It performed just as well as the best existing models on general language tasks (like answering questions or understanding sentences).
Deep Understanding: When they tested how well the model understood specific linguistic details (like grammar, logic, or discourse), the new model won in 49 out of 65 categories.
Why? Because the model didn't have to sacrifice its understanding of where it was in the text to understand what the text meant.

Summary

The paper shows that by giving "Position" and "Meaning" their own dedicated lanes on the highway, the AI doesn't crash. The position information organizes itself into a simple, efficient map, and the model becomes better at understanding the structure of long documents without losing its ability to understand the words.

Technical Summary: Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

Problem Statement

Positional Encoding (PE) is fundamental to how permutation-invariant Transformers represent sequence order, yet the internal mechanisms of how positional information is processed and stored remain poorly understood. Modern approaches, particularly Relative Positional Encoding (RPE) methods like Rotary Positional Encoding (RoPE), often struggle with long-context understanding and retrieval tasks. Furthermore, existing RPE methods rely on heuristics (e.g., long-term decay) that may create positional biases and bottleneck performance. While evidence suggests that positional and semantic signals occupy nearly orthogonal subspaces in trained Transformers, standard architectures entangle these signals, making it difficult to isolate their specific roles or optimize them independently. The authors posit that explicitly separating these streams could provide mechanistic insights and lead to better positional representations.

Methodology

The authors propose a novel architecture, DSTG-NeoBERT, which explicitly disentangles three information streams: Semantic, Absolute Positional (AP), and Relative Positional (RP).

Architectural Modifications

Based on the NeoBERT architecture, the model processes these streams through distinct modifications:

Input Representations: Each token is represented by two separate embeddings: a $d_{AP}$ -dimensional AP embedding and a $d_{sem}$ -dimensional semantic embedding. The total hidden size is $d_{model} = d_{AP} + d_{sem}$ .
Attention Mechanism:
- Queries and Keys are computed independently for each stream within each attention head.
- The attention logit is the sum of the Relative Positional (RP) bias, the semantic attention weight, and the AP attention weight.
- Values are computed separately for each stream and remain in their respective spaces.
- A key design choice projects both streams to the same per-head size to ensure variance matching, preventing the higher-dimensional space from dominating.
Layer Normalization: Separate RMSNorm layers are applied to the AP and semantic embeddings before and after attention to avoid inter-dependence.
Feed-Forward Network (SwiGLU): The semantic and positional embeddings are concatenated and projected into a hidden space. Crucially, the down-projection re-separates the information into distinct AP and semantic streams. This is the only component where the streams directly interact, allowing for semantic-conditioned structural updates.
Training Objective (MLM Decoding): The Masked Language Modeling (MLM) decoding head is applied only to the semantic part of the final hidden state. The AP component of the final layer receives no gradient from the MLM objective, effectively freeing the positional subspace from prediction pressure.
Baselines: The authors compare DSTG-NeoBERT against three baselines trained on the same data (FineWeb) with identical hyperparameters:
- RoPE-NeoBERT: Uses Rotary Positional Encoding.
- AP-NeoBERT: Uses learned Absolute Positional embeddings (entangled).
- RP-NeoBERT: Uses a learned Relative Positional bias.

Key Findings and Contributions

1. Emergence of a Low-Frequency Structural Manifold

The isolated AP subspace in the disentangled model spontaneously collapses into a low-frequency two-dimensional manifold.

Dimensionality Reduction: PCA on the learned AP embedding matrix reveals that the first two principal components capture 94.1% of the total spatial variance.
Frequency Analysis: A Type-II Discrete Cosine Transform confirms these components operate at low frequencies, concentrating nearly all spectral power in the first four bins.
Contrast: In contrast, the entangled AP baseline (AP-NeoBERT) remains high-dimensional and dominated by high-frequency signals, capturing only 23.4% of variance in the first two PCs.

2. Specialization of Attention Heads

The disentangled architecture reveals a clear functional taxonomy where attention heads specialize almost exclusively into two groups:

Structure-Oriented Heads: Heavily influenced by AP information.
Semantic-Oriented Heads: Driven by semantic content.
Role of RP: The authors found no purely RP-oriented heads. Instead, the RP bias acts as a localized support mechanism for semantic heads (e.g., preventing self-attention to focus on distant context), while AP serves as a distinct structural information stream.

3. Preservation of Macroscopic Structure

Standard positional encodings fail to robustly retain macroscopic document structure:

Entangled AP: While AP-NeoBERT encodes structural information in intermediate layers, it loses most of this information in the final layer due to interference from the MLM prediction objective (positional dimensions are repurposed for semantic prediction).
Relative Methods (RoPE/RP): These methods largely fail to encode macroscopic document structure in their hidden states under MLM pretraining.
Disentangled Solution: By offloading position to an isolated subspace and excluding it from the MLM head, DSTG-NeoBERT preserves structural information (sentence/paragraph boundaries) throughout the network, including the final layer.

4. Probing Results

The authors evaluated the models using linear probes on three levels of hierarchy and the Flash-Holmes benchmark:

Token-Level & Segment-Level AP: DSTG-NeoBERT maintains high $R^2$ scores for absolute and segment-level position in the final layer, whereas entangled AP and relative baselines show significant degradation.
Intra-Segment Position: Interestingly, intra-segment position information is retrievable from the semantic subspace in all models, suggesting this specific local information does not require the AP stream.
Linguistic Fidelity (Flash-Holmes): The disentangled architecture improves probing results on 49 out of 65 linguistic phenomena across five fields (Morphology, Syntax, Semantics, Discourse, Reasoning). Notably, it outperforms all baselines in Syntax, Semantics, and Discourse.

Significance and Claims

The paper claims that explicitly disentangling positional and semantic streams provides a "clean mechanistic study" of Transformer internals. The primary significance lies in demonstrating that:

Positional information is inherently low-dimensional and structural, organizing into a simple manifold when not forced to compete with semantic prediction.
The loss of structural information in standard models is a direct result of the MLM objective overwriting positional dimensions in the final layers.
Decoupling these streams allows for richer embeddings that preserve macroscopic document structure, leading to improved linguistic representations without degrading standard downstream performance (GLUE, MTEB, SQuAD results remain comparable to baselines).

The authors suggest this approach offers a new direction for designing positional encodings that do not interfere with semantic representations and could enable practical applications such as caching position-independent semantic representations for RAG or manipulating the low-dimensional AP manifold for length extrapolation. However, they note that these findings are currently limited to encoder-only models and a relatively small scale (6 layers, 22B tokens).