🤖 machine learning

Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General

This paper introduces Parabolic Position Encoding (PaPE), a principled, vision-centric position encoding method that leverages parabolic functions to achieve superior extrapolation and generalization across diverse visual modalities compared to existing baselines.

Original authors: Christoffer Koo Øhrstrøm, Rafael I. Cabral Muchacho, Yifei Dong, Filippos Moumtzidellis, Ronja Güldenring, Florian T. Pokorny, Lazaros Nalpantidis

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Christoffer Koo Øhrstrøm, Rafael I. Cabral Muchacho, Yifei Dong, Filippos Moumtzidellis, Ronja Güldenring, Florian T. Pokorny, Lazaros Nalpantidis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize objects in a room. You show it a picture of a cat sitting on a table. The robot needs to know two things: what it is looking at (a cat) and where it is looking (on the table, not on the floor).

In the world of AI, the "what" is easy. The "where" is tricky. Most robots today use a position system borrowed from language learning, like a ruler that just counts "1, 2, 3" across a line. But the world isn't a line; it's a 3D space with up, down, left, right, and diagonals.

This paper introduces a new tool called Parabolic Position Encoding (PaPE). Think of PaPE as giving the robot a smart, flexible map instead of a rigid ruler.

Here is how it works, using simple analogies:

1. The Problem with Old Maps

Old methods (like the ones used for language) are like a straight ruler. They tell the robot, "This token is 5 steps away from that one." But they struggle when the robot needs to understand:

Direction: Is the cat above the table or below it?
Distance: Is the cat close to the table or far away?
Rotation: If you turn the picture sideways, does the robot still know it's a cat?

2. The PaPE Solution: A "Smart Bounce"

The authors designed PaPE based on five simple rules that make sense for vision:

Translation Invariance: It doesn't matter if the cat is in the top-left or bottom-right; the relationship between the cat and the table stays the same.
Rotation Invariance: If you spin the room, the robot should still understand the layout.
Distance Decay: Things that are close should "talk" to each other more loudly than things far away.
Directionality: The robot needs to know if something is to the left or right, not just how far.
Context Awareness: The robot should be able to decide, "Hey, for this specific object, I need to look far away," or "For this one, I only care about what's right next to me."

The Analogy: Imagine you are throwing a ball at a target.

Old methods are like a rigid tape measure. They just tell you the distance.
PaPE is like a bouncy trampoline. The shape of the trampoline (the "parabola") changes depending on who is throwing the ball (the content of the image).
- If the ball is heavy (a complex object), the trampoline might be stiff, keeping the focus tight (local).
- If the ball is light, the trampoline might be loose, letting the ball bounce far (global).
- It naturally curves to show direction (left/right) and gets weaker as you move away (distance decay).

3. The "Magic" of Extrapolation (The Zoom Test)

One of the biggest tests for these robots is extrapolation. Imagine you train the robot to recognize cats in small 224x224 pixel pictures. Then, you suddenly show it a giant 1024x1024 pixel picture without retraining it.

Old robots get confused. They think the cat is huge or in the wrong place. Their accuracy crashes.
PaPE robots are like a zoom lens. They handle the giant picture almost as well as the small one.
- The paper shows that at the highest resolution, PaPE was 10.5% more accurate than the next-best method. It's like the robot didn't even notice the picture got bigger.

4. Does it work everywhere?

The authors tested this "smart map" on eight different types of vision tasks, like:

Videos: Watching people move (UCF101).
Event Cameras: Cameras that only see changes in light (like a fire alarm seeing smoke).
3D Point Clouds: Digital clouds of dots representing 3D objects (like a car or a chair).
Standard Photos: Recognizing objects in images (ImageNet).

The Result: PaPE was the winner or tied for the winner in 5 out of 8 tests, and it beat everyone else in 2 of them. It proved to be a "universal" map that works for videos, 3D shapes, and photos alike.

5. The Trade-off

Is there a catch? Yes, but a small one.
To make this smart map, the robot needs to carry a slightly heavier backpack. PaPE adds a little bit of extra data (about 7% to 20% more memory and computing power depending on the settings). However, the authors say this cost is small compared to the huge gain in accuracy and the ability to handle different resolutions without retraining.

Summary

The paper proposes PaPE, a new way to teach AI where things are located in a picture. Instead of using a rigid, language-based ruler, it uses a flexible, curved "parabola" that understands distance, direction, and context.

It's robust: It works even when you zoom in or out drastically.
It's general: It works on videos, 3D scans, and photos.
It's efficient: It fits into modern AI chips without slowing them down too much.

In short, PaPE gives the robot a better sense of space, making it smarter and more adaptable to the real world.

Technical Summary: Parabolic Position Encoding (PaPE)

Problem Statement

Vision Transformers (ViTs) have become dominant in computer vision, yet their position encodings are often adapted from natural language processing (NLP) architectures. While language is inherently 1D and largely indifferent to geometric direction, vision modalities (images, videos, event streams, point clouds) possess distinct spatial characteristics. Existing adaptations of position encodings to $n$ -dimensional vision structures often fail to fully account for these characteristics, such as translation invariance, rotation invariance, distance decay, directionality, and context awareness. Furthermore, many current methods struggle to extrapolate effectively to resolutions significantly higher than those seen during training, limiting their robustness and applicability in multi-resolution scenarios.

Methodology: Parabolic Position Encoding (PaPE)

The authors propose Parabolic Position Encoding (PaPE), a position encoding designed specifically for vision modalities based on five distilled design principles:

Translation Invariance: The encoding depends on the relative position between tokens, not their absolute coordinates.
Rotation Invariance (PaPE-RI): A specific instantiation where the encoding remains unchanged under rotation (though this comes at the cost of directionality).
Distance Decay: Attention strength diminishes as the distance between tokens increases.
Directionality: The encoding accounts for the specific direction between tokens (e.g., up, down, diagonal), which is crucial for vision but less relevant in 1D language.
Context Awareness: The strength of distance decay and directionality is modulated by the content of the tokens themselves, allowing the model to adapt to local versus long-range interactions.

Mathematical Formulation

PaPE models the relative position $\Delta r_{ij} = W_p(r_j - r_i)$ as the dependent variable in a sum of $m$ learnable parabolas. The attention score $S_{ij}$ is decomposed into three components:
$S_{ij} = \langle a_i, \Delta r_{ij}^{\odot 2} \rangle + \langle b_i, \Delta r_{ij} \rangle + \langle q_i, k_j \rangle$
Where:

$\langle a_i, \Delta r_{ij}^{\odot 2} \rangle$ encodes distance decay. The coefficient $a_i$ is derived from the token content $x_i$ via a learnable projection and a softplus activation (negated) to ensure $a_i < 0$ , enforcing concavity.
$\langle b_i, \Delta r_{ij} \rangle$ encodes directionality. The coefficient $b_i$ is also derived from $x_i$ .
$\langle q_i, k_j \rangle$ represents the standard semantic attention.

Rotation Invariant Variant (PaPE-RI)

To achieve rotation invariance, PaPE-RI imposes constraints: $b_i = 0$ (removing directionality), $a_i$ is a scalar $\alpha_i < 0$ , and the projection matrix $W_p$ is a scaled identity matrix. This reduces the distance term to a function of the squared Euclidean distance, which is rotation invariant.

Compatibility with Efficient Kernels

A critical technical contribution is the derivation of separate query and key transformations ( $f_q$ and $f_k$ ) that allow PaPE to be computed without materializing the full $N \times N$ attention matrix. This ensures compatibility with efficient attention kernels like FlashAttention. The transformations inject positional information directly into the query and key vectors, increasing the head dimension by $p^2 + 2p + 2$ (where $p$ is the spatial dimension), which is a moderate overhead for $p=2$ or $p=3$ .

Key Contributions

Novel Encoding: Introduction of PaPE and PaPE-RI, the first position encodings derived from the combined principles of translation/rotation invariance, distance decay, directionality, and context awareness.
Efficiency: Demonstration that PaPE is compatible with efficient attention kernels via separate query-key transformations, avoiding the computational bottleneck of full attention matrix materialization.
Extrapolation: Evidence that PaPE exhibits superior extrapolation capabilities for classification tasks, maintaining performance at resolutions far exceeding training resolution without additional training or interpolation.
Generality: Validation of PaPE across 8 datasets spanning 4 vision modalities (videos, event cameras, images, and point clouds), showing it matches or exceeds state-of-the-art baselines.

Experimental Results

Classification Extrapolation

On ImageNet-1K, models trained at $224^2$ resolution were evaluated up to $1024^2$ without retraining or position interpolation.

PaPE achieved an accuracy of 74.9% at $1024^2$ , outperforming the next-best baseline (LookHere) by 10.5% (LookHere achieved 64.4%).
In contrast, standard baselines like nD-sincos and RoPE variants suffered significant performance drops at higher resolutions.
The authors attribute this success to PaPE's context-aware nature, which allows the model to adapt positional decay dynamically.

General Vision Position Encoding

Evaluation across 8 datasets (UCF101, DvsGesture, GEN1, ImageNet-1K, COCO, ScanNet, ModelNet40, nuScenes) yielded the following:

Average Performance: PaPE achieved the highest average score (66.3), surpassing the next-best method (RoPE) by 1.0 point.
Dataset Wins: PaPE matched the best baseline on 5 datasets and exceeded all baselines on 2 datasets (UCF101 and GEN1).
Spatio-Temporal Data: PaPE showed particularly strong gains on UCF101 (49.5% vs. 43.9% for RoPE) and GEN1, suggesting its directionality and context awareness are vital for fast-moving objects and event-based data.
Point Clouds: On ScanNet, PaPE underperformed compared to absolute encodings (nD-sincos), suggesting that for indoor scenes with characteristic absolute sizes, absolute position information may be beneficial. Conversely, on ModelNet40 (isolated objects), translation-invariant encodings outperformed absolute ones.

Robustness and Efficiency

Out-of-Distribution (OOD): PaPE did not degrade more than baselines on ImageNetV2 and ImageNet-Renditions, confirming that context awareness does not harm generalization.
Parameter Matching: When baselines were adjusted to match PaPE's parameter count, PaPE retained its performance advantage, indicating the gains are not solely due to increased model capacity.
Efficiency: PaPE introduces a moderate resource increase (7–20% in VRAM and FLOPS for $m=50$ ) but remains compatible with standard efficient attention implementations.

Significance and Claims

The paper claims that PaPE represents a general vision position encoding that bridges the gap between language-derived encodings and the specific geometric needs of vision.

Extrapolation: The authors claim PaPE offers "remarkable" extrapolation, improving absolute terms by up to 10.5% over the next-best method at high resolutions, a critical factor for reducing training costs and enabling robust multi-resolution inference.
Principled Design: By explicitly designing for translation invariance, distance decay, directionality, and context awareness, PaPE addresses the "incomplete" coverage of vision characteristics in prior adaptations.
Versatility: The results across diverse modalities (from 2D images to 3D point clouds and asynchronous event streams) suggest that PaPE provides a unified, robust solution for vision Transformers, paving a new path for position encoding research.

The authors remain modest regarding limitations, noting the moderate increase in parameters and resource usage dependent on the hyperparameter $m$ , and suggest future work could focus on reducing $m$ or integrating PaPE more deeply into efficient kernels.