URoPE: Universal Relative Position Embedding across Geometric Spaces

1. Problem Statement

Transformers have become the dominant architecture for computer vision tasks, including multi-view perception and generation. However, a critical bottleneck exists in encoding spatial relationships between tokens originating from different viewpoints, coordinate systems, or geometric modalities (e.g., 2D images vs. 3D points).

• Limitations of Existing Methods:
  • Standard RoPE (Rotary Position Embedding): Operates only within a single flat coordinate space (1D sequences or 2D grids). It fails in cross-view scenarios because pixels that are close in 3D space may be far apart in their respective 2D image grids.
  • Ray-based Embeddings (e.g., Plücker): Often treat rays as absolute encodings or rely on global coordinate systems, lacking the relative bias properties crucial for attention generalization.
  • Hybrid Approaches: Recent methods often decouple inter-camera geometry from intra-image spatial positions or rely on complex matrix multiplications that are incompatible with efficient attention kernels like FlashAttention.
• Core Challenge: How to model the relative position of a key token from one view relative to a query token in another view without relying on a fixed global coordinate system or learnable depth predictors, while maintaining compatibility with standard attention mechanisms.

2. Methodology: URoPE

The authors propose URoPE, a universal extension of RoPE that utilizes projective geometry to express cross-view correspondences directly in the query image plane.

Key Mechanisms:

1. Camera Rays as a Bridge:
   Instead of abstract ray coordinates, URoPE treats each image pixel as a camera ray defined by the camera center ($o_i$) and direction ($r_i$). This bridges 2D image coordinates and 3D space.

2. Depth-Anchored Lifting:
   Since true depth is unknown during attention, URoPE lifts each key pixel $(u, v)$ along its camera ray to a set of fixed depth anchors $D = \{d_h\}$. This creates a set of 3D points:
   $$p_i^h(u, v) = o_i + d_h \cdot r_i(u, v)$$

3. Cross-View Projection:
   These lifted 3D points are projected into the query camera's image plane using the relative camera transform and the query camera's intrinsics. This yields a set of projected pixel coordinates $(u^h, v^h)$ in the query view.

   • Result: The correspondence between a key in view $i$ and a query in view $j$ is now represented as a 2D relative position within the query image plane.

4. Depth-Anchored Multi-Head Attention:

   • Head Assignment: Different attention heads (or head groups) are assigned different fixed depth anchors. This allows the model to jointly cover near-to-far field correspondences (epipolar lines) without learning depth.
   • RoPE Application: Standard 2D RoPE is applied between the query location $(u', v')$ and the projected key location $(u^h, v^h)$ within the query image plane.
   • Compatibility: By operating entirely through RoPE on projected coordinates, URoPE remains parameter-free, intrinsics-aware, and fully compatible with FlashAttention and other optimized kernels.

5. Handling 2D-3D Interaction:
   For tasks involving 3D queries (e.g., 3D object detection), the method skips the image-plane projection and measures relative positions directly in 3D space between the 3D query and the lifted 2D points.

3. Key Contributions

• Universal Geometric Encoding: URoPE is the first relative position embedding that unifies cross-view (2D-2D) and cross-dimensional (2D-3D) reasoning within a single, parameter-free framework.
• Projective Geometry Integration: It replaces abstract ray embeddings with explicit projective geometry, mapping cross-view correspondences into a shared 2D coordinate system where standard RoPE can be applied.
• Depth-Agnostic Multi-Head Design: By assigning fixed depth anchors to different attention heads, it resolves the inherent depth ambiguity of epipolar geometry without requiring learnable depth prediction modules.
• Efficiency and Compatibility: The method introduces no asymptotic computational overhead and works seamlessly with existing RoPE-optimized attention kernels.
• Invariance: The encoding is invariant to the choice of global coordinate systems (SE(3) invariant) and aware of camera intrinsics.

4. Experimental Results

URoPE was evaluated as a plug-in replacement for position embeddings across three distinct tasks:

• Novel View Synthesis (2D $\to$ 2D):

  • Datasets: Objaverse and RealEstate10k.
  • Baselines: Plücker Ray, 6D RoPE, P-RoPE, RayRoPE.
  • Results: URoPE achieved state-of-the-art performance (e.g., 26.02 PSNR on RealEstate10k vs. 25.28 for P-RoPE), demonstrating superior ability to render fine-grained details and handle varying camera intrinsics.

• 3D Object Detection & Tracking (2D $\to$ 3D):

  • Datasets: nuScenes.
  • Baselines: PETR and StreamPETR.
  • Results: Consistent improvements in NDS (37.3 $\to$ 37.3+ for PETR; 47.6 $\to$ 50.6 for StreamPETR) and tracking metrics (AMOTA). URoPE effectively lifted 2D features into 3D space, improving detection of small objects and trajectory coherence.

• Stereo Depth Estimation (2D $\to$ 2D):

  • Datasets: RGBD, Scenes11, SUN3D.
  • Baselines: UniMatch with P-RoPE.
  • Results: URoPE consistently outperformed baselines (e.g., lower Abs Rel and RMSE), proving its versatility even in simple stereo setups.

• Robustness & Scalability:

  • Out-of-Distribution: URoPE maintained robust performance when tested with different camera focal lengths and varying numbers of context views, despite being trained with fixed parameters.
  • Scaling: Performance gains persisted when scaling the model size by 50x, indicating strong scalability.
  • Ablation: Head-wise splitting of depth anchors was found superior to channel-wise splitting. Fixed anchors outperformed learned depth modules, confirming that explicit geometric priors are more effective than learning depth from shallow features.

5. Significance

URoPE represents a paradigm shift in how Transformers handle geometric reasoning. By grounding relative position embedding in projective geometry, it eliminates the need for task-specific architectural changes or complex attention masks.

• Generality: It serves as a "universal" plug-in for any Transformer-based vision task involving multiple views or modalities.
• Practicality: Its parameter-free nature and compatibility with efficient kernels make it immediately deployable in production systems.
• Future Impact: It establishes a new standard for geometric reasoning in vision Transformers, potentially extending to uncalibrated settings and other 3D reconstruction tasks.

In summary, URoPE solves the fundamental disconnect between 2D image grids and 3D world geometry by projecting 3D hypotheses back into the 2D query plane, allowing standard attention mechanisms to "see" the 3D structure of the scene.