FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception

1. Problem Statement

Current Vision Foundation Models (VFMs) and Bird's Eye View (BEV) perception pipelines are fundamentally designed for pinhole cameras, assuming a rectilinear (Cartesian) geometry where equal pixel distances correspond to equal spatial offsets. This assumption fails for fisheye cameras, which are standard in autonomous vehicles for 360° surround-view coverage but exhibit severe radial distortion.

The paper identifies two primary barriers preventing the adoption of VFMs in fisheye perception:

1. Data Scarcity: Large-scale, annotated fisheye datasets are orders of magnitude smaller than perspective datasets (e.g., nuScenes, KITTI), making it impractical to retrain foundation models from scratch.
2. Geometric Mismatch: Existing architectures use position encodings (learned, sinusoidal, or standard 2D Rotary) that encode pixel distances. In fisheye projections, a fixed pixel offset near the image center subtends a much smaller angular extent than the same offset at the periphery (3–5× difference). This causes standard attention mechanisms to misinterpret spatial relationships, leading to geometric inconsistencies in both self-attention and cross-attention (BEV lifting).

2. Methodology

The authors propose FishRoPE, a lightweight framework that adapts frozen VFMs to fisheye geometry without requiring full retraining. The architecture consists of three main components:

A. Backbone: Frozen DINOv2 with LoRA

• Base Model: The authors use a frozen DINOv2 (ViT-B/14) backbone, chosen for its robust, self-supervised features that generalize well across domain shifts and its global self-attention mechanism, which is naturally suited to handle the non-local relationships in distorted fisheye images.
• Adaptation: To adapt this frozen model to the fisheye domain with minimal parameters, Low-Rank Adaptation (LoRA) modules (rank $r=16$) are injected into the query and value projections of the self-attention layers. This adds only ~3M trainable parameters to the 86M frozen backbone.
• Feature Extraction: Multi-scale features are extracted from intermediate ViT layers (3, 6, 9, 12) and fused via a Feature Pyramid Network (FPN).

B. Core Innovation: Fisheye Rotary Position Embedding (FishRoPE)

This is the primary technical contribution. FishRoPE replaces the implicit Cartesian grid assumption with a spherical coordinate system $(\theta, \phi)$ derived from the fisheye lens model (specifically the Kannala–Brandt projection).

• Coordinate Transformation: For each image patch at pixel coordinates $(u, v)$, the system computes the incidence angle $\theta$ and azimuthal angle $\phi$ using the inverse Kannala–Brandt projection.
• Embedding Mechanism: The embedding dimension $d$ is split into two subspaces ($d/2$ for $\theta$ and $d/2$ for $\phi$). Standard Rotary Position Embedding (RoPE) rotation matrices are applied to these subspaces based on the angular coordinates rather than pixel indices.
• Key Properties:
  • Angular Relative Position: Attention logits depend on the relative angular separation between tokens, not pixel distance. This preserves the relative-position property of RoPE in a geometrically meaningful space.
  • Peripheral Disambiguation: Tokens at the periphery (which cover large angular extents in few pixels) receive distinct position codes, resolving spatial ambiguity.
  • Degeneracy: In the paraxial regime (near the optical center), FishRoPE naturally reduces to standard 2D RoPE, ensuring compatibility with pinhole cameras.

C. Task-Specific Heads

The framework supports two tasks using the same backbone and FishRoPE-enhanced features:

1. 2D Object Detection: An anchor-free CenterNet-style head predicts class heatmaps, box dimensions, and orientation.
2. BEV Semantic Segmentation: A view transformation module projects image features to a top-down occupancy grid. Unlike previous methods that require intermediate rectification, this module uses the forward Kannala–Brandt projection to map BEV grid queries directly to image keys. FishRoPE is applied to both the BEV queries and image keys, ensuring cross-attention operates on angular proximity rather than distorted pixel distance.

3. Key Contributions

1. Geometric Adapter for VFMs: Demonstrates that frozen VFMs (DINOv2) can be effectively adapted to fisheye imagery using lightweight LoRA, avoiding the need for massive fisheye-specific pretraining.
2. FishRoPE: Introduces the first position embedding that operates in the spherical coordinate system of fisheye lenses. It corrects the geometric mismatch in attention mechanisms, allowing the model to reason about spatial relationships accurately under radial distortion.
3. Architecture Agnosticism: The method is plug-and-play, adding negligible computational overhead and degrading gracefully to standard RoPE for pinhole cameras.
4. State-of-the-Art Performance: Achieves new benchmarks on both 2D detection and BEV segmentation without requiring complex architectural changes or heavy retraining.

4. Experimental Results

The model was evaluated on two benchmarks: WoodScape (2D detection) and SynWoodScapes (BEV segmentation).

• 2D Object Detection (WoodScape):
  • FishRoPE (DINOv2-B + FishRoPE) achieved 54.2 mAP, outperforming the previous best (FisheyeDetNet at 49.5 mAP) and the baseline using DINOv2 with standard 2D RoPE (52.5 mAP).
  • Even with a ResNet-18 backbone, FishRoPE (51.4 mAP) outperformed prior specialized fisheye detectors.
• BEV Segmentation (SynWoodScapes):
  • FishRoPE achieved 65.1 mIoU, surpassing the previous state-of-the-art FishBEV (64.22 mIoU), despite FishBEV using a larger ViT-L backbone while FishRoPE used ViT-B.
• Ablation Studies:
  • Position Encoding: FishRoPE consistently outperformed learned absolute, sinusoidal, and standard 2D RoPE encodings.
  • Backbone: DINOv2-B significantly outperformed Swin-T and ResNet variants, confirming the value of frozen foundation models for this domain.
  • Components: Using both $\theta$ and $\phi$ in the embedding provided marginal but consistent gains over using $\theta$ alone.

5. Significance

FishRoPE addresses a critical gap in autonomous driving perception: the inability to leverage powerful, general-purpose Vision Foundation Models for fisheye cameras. By decoupling the feature extraction (via frozen VFMs) from the geometric reasoning (via FishRoPE), the paper provides a scalable solution that:

• Eliminates the need for massive fisheye datasets to train foundation models from scratch.
• Corrects fundamental geometric errors in attention mechanisms caused by radial distortion.
• Unifies perception pipelines, allowing the same architecture to handle both pinhole and fisheye inputs seamlessly.

The work suggests a future direction where perception models are not tied to specific camera geometries but are instead adapted via geometrically aware position embeddings, making them robust to diverse sensor configurations.