Technical Summary: RayTun3R

Problem Statement

Recent 3D foundation models (e.g., DUSt3R, MASt3R, VGGT, $\pi^3$, Depth Anything 3) excel at feed-forward depth and pose estimation on standard pinhole imagery. However, their performance degrades significantly when applied to fisheye or wide-angle inputs. The authors identify that this failure stems from a pinhole camera bias inherent in the positional encodings of these pretrained models.

While standard parameter-efficient fine-tuning (PEFT) methods like LoRA or adapter layers modify generic feature transformations, they fail to address the specific geometric mismatch: pinhole and fisheye cameras assign different ray geometries to the same image-grid locations. Existing workarounds, such as splitting fisheye images into multiple virtual pinhole crops (e.g., Center-PH, Multi-PH), introduce computational overhead and require complex prediction fusion without directly modeling the fisheye geometry.

Methodology: RayTun3R

RayTun3R is a lightweight, online camera adaptation approach designed to correct the pinhole bias in frozen 3D foundation models without retraining the backbone. The core hypothesis is that adapting the components linking token locations to camera rays is more effective than adapting generic features.

Key Components

1. Frozen Backbone: The pretrained 3D model (weights $\theta$) remains completely fixed.
2. Lightweight Adapter: Only specific components tied to token position and camera geometry are updated. The total number of trainable parameters is minimal (10,752 for DA3-Small).
   • Positional Embedding Correction: The authors parameterize corrections for absolute positional embeddings ($P_A$) in polar coordinates (radius $\rho$ and angle $\theta$) relative to the principal point. These are implemented as learnable radial and angular lookup tables.
   • Rotary Positional Encoding (RoPE) Correction: For models using RoPE, a learnable radial correction is applied to the rotary angles.
   • Parameter-Free Corrections:
     • Prediction-Grid Coordinates: Replaces regular grid coordinates with camera-aware coordinates obtained by undistorting fisheye grid locations.
     • Patch Tokenization: Patches outside the valid fisheye lens are replaced with the mean valid token, and patches are locally undistorted before tokenization to approximate pinhole crops.

Training Strategy

The adapter is trained online on a short temporal segment of fisheye frames using self-supervised geometric losses, without requiring ground-truth depth or pose:

• Reprojection Loss ($L_{reproj}$): Enforces consistency between predicted 3D points and correspondences from an off-the-shelf matcher (UFM).
• Pose Regularization ($L_{pose}$): Uses a fixed relative pose estimate (from MAGSAC++) as a pseudo-label to stabilize early optimization.
• Smoothness Priors: Includes edge-aware depth smoothness and L2/Total Variation regularization to ensure the positional corrections remain small and smooth relative to the pretrained table.

Key Contributions

1. Analysis of Pinhole Bias: The authors demonstrate that the local Jacobian of pretrained positional embeddings follows the constant geometry of a pinhole camera. They show that after adaptation, this Jacobian becomes radius-dependent, aligning with fisheye geometry.
2. Lightweight Adapter Design: Introduction of RayTun3R, which adapts only positional components (absolute and rotary) while keeping the backbone frozen. This approach uses $\sim14\times$ fewer trainable parameters than LoRA and adds negligible inference overhead.
3. Empirical Validation: The method is validated across diverse fisheye datasets (FOV 110°–200°) and three different frozen backbones (Depth Anything 3, VGGT, $\pi^3$).

Results

Experiments on five datasets (ETH3D, KITTI-360, TUM-VI, ScanNet++, FIORD) show that RayTun3R consistently outperforms baselines:

• Pose Accuracy: RayTun3R reduces rotation error by 2–12$\times$ compared to the unadapted model and significantly outperforms LoRA and calibration tokens (CalTok). For example, on KITTI-360, it achieves a rotation error of 0.84° compared to 1.69° (Vanilla) and 1.37° (LoRA).
• Depth Accuracy: While projection-based methods (Center-PH) sometimes yield competitive depth on moderate FOVs by mimicking the training distribution, RayTun3R achieves competitive depth accuracy while utilizing the full fisheye field of view.
• Efficiency: Unlike Multi-PH, which incurs a 300% inference cost by processing multiple views, RayTun3R maintains the original inference speed (100ms/frame) with zero overhead. It uses only 10.8K trainable parameters compared to 147.5K for LoRA.

Significance and Claims

The paper claims that the failure of 3D foundation models on fisheye imagery is fundamentally a mismatch in positional representation rather than a lack of feature capacity. By isolating and correcting the camera-dependent mapping between image tokens and viewing rays, RayTun3R enables these models to generalize to wide-angle inputs efficiently.

The authors position this work as a solution that avoids the computational costs of multi-view fusion and the architectural changes required for camera-aware training. They emphasize that their method is "parameter-efficient" and "runtime-efficient," making it suitable for online adaptation scenarios.

Limitations Acknowledged by the Authors

• Camera Specificity: The correction is camera-specific; a different fisheye model or distortion profile requires re-adaptation.
• Distortion Assumptions: The method assumes a principal point and mostly radial distortion, not explicitly covering strong tangential or non-radial optics.
• Calibration Dependency: It requires known camera parameters (though predicted calibration is shown to be viable).
• Motion Requirements: The self-supervised training requires sufficient inter-frame displacement; degenerate motion can weaken constraints.
• Scope: The current work focuses on fisheye images, with panoramic/equirectangular inputs left for future work.