💻 computer science

Wid3R: Wide Field-of-View 3D Reconstruction via Camera Model Conditioning

Wid3R is a novel feed-forward neural network that enables wide field-of-view 3D reconstruction by directly modeling wide-angle imagery through ray-based representation and camera model conditioning, eliminating the need for explicit calibration or undistortion while achieving significant performance gains on 360-degree scenes.

Original authors: Dongki Jung, Jaehoon Choi, Adil Qureshi, Somi Jeong, Dinesh Manocha, Suyong Yeon

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongki Jung, Jaehoon Choi, Adil Qureshi, Somi Jeong, Dinesh Manocha, Suyong Yeon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a room using only a stack of 2D photos.

Most computer vision systems today are like highly specialized architects who only know how to read blueprints drawn with a straight ruler (standard "pinhole" cameras). If you hand them a photo taken with a fisheye lens (which makes straight lines look curved) or a 360-degree camera (which wraps the whole world into a single ball), they get confused. They try to force the curved lines to be straight, often resulting in a warped, broken 3D model. To fix this, engineers usually have to manually "un-curve" the photos first, a tedious process that often loses important details.

Wid3R is like a super-flexible architect who doesn't need blueprints to be straight. It can look at a distorted, curved photo and instantly understand, "Ah, this is a fisheye view," or "This is a 360-degree wrap," and build the 3D model correctly without needing to fix the photo first.

Here is how it works, broken down with simple analogies:

1. The "Ray" vs. The "Point"

Old Way (The Point): Traditional systems look at a pixel and ask, "How far away is this point?" They try to guess a specific coordinate in space. This works great for normal photos but fails miserably when the image is curved because the "straight line" logic breaks.
Wid3R's Way (The Ray): Instead of guessing a point, Wid3R imagines a laser beam (a ray) shooting out from the camera for every single pixel. It asks, "In which direction is this laser pointing, and how far does it travel?"
- Analogy: Imagine you are in a dark room with a flashlight. Instead of trying to guess exactly where a dust mote is floating, you just shine the light. Wid3R is smart enough to know that if the flashlight beam is curved (due to a fisheye lens), the dust mote is still somewhere along that curved beam.

2. The "Language Translator" (Camera Model Tokens)

Different cameras speak different "languages" of distortion. A GoPro speaks "Fisheye," while a Matterport 360 camera speaks "Spherical."

The Problem: If you feed a Fisheye photo into a system trained only on normal photos, it's like speaking English to someone who only understands French.
The Solution: Wid3R uses a special "Camera Model Token." Think of this as a name tag or a translator that the camera wears. Before Wid3R starts looking at the photo, it checks the name tag.
- If the tag says "Fisheye," Wid3R switches its brain to "Fisheye Mode."
- If the tag says "360," it switches to "Spherical Mode."
- This allows one single brain (the neural network) to handle any type of camera without getting confused.

3. The "Spherical Harmonics" (The Map)

To describe the direction of those laser beams (rays) in a curved world, Wid3R uses a mathematical tool called Spherical Harmonics.

Analogy: Imagine trying to describe the shape of a globe. You could try to draw it on a flat piece of paper (which stretches and tears the map), or you could use a globe. Spherical Harmonics are like the mathematical grid lines on a globe. They allow Wid3R to describe the direction of a laser beam perfectly, no matter how wide or curved the camera's view is.

Why is this a big deal?

No More "Un-distorting": You don't need to spend time manually fixing the photos before feeding them to the AI. Wid3R handles the messiness directly.
Better for Robots and VR: Real-world robots, self-driving cars, and VR headsets often use wide-angle or 360-degree cameras to see everything around them. Wid3R lets these devices build accurate 3D maps of their surroundings instantly, without getting dizzy from the distortion.
Speed: Because it's a "feed-forward" network (it looks at the image once and gives an answer), it's incredibly fast. It can build a 3D map of a large room in about 3 seconds, whereas older methods that try to stitch photos together mathematically can take minutes.

In a Nutshell

Wid3R is the first AI that treats curved, distorted, and 360-degree photos as first-class citizens. Instead of forcing these images to look like normal photos, it learns to "speak" the language of distortion, using laser beams (rays) and name tags (tokens) to build accurate 3D worlds from any camera you throw at it.

1. Problem Statement

Existing learning-based multi-view 3D reconstruction methods (e.g., VGGT, $\pi^3$ , Dust3R) predominantly assume input images are captured by pinhole cameras or have been pre-processed (undistorted/rectified) to fit a pinhole model. This creates a significant bottleneck for real-world applications involving Wide Field-of-View (FoV) cameras, such as:

Fisheye cameras (common in robotics and autonomous navigation).
360°/Spherical cameras (used in AR/VR and large-scale mapping).

Key Challenges:

Distortion Sensitivity: Standard models fail to generalize to the complex non-linear projection effects of wide-angle lenses, leading to poor depth and pose estimation.
Preprocessing Overhead: Traditional pipelines require explicit camera calibration and image rectification, which introduces engineering complexity and geometric information loss.
Data Sparsity: High-quality 3D ground truth for 360° imagery is scarce compared to perspective data, making it difficult to train specialized models.

2. Methodology: Wid3R

Wid3R is a feed-forward neural network designed to perform multi-view 3D reconstruction directly on distorted wide-angle images without explicit undistortion.

Core Architecture Components

Ray-Based Representation (Angular & Radial Modules):
- Instead of regressing point maps directly in Cartesian coordinates (which assumes a pinhole model), Wid3R decomposes 3D reconstruction into Ray Direction and Radial Distance.
- Angular Module: Predicts the direction of the camera ray for each pixel using Spherical Harmonics (SH). It encodes the ray direction $\mathbf{R}_i(\theta, \phi)$ as a linear combination of SH basis functions, allowing the network to implicitly model arbitrary camera projection geometries.
- Radial Module: Predicts the distance $D_i$ along the predicted ray.
- The final 3D point is computed as $\hat{P} = \hat{R} \odot \hat{D}$ .
Camera Model Conditioning (Camera Model Token):
- The network accepts a learnable Camera Model Token ( $C_i$ ) as an input condition.
- This token acts as a prompt, informing the network of the specific projection characteristics (e.g., Fisheye, 360°, Pinhole) of the input image.
- This allows a single unified architecture to adapt to diverse camera types without retraining or architectural changes.
Per-View Local Geometry & Scale Alignment:
- Inspired by $\pi^3$ , the network predicts point maps in local camera coordinate frames rather than a global frame to handle permutation equivariance.
- To resolve scale ambiguity across views, a global optimal scale factor $s^*$ is solved via an $L_1$ loss minimization between predicted and ground-truth point maps, weighted by radial distance.
Training Strategy:
- Datasets: Trained on a diverse mix of datasets including Pinhole (Hypersim, vKITTI), Fisheye (ASE, ScanNet++, KITTI-360), and 360° (TartanAirV2, Matterport3D, 360Loc).
- Data Augmentation: Includes geometric transformations and a specific strategy to transform pinhole images into fisheye models via reprojection to increase data diversity.
- Loss Functions: Combines point map loss, normal loss, pose loss, and specific ray losses (asymmetric angular loss and radial distance loss).

3. Key Contributions

First Multi-View Feed-Forward Model for Distorted Images: Wid3R is the first method capable of jointly estimating dense 3D point maps and camera poses from wide FoV images (fisheye and 360°) in a single feed-forward pass, eliminating the need for explicit calibration or rectification.
Unified Ray-Based Representation: Introduces a novel formulation using Spherical Harmonics to represent ray directions, enabling the model to reason about scene geometry across diverse projection models within a unified network.
Camera Model Tokens: Proposes a lightweight conditioning mechanism that explicitly encodes camera projection types, significantly improving generalization and robustness to distortion.
Scalability to 360° Scenes: Demonstrates that conditioning on diverse camera types (even with limited 360° training data) alleviates data sparsity issues and improves performance on 360° scenes.

4. Experimental Results

Wid3R was evaluated on multiple benchmarks, showing significant improvements over state-of-the-art methods (VGGT, $\pi^3$ , Dust3R) and traditional SfM pipelines.

Camera Pose Estimation:
- On Zip-NeRF (Fisheye), Wid3R improved AUC@30° by +77.33% compared to $\pi^3$ .
- On Stanford2D3D (360°), it achieved a 48.2% accuracy improvement in point map reconstruction.
- It achieved near-perfect Relative Rotation Accuracy (RRA) on FIORD and Zip-NeRF datasets.
Large-Scale Localization:
- On Matterport3D, Wid3R achieved comparable localization accuracy to complex SfM pipelines (like EDM + IM360) but with a massive speedup (processing time reduced from 30 minutes to **3 seconds**).
Monocular Depth:
- Despite being a multi-view model, Wid3R achieved competitive performance in single-view 360° depth estimation, outperforming specialized monocular 360° depth models.
Ablation Studies:
- Confirmed that training with diverse camera models (Pinhole + Fisheye + 360°) significantly boosts performance on 360° test sets compared to training solely on 360° data.
- Validated that the Camera Model Token is essential for performance gains.

5. Significance and Impact

Real-World Applicability: By removing the dependency on rectification and pinhole assumptions, Wid3R bridges the gap between academic 3D reconstruction models and real-world hardware (drones, robots, VR headsets) that rely on wide-angle lenses.
Efficiency: The feed-forward nature allows for real-time 3D reconstruction in large-scale environments, making it viable for SLAM and autonomous navigation where iterative optimization (like Bundle Adjustment) is too slow.
Foundation for Future Work: The paper suggests that Wid3R's geometric tokens can serve as priors for generative AI (e.g., generating consistent 3D scenes from text or panoramic images), potentially enhancing the geometric consistency of synthetic data generation.

In summary, Wid3R represents a paradigm shift in 3D vision, moving from "distortion-free" assumptions to "distortion-aware" learning, enabling robust, fast, and accurate 3D reconstruction directly from the raw outputs of modern wide-field cameras.