3D Gaussian Splatting with Fisheye Images: Field of View Analysis and Depth-Based Initialization

This paper presents the first evaluation of 3D Gaussian Splatting on real 200° fisheye imagery, demonstrating that 160° field-of-view yields optimal reconstruction quality and introducing a UniK3D-based depth initialization method that overcomes Structure-from-Motion failures in extreme wide-angle, distorted scenes.

The Gist

Imagine you are trying to build a 3D model of a room using only flat photographs. Usually, you take many photos with a normal camera (like on your phone) and a computer stitches them together. This is called 3D Gaussian Splatting. It's like taking thousands of tiny, fluffy cotton balls (the "Gaussians") and arranging them in 3D space until they look exactly like the real room.

But what if you want to use a fisheye lens? You know, those crazy wide-angle lenses that make everything look curved and wrap around you, like a fish seeing the world? These lenses are great because you can see almost everything in one shot (200 degrees!), but they are a nightmare for computers because the straight lines in the real world look like bent spaghetti in the photo.

This paper is the first big test of trying to build 3D models using these "fisheye" photos. Here is the story of what they found, broken down simply:

1. The Problem: The "Curved World" Confusion

Most 3D software expects photos to look like a standard window (flat). When you feed it a fisheye photo, the software gets dizzy. The edges of the photo are so distorted that the computer struggles to figure out where objects actually are.

The researchers tested two new "translation tools" (methods) designed to understand these curved photos:

• Fisheye-GS: A method that tries to straighten the world out mathematically.
• 3DGUT: A more complex method that tries to handle the curves directly without straightening them first.

2. The "Sweet Spot" Experiment: How Wide is Too Wide?

The researchers took photos at three different "zoom levels" of the fisheye lens:

• 200° (The Full Fish): Seeing everything, including the ceiling and floor in one go.
• 160° (The Balanced View): Cropping the edges slightly.
• 120° (The Normal View): Cutting off most of the curve.

The Discovery:
They found that 200° was actually too much! The distortion at the very edges was so strong that the 3D models got blurry and messy.
160° was the "Goldilocks" zone. It was wide enough to see the whole room but narrow enough that the edges weren't too warped. It was the perfect balance between seeing the whole scene and keeping the picture sharp.

3. The "GPS" Problem: How to Start Building?

To build a 3D model, the computer needs a rough map to start with. Usually, it uses a technique called SfM (Structure-from-Motion), which is like a detective looking for matching dots between many photos to figure out the 3D shape.

• The Issue: On fisheye photos, this detective gets confused. The dots don't match up well because of the distortion, so the map is broken.

The New Solution: The "Magic Eye" (UniK3D)
Instead of using the detective (SfM), the researchers tried a new AI called UniK3D. Think of UniK3D as a psychic who can look at a single photo and guess the depth and shape of the room without needing to compare it to other photos.

• The Catch: This AI was mostly trained on normal photos and fake computer-generated images, not real fisheye photos.
• The Result: Surprisingly, the "psychic" worked! Even though it wasn't trained on these specific crazy lenses, it could guess the 3D shape well enough to build a great model. In fact, for some scenes, it was faster and just as good as the traditional detective method.

4. The Verdict: What Works Best?

• For small, cluttered rooms: The complex method (3DGUT) did a great job, but only if the "psychic" (UniK3D) helped start the process.
• For big, open spaces: The simpler method (Fisheye-GS) was more stable and reliable.
• The Initialization: Using the AI "psychic" (UniK3D) to start the process is a game-changer. It takes seconds instead of hours, and it works even in tricky conditions like fog, glare, or dark nights where the traditional method fails.

The Big Takeaway

This paper proves that you can build amazing 3D models using ultra-wide fisheye cameras, which is huge for robots, self-driving cars, and VR.

• Don't use the full 200° view if you want the sharpest image; crop it slightly to 160°.
• You don't need a slow, complex detective to start the process; a fast AI guess (UniK3D) is often good enough and much quicker.

It's like realizing you don't need to measure every inch of a room with a tape measure to build a model; sometimes, a really good guess from a smart AI is all you need to get started!

Technical Summary

1. Problem Statement

3D Gaussian Splatting (3DGS) has revolutionized real-time 3D reconstruction but is predominantly optimized for narrow Field-of-View (FoV) perspective cameras. Fisheye cameras offer ultra-wide FoVs (often >180°), which are crucial for applications like autonomous driving, robotics, and VR/AR to reduce sensor counts and capture time. However, applying 3DGS to fisheye imagery presents two major challenges:

• Severe Distortion: Non-linear projection models cause significant radial distortion, particularly at the periphery, which standard 3DGS pipelines struggle to handle.
• Initialization Failure: Standard Structure-from-Motion (SfM) pipelines (e.g., COLMAP) often fail or produce sparse, inaccurate point clouds when initialized on heavily distorted fisheye images, especially at extreme FoVs (e.g., 200°).
• Lack of Evaluation: While recent methods (Fisheye-GS, 3DGUT) have adapted 3DGS for fisheye lenses, there is no comprehensive evaluation of their performance on real-world data with FoVs exceeding 180°, nor is there a validated alternative to SfM for initialization in this domain.

2. Methodology

Dataset and Setup

The authors utilized the FIORD dataset, comprising 10 real-world scenes (5 indoor, 5 outdoor) captured with an Insta360 One RS 1-Inch camera.

• Input: 200° fisheye images (3264×3264 resolution).
• Calibration: Lenses were calibrated using the Camera Calibration Toolbox for Generic Lenses, with parameters converted to OpenCV's fisheye model for compatibility.
• Metrics: Evaluation used PSNR, SSIM, and LPIPS on held-out test views.

Evaluated Reconstruction Methods

The study benchmarked two state-of-the-art fisheye-adapted 3DGS methods:

1. Fisheye-GS: Uses an equidistant projection model. It reprojects images to remove distortion terms ($k_1$) to maintain a distortion-free assumption but is theoretically limited to 180°.
2. 3DGUT: Uses an Unscented Transform to propagate sigma points through arbitrary non-linear camera models without reprojection. It attempts to model the full 200° distortion directly.

Field of View (FoV) Analysis

To analyze the trade-off between scene coverage and distortion, the authors generated subsets of the data at 160° and 120° by reprojecting the 200° inputs. This allowed them to test how reducing the angular input affects reconstruction quality.

Depth-Based Initialization (UniK3D)

To address SfM failures, the authors introduced UniK3D (a transformer-based monocular depth estimator) as an initialization source.

• Process: Depth maps and ray directions were predicted from just 2–3 fisheye images per scene.
• Alignment: The predicted points were converted to the COLMAP world frame, fused, and downsampled via voxel sampling to match the point count of SfM for a fair comparison.
• Novelty: This is the first application of UniK3D to real fisheye imagery with FoVs >200°, despite the model not being trained on such extreme real-world data.

3. Key Contributions

1. First Real-World Evaluation: Provided the first systematic evaluation of Fisheye-GS and 3DGUT on real images with FoVs >180° across diverse indoor and outdoor environments.
2. FoV Trade-off Analysis: Empirically demonstrated that 160° offers the optimal balance between scene coverage and distortion management, outperforming both 200° (too much distortion) and 120° (loss of context).
3. SfM Alternative: Validated UniK3D as a viable, lightweight alternative to SfM for 3DGS initialization in fisheye scenarios, achieving comparable or superior geometric accuracy in many cases with significantly faster preprocessing.
4. Benchmarking: Established a benchmark for wide-angle reconstruction, highlighting the specific failure modes of current methods at extreme peripheries.

4. Key Results

Impact of Field of View

• 160° is Optimal: Both methods achieved their best performance (highest SSIM, lowest LPIPS) at 160°.
  • At 200°, performance degraded due to projection ambiguity near the image boundary (especially for 3DGUT) and severe distortion.
  • At 120°, performance dropped because excessive cropping removed essential scene context, lowering PSNR.
• Method Comparison:
  • 3DGUT excelled in compact, bounded scenes (e.g., Kitchen, Hall) where its non-linear distortion handling preserved peripheral details.
  • Fisheye-GS proved more stable in large-scale scenes (e.g., Road, Bridge) and outdoor environments, likely due to its tighter coupling with COLMAP's sparse initialization and simpler equidistant model.

Impact of Initialization (SfM vs. UniK3D)

• Fisheye-GS: Depth-based initialization (UniK3D) performed competitively with or better than SfM in most scenes. It provided more accurate geometric coverage from fewer views and reduced preprocessing time from ~1 hour (SfM) to ~10 seconds.
• 3DGUT: Depth-based initialization yielded mixed results. While it worked well in central regions, it often degraded performance at the periphery due to projection ambiguity amplifying errors in the Unscented Transform.
• General Finding: UniK3D successfully generated geometrically accurate point clouds from just 2–3 images, even in challenging conditions like fog, glare, and open sky, despite lacking specific training on 200° real fisheye data.

5. Significance and Conclusion

This paper establishes that fisheye-based 3D Gaussian Splatting is feasible without heavy, error-prone SfM preprocessing.

• Practicality: The study demonstrates that monocular depth estimation (UniK3D) can replace traditional SfM pipelines, drastically reducing computational costs and initialization time while maintaining reconstruction quality.
• Design Guidelines: It provides critical design guidelines for future systems:
  • Limit effective FoV to ~160° to balance coverage and distortion.
  • Choose 3DGUT for small, distortion-heavy indoor scenes and Fisheye-GS for large-scale or outdoor environments.
• Future Work: The authors suggest future research could focus on regressing Gaussian parameters directly from monocular estimators and extending these methods to even larger-scale scenes.

In summary, the work bridges a critical gap in 3D reconstruction by validating the use of ultra-wide fisheye cameras with 3DGS and offering a robust, depth-based initialization strategy that overcomes the limitations of traditional SfM in highly distorted environments.