FullCircle: Effortless 3D Reconstruction from Casual 360$^\circ$ Captures

1. Problem Statement

The paper addresses the bottleneck in scalable 3D scene reconstruction: the difficulty of capturing high-quality data casually (i.e., without strict protocols or expert operators).

• Limitations of Perspective Cameras: Traditional 3D reconstruction relies on perspective cameras with narrow fields of view (FOV). Achieving sufficient coverage and feature correspondences requires dense, redundant capture (often 10–30× more images), making casual data collection inefficient.
• Limitations of 360° Cameras: While 360° cameras offer superior coverage and calibration constraints, existing reconstruction methods for them face two major hurdles:
  1. Preprocessing Requirements: Many methods require undistorting images into cubemaps or equirectangular projections, which introduces stitching artifacts and information loss.
  2. The "Capturer" Distractor: In casual handheld 360° capture, the camera operator is inevitably visible in the frame. This introduces photometric inconsistencies that confuse standard radiance field optimizers (like 3DGS), leading to severe artifacts or the reconstruction of the operator themselves. Existing robust methods often fail when the operator remains static for extended periods.

2. Methodology: FullCircle Pipeline

The authors propose a training-free, robust pipeline that reconstructs 3D scenes directly from raw dual-fisheye images without stitching or undistortion preprocessing.

A. Raw Fisheye Input Strategy

Instead of using stitched equirectangular images (which suffer from misalignment and stitching artifacts), the method uses the raw, un-stitched dual-fisheye frames directly.

• Advantage: This preserves the full optical information and avoids artifacts at the stitching boundary.
• Implementation: The pipeline uses 3D Gaussian Ray Tracing (3DGRT), a variant of 3D Gaussian Splatting (3DGS) capable of rendering non-pinhole (fisheye) images without prior undistortion.

B. Capturer Mask Estimation (The Core Innovation)

To handle the visible camera operator, the authors introduce a two-stage, training-free masking strategy leveraging pre-trained vision foundation models (YOLOv8 and SAMv2):

1. Initial Detection:
   • The 360° image is projected into 16 overlapping virtual 90° pinhole views.
   • YOLOv8 detects the "person" in these views.
   • SAMv2 generates an initial mask around the detected person.
   • The global direction of the capturer is estimated by averaging the coordinates of the detected pixels.
2. Synthetic Re-centering & Refinement:
   • The system synthesizes a new 180° fisheye view centered on the detected capturer.
   • SAMv2 is re-run on this synthetic view using the image center as a prompt. Because the capturer is now centered, the distortion is minimized, and the segmentation is highly accurate.
   • Temporal Consistency: The mask is propagated through time. Since the capturer is centered in the synthetic view, the segmentation remains stable even if the operator is static.
   • The refined masks are mapped back to the original dual-fisheye inputs.
3. Boundary Masking: A dilated mask is applied to the outer edges of the fisheye lenses to remove chromatic aberration and distortion artifacts.

C. Reconstruction

• Camera Pose Estimation: COLMAP is used to estimate poses directly from the raw dual-fisheye images (using SphereSfM for omnidirectional data).
• Training: The masked raw fisheye frames are fed into the 3DGRT model. The masks prevent the optimizer from learning the geometry of the human operator.

3. Key Contributions

1. Robust Casual 360° Pipeline: A complete system for 3D reconstruction from raw dual-fisheye videos that requires no special capture protocols and automatically removes the camera operator.
2. Raw Fisheye Optimization: Demonstrates that training directly on raw fisheye images (via 3DGRT) yields higher fidelity than methods that undistort images to perspective views first, due to better coverage and lack of stitching artifacts.
3. Training-Free Operator Removal: A novel masking strategy using off-the-shelf foundation models (YOLOv8 + SAMv2) with synthetic re-centering, avoiding the need for costly data collection and fine-tuning of segmentation models for distorted imagery.
4. New Benchmark Dataset: Introduction of a multi-tiered dataset of 9 diverse scenes captured with a consumer-grade Insta360 X4. It includes:
   • Raw dual-fisheye video with the operator visible.
   • Dedicated "golden" test sets captured on tripods (operator-free) for unbiased evaluation.
   • Scenes ranging from "Easy" (static indoor) to "Hard" (dynamic, low-light, multiple distractors).

4. Experimental Results

The authors evaluated FullCircle against state-of-the-art baselines, including vanilla 3DGS, robust perspective methods (SLS-MLP, NOTG), and robust 360° methods (OmniLocalRF, 3DGRT).

• Dual-Fisheye vs. Perspective: Experiments showed that using raw dual-fisheye images for both calibration and reconstruction significantly outperforms using simulated perspective images, especially for novel views outside the training trajectory.
• Raw vs. Undistorted: Training on raw fisheye images (FullCircle) achieved higher PSNR and visual quality compared to baselines that required undistorting images to perspective views (which introduced resampling artifacts).
• Distractor Handling:
  • Static Operator: FullCircle successfully handled cases where the operator stood still for long periods, a scenario where methods like OmniLocalRF and SLS-MLP failed (reconstructing the operator as part of the scene).
  • Quantitative Performance: FullCircle achieved the highest mean PSNR (29.6 dB) across the dataset, outperforming the next best robust baseline (SLS-MLP at 28.3 dB) and non-robust baselines.
• Ablation Studies:
  • Removing the "synthetic re-centering" step caused SAMv2 to fail in highly distorted regions, leading to noisy reconstructions.
  • Comparing against OmniSAM (a specialized omnidirectional segmenter), FullCircle's approach was more robust and did not require retraining.

5. Significance and Impact

• Scalability: By enabling "effortless" capture with consumer 360° cameras, this method paves the way for massive, scalable 3D dataset collection, which is crucial for training generalizable feed-forward 3D reconstruction models.
• Workflow Simplification: It removes the need for tedious capture protocols (e.g., ensuring no static periods, avoiding the operator in the frame) and complex preprocessing (stitching/undistortion).
• Benchmarking: The released dataset fills a critical gap in the literature, providing the first public benchmark for robust 360° reconstruction from casual, handheld captures with visible operators.
• Future Directions: The work highlights that while static distractors are a challenge, the redundancy of 360° data allows for robust handling of dynamic distractors, suggesting a path toward fully automated 3D content creation.

Conclusion: FullCircle demonstrates that 360° cameras, when paired with a robust pipeline that respects raw optical data and leverages foundation models for masking, are superior to perspective cameras for casual 3D reconstruction, offering wider coverage, better calibration, and higher-quality novel view synthesis.