💻 computer science

Pose-Free Omnidirectional Gaussian Splatting for 360-Degree Videos with Consistent Depth Priors

This paper introduces PFGS360, a pose-free omnidirectional 3D Gaussian Splatting method that reconstructs 3D scenes from unposed 360-degree videos by leveraging spherical consistency for camera pose estimation and depth-inlier-aware densification for high-fidelity novel view synthesis.

Original authors: Chuanqing Zhuang, Xin Lu, Zehui Deng, Zhengda Lu, Yiqun Wang, Junqi Diao, Jun Xiao

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chuanqing Zhuang, Xin Lu, Zehui Deng, Zhengda Lu, Yiqun Wang, Junqi Diao, Jun Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect, 3D digital twin of a room you just walked through, but you have a very specific problem: you don't have a GPS tracker on your camera. You just have a video of the room, and you don't know exactly where the camera was at any given moment.

This is the challenge the paper PFGS360 solves. It's like trying to assemble a giant 3D puzzle while blindfolded, where the pieces (the 3D objects) keep changing shape as you move.

Here is the breakdown of how they did it, using some everyday analogies.

The Problem: The "Blindfolded Builder"

Usually, to build a 3D model from a video, computers use a technique called SfM (Structure from Motion). Think of this as a super-smart surveyor who walks around, counts steps, and uses a laser to measure distances before building anything. It's accurate, but it's slow and requires a lot of pre-processing.

The authors wanted to skip the surveyor. They wanted to build the 3D world while watching the video, without knowing the camera's location in advance.

The Catch: When you look at a 360-degree video (like a VR headset view), the image is stretched like a map of the world. If you try to guess where you are just by looking at the picture, the math gets messy near the "North and South Poles" of the image, causing the 3D model to wobble and fall apart.

The Solution: PFGS360

The authors built a system that acts like a smart, self-correcting construction crew. They introduced two main "tools" to fix the wobbling and build a perfect model.

Tool 1: The "Consistency Detective" (Spherical Consistency-Aware Pose Estimation)

The Analogy: Imagine you are walking through a museum, looking at paintings. You take a photo, then turn around and take another. To know where you are, you look for things that match.

The Old Way: Previous methods tried to guess your position by looking at the colors of the walls. But if the lighting changes or the wall is blurry, they get confused.
The New Way (PFGS360): This tool is a "Consistency Detective." It doesn't just look at colors; it checks if the depth (how far away things are) makes sense from different angles.
- It asks: "If I look at this chair from the left, and then from the right, does the distance match up?"
- If the math says "No, that's impossible," it ignores that part of the image.
- By only trusting the parts where the depth "checks out" across the whole 360-degree sphere, it can figure out exactly where the camera is, even without a GPS. It's like solving a maze by only walking on the path that connects perfectly to the next step.

Tool 2: The "Quality Control Filter" (Depth-Inlier-Aware Densification)

The Analogy: Imagine you are trying to fill a swimming pool with water (the 3D model), but you have a hose that sometimes sprays clean water and sometimes sprays muddy sludge (bad data).

The Problem: When the computer guesses the depth of objects, it sometimes gets it wrong (the "mud"). If you let that mud into your 3D model, the walls will look wavy and the floor will look bumpy.
The New Way (PFGS360): This tool is a Quality Control Filter.
- It looks at the "muddy" guesses and the "clean" guesses.
- It keeps the "clean" water (the Depth Inliers)—the parts where the depth estimate is consistent and reliable.
- It throws away the "mud" (the Gaussian Outliers)—the parts where the computer is hallucinating or getting confused.
- It then uses the clean water to fill in the gaps in the 3D model, making it dense and detailed.

The Result: A Crystal Clear 360° World

By combining these two tools, the system can:

Figure out where the camera is just by watching the video (no GPS needed).
Build a 3D model that is so accurate you can walk around inside it virtually.

Why is this a big deal?

Speed: It skips the slow "surveyor" step.
Quality: It creates better 3D videos than previous methods that didn't use GPS, and it's even beating methods that did use GPS in some cases.
Realism: You can look around a virtual room, and the textures (like the grain of wood or the pattern on a rug) look sharp and real, not blurry or warped.

In a Nutshell

Think of PFGS360 as a self-driving 3D camera. Instead of needing a map to know where it is, it looks at the world, checks if its own measurements make sense, throws out the bad guesses, and builds a perfect, photorealistic 3D world right in front of your eyes. It turns a shaky, untracked 360-degree video into a stable, explorable digital universe.

1. Problem Statement

Omnidirectional 3D Gaussian Splatting (3DGS) is a powerful technique for 3D scene representation, particularly for VR applications like virtual tourism. However, existing methods face two critical limitations:

Reliance on SfM: Most current omnidirectional 3DGS methods depend on Structure-from-Motion (SfM) to provide camera poses and sparse point priors. SfM is computationally expensive, slow, and often fails in textureless or dynamic scenes.
Instability of Pose-Free Methods: While pose-free methods exist for perspective images, they fail when applied directly to 360-degree panoramas. The spherical projection of panoramas introduces:
- Jacobian Amplification: Errors near the poles are significantly amplified during rasterization, destabilizing pose gradients.
- Projection Inconsistencies: Spherical affine approximations cause varying projection errors across different regions, especially with large viewpoint changes.
- Incompatible Priors: Existing 3D Visual Foundation Models (VFMs) and monocular depth estimators are trained on perspective images and struggle to provide consistent 2D-3D correspondences for panoramic inputs.

Goal: To develop a framework that reconstructs high-fidelity 3D Gaussians and estimates accurate camera poses from unposed 360-degree videos without relying on SfM.

2. Methodology: PFGS360 Framework

The authors propose PFGS360, a pose-free framework that operates in a frame-by-frame manner. It consists of two core modules designed to handle the unique challenges of spherical geometry and noisy depth priors.

A. Spherical Consistency-Aware Pose Estimation (SCA-PE)

To estimate camera poses without SfM, the method avoids direct gradient back-propagation through the rasterizer (which is unstable for panoramas) and instead uses a PnP (Perspective-n-Point) based solver enhanced with spherical consistency.

2D-3D Correspondence: The system renders depth maps from the current reconstructed Gaussians ( $G_t$ ) and establishes correspondences with the unposed input image ( $I_{t+1}$ ).
Consistency Masking: To mitigate geometric inaccuracies caused by photometric overfitting, the method computes a spherical reprojection consistency map. It projects pixels between source and reference frames using depth maps and checks for:
- Tangential angular error ( $\mathcal{L}_{tan}$ ).
- Relative depth error.
- Pixels satisfying these thresholds form a "consistent mask."
Pose Solver: The camera pose for the new frame is estimated by minimizing a spherical consistency-aware reprojection error, weighted by the consistency mask and a spherical balancing coefficient ( $\lambda = \sin \theta$ ).
Post-Refinement: A photometric loss refinement step is applied using an adjacent-frame consistency mask to weight the loss, reducing noise from geometric inaccuracies.

B. Depth-Inlier-Aware Densification (DIA-Densify)

To improve the fidelity of the 3D reconstruction, the method densifies the Gaussian model using monocular depth priors while filtering out noise.

Depth Inlier Merging:
- Monocular depth maps are aligned to the 3DGS scale using the consistency masks.
- A Normalized Cross-Correlation (NCC) metric is used to identify high-confidence depth inliers by comparing warped images.
- Regions identified as consistent inliers across multiple frames are merged into the Gaussian model as new seed points.
Gaussian Outlier Pruning:
- The system accumulates consistency and inconsistency masks onto existing Gaussians.
- Gaussians with high inconsistency scores (outliers) are either replaced by the new depth inliers or have their opacity reset to allow for re-optimization.
- This effectively prunes noisy geometry while preserving reliable structures.

3. Key Contributions

First Pose-Free Omnidirectional 3DGS: The paper introduces the first framework capable of reconstructing 3D Gaussians and estimating camera poses from unposed 360-degree videos without any SfM priors.
Spherical Consistency-Aware Pose Solver: A novel module that establishes robust 2D-3D correspondences using spherical reprojection consistency, overcoming the instability of spherical rasterization gradients.
Depth-Inlier-Aware Densification: A mechanism that aggregates geometrically consistent depth priors from monocular models while actively pruning Gaussian outliers, solving the issue of inconsistent geometric priors in panoramic videos.
State-of-the-Art Performance: The method eliminates the need for ground-truth poses or SfM initialization while outperforming existing pose-aware methods that rely on them.

4. Experimental Results

The method was evaluated on the OB3D (synthetic) and Ricoh360 (real-world) datasets.

Novel View Synthesis (NVS)

OB3D Dataset: PFGS360 achieved the highest PSNR, SSIM, and lowest LPIPS across all categories (Egocentric/Non-Egocentric, Indoor/Outdoor).
- It outperformed the second-best pose-free method (3R-GS) by 4.42 dB in Egocentric trajectories and 4.06 dB in Non-Egocentric trajectories.
- Crucially, it surpassed pose-aware methods (ODGS, OmniGS) that rely on ground-truth SfM priors, demonstrating that the proposed consistency priors are more effective than sparse SfM points in these scenarios.
Ricoh360 Dataset: In challenging real-world scenarios with motion blur and occlusion, PFGS360 achieved 28.05 dB PSNR, outperforming the runner-up by 1.78 dB.

Camera Pose Estimation

Accuracy: PFGS360 achieved significantly lower Relative Pose Error (RPE) and Absolute Trajectory Error (ATE) compared to other pose-free methods.
- On OB3D Non-Egocentric (large motion), PFGS360 achieved an RPE_t of 0.040, whereas competitors like 3R-GS failed to recover accurate poses (RPE_t > 2.0).
Visual Quality: Visualizations showed that PFGS360 eliminated artifacts (e.g., floating pillars, blurred textures) common in other methods, producing sharp, photorealistic panoramic views even in complex outdoor scenes.

Ablation Studies

Removing the SCA-PE module resulted in significant pose errors and rendering artifacts.
Removing the DIA-Densify components (Depth Inlier Merging and Outlier Pruning) led to noisy geometry and incomplete scene reconstruction, confirming the necessity of filtering unreliable depth priors.
The method showed robustness across different monocular depth models (UniK3D, DA2, DepthAnywhere), with UniK3D providing the best absolute depth consistency.

5. Significance

This work represents a major step forward in 360-degree video processing and VR/AR content creation.

Efficiency: By removing the dependency on slow, error-prone SfM pipelines, PFGS360 enables faster, more scalable 3D reconstruction from raw video footage.
Robustness: It solves the fundamental instability of applying perspective-based pose-free methods to spherical projections, making high-fidelity 3D reconstruction accessible for real-world 360-degree videos where ground truth is unavailable.
Superiority: The fact that it outperforms methods using ground-truth poses suggests that dense, consistent geometric priors (derived from the method's own consistency checks) are more valuable for reconstruction than sparse SfM points, especially in large-viewpoint-change scenarios.

The code is publicly available, facilitating further research in pose-free omnidirectional 3D reconstruction.