💻 computer science

Holo360D: A Large-Scale Real-World Dataset with Continuous Trajectories for Advancing Panoramic 3D Reconstruction and Beyond

The paper introduces Holo360D, a novel large-scale dataset featuring continuous panoramic sequences paired with high-quality 3D geometry, designed to overcome the limitations of discrete trajectories and spherical distortions in advancing feed-forward panoramic 3D reconstruction.

Original authors: Jing Ou, Zidong Cao, Yinrui Ren, Zhuoxiao Li, Jinjing Zhu, Tongyan Hua, Shuai Zhang, Hui Xiong, Wufan Zhao

Published 2026-04-27

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jing Ou, Zidong Cao, Yinrui Ren, Zhuoxiao Li, Jinjing Zhu, Tongyan Hua, Shuai Zhang, Hui Xiong, Wufan Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to navigate a room using only a 360-degree camera (the kind that sees everything at once, like a bubble).

Currently, most robots are trained using "normal" photos—the kind you take with your phone. But 360-degree photos are tricky; they are stretched and distorted, like looking at a person through a funhouse mirror. Because of this, most robots get "motion sickness" or become very confused when they try to use 360-degree views to understand the 3D world.

The researchers who wrote this paper have created a "Super-Training Manual" for these robots, which they call Holo360D.

Here is the breakdown of why this is a big deal, using some simple analogies:

1. The Problem: The "Stuttering" Map

Imagine you are trying to learn a hiking trail, but instead of a smooth video of the path, someone gives you a series of disconnected snapshots taken every 100 yards. You’d have a hard time understanding the flow of the trail.

Existing datasets for 360-degree cameras are like those disconnected snapshots. They are "jumpy." The researchers realized that to truly understand 3D space, a robot needs to see a continuous movie of the world, not just a slideshow of random spots. Holo360D provides this smooth, continuous "movie" of movement.

2. The Solution: The "High-Definition Blueprint"

To learn 3D, a robot needs to know exactly how far away things are. Most current datasets provide "blurry" or "holey" information. It’s like trying to learn to build a Lego castle, but the instruction manual has missing pages and smudged pictures. If the manual says a wall is there, but the picture shows a hole, the robot gets confused.

The researchers used professional-grade laser scanners (LiDAR) to create a perfect, high-definition "blueprint" of the world. They even went through a "digital cleaning" process:

Denoising: Like erasing pencil smudges on a drawing.
Hole Filling: Like using digital "spackle" to fix cracks in a wall so the robot doesn't think there's a bottomless pit where a window should be.
Remeshing: Like smoothing out a wrinkled sheet so the robot can see the true shape of a chair or a table.

3. The Result: A Smarter Robot

The researchers took existing "AI brains" (models like $\pi3$ and VGGT) and gave them this new, high-quality training manual.

The results were like taking a student who was struggling with a textbook and giving them a high-definition, interactive 3D hologram instead. The robots became much better at:

Knowing where they are: They stopped getting lost.
Seeing through the "trickery": They became much better at understanding shapes, even when looking at tricky things like glass windows or complex furniture.

Summary in a Nutshell

If learning 3D reconstruction is like learning to draw a perfect sphere, previous datasets were like giving you a crumpled, blurry photocopy of a circle. Holo360D is like giving you a crisp, high-resolution, 3D model to trace. It provides the smoothness (continuous movement) and the clarity (perfect depth) that AI needs to finally master the 360-degree world.

Technical Summary: Holo360D Dataset

1. Problem Statement

The paper identifies a critical gap in the development of feed-forward 3D reconstruction models (e.g., DUST3R, VGGT) when applied to panoramic imagery. While these models excel at perspective images, they struggle with panoramas due to:

Spherical Distortion: The equirectangular projection causes non-uniform sampling and severe stretching near the poles, invalidating geometric priors learned from perspective views.
Dataset Limitations: Existing panoramic datasets (like Matterport3D or KITTI-360) suffer from three main issues:
1. Small Scale: Insufficient sample sizes for effective model fine-tuning.
2. Poor Depth Quality: Incomplete depth maps due to occlusions, glass reflections, and poor alignment between RGB and depth data.
3. Trajectory Discontinuity: Most datasets use discrete, fixed camera locations, which prevents the training of models capable of handling continuous, multi-view trajectories.

2. Methodology

The authors propose Holo360D, a large-scale dataset designed to overcome these hurdles through a rigorous data acquisition and processing pipeline.

A. Data Collection:

Hardware: A handheld system combining a 3D laser scanner (LiDAR, RTK-GNSS, IMU, pinhole cameras) with a 360° camera.
Strategy: To ensure high completeness, the team used a continuous traversal strategy with overlapping trajectories, moving at controlled speeds (0.3–0.6 m/s). This resulted in 109,495 panoramas and 31.5 km of trajectory.

B. Reconstruction & Post-Processing Pipeline:

Offline Reconstruction: Raw data is processed via a high-precision offline SLAM system and global bundle adjustment to align camera poses and point clouds. Poisson surface reconstruction is then used to create dense meshes.
Data Denoising: A three-step process (manual cropping, radius outlier removal, and visual inspection) removes motion artifacts and specular reflections.
Mesh Hole Filling: To fix gaps caused by glass or occlusions, the authors use a curvature-preserving triangulation method. Small holes are filled automatically, while large holes are subdivided into smaller segments to minimize geometric distortion.
Region-specific Remeshing: To preserve fine details (like furniture) without excessive computational cost, the system retains general meshes for flat surfaces (walls/floors) but reconstructs complex regions using high-resolution original point clouds.
Depth Map Creation: High-quality depth maps are rendered by reprojecting the processed meshes and point clouds into the 360° image space.

3. Key Contributions

Holo360D Dataset: The first large-scale real-world panoramic 3D dataset providing continuous trajectories and accurately aligned, high-completeness depth maps.
Advanced Post-processing Pipeline: A specialized workflow (denoising, hole filling, remeshing) that ensures the ground truth geometry is robust enough for high-quality supervision.
New Benchmark & Insights: The establishment of a fine-tuning benchmark that provides empirical guidance on how to adapt perspective-based models to the panoramic domain.

4. Results and Key Findings

The authors evaluated the dataset by fine-tuning state-of-the-art models (specifically $\pi3$ , VGGT, and FLARE).

Optimal Fine-tuning Strategies:
- Input Representation: Splitting panoramas into multiple perspective views (split views) significantly outperforms using raw panoramic images, as it mitigates spherical distortion.
- View Decomposition: An 8-view configuration (horizontal coverage) is superior to a 10-view configuration; adding vertical (up/down) views introduces noise from low-texture ceilings or dynamic floor movements.
- Supervision Type: Mesh-based depth provides stronger geometric constraints and better pose estimation than sparse point-based depth.
Performance Gains: Fine-tuning on Holo360D consistently improved camera pose estimation (ATE, RPE) and point cloud reconstruction (Accuracy and Completeness) across all tested models.
Superiority over Existing Data: Compared to Matterport3D, Holo360D-trained models showed much higher accuracy and completeness, particularly in challenging areas like transparent glass surfaces.

5. Significance

Holo360D addresses a fundamental bottleneck in computer vision: the transition from perspective-based 3D reconstruction to seamless, continuous panoramic reconstruction. By providing high-fidelity, continuous, and large-scale training signals, this work enables the development of next-generation models capable of understanding complex, real-world environments through a 360° lens, which is vital for robotics, AR/VR, and autonomous navigation.