Technical Summary: H-OmniStereo

Problem Statement

Stereo matching on top-bottom equirectangular images offers a cost-effective framework for full-surround 3D perception, leveraging vertically aligned epipolar lines to utilize advanced perspective stereo architectures. However, the performance of existing adaptations is severely constrained by two primary factors:

1. Data Scarcity: There is a lack of large-scale, diverse, and accurately calibrated omnidirectional stereo datasets. Existing datasets (e.g., Deep360, 360SD-Net) are limited in scene variety, camera configurations, and scale, leading to poor zero-shot generalization.
2. Prior Degradation: Standard monocular depth foundation models, trained on perspective images, suffer from spherical distortions when applied to equirectangular inputs. Furthermore, recent equirectangular depth models predict radial distance to the sphere center, resulting in inconsistent depth values for the same 3D point across top and bottom views. This inconsistency renders their features ill-suited as priors for establishing reliable stereo correspondences.

Methodology

The authors propose H-OmniStereo, a zero-shot omnidirectional stereo matching framework designed to address these limitations through a large-scale synthetic dataset and a novel geometric prior.

1. Large-Scale Synthetic Dataset

To overcome data scarcity, the authors constructed a high-quality synthetic dataset using NVIDIA Isaac Sim, comprising over 2.8 million top-bottom equirectangular stereo pairs.

• Diversity: The dataset spans over 1,000 scenes and 800,000 diverse 3D assets. It includes both fixed layouts (from GRUtopia and HM3D) and randomized layouts (chaotic and realistic setups with varied lighting, materials, and object placements).
• Configuration: Camera baselines are randomized (0.05m–0.5m), and poses include randomized roll and pitch (±45°).
• Scale: This dataset is approximately 70 times larger than the largest prior omnidirectional stereo dataset and offers significantly higher diversity.

2. Heading-Aligned Normal Priors

Instead of relying on depth priors, the framework utilizes a monocular normal estimator operating in a heading-aligned coordinate system.

• Coordinate System: Unlike standard normal estimation which uses a fixed camera coordinate system, this approach defines normals in a longitude-varying system that rotates with the pixel's longitude.
• Benefits: This design ensures that identical local patterns correspond to the same normal vector regardless of heading rotation. It improves training efficiency, provides distortion-robust priors, and naturally accommodates train-test Field-of-View (FoV) mismatches and random cropping during training.
• Training: The normal estimator is trained on 8.4 million images, combining the synthetic stereo data with additional monocular images from existing datasets (Structured3D, TartanAir V2, SpatialGen).

3. Network Architecture

The H-OmniStereo pipeline follows an iterative optimization approach:

• Feature Extraction: A ViT backbone extracts multi-scale features, which are fused with spherical-aware ray embeddings via cross-attention. The heading-aligned normal estimator provides geometric priors, which are integrated with image features via a Side-Tuning Adapter.
• Iterative Matching: A 4D hybrid cost volume is constructed and filtered using Attentive Hybrid Cost Filtering. Disparity is refined iteratively using ConvGRU blocks.
• Uncertainty Estimation: An auxiliary ConvGRU module runs in parallel to the disparity refinement, explicitly quantifying the reliability of predictions to filter unreliable keypoints.

Key Contributions

1. Dataset Construction: The creation of a massive, diverse synthetic dataset (2.8M pairs) that significantly outperforms existing benchmarks in scale and variety, serving as a foundation for zero-shot generalization.
2. Heading-Aligned Normal Estimator: The proposal of a novel normal estimation method in a heading-aligned coordinate system. This approach eliminates the burden of distinguishing spatially varying normals for identical patterns, enhancing robustness to FoV mismatches and spherical distortions.
3. Zero-Shot Generalization: The demonstration that a single model trained on this synthetic data and utilizing these specific priors can achieve high accuracy on out-of-domain datasets and real-world consumer camera setups without fine-tuning.

Experimental Results

Extensive experiments validate the framework's superiority over existing methods (e.g., 360SD-Net, MODE, 360-IGEV-Stereo, DFI-OmniStereo):

• Stereo Matching: On out-of-domain datasets (3D60, 3D60 Warp, MVS GI), H-OmniStereo achieves significantly lower Mean Absolute Error (MAE) and higher outlier percentages (BP-1, D1) compared to baselines. For instance, on the 3D60 dataset, it reduced MAE from ~0.679 (360SD-Net) to 0.103.
• Generalization: The model successfully generalizes to real-world consumer 360° camera setups, reconstructing finer details and more precise spatial layouts than baselines.
• Ablation Studies:
  • Replacing the heading-aligned normal prior with perspective depth models (DepthAnythingV2) or equirectangular depth models (DA2) resulted in degraded performance, confirming the necessity of consistent geometric priors.
  • Scaling the training data from 522K to 2.8M pairs consistently improved accuracy.
  • The heading-aligned representation showed superior robustness to train-test FoV mismatches compared to standard camera-coordinate normals.
• Application: Integrated into a visual odometry pipeline, the method achieved lower Relative Translation Error (RTE) and Relative Rotation Error (RRE) compared to a state-of-the-art pinhole stereo baseline (FoundationStereo), attributed to the utilization of full 360° information and uncertainty filtering.

Significance

The paper positions H-OmniStereo as a significant step forward in overcoming the data and prior limitations that have historically hindered omnidirectional stereo matching. By decoupling the reliance on distorted depth priors and instead utilizing robust, consistent normal priors within a heading-aligned framework, the authors demonstrate that high-quality 3D perception can be achieved in a zero-shot manner. The work establishes that with sufficient synthetic data diversity and geometrically consistent priors, models can generalize effectively to unseen real-world scenarios, bridging the gap between synthetic training and consumer-grade omnidirectional perception.