Technical Summary: Unified Panoramic Geometry Estimation via Multi-View Foundation Models (PaGeR)

Problem Statement

While monocular geometry estimation from perspective images has matured significantly through foundation models capable of zero-shot 3D reconstruction, extending these capabilities to panoramic (360°) imagery remains challenging. Existing panoramic approaches face two primary limitations:

1. Data Scarcity and Overfitting: High-quality panoramic datasets with metrically accurate depth and surface normals are laborious to collect and scarce, causing models to overfit to small datasets and struggle with generalization.
2. Geometric Distortion: Most methods rely on Equirectangular Projection (ERP), which introduces severe nonlinear distortions (extreme stretching near poles). This domain shift degrades the efficacy of translation-invariant architectures trained on perspective data and hinders the transfer of powerful perspective priors to the panoramic domain.

Methodology

The authors introduce PaGeR (Panoramic Geometry Reconstruction), a framework designed to lift state-of-the-art perspective foundation models (specifically Depth Anything 3, or DA3) to the panoramic domain without requiring massive retraining or custom architectures.

Core Strategy: Cubemap Representation

Instead of processing panoramas as distorted equirectangular images, PaGeR represents the input as a cubemap—six 90° field-of-view perspective images mapped to the faces of a unit cube. This approach:

• Aligns with the inductive priors of perspective-trained models.
• Provides uniform sampling of the ray space.
• Avoids the geometric distortions inherent in ERP.

Architectural Adaptations

To adapt the multi-view transformer backbone (DA3) for holistic 360° estimation, the authors implement several key modifications:

1. Implicit Encoder Synchronization: The Vision Transformer (ViT) encoder is fine-tuned on panoramic data without structural changes. By supplying fixed camera matrices and axis-aligned extrinsics for the six faces, the network learns to route context and synchronize features across adjacent cubemap faces via cross-view attention layers.
2. Spherically Aware Decoder Padding: To prevent geometric and photometric discontinuities at the boundaries of the cubemap faces during dense upsampling, the decoder employs cross-face valid padding. Instead of standard zero-padding, this layer dynamically extracts features from geometrically adjacent faces, ensuring seamless transitions across the spherical manifold.
3. Hybrid Training Regime: To preserve the rich priors of the pre-trained perspective model and mitigate the sim-to-real domain gap, the model undergoes mixed training. It alternates between:
   • Panoramic batches: Processing the full six-face configuration with active cross-face padding.
   • Perspective batches: Processing single real-world images (warped to 90° FoV) with standard zero-padding.
     This strategy prevents catastrophic forgetting while teaching the model to handle continuous spherical observations.

Unified Multi-Task Decoding

PaGeR utilizes a shared transformer backbone with specialized, coupled task heads to decode multiple geometric modalities in a single forward pass:

• Scale-Invariant (SI) Depth: Predicts local log-planar depth to compress metric variance. It uses a composite loss balancing pixel precision, gradient preservation, and normal consistency.
• Surface Normals: A parallel branch initialized with pre-trained depth weights, optimized with a joint objective combining cosine similarity and perceptual loss to prevent over-smoothing.
• Metric Scale: A coarse, low-resolution decoder predicts a global scale factor by aligning the SI depth with a coarse metric log-depth map. This decouples absolute scale estimation from high-resolution structural reconstruction.
• Sky Segmentation: A lightweight branch segments the sky to explicitly decouple infinite depth regions, stabilizing metric regression for finite structures.

Key Contributions

1. Novel Adaptation Strategy: A method to transfer perspective foundation models to the 360° domain using cubemap representation and hybrid training, sidestepping ERP distortions.
2. PaGeR Framework: A unified model featuring a shared transformer backbone and specialized heads for simultaneous prediction of SI depth, metric depth, surface normals, and sky masks.
3. ZüriPano Benchmark: A new dataset of real-world outdoor urban panoramas with high-accuracy LiDAR scans, curated to address the lack of rigorous evaluation data for long-range outdoor scenarios.
4. State-of-the-Art Performance: Demonstrated zero-shot generalization across diverse indoor and outdoor environments, outperforming existing methods limited to specific settings.

Experimental Results

The authors evaluated PaGeR on both indoor (Matterport3D360, Stanford2D3DS) and outdoor (ZüriPano) datasets.

• Quantitative Performance: PaGeR achieved state-of-the-art results across all benchmarks. On the challenging outdoor ZüriPano dataset, it reduced the Absolute Relative Error (AbsRel) from 18.27 (previous best, RPG360) to 9.36. In metric depth estimation, it achieved an RMSE of 530.85 on ZüriPano, significantly outperforming the next best method (716.38).
• Surface Normals: On the Structured3D dataset, PaGeR achieved a Mean Angular Error of 5.49° and an MSE of 174.9, surpassing specialized architectures like PanoNormal and HyperSphere.
• Qualitative Results: Visual comparisons show PaGeR recovers sharper boundaries and more accurate global structures compared to baselines like DAP and DA2, which often suffer from over-smoothing. Point cloud reconstructions demonstrate the framework's ability to preserve intricate indoor structures and complex outdoor topographies without the boundary artifacts common in other methods.
• Efficiency: The model processes a full 2K panorama in 0.5 seconds on a single consumer-grade GPU, consuming 12.8 GB of memory.

Significance and Claims

The paper claims that PaGeR establishes a new state of the art for zero-shot panoramic geometry reconstruction. Its primary significance lies in demonstrating that powerful perspective foundation models can be effectively adapted to the spherical domain through minimal architectural changes (cubemap representation and cross-face padding) and a hybrid training strategy.

The authors emphasize that their approach:

• Retains Rich Priors: By leveraging the pre-trained DA3 backbone, the model maintains a strong understanding of 3D surface structure without needing to relearn from scratch.
• Ensures Global Consistency: The cross-face padding and synchronized encoder allow for seamless reconstruction across the entire 360° view, resolving the boundary discontinuity issues of previous multi-face approaches.
• Generalizes Robustly: The framework performs exceptionally well in unseen environments, including long-range outdoor scenes where previous methods often fail.

The paper concludes that while the current instantiation uses DA3, the architectural principles are compatible with other geometric transformers, offering a generalized methodology for bridging the domain gap between perspective and panoramic 3D scene understanding. The authors acknowledge limitations common to monocular estimation, such as difficulties with specular or transparent surfaces, and note that subtle boundary misalignments can still occur in rare, structurally complex scenes.