Technical Summary: GeoStereo

Problem Statement

Stereo matching and surface normal estimation are fundamental tasks in 3D vision, providing complementary descriptions of scene geometry. While recent foundation models have improved stereo matching accuracy, existing feed-forward approaches struggle in challenging scenarios such as low-light environments, highly reflective surfaces, transparent objects, and weakly textured regions. These failures are attributed to the long-tail distribution of training data and the inherent limitations of data-hungry feed-forward paradigms in handling ill-posed geometric regions. Furthermore, while diffusion models have shown promise in monocular depth and normal estimation, their integration with stereo geometry for joint disparity and normal prediction remains underexplored.

Methodology

The authors propose GeoStereo, a unified framework that couples a feed-forward stereo matching pipeline with a diffusion-based normal estimation branch. The core philosophy is to leverage the strong structural priors of diffusion models to enhance disparity estimation in ambiguous regions, while utilizing the geometric constraints of stereo disparity to guide normal prediction.

Framework Architecture

As illustrated in the paper, the framework operates through two main interconnected pathways:

1. Stereo Matching Branch:

   • Utilizes a weight-shared feature extractor (initialized from strong monocular priors like DepthAnythingV2) to process rectified stereo pairs ($I_l, I_r$).
   • Constructs a hybrid cost volume and employs a soft argmin operation to generate an initial disparity ($D_{init}$).
   • Refines this initial prediction using a ConvGRU-based update module to produce the final disparity ($D_{final}$).

2. Disparity-to-Normal Initialization (D2N):

   • To bridge the two tasks, $D_{init}$ is converted into a coarse, geometry-aware surface normal prior ($N_{init}$).
   • This is achieved by transforming disparity into a relative depth representation ($Z$), constructing a pseudo-3D point map, and computing local tangential directions via central differences to derive the initial normal vector.
   • This step provides a structured geometric starting point for the diffusion process, reducing ambiguity.

3. Diffusion-Based Normal Estimation Branch:

   • Input Conditioning: The branch receives two key inputs:
     • The geometry-aware prior $N_{init}$ derived from disparity.
     • A Warp-to-Left-View condition ($I_{r \to l}$), where the right image is warped into the left-view coordinate system using $D_{init}$. This ensures spatial alignment and provides complementary cross-view appearance information.
   • Denoising Process: A U-Net (based on Stable Diffusion v2.1) performs iterative denoising. The model is trained using $x_0$-reparameterization to predict the clean latent directly.
   • Output: The refined latent is decoded via a VAE decoder to produce the final high-fidelity surface normal map ($N_{final}$).

Optimization Strategy

The framework employs a consistency training strategy with a two-stage approach:

• Stage 1: The disparity and normal branches are optimized separately to obtain stable initial predictions.
• Stage 2: The entire framework is jointly optimized. The total loss function combines the disparity loss ($L_{disp}$) and the normal estimation loss ($L_{norm}$).
• Gradient Flow: Crucially, both the D2N initialization and the warp-to-left-view operations are differentiable with respect to $D_{init}$. This allows gradients from the normal objective to propagate back to the stereo branch, enabling the diffusion priors to regularize disparity estimation while the stereo geometry guides normal refinement.

Key Contributions

1. Unified Framework: The proposal of GeoStereo, which integrates feed-forward stereo matching with diffusion-based normal estimation for simultaneous disparity and surface normal prediction.
2. Coupling Mechanism: The design of an effective interaction mechanism where disparity informs normal initialization (D2N) and provides aligned cross-view conditions (Warp-to-Left), while the diffusion branch provides structural priors to enhance disparity in ill-posed regions.
3. Zero-Shot Generalization: Demonstrated robust performance in challenging, unseen scenarios (low-light, reflections, transparency) without target-domain fine-tuning.

Experimental Results

The authors evaluated GeoStereo on multiple zero-shot benchmarks, comparing it against state-of-the-art (SOTA) feed-forward and diffusion-based methods.

• Disparity Estimation: On benchmarks including KITTI, NYUv2, ScanNet, Hypersim, and DREDS, GeoStereo achieved Rank-1 or highly competitive performance. For instance, on KITTI, it achieved an EPE of 0.8642 and a D1 outlier rate of 2.88%, outperforming FoundationStereo (0.8679 EPE, 2.96% D1) and other SOTA models.
• Surface Normal Estimation: On indoor benchmarks (iBims-1, NYUv2, DIODE, ScanNet), the method delivered the best or highly competitive normal estimation accuracy. On iBims-1, it achieved a mean angular error of 17.00° and 68.92% of pixels with error < 11.25°, surpassing Marigold and StableNormal.
• Qualitative Performance: Visual comparisons (Fig. 5) highlight significant improvements in challenging regions such as reflective surfaces, transparent objects, and low-light areas, where the method produces more reliable disparity and sharper normal boundaries compared to baselines.
• Ablation Studies:
  • Architecture: The parallel design combining diffusion and feed-forward networks outperformed sequential or purely feed-forward parallel designs.
  • Warp-to-Left-View: Using the warped right-view condition yielded better results than using the raw right view or left view alone, confirming the value of spatially aligned cross-view cues.
  • D2N Initialization: Removing the disparity-to-normal initialization degraded performance in both tasks, validating the necessity of the geometric prior for the diffusion process.

Significance and Claims

The paper claims that GeoStereo represents a significant step forward in robust 3D geometry estimation by effectively unifying two complementary tasks. Its primary significance lies in:

• Overcoming Feed-Forward Limitations: By integrating diffusion priors, the framework addresses the failure modes of traditional stereo matching in ill-posed regions (e.g., reflections, transparency) where correspondence is ambiguous.
• Mutual Reinforcement: The design demonstrates that disparity and normal estimation can mutually benefit; the diffusion branch provides structural constraints for disparity, while stereo geometry offers reliable initialization for normal refinement.
• Generalization: The method achieves strong zero-shot generalization across diverse synthetic and real-world datasets, suggesting that the learned geometric priors are robust to domain shifts.

The authors note that while the method is effective, the inclusion of the diffusion branch introduces computational overhead, leaving room for future work on efficiency improvements and extension to other intrinsic geometric tasks like albedo and material estimation.