MLG-Stereo: ViT Based Stereo Matching with Multi-Stage Local-Global Enhancement

1. Problem Statement

Stereo matching aims to estimate dense pixel-wise disparity (depth) between rectified image pairs. While Convolutional Neural Networks (CNNs) have excelled at preserving local details and handling arbitrary resolutions, they suffer from limited receptive fields, leading to poor performance in textureless regions and weak zero-shot generalization. Conversely, Vision Transformer (ViT) based methods offer robust global context understanding and strong zero-shot capabilities but face two critical limitations:

1. Resolution Sensitivity: They struggle with ultra-high-resolution images and often require resizing, causing a loss of fine geometric details.
2. Local Detail Loss: Their reliance on global tokens often compromises fine-grained local predictions compared to CNNs.
3. Pipeline Limitations: Existing ViT-based approaches typically restrict global modeling to the encoder stage. Decoders often rely solely on local cost volumes, leading to optimization processes prone to local optima and requiring excessive iterations to converge.

2. Methodology: MLG-Stereo

The authors propose MLG-Stereo, a unified pipeline that extends local-global enhancement across all stages of the stereo matching process (encoding, matching, and decoding). The framework consists of three core components:

A. Multi-Granularity Feature Network (MGFN)

• Goal: To bridge the gap between global semantic understanding and local geometric details while supporting arbitrary input resolutions.
• Mechanism:
  • Dual-Path Encoding: The network processes images via two parallel paths using a shared-weight DINOv2 backbone (a Vision Foundation Model):
    1. Full Image Encoder: Processes the entire image to capture global context.
    2. Patch Image Encoder: Processes overlapping patches (at full and half resolutions) to capture fine-grained local details.
  • Feature Fusion: Features extracted from different scales and depths are merged using a Feature Fusion Network to create a multi-scale feature pyramid.
  • Outcome: This allows the model to extract high-quality features for inputs of arbitrary scales, effectively combining the global priors of VFMs with the local precision of patch-based processing.

B. Local-Global Cost Volume (LGCV)

• Goal: To construct a cost volume that captures both local correlations and long-range global dependencies without incurring prohibitive memory costs.
• Mechanism:
  • Local Cost Volume (LCV): Generated using Group-Wise Correlation (GWC) and 3D hourglass networks to preserve fine local matching details.
  • Global Cost Volume (GCV):
    • The Grouped All-Pairs Correlation (GAPC) is compressed into a fixed-length latent space using learnable latent tokens to reduce memory overhead for large disparities.
    • Bidirectional Attention Blocks are applied to these latent tokens to aggregate global geometric and contextual information.
    • Locally-grouped Self-Attention (LSA) and Global Sub-sample Attention (GSA) are used to further refine the global matching information.
  • Outcome: The LGCV provides a rich representation that integrates local precision with global awareness, overcoming the limited receptive field of 3D CNNs.

C. Local-Global Guided Recurrent Unit (LGRU)

• Goal: To guide the iterative disparity refinement process using global priors, ensuring faster convergence and avoiding local optima.
• Mechanism:
  • Cross-Attention Guidance: At each iteration, the current disparity is used to lookup the Local Cost Volume (LCV). This local information is then fused with the Global Cost Volume (GCV) via a Cross-Attention mechanism.
  • Iterative Update: The resulting enhanced feature, combined with the local cost and current disparity, is fed into a Multi-level ConvGRU block to update the hidden state and predict residual disparity.
  • Outcome: Global matching priors explicitly steer the local optimization, correcting errors in ill-posed regions (e.g., occlusions, weak textures) and accelerating convergence.

3. Key Contributions

1. Unified Pipeline Design: MLG-Stereo is the first framework to systematically extend local-global enhancement beyond the encoder, integrating it into the cost volume construction and iterative decoding stages.
2. Multi-Granularity Feature Network (MGFN): Overcomes the fixed-resolution constraints of Vision Foundation Models (VFMs) by fusing global full-image features with multi-scale patch features, enabling robust performance on arbitrary resolutions.
3. Efficient Global Matching: Introduces the LGCV and LGRU, which utilize latent token compression and cross-attention to inject global guidance into the iterative optimization process, significantly improving robustness in ambiguous regions.
4. State-of-the-Art Performance: The method achieves leading results on major benchmarks while maintaining strong zero-shot generalization capabilities.

4. Experimental Results

The model was evaluated on Middlebury, KITTI-2012, and KITTI-2015 benchmarks.

• Middlebury Benchmark:
  • Achieved a Bad 2.0 error of 1.76% (using DINOv2-G), outperforming the previous leader, FoundationStereo (1.84%), by 4.35%.
  • Demonstrated superior performance on high-resolution images where other ViT-based methods degrade.
• KITTI Benchmarks:
  • KITTI-2015: Achieved a D1-all error of 1.36%, outperforming AIO-Stereo (1.54%) and Monster (1.41%).
  • KITTI-2012: Achieved a 2-all error of 1.71%, showing significant gains over recent baselines.
• Zero-Shot Generalization:
  • When trained only on the synthetic SceneFlow dataset and tested on Middlebury, MLG-Stereo achieved state-of-the-art results (EPE 0.53 on Middlebury Q), proving the robustness of the VFM-based architecture.
• Efficiency:
  • While computationally heavier than pure CNNs due to the VFM backbone, MLG-Stereo is significantly more efficient than other VFM-based methods (e.g., FoundationStereo) in terms of inference time (8.6s vs 12.4s) and memory usage (26GB vs 44GB) on high-resolution inputs, thanks to the latent token compression strategy.
• Convergence: The model converges faster, achieving the baseline's performance in only 4 iterations compared to the baseline's requirement for more steps.

5. Significance and Future Work

Significance:
MLG-Stereo represents a paradigm shift in stereo matching by successfully marrying the global context capabilities of Vision Transformers with the local detail preservation of CNNs. It solves the critical trade-off between resolution sensitivity and global modeling, offering a solution that is both highly accurate and robust to varying input scales and domain shifts (zero-shot).

Limitations & Future Directions:

• Computational Cost: The reliance on large-scale VFMs (DINOv2) results in high inference latency and memory usage, making deployment on edge devices challenging. Future work will focus on knowledge distillation and model sparsification.
• Ill-Posed Regions: While improved, extreme cases like thin railings or mutual occlusions still pose challenges. The authors plan to integrate depth-specific foundation models (e.g., Depth Anything) to provide stronger geometric priors for these ambiguous regions.