ESMStereo: Enhanced ShuffleMixer Disparity Upsampling for Real-Time and Accurate Stereo Matching

Technical Summary: ESMStereo

Problem Statement
Stereo matching is a critical component for autonomous systems, yet developing deep learning models that simultaneously deliver high accuracy and real-time performance remains a significant challenge. In cost-volume-based stereo matching, accurate disparity estimation typically relies on large-scale cost volumes. However, these large volumes store substantial redundant information and necessitate computationally intensive 3D aggregation units, rendering real-time performance unattainable on resource-constrained devices. Conversely, constructing small-scale cost volumes with lightweight aggregation units facilitates real-time inference but often results in significant accuracy degradation due to insufficient matching information and the loss of geometric details. Existing strategies to mitigate this trade-off, such as edge-aware upsampling or feature-guided refinement, often struggle to recover high-frequency details effectively or introduce excessive computational overhead.

Methodology
The authors propose ESMStereo (Enhanced ShuffleMixer Stereo), a real-time stereo matching architecture designed to balance speed and accuracy by shifting computational complexity from the 3D aggregation stage to a 2D post-disparity regression stage. The architecture consists of three primary components:

1. Compact Cost Volume and Lightweight Aggregation: The network constructs cost volumes at reduced resolutions (1/4, 1/8, or 1/16 of the original image) using efficient feature extractors (EfficientNet-B2 for larger variants, MobileNetV2 for the small variant). These volumes are processed by a lightweight 3D hourglass network to generate an initial low-resolution disparity map.
2. Enhanced ShuffleMixer (ESM) Module: This is the core contribution of the paper. The ESM module addresses the information loss inherent in small-scale cost volumes through a multi-stage upsampling process:
   • Feature Fusion: It extracts features from the initial low-resolution disparity map and fuses them with contextual features from the left image.
   • Enhanced ShuffleMixer: The fused features are processed through FMBlocks (Feature Mixer Blocks). These blocks utilize a Shuffle Mixing Layer that applies LayerNorm, a channel-wise split-point MLP with residual connections, and depthwise convolutions. This mechanism mixes features via shuffling and layer splitting to focus on local correlations efficiently.
   • Pixel Shuffle and Refinement: The mixed features are upsampled using the computationally efficient pixel shuffle operation. To compensate for the lack of inherent smoothing in pixel shuffle, a compact, feature-guided hourglass refinement network is employed. This network recovers fine scene geometry and smooths the disparity estimation by integrating global and local contexts.
3. Progressive Upsampling: The disparity map is upsampled in multiple stages (e.g., from 1/16 to 1/4 to 1/1) using the ESM module, allowing the network to progressively refine the disparity map while maintaining real-time speeds.

Key Contributions

• Enhanced ShuffleMixer (ESM): The introduction of a fast, feature-guided upsampler that integrates feature fusion, local correlation focusing via ShuffleMixer layers, and a lightweight hourglass refinement network to recover scene details lost during the initial low-resolution aggregation.
• Computational Efficiency Strategy: A novel architecture design that minimizes the computational burden of 3D cost volumes and aggregation modules by transferring the workload to the post-disparity regression stage, where efficient 2D modules recover stereo scene details.
• Performance Benchmarks: The demonstration of competitive real-time performance across multiple scales (ESMStereo-S, M, and L), achieving high accuracy on standard benchmarks while maintaining low inference times.

Experimental Results
The authors evaluated ESMStereo on the SceneFlow, KITTI 2012, KITTI 2015, ETH3D, and Middlebury 2014 datasets.

• SceneFlow: The ESMStereo-L-gwc variant achieved an End-Point Error (EPE) of 0.53 pixels with an inference time of 26 ms on an RTX 4070 Super. The lightweight ESMStereo-S-gwc variant achieved 1.10 pixels EPE at 116 FPS.
• KITTI Benchmarks: On KITTI 2012, ESMStereo-L-gwc achieved a D1 error of 5.4%, and on KITTI 2015, it achieved 5.5%. These results place the model competitively against state-of-the-art real-time methods, often outperforming them in accuracy while maintaining similar or faster inference speeds.
• Edge Deployment: On the NVIDIA Jetson AGX Orin, the ESMStereo-S-gwc model achieved 91 FPS with a D1 error of 10.8%, demonstrating suitability for edge devices.
• Generalization: Models trained exclusively on the synthetic SceneFlow dataset showed strong generalization to real-world datasets (KITTI, ETH3D, Middlebury), indicating robust adaptability to unseen scenarios.
• Ablation Studies: Experiments confirmed that the group-wise correlation cost volume outperforms norm-correlation, and that the combination of the FMBlock and the feature-guided hourglass refinement significantly reduces EPE compared to baseline configurations.

Significance and Claims
The paper claims that ESMStereo successfully addresses the trade-off between speed and accuracy in stereo matching by mitigating the information loss associated with small-scale cost volumes. By utilizing the Enhanced ShuffleMixer, the network effectively restores critical details and scene geometry without relying on computationally expensive 3D operations during the upsampling phase. The authors assert that this design allows for the integration of both global and local contexts, resulting in highly accurate disparity maps in real-time. The work is positioned as a significant step forward for real-time autonomous systems, offering a compact, efficient, and accurate solution that generalizes well across diverse datasets and hardware platforms, including resource-constrained edge devices. The source code is made available to facilitate further research and deployment.