💻 computer science

Lite Any Stereo V2: Faster and Stronger Efficient Zero-Shot Stereo Matching

This paper introduces Lite Any Stereo V2 (LAS2), an ultra-fast stereo matching model series that challenges the trade-off between efficiency and zero-shot generalization by employing a latency-optimized 2D-only cost aggregation architecture and a novel three-stage training strategy to achieve state-of-the-art accuracy with significantly lower inference latency on resource-constrained platforms.

Original authors: Junpeng Jing, Ronglai Zuo, Zhelun Shen, Shangchen Zhou, Rolandos Alexandros Potamias, Stefanos Zafeiriou, Krystian Mikolajczyk, Jiankang Deng

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junpeng Jing, Ronglai Zuo, Zhelun Shen, Shangchen Zhou, Rolandos Alexandros Potamias, Stefanos Zafeiriou, Krystian Mikolajczyk, Jiankang Deng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a pair of eyes (two cameras) trying to figure out how far away objects are in a scene. This is called stereo matching. For a long time, the best "eyes" were huge, heavy, and slow—like a giant supercomputer trying to solve a puzzle. They were accurate but too heavy to carry around in a robot or a self-driving car. On the other hand, the "lightweight" eyes were fast and easy to carry, but they often got confused when they saw something new they hadn't practiced on before.

This paper introduces Lite Any Stereo V2 (LAS2), a new family of "eyes" that claims to be both super fast and surprisingly smart, even when looking at completely new scenes without any prior practice (a concept called "zero-shot").

Here is how they did it, broken down into simple analogies:

1. The New Blueprint: A 2D Shortcut

Previous fast models tried to be efficient by doing less math, but they still used a complex 3D "scaffolding" to build their understanding of depth. The authors realized that on real-world chips (like those in phones or cars), this 3D scaffolding is actually slow, even if the math looks simple on paper.

The Analogy: Imagine trying to organize a library. The old fast models were like trying to stack books in 3D towers to save space, but it took forever to climb up and down the ladders. LAS2 says, "Let's just lay the books flat on 2D shelves."
The Result: By switching to a 2D-only framework, they removed the heavy climbing. This made the model significantly faster on real hardware without losing its ability to see depth clearly.

2. The Three-Stage Training Camp

To make these lightweight eyes smart enough to handle the real world, the authors didn't just feed them data; they put them through a strict three-stage training camp:

Stage 1: The Classroom (Synthetic Data).
The model starts by studying in a perfect, computer-generated world (like a video game) where the answers are 100% correct. This gives it a solid foundation.
Stage 2: The Stress Test (Self-Distillation).
Now, the model has to learn to stay calm when things get messy. The teachers (the model itself) show the student the same picture but with weird filters, blurs, or color changes. The goal is to teach the model to recognize the shape of things, not just the specific colors or lighting. It's like learning to recognize a friend's face whether they are wearing sunglasses, a hat, or standing in the dark.
Stage 3: The Real World Internship (Real-World Pseudo-Labels).
This is the big leap. The model is sent out into the real world to look at unlabeled photos. Since there are no answer keys, a "super-smart teacher" (a massive, slow AI) guesses the answers first.
- The Filter: But the super-smart teacher sometimes makes mistakes, especially on tricky things like shiny windows or the sky. LAS2 uses a filter to throw out the teacher's bad guesses before the student learns from them.
- The Safety Net: If the student tries to learn from a really hard, confusing picture, the system puts a "cap" on how much the student can change their mind. This prevents the student from getting confused by one bad example and forgetting everything else.

3. The Results: Fast and Strong

The paper claims that this new approach creates a family of models (from small "S" to large "H") that beat the competition:

Speed: On powerful servers and on small edge devices (like the NVIDIA Orin chip found in robots), LAS2 runs 1.6x to 2.7x faster than the previous best fast models.
Accuracy: Even though it is fast, it is more accurate than other fast models when looking at real-world scenes it has never seen before. It even comes close to the accuracy of the giant, slow super-models, but runs much faster.
Visual Quality: When looking at tricky scenes (like reflections on a car or a long hallway), LAS2 produces cleaner, less "noisy" depth maps compared to older methods.

What It Can't Do (The Limitations)

The authors are honest about the limits. Even with these improvements:

The Gap: It still isn't quite as perfect as the massive, slow models that use huge "foundation" knowledge.
The Data Bottleneck: It is limited by the lack of high-quality, real-world stereo data. There just aren't enough labeled photos of the real world to train it to be perfect.
The "Impossible" Scenes: Like all current technology, it still struggles with extremely difficult situations, such as looking through a mirror, seeing a transparent glass wall, or dealing with blindingly bright lights.

In summary: LAS2 is a new way of building "smart eyes" that are light enough to carry in your pocket but sharp enough to see the world clearly, even in places they've never visited before. They achieved this by simplifying the internal structure and teaching the model to learn from a mix of perfect simulations and filtered real-world guesses.

Technical Summary: Lite Any Stereo V2 (LAS2)

Problem Statement

Recent advancements in stereo matching have achieved remarkable accuracy, particularly through large-scale foundation models and iterative refinement pipelines. However, these high-performance methods often rely on computationally expensive 3D cost aggregation, heavy backbones, or additional monocular depth priors, making them difficult to deploy on resource-constrained platforms like edge devices. Conversely, existing efficient stereo models offer faster inference but are generally perceived as lacking the capacity for strong zero-shot generalization, often requiring domain-specific fine-tuning. This paper challenges the assumption that lightweight models cannot achieve strong zero-shot generalization without sacrificing efficiency, aiming to bridge the gap between high accuracy and low latency.

Methodology

1. Architecture Design: 2D-Only Cost Aggregation

The authors propose Lite Any Stereo V2 (LAS2), a model series developed with practical deployment latency as the primary optimization target, rather than theoretical Multiply-Accumulate Operations (MACs) alone.

Feature Extraction: Instead of leveraging heavy monocular depth priors (e.g., DepthAnything) which introduce significant overhead, LAS2 utilizes a conventional ImageNet-pretrained backbone. Specifically, FasterNet is chosen over MobileNet variants because, despite slightly higher MACs, it demonstrates superior real-world inference speed on GPUs and edge devices.
Cost Aggregation: A key architectural shift is the removal of the 3D cost aggregation component common in accurate models. LAS2 adopts a purely 2D cost aggregation framework using a U-Net-style design with strided residual layers and transposed convolutions. This design avoids the computational bottlenecks of 3D convolutions while maintaining geometric reasoning capabilities.
Model Variants:
- LAS2-S/M/L: Feed-forward variants with increasing capacity (varying depths of the aggregation module) designed for different efficiency budgets.
- LAS2-H: An iterative variant that uses the feed-forward LAS2-M to generate an initial disparity and intermediate representations. These are then refined by a context-guided recurrent update module (ConvGRU) to achieve higher accuracy.

2. Training Strategy: Three-Stage Pipeline

To enable robust zero-shot generalization without real-world ground truth, the authors introduce a three-stage training strategy:

Stage 1 (Synthetic Supervision): The model is trained from scratch on a large mixture of synthetic datasets (approx. 1.8M pairs, including SceneFlow, FallingThings, FSD, etc.) using standard supervised disparity regression loss.
Stage 2 (Self-Distillation): To improve robustness to input perturbations and learn domain-invariant representations, a self-distillation strategy is employed. A "teacher" model (trained on clean data) guides a "student" model (trained on perturbed data) via feature alignment loss. The authors found that a fixed-teacher strategy yields the best results.
Stage 3 (Real-World Knowledge Distillation): The model is fine-tuned on 0.5M unlabeled real-world stereo pairs. Pseudo-labels are generated by a high-capacity teacher model (FoundationStereo). To ensure reliability, the authors introduce:
- Pseudo-Label Filtering: A combination of Left-Right (LR) consistency checks, edge-aware masks (suppressing artifacts in textureless regions), and sky segmentation masks to filter out unreliable supervision.
- Error Clamping: An operation that truncates the per-pixel disparity loss at a threshold ( $\tau_{clamp}$ ). This prevents high-error pixels (common in difficult real-world regions) from dominating the gradients and destabilizing the synthetic-to-real transfer.

Key Contributions

LAS2 Model Series: An ultra-fast stereo matching family (S, M, L, H) that achieves state-of-the-art (SOTA) accuracy among efficient methods while maintaining significantly lower latency.
Deployment-Oriented Architecture: A systematic redesign of stereo architecture that prioritizes practical latency over theoretical MACs, introducing a 2D-only cost aggregation framework that outperforms hybrid 3D-2D designs on edge hardware.
Three-Stage Training Strategy: A novel pipeline combining synthetic supervision, self-distillation, and real-world knowledge distillation that enables efficient models to generalize across diverse unseen scenarios.
Reliability Mechanisms: The introduction of pseudo-label filtering and error-clamping operations, which significantly improve the stability and effectiveness of training on unlabeled real-world data.

Experimental Results

Extensive experiments were conducted on four real-world benchmarks (KITTI 2012, KITTI 2015, ETH3D, Middlebury) and a weather-specific subset (DrivingStereo).

Accuracy vs. Latency:
- LAS2-M (feed-forward) consistently outperforms the previous SOTA efficient method LAS across all four benchmarks, reducing errors while running 1.6× faster on H200 and 1.9× faster on Orin 8G.
- LAS2-H (iterative) achieves stronger overall zero-shot performance than the iterative method Fast-FoundationStereo, with 1.8× and 2.7× faster inference on H200 and Orin, respectively.
Zero-Shot Generalization: LAS2 models demonstrate strong generalization to in-the-wild images, producing cleaner disparity maps with fewer artifacts (e.g., in reflections, low-texture regions, and complex layouts) compared to existing efficient baselines.
Ablation Studies: The authors confirm that the 2D aggregation design is critical for latency, FasterNet is superior to MobileNet for real-world speed, and the three-stage training strategy (particularly the filtering and clamping in Stage 3) is essential for performance gains.
Leaderboard Performance: On the KITTI 2015 leaderboard, the finetuned LAS2-M achieved the best results among all published efficient methods at the time of submission.

Significance and Claims

The paper claims that lightweight stereo models can achieve strong zero-shot generalization without sacrificing deployment efficiency, effectively narrowing the performance gap between lightweight and accuracy-oriented models. By shifting the focus from theoretical complexity (MACs) to practical inference latency and employing robust training strategies for unlabeled data, LAS2 offers a practical, off-the-shelf solution for stereo matching on resource-constrained hardware. The authors position LAS2 as a significant step toward deploying high-quality stereo vision in real-world applications such as robotics and autonomous driving, where both speed and generalization are critical.

The paper acknowledges limitations, noting that a performance gap remains compared to prior-based high-accuracy methods and that extremely challenging scenes (e.g., severe reflections, transparent surfaces) remain an open challenge. However, it asserts that the proposed approach provides a balanced and deployable solution for efficient stereo matching.