💻 computer science

Match Stereo Videos via Bidirectional Alignment

This paper introduces BiDAStereo, a novel framework utilizing a bidirectional alignment mechanism to achieve state-of-the-art video stereo matching with improved temporal consistency, alongside the release of new synthetic and real-world datasets focused on natural and urban scenes to address current limitations in data and methods.

Original authors: Junpeng Jing, Ye Mao, Anlan Qiu, Krystian Mikolajczyk

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junpeng Jing, Ye Mao, Anlan Qiu, Krystian Mikolajczyk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a 3D photo of the world using two eyes (or two cameras). This is called stereo matching. It's like your brain figuring out how far away a tree is by comparing what your left eye sees versus your right eye.

Now, imagine doing this not for a single photo, but for a video. That's the challenge this paper tackles.

The Problem: The "Flickering" Video

When you take a 3D photo of a single moment, it's usually pretty clear. But when you try to do this for a whole video, existing computer programs get confused. They look at each frame one by one, like a person flipping through a photo album too fast.

Because they don't talk to each other, the depth estimates jump around wildly from one frame to the next.

The Analogy: Imagine a movie where the actors suddenly teleport a few inches left or right every time the camera cuts. The background looks like it's vibrating or "flickering" like a broken neon sign. This is what happens to 3D videos made by old methods.

The Solution: The "Handshake" and the "Memory"

The authors of this paper, Junpeng Jing and his team, came up with two main ideas to fix this flickering.

1. The "Handshake" (Bidirectional Alignment)

In a video, objects move. If a car drives from left to right, the pixels representing that car shift. Old methods try to match pixels without accounting for this movement, which is like trying to catch a ball while running backward.

The authors introduced a Bidirectional Alignment.

The Analogy: Imagine you are trying to compare two photos of a moving car. Instead of just looking at them, you first "slide" the car in the second photo so it lines up perfectly with the first one. You do this for the frame before and the frame after the current one.
Why it works: By "handshaking" (aligning) the frames first, the computer knows exactly where the same object is in time. This stops the jitter.

2. The "Memory" (Global Propagation)

Even with alignment, looking at just the immediate neighbors (the frame before and after) isn't enough. It's like trying to understand a story by only reading two sentences at a time. You miss the big picture.

They created a system that remembers the "state" of the whole video.

The Analogy: Think of a relay race. Instead of just passing the baton to the person next to you, the runner carries a "memory" of the entire race so far. If a runner stumbles, the memory helps them correct their path based on where they were 10 seconds ago, not just 1 second ago.
The Result: This creates a smooth, consistent 3D video where the depth doesn't jump around, even if the camera is shaking or objects are moving fast.

The New Tools: A Plugin and a New Playground

The paper also introduces two very practical tools:

A. The "Stabilizer Plugin" (BiDAStabilizer)

The Problem: Many people have already built great 3D photo cameras (image-based AI), but they are terrible at video. Retraining them from scratch is expensive and slow.
The Fix: The authors built a "plugin" (like a software add-on). You can take an existing 3D photo camera, plug this stabilizer in, and suddenly it can handle video without flickering. It's like putting a stabilizer gimbal on a shaky handheld camera.

B. The New Playgrounds (New Datasets)
To teach these computers better, you need better practice fields.

The Old Fields: Most existing training data was like a toy box with plastic blocks (indoor scenes) or abstract cartoons. They didn't look like the real world.
The New Fields:
1. Infinigen SV: A massive, hyper-realistic computer-generated world of mountains, forests, and deserts. It's like a video game with perfect physics and lighting, used to train the AI on nature.
2. SouthKen SV: Real-world footage filmed in London (South Kensington) with a special 3D camera. It includes rainy days, busy streets, and people walking. It's the "real life" test drive.

Why Does This Matter?

This isn't just about making pretty 3D movies.

Robotics: Robots need to walk through a forest without tripping over a rock that looks like it's floating.
Self-Driving Cars: They need to know if a pedestrian is 10 meters away or 12 meters away, and that distance needs to be consistent as the car moves.
Virtual Reality (VR): If the 3D world flickers, it makes you sick. This technology makes VR worlds feel solid and real.

Summary

In short, this paper says: "To make 3D videos smooth, we need to align the frames so objects line up, and we need a memory system to remember the whole video, not just the current moment. We also built a plugin to fix old cameras and created new training worlds (both fake and real) to teach the AI how to see the world clearly."

1. Problem Statement

Video Stereo Matching aims to estimate consistent disparity maps from rectified stereo video sequences. While deep learning has advanced stereo matching for static image pairs, applying these methods to videos introduces significant challenges:

Temporal Inconsistency: Frame-by-frame processing of image-based methods leads to severe flickering in disparity maps because temporal information is ignored.
Limitations of Existing Video Methods: Current video approaches often rely on sliding window operations (local temporal receptive fields). This restricts the model to local information, causing low-frequency oscillations in the output and failing to utilize global sequence context.
Misalignment: In dynamic scenes, matching points shift over time. Directly applying temporal attention to raw frames without alignment is inaccurate.
Data Scarcity: Existing datasets lack realistic outdoor natural scenes (most are indoor or synthetic objects) and high-quality real-world benchmarks with ground truth or reliable qualitative data.

2. Methodology

The authors propose two main components: a novel video processing framework (BiDAStereo) and a plugin stabilizer network (BiDAStabilizer), both centered around a Bidirectional Alignment mechanism.

A. Core Mechanism: Bidirectional Alignment

Instead of processing frames independently or using a simple sliding window, the method aligns features from adjacent frames ( $t-1, t+1$ ) to the center frame ( $t$ ) using optical flow. This ensures that corresponding points across time are spatially consistent before correlation or aggregation occurs.

B. BiDAStereo (Video Framework)

A dedicated network designed to output a temporally consistent disparity sequence directly.

Architecture: Comprises feature extraction, optical flow estimation, and an iterative update module.
Triple-Frame Correlation Layer (TFCL):
- Aligns right-frame features from $t-1$ and $t+1$ to the center frame $t$ using bidirectional optical flow.
- Constructs cost volumes between the center left frame and the aligned right frames (triple-frame correlation).
- This enriches scene perception by utilizing occluded points visible in adjacent frames.
Motion-Propagation Recurrent Unit (MRU):
- Introduces a motion hidden state for each frame to cache global temporal context.
- Performs global propagation: Neighboring hidden states are aligned to the center state and fused to update the center frame's state.
- This allows information to propagate across the entire sequence, not just within a local window, effectively expanding the temporal receptive field.
Iterative Refinement: Uses a "Super Kernel Updater" with a large horizontal kernel ( $1 \times 1 \times 15$ ) to refine disparities iteratively.

C. BiDAStabilizer (Plugin Network)

A lightweight network designed to retrofit existing pre-trained image-based stereo models (e.g., RAFTStereo, IGEVStereo) for video tasks without retraining the base model.

Workflow:
1. The frozen image model generates initial (inconsistent) disparities frame-by-frame.
2. Bidirectional optical flow is estimated.
3. Adjacent disparities are aligned to the center frame.
4. A Propagation Encoder integrates forward and backward hidden states with the aligned disparity features to predict a residual disparity ( $\Delta d$ ).
5. The final output is the sum of the original disparity and the predicted residual, ensuring temporal consistency.

3. Key Contributions

Bidirectional Alignment Mechanism: Proposed as a fundamental operation to enforce temporal consistency, enabling accurate feature matching across time steps.
BiDAStereo Framework: A novel architecture utilizing triple-frame correlation and global motion propagation to leverage both local and global temporal information.
BiDAStabilizer: A plugin network that enhances the temporal consistency of any existing image-based stereo method, offering a resource-efficient solution.
New Datasets:
- Infinigen Stereo Video (ISV): A large-scale, high-fidelity synthetic dataset focusing on outdoor natural scenes (mountains, forests, rivers) with dense ground truth, addressing the lack of nature scenarios in existing benchmarks.
- South Kensington Stereo Video (SouthKen SV): A real-world dataset captured in diverse urban environments (indoor/outdoor, various weather) using a stereo camera, serving as a qualitative benchmark.
State-of-the-Art Performance: Achieved SOTA results on multiple benchmarks (Sintel, Dynamic Replica, KITTI Depth, Infinigen SV) in both in-domain and out-of-domain settings.

4. Experimental Results

The paper evaluates performance using spatial accuracy (EPE, $\delta_{n-px}$ ) and temporal consistency metrics (TEPE, OPW, RTC, TCC, TCM).

In-Domain Performance:
- BiDAStabilizer significantly improved the temporal consistency of RAFTStereo and IGEVStereo. For example, on the Dynamic Replica dataset, it reduced TEPE by 43.9% and 43.4% respectively.
- BiDAStereo outperformed the previous SOTA (DynamicStereo) across most metrics, particularly in reducing flickering and oscillation.
Out-of-Domain (Generalization):
- When trained on synthetic data (SceneFlow) and tested on Sintel, the proposed methods showed superior generalization. BiDAStabilizer improved TEPE by 8.10% (Sintel Final) over the baseline.
- Training with the new Infinigen SV dataset further boosted generalization capabilities, proving the value of natural scene data.
Robustness:
- Models trained on mixed datasets maintained high performance across different benchmarks, demonstrating robustness to domain shifts.
Qualitative Results:
- Visualizations on real-world data (SouthKen SV) showed that BiDAStabilizer effectively eliminated flickering in dynamic scenes (e.g., moving cars, pedestrians) and transparent objects (windows), producing stable 3D reconstructions.

5. Significance

Paradigm Shift: Moves beyond local sliding-window approaches to a global propagation strategy via bidirectional alignment, solving the issue of low-frequency oscillations in video stereo.
Practical Utility: The BiDAStabilizer allows researchers and practitioners to upgrade existing image-based models to video-capable ones with minimal computational overhead and no need to retrain the heavy backbone.
Data Advancement: The release of Infinigen SV and SouthKen SV fills critical gaps in the field, providing necessary resources for training on natural outdoor environments and evaluating real-world robustness.
Broad Impact: The framework is compatible with depth priors and outperforms monocular video depth methods by an order of magnitude in accuracy, highlighting the continued importance of stereo geometry in video understanding.