💻 computer science

DreamStereo: Towards Real-Time Stereo Inpainting for HD Videos

DreamStereo addresses the challenges of real-time stereo video inpainting by introducing Gradient-Aware Parallax Warping, Parallax-Based Dual Projection, and Sparsity-Aware Stereo Inpainting to generate geometrically consistent results and achieve a 10.7x speedup, enabling 25 FPS processing of HD videos on a single A100 GPU.

Original authors: Yuan Huang, Sijie Zhao, Jing Cheng, Hao Xu, Shaohui Jiao

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuan Huang, Sijie Zhao, Jing Cheng, Hao Xu, Shaohui Jiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, high-definition movie playing on your phone. It looks great, but it's flat—like a painting on a wall. You want to turn it into a 3D movie so you can watch it with VR glasses or on a special 3D TV, where things pop out and look real.

The problem is, when you try to create that "second eye's view" (the right side of the 3D image), some parts of the picture get hidden behind objects. It's like looking at a tree; you can see the leaves, but you can't see the sky behind the leaves. In a 3D view, that hidden sky needs to be "painted in" so the brain doesn't see a black hole.

This paper, DreamStereo, is like a super-smart, super-fast artist that can fill in those missing holes in real-time, making your flat videos look like immersive 3D experiences.

Here is how they did it, broken down into three simple tricks:

1. The "Smooth Painter" (Gradient-Aware Parallax Warping)

The Problem:
Imagine trying to copy a photo onto a new canvas by stretching it. If you just stretch it quickly (like the old methods did), the edges get messy. You end up with "scattered pixels"—little dots of color floating in the wrong places, like a spilled paint bucket. This makes the new 3D view look glitchy and broken.

The Solution:
The authors invented a new way to stretch the image called GAPW. Think of it like using a smart, flexible ruler instead of a rubber band.

Instead of just pulling pixels, this method looks at the "slope" or "gradient" of the image.
It ensures that when an object moves, the background behind it flows smoothly, like water filling a bucket, rather than splashing everywhere.
The Result: The "holes" that need to be filled have clean, smooth edges, making the painting job much easier and the final result much sharper.

2. The "Magic Mirror" (Parallax-Based Dual Projection)

The Problem:
To teach an AI how to fill in these holes, you need a teacher. Usually, you'd need a teacher who has two cameras recording the same scene at the same time (a stereo video). But those cameras are expensive, rare, and hard to find. Most of the internet is just flat, single-camera videos.

The Solution:
The authors created a strategy called PBDP, which is like a magic mirror trick.

They take a normal, flat video.
They use a "smart guesser" (an AI depth estimator) to figure out how far away everything is.
Then, they use their "Smooth Painter" (GAPW) to simulate what the other eye would see.
Crucially, they bounce the image back and forth (Dual Projection) to check their work. If the image looks weird when bounced back, they know where the "holes" are.
The Result: They can create thousands of perfect "training examples" using just regular YouTube videos. They don't need expensive 3D cameras anymore!

3. The "Lazy Artist" (Sparsity-Aware Stereo Inpainting)

The Problem:
Even with a great teacher, painting every single pixel in a 4K video is slow. Imagine an artist who stops to paint every single blade of grass in a field, even the ones that are already perfect green. That's a waste of time. In a 3D video, 90% of the picture is already correct; only the tiny "holes" behind objects need fixing.

The Solution:
They introduced SASI, which is like hiring a lazy but brilliant artist who only paints what's necessary.

The AI looks at the video and says, "Hey, 70% of these pixels are fine. I'm going to ignore them."
It focuses its energy only on the scattered holes that need filling.
The Result: This makes the process 10 times faster. Instead of taking minutes to process a clip, it does it in a blink (25 frames per second). This means you can watch 3D movies in real-time on a single powerful computer chip.

The Big Picture

Before this paper, turning flat videos into 3D was either:

Slow: Taking forever to compute.
Messy: Full of glitches and floating pixels.
Expensive: Requiring special cameras to train the AI.

DreamStereo changes the game by:

Making the edges smooth (no more glitches).
Learning from regular videos (no expensive cameras needed).
Ignoring the parts that don't need work (super fast).

In a nutshell: They built a system that can take your boring flat videos and turn them into immersive 3D experiences instantly, making the future of AR/VR and 3D entertainment much more accessible.

1. Problem Statement

The paper addresses the challenge of Stereo Video Inpainting, which involves filling occluded regions in warped stereo videos with visually coherent content while maintaining temporal consistency. The task is critical for converting the vast amount of existing monocular (2D) video content into immersive stereo (3D) formats for AR/VR applications.

The authors identify two primary bottlenecks in current approaches:

Data Scarcity and Quality: Existing methods struggle due to a lack of high-quality stereo inpainting datasets. Acquiring real stereo videos is expensive, and synthetic datasets generated via forward warping (common in prior work like TrajectoryCrafter) suffer from scattered pixels and misaligned artifacts at object boundaries, degrading model learning.
Computational Redundancy: Stereo inpainting only requires modifying a small fraction of pixels (the occluded regions). However, existing diffusion-based methods process all visual tokens uniformly. Since the computational complexity of Diffusion Transformers (DiT) scales quadratically ( $O(N^2)$ ) with the number of tokens, this results in massive redundant computation and slow inference speeds, making real-time HD video processing impossible.

2. Methodology

The proposed framework, DreamStereo, consists of three interconnected components designed to solve the data quality and inference speed issues simultaneously.

A. Gradient-Aware Parallax Warping (GAPW)

To generate high-quality training data without real stereo videos, the authors propose a novel warping technique.

Mechanism: Unlike standard forward warping (which maps source pixels to target coordinates, causing holes and scattered artifacts), GAPW utilizes backward warping. It calculates the inverse mapping from the target view to the source view.
Gradient Awareness: It incorporates the Jacobian matrix (gradients of the coordinate mapping function) to detect deformation. Regions with high deformation (where $|\frac{\partial x'}{\partial x}| > \delta$ ) are identified as occlusions.
Result: This produces smooth, continuous occlusion masks and avoids the scattered pixel artifacts common in forward warping, ensuring geometric consistency between the warped view and the original.

B. Parallax-Based Dual Projection (PBDP)

This is a data construction strategy that leverages GAPW to create stereo inpainting datasets from monocular videos only.

Process:
1. Take a monocular video ( $V_1$ ) and estimate its depth/disparity ( $D_1$ ).
2. Use GAPW to warp $V_1$ to a virtual second view ( $V_2$ ) and generate its disparity ( $D_2$ ).
3. Warp $V_2$ back to the original view ( $V'_1$ ) using GAPW to reconstruct the original view and generate a precise occlusion mask ( $M_1$ ) for the input view.
Outcome: This creates a training pair $(V_1, M_1)$ where the model learns to inpaint the occluded regions of $V_1$ using information from the synthesized $V_2$ . This eliminates the need for expensive stereo video collection.

C. Sparsity-Aware Stereo Inpainting (SASI)

To achieve real-time performance, the authors introduce a token-sparsity mechanism within the Diffusion Transformer (DiT) architecture.

Token Selection: Since only occluded regions need inpainting, the method employs Mask-Based Token Selection.
- The occlusion mask is downsampled to the latent space.
- A dilation function expands the mask slightly to include boundary context.
- Only tokens falling within the masked (and dilated) regions are selected for the DiT attention mechanism; the remaining ~70% of tokens are pruned.
Architecture: Built upon WanVideo (a DiT-based video generation model) with a lightweight 3D VAE. The denoiser is trained to predict the velocity of the latent representation, conditioned on the sparse tokens and the mask.
Efficiency: By reducing the number of active tokens to ~25.6%, the inference speed is drastically improved due to the reduction in quadratic attention complexity.

3. Key Contributions

GAPW (Gradient-Aware Parallax Warping): A warping technique that combines backward warping with gradient analysis to generate smooth occlusion masks and eliminate scattered pixel artifacts, significantly improving dataset quality.
PBDP (Parallax-Based Dual Projection): A scalable data generation strategy that synthesizes high-quality stereo inpainting pairs from monocular videos, removing the dependency on costly stereo video datasets.
SASI (Sparsity-Aware Stereo Inpainting): A real-time inference framework that eliminates >70% of redundant visual tokens. This achieves a 10.7× speedup in DiT inference while maintaining performance comparable to full-computation models.

4. Experimental Results

The method was evaluated on HD (768 × 1280) videos using an NVIDIA A100 GPU.

Performance Metrics:
- Speed: Achieves 25 FPS (40.1 ms latency) with only 1 Noise Function Evaluation (NFE), enabling real-time processing.
- Quality: Achieves a PSNR of 30.48 dB and SSIM of 0.900 on the HD-100 benchmark.
- Comparison: Outperforms state-of-the-art methods like StereoCrafter, ZeroStereo, and ProPainter. Notably, while ProPainter is fast, it produces blurry outputs; ZeroStereo requires many NFEs (slow) to achieve decent quality. DreamStereo achieves the best accuracy with the fastest speed.
Ablation Studies:
- Data Construction: Models trained on PBDP-generated data significantly outperform those trained on TrajectoryCrafter (forward warping) data, confirming the importance of clean masks.
- Token Sparsity: Reducing token retention to ~25% (via dilation and stride) reduces latency by ~10× with negligible quality loss (PSNR drop < 0.5 dB).
- Robustness: The model remains robust even with wide baselines (large disparities up to 0.08), maintaining geometric accuracy where other methods degrade.

5. Significance

DreamStereo represents a significant leap forward in 2D-to-3D video conversion:

Real-Time Capability: It is the first method to enable real-time (25 FPS) stereo inpainting for HD videos on a single consumer-grade GPU (A100), making it viable for interactive AR/VR applications.
Data Efficiency: By solving the data scarcity issue through PBDP, it allows the training of high-fidelity models without the prohibitive cost of collecting stereo video datasets.
Computational Efficiency: The SASI approach demonstrates that diffusion models can be made highly efficient by exploiting the inherent sparsity of the inpainting task, offering a blueprint for accelerating other vision tasks where only a subset of pixels requires modification.

In summary, DreamStereo bridges the gap between high-quality stereo generation and real-time performance, making the conversion of the world's monocular video library into immersive 3D content feasible.