Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion

1. Problem Statement

Video Super-Resolution (VSR) aims to reconstruct High-Resolution (HR) videos from Low-Resolution (LR) inputs. While existing methods have made progress, they face a critical trade-off between perceptual quality and latency:

• CNN/Transformer-based methods: Offer fast inference and low latency but often lack the generative priors needed for high perceptual detail and realistic textures.
• Diffusion-based methods: Deliver superior perceptual quality but are computationally expensive (requiring 50+ denoising steps) and typically rely on bidirectional (future frame) information. This reliance on future frames makes them unsuitable for online/streaming scenarios, resulting in prohibitive end-to-end latency (e.g., waiting for an entire video sequence to process before outputting the first frame).

The core challenge is to develop a diffusion-based VSR framework that operates strictly causally (using only past frames) while maintaining high perceptual quality and achieving inference speeds comparable to real-time online models.

2. Methodology: Stream-DiffVSR

The authors propose Stream-DiffVSR, a causally conditioned diffusion framework designed for efficient online VSR. The architecture consists of three core components:

A. Distilled Few-Step U-Net (Acceleration)

• Problem: Standard diffusion models require 50+ steps, causing high latency.
• Solution: The authors distill a pre-trained Stable Diffusion (SD) ×4 Upscaler into a 4-step denoising variant.
• Technique: They employ Rollout Distillation. Unlike standard distillation that supervises random timesteps, this method forces the U-Net to perform the full 4-step denoising trajectory in every training iteration. Loss is applied only to the final denoised latent. This ensures the training trajectory perfectly mirrors the inference process, significantly reducing the training-inference gap and stabilizing the 4-step generation.

B. Auto-Regressive Temporal Guidance (ARTG)

• Problem: Generating frames independently leads to flickering and temporal inconsistency.
• Solution: A mechanism that injects motion-aligned cues from previous frames into the latent denoising process.
• Mechanism:
  1. The previously generated High-Quality (HQ) frame is warped to the current frame's perspective using optical flow.
  2. This warped frame serves as a conditioning input to the U-Net alongside the current noisy latent.
  3. This guides the denoising process to maintain temporal coherence without requiring future frames.

C. Auto-Regressive Temporal-aware Decoder

• Problem: The latent features from the diffusion model may lack fine spatial details or temporal alignment when decoded to RGB space.
• Solution: A lightweight decoder enhanced with a Temporal Processor Module (TPM).
• Mechanism:
  1. The decoder takes the denoised latent and features from the warped previous HQ frame.
  2. The TPM is inserted after spatial convolution layers. It aligns and fuses current spatial features with temporal features from the previous frame via interpolation, 1D convolutions, and weighted summation.
  3. This explicitly injects temporal context into the final RGB reconstruction, enhancing detail recovery and stability.

D. Training Pipeline

The model is trained in three sequential, decoupled stages to ensure stability:

1. Stage 1 (U-Net Distillation): Distills the 50-step SD model into a 4-step model using LQ/HQ pairs.
2. Stage 2 (Decoder Training): Trains the Temporal Processor Module (TPM) within the decoder using ground-truth frames to learn temporal fusion.
3. Stage 3 (ARTG Training): Trains the Auto-regressive Temporal Guidance module using flow-warped previous outputs while freezing the U-Net and decoder.

3. Key Contributions

1. First Strictly Causal Diffusion VSR: Stream-DiffVSR is the first diffusion-based VSR method designed for strictly online, frame-by-frame inference, eliminating the need for future frames.
2. Drastic Latency Reduction: By combining 4-step distillation with a causal design, the method reduces the "time-to-first-frame" from >4600 seconds (for bidirectional diffusion models like StableVSR) to 0.328 seconds per frame on an RTX 4090. This represents a reduction of over 3 orders of magnitude.
3. Superior Perceptual Quality: Despite being an online method, it outperforms existing online CNN/Transformer baselines (e.g., TMP, RealViformer) in perceptual metrics (LPIPS) while maintaining comparable inference speeds.
4. Efficient Architecture: The framework operates within a manageable memory footprint (~19.58 GB), avoiding the Out-of-Memory (OOM) issues common in other diffusion-based VSR methods that require processing entire video chunks.

4. Experimental Results

The method was evaluated on standard benchmarks: REDS4, Vimeo-90K-T, VideoLQ, and Vid4.

• Perceptual Quality: Stream-DiffVSR achieves the best LPIPS scores among online methods.
  • On REDS4: LPIPS 0.099 (vs. 0.194 for TMP and 0.123 for MIA-VSR).
  • On Vimeo-90K-T: LPIPS 0.056 (vs. 0.109 for TMP).
• Temporal Consistency: It significantly outperforms other online methods in temporal metrics (tLP and tOF), reducing flickering artifacts.
• Inference Speed & Latency:
  • Runtime: 0.328s/frame (720p) on RTX 4090.
  • Max Latency: 0.328s (single frame delay).
  • Comparison: It is ~130x faster in latency than the online state-of-the-art TMP and >10,000x faster in initial delay than offline diffusion models like StableVSR.
• Memory Efficiency: Peak memory usage is ~19.58 GB, whereas other diffusion baselines (e.g., Upscale-A-Video, SeedVR2) often exceed 40-60 GB or fail with OOM errors on similar hardware.

5. Significance

Stream-DiffVSR bridges the gap between the high-quality generative capabilities of diffusion models and the strict latency requirements of real-world applications.

• Practical Deployment: It makes diffusion-based VSR viable for live broadcasting, video conferencing, AR/VR, and autonomous driving, where waiting for future frames is impossible.
• Paradigm Shift: It demonstrates that diffusion models do not need to be slow or offline; with proper distillation and causal temporal modeling, they can achieve real-time performance without sacrificing perceptual fidelity.
• Future Impact: This work paves the way for high-fidelity, low-latency generative video processing in interactive and streaming environments.