🤖 machine learning

Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models

The paper introduces AVIS, an autoregressive diffusion framework that enables real-time video inverse problem solving by streaming restoration and optimizing measurement consistency, significantly reducing latency and boosting throughput while maintaining superior quality compared to existing non-autoregressive solvers.

Original authors: Taesung Kwon, Jonghyun Park, Hyungjin Chung, Jong Chul Ye

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Taesung Kwon, Jonghyun Park, Hyungjin Chung, Jong Chul Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very old, scratched, and blurry home movie. You want to restore it to look crisp and clear again. This is what computer scientists call a "video inverse problem."

For a long time, the best tools to fix these videos (called Diffusion Models) worked like a meticulous, slow-motion painter. To fix a 10-second clip, the painter had to look at the entire 10 seconds at once, figure out the whole picture, and then paint the first frame. You couldn't see the first frame until the painter finished the very last one. This caused a huge delay (latency) and made it impossible to watch the video while it was being fixed.

The paper introduces a new system called AVIS (and its super-fast cousin, AVIS Flash) that changes the game. Here is how it works, using simple analogies:

1. The Old Way: The "Whole-Picture" Painter

Think of the old methods as an artist trying to paint a mural of a whole city. They can't start painting the left side until they know exactly what the right side looks like. They have to plan the whole thing in their head before making a single stroke.

The Problem: You have to wait for the entire mural to be finished before you can see the first building. If the video is long, you wait a long time. Also, the artist has to constantly check their work against the original blurry photo, which is a slow, repetitive process.

2. The New Way (AVIS): The "Streaming" Storyteller

The authors realized that video is naturally sequential (one frame follows another). Instead of painting the whole mural at once, AVIS acts like a storyteller telling a story one chapter at a time.

How it works: As soon as the first "chapter" (a few seconds of video) is restored, it is shown to you immediately. Then, the system moves to the next chapter, using the previous one as a guide.
The Benefit: There is no waiting. You see the video stream in real-time, just like watching a live broadcast.

3. The Secret Sauce: The "Rough Draft" Start

Even with the new storytelling method, the system still had to do a lot of "cleaning up" to make sure the video matched the original blurry measurements. This was still slow.

The authors added a clever trick: The Rough Draft.

The Analogy: Imagine you are trying to fix a torn map. Instead of starting with a blank piece of paper and guessing where the roads go, you first quickly sketch a rough, imperfect map that fits the torn edges. Then, you use your artistic skills to refine that sketch into a perfect map.
In the paper: AVIS creates a quick, "good enough" version of the video first. Because it starts with this rough draft, the AI doesn't have to guess from scratch. It only needs to do a few quick "polishing" steps to make it perfect. This cuts the time down dramatically.

4. The Super-Fast Version (AVIS Flash): The "Trust the Flow" Method

The authors noticed something interesting: Once the first few seconds of the video were fixed perfectly, the AI was so good at predicting what comes next that it didn't need to double-check the rest of the video against the blurry original.

The Analogy: Think of a river. If you fix the source of the river (the first chunk), the water flows downstream naturally. You don't need to stop and measure the water at every single bend; you just let it flow.
The Result: AVIS Flash only does the heavy "double-checking" on the very first chunk of the video. For everything that follows, it just lets the AI's natural prediction flow forward.
The Speed: This makes the system incredibly fast. The paper claims it can restore video at 5.91 frames per second on a standard high-end computer, which is fast enough to be considered "real-time."

Summary of Results

The paper tested this on various tasks like making blurry videos sharp, filling in missing parts (like a scratched film), and removing noise.

Speed: The old methods took over 100 seconds to show you the first frame. AVIS shows it in 4 seconds. AVIS Flash is even faster.
Quality: Despite being so much faster, the restored videos looked just as good (or better) than the slow, old methods.
Versatility: It works on different types of video damage (blur, missing parts, low resolution) without needing to be retrained for each specific problem.

In short, the paper presents a way to fix broken videos that is no longer a slow, batch-process job, but a fast, streaming experience that feels like watching a live restoration.

Technical Summary: Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models

Problem Statement

Video inverse problems aim to reconstruct a clean video $x$ from a degraded measurement $y$ (e.g., low-resolution, masked, or blurred) defined by $y = A(x) + n$ . While diffusion and flow-based models have emerged as powerful zero-shot priors for these tasks, their real-time deployment is currently hindered by two critical inefficiencies:

High Initial Latency: Existing Diffusion model-based Video Inverse problem Solvers (DVIS) typically restore videos holistically, sampling the entire sequence simultaneously. Consequently, the first frame cannot be displayed until the entire video is restored, creating a latency bottleneck equal to the total restoration time.
Low Throughput: Modern video diffusion models often operate in the latent space of a Variational Autoencoder (VAE). To enforce measurement consistency, current solvers require multiple VAE passes (encoding and decoding) during the restoration process, restricting throughput to below 1 FPS.

Methodology

The authors propose the Autoregressive Video Inverse problem Solver (AVIS) and its accelerated variant, AVIS Flash, which leverage autoregressive (AR) video diffusion models to address these bottlenecks.

Core Framework: AVIS

Unlike non-autoregressive solvers that process the entire video jointly, AVIS restores videos in a streaming manner, generating video chunks sequentially. This architecture naturally eliminates the initial latency bottleneck. To address the throughput issue and the inherent slowness of diffusion models, AVIS introduces a measurement-consistent initialization:

Initialization: Instead of starting reverse diffusion from pure Gaussian noise, AVIS first computes a coarse, measurement-consistent estimate $x_{init}$ in pixel space by minimizing the objective $\|y - A(x)\|^2$ using a multi-step optimization algorithm (Conjugate Gradient).
Accelerated Sampling: This estimate is encoded into the latent space and diffused to an intermediate timestep $t_0$ . The reverse diffusion process then begins from $t_0$ rather than $t=1$ , significantly reducing the number of required sampling steps.
Iterative Guidance: For every video chunk, AVIS enforces measurement consistency at each denoising step using the Decomposed Diffusion Sampling (DDS) framework. This involves decoding the latent estimate, solving a proximal optimization problem to align with measurements, and re-encoding, all without requiring expensive Jacobian computations.

Accelerated Variant: AVIS Flash

To further boost efficiency, AVIS Flash modifies the guidance strategy:

Single-Chunk Guidance: It enforces measurement consistency (via iterative VAE passes) only on the first video chunk.
Autoregressive Propagation: Subsequent chunks ( $n \ge 2$ ) are generated solely through autoregressive propagation from the corrected prefix (using KV caching), bypassing explicit measurement guidance.
Theoretical Justification: The authors provide an error decomposition analysis (Proposition 1), showing that the final error is bounded by the initial error (mitigated by the initialization) and the context error propagated from previous chunks. They observe that the context error acts as an additive perturbation rather than a multiplicative amplifier, allowing the system to maintain fidelity without continuous guidance.
Long-Video Stability: For very long sequences, the framework can periodically re-inject measurement consistency (e.g., every $N$ chunks) to prevent temporal drift.

Key Contributions

AVIS Framework: The investigation of autoregressive video diffusion models for zero-shot video restoration. AVIS accelerates restoration by initializing the reverse diffusion process with a measurement-consistent estimate, reducing sampling steps while maintaining consistency for every chunk.
AVIS Flash: A highly accelerated variant that enforces measurement consistency solely on the first chunk. This approach substantially increases throughput while maintaining competitive performance, offering a favorable efficiency–performance trade-off.
Performance Gains: The paper demonstrates that these methods drastically reduce initial latency and increase throughput compared to leading non-autoregressive solvers, paving the way for real-time deployment.

Experimental Results

The methods were evaluated on 100 high-resolution videos across five inverse problems: Super-Resolution (4×), Random Inpainting (50% mask), Gaussian Deblur, Temporal Average, and Spatio-Temporal Average.

Efficiency:
- Latency: AVIS reduces initial latency from 114s (LVTINO) to 4s.
- Throughput: AVIS increases throughput from 0.71 FPS to 1.18 FPS. AVIS Flash achieves 5.91 FPS on a single RTX 4090 GPU (and 10.2 FPS on an H100), representing an 8× to 12× speedup over baselines.
Restoration Quality:
- AVIS achieves superior or highly competitive results across fidelity metrics (PSNR, SSIM, LPIPS, FVD) and perceptual metrics (VBench) compared to baselines like DiffIR2VR-Zero, VISION-XL, and LVTINO.
- AVIS Flash maintains competitive performance, with only a slight drop in specific metrics compared to AVIS, but offers a dramatic improvement in speed.
Ablation Studies:
- Removing the autoregressive KV cache degrades performance, confirming the value of context propagation.
- Starting the reverse process from pure noise (without initialization) leads to drift; the measurement-consistent initialization is crucial for fidelity.
- A start time $t_0 = 0.1$ and $K=2$ sampling steps were found to offer the best balance between speed and quality.

Significance and Claims

The paper claims that AVIS and AVIS Flash establish a strong foundation for the practical, real-time deployment of diffusion-based video inverse problem solvers. By leveraging autoregressive generation and a novel initialization strategy, the framework bridges the gap between the high-quality generative priors of diffusion models and the strict latency/throughput requirements of real-world applications.

The authors note that while the methods significantly advance the state-of-the-art, they still rely on multiple VAE passes for the initial pixel-space guidance. Future work could explore enforcing measurement consistency directly in the latent space to further accelerate the process. Additionally, the capability to restore high-fidelity videos from degraded inputs raises ethical considerations regarding misinformation and the reconstruction of sensitive data, which the authors acknowledge as a broader impact to consider.