Next-Frame Decoding for Ultra-Low-Bitrate Image Compression with Video Diffusion Priors

1. Problem Statement

Ultra-Low-Bitrate Image Compression (ULB-IC) aims to reconstruct high-fidelity images from extremely small bitstreams. While traditional Learned Image Compression (LIC) methods (based on VAEs) struggle with perceptual quality at low bitrates, recent Generative Image Compression approaches using Diffusion Models have shown promise. However, existing diffusion-based methods face three critical challenges:

1. Semantic Drift & Faithfulness: Standard image diffusion models often start from Gaussian noise. The uncertainty of this initialization and the influence of conditional signals can lead to "hallucinations" where the reconstructed image loses faithfulness to the source content (e.g., changing object identities or layouts).
2. Computational Latency: Iterative denoising processes in diffusion models are computationally expensive, leading to high decoding latency that hinders practical deployment.
3. Domain Gap: Existing methods treat image reconstruction as a static generation task, ignoring the potential of temporal dynamics found in video generation models to guide detail synthesis.

2. Methodology: NeFIC (Next-Frame Image Compression)

The authors propose NeFIC, a novel paradigm that reinterprets single-image generative decoding as a virtual temporal transition. Instead of generating an image from noise, the method evolves a "compact anchor frame" into the final high-fidelity image using a Video Diffusion Model (VDM).

Core Concept: Virtual Temporal Evolution

• The Anchor Frame: An explicit intermediate state encoded by a VAE-based Anchor Codec. It preserves scene geometry, semantic layout, and coarse appearance but discards high-frequency details.
• The Transition: The decoding process is framed as a next-frame prediction task. The anchor frame is treated as Frame $t=0$, and the target image is Frame $t=1$. The VDM predicts the transition from the blurry anchor to the detailed target.

Two-Stage Training Scheme

To bridge the gap between video generation and image compression, and to eliminate iterative denoising, the authors propose a two-stage training pipeline:

Stage I: Next-Frame Decoding Adaptation

• Goal: Adapt a pretrained VDM (specifically CogVideoX-1.5) to perform single-interval next-frame prediction conditioned on the anchor.
• Mechanism:
  • The model is fine-tuned using LoRA (Low-Rank Adaptation) on the DiT (Diffusion Transformer) blocks.
  • Tokenization: The input to the transformer consists of three concatenated token sequences: Text embeddings (universal prompt), Anchor tokens (encoded from the decoded anchor frame), and Noisy target tokens.
  • Loss: The model learns to predict the clean latent of the target frame from the noisy latent, conditioned on the anchor. An auxiliary anchor loss ensures the anchor remains within the VDM's training distribution.
• Outcome: The VDM learns to synthesize high-frequency details (textures, edges) based on the coarse anchor, effectively learning a "defocus-to-focus" temporal prior.

Stage II: One-Step Generative Bypass

• Goal: Collapse the multi-step diffusion process into a single-step generation to drastically reduce latency and remove stochastic ambiguity.
• Mechanism:
  • Conditional Anchor Encoding: The Anchor Encoder is conditioned on the Video-VAE latents of the original image. This aligns the compression latent space with the generative latent space, preventing semantic drift (e.g., preventing extra teeth from appearing).
  • Bypass Refinement: A learned Transformer module maps the decoded anchor features into a bypass latent. This bypass latent serves as the initial state for the generation, replacing the standard Gaussian noise.
  • One-Step Inference: The VDM performs a single denoising step (at $t^* = 500$) starting from the bypass latent and conditioned on the anchor to produce the final reconstruction.
• Outcome: This creates a "latent bypass" that regularizes the generation space, enabling faithful, one-step decoding without iterative sampling.

3. Key Contributions

1. Novel Paradigm: Reinterprets image compression as a virtual temporal evolution (anchor $\to$ target) rather than static generation, leveraging the "defocus-to-focus" priors inherent in video diffusion models.
2. Efficient One-Step Decoding: Introduces a Two-Stage Training and Bypass Refinement mechanism that transforms a multi-step video diffusion model into a single-step generative decoder, achieving up to 5$\times$ speedup in decoding compared to iterative methods.
3. Superior Performance: Achieves state-of-the-art results in both perceptual realism (FID, KID) and structural fidelity (LPIPS, DISTS, PSNR) while significantly reducing bitrate requirements.

4. Experimental Results

The method was evaluated on Kodak, DIV2K, and CLIC2020 test sets against recent SOTA codecs (DiffC, StableCodec, PerCo, etc.).

• Bitrate Savings: On the CLIC2020 test set, NeFIC achieves >50% bitrate savings across LPIPS, DISTS, FID, and KID compared to DiffC.
• Distortion Metrics: Unlike many generative codecs that sacrifice PSNR for realism, NeFIC improves distortion metrics as well. On Kodak, it outperforms StableCodec by 18.27% in PSNR BD-rate.
• Efficiency:
  • Decoding Speed: NeFIC decodes images in 0.88s (Kodak) and 6.83s (DIV2K), which is 5.5$\times$ to 8.6$\times$ faster than DiffEIC and significantly faster than multi-step diffusion baselines.
  • Latency: The one-step bypass eliminates the need for 50+ iterative denoising steps.
• Ablation Studies:
  • Removing Stage I (adaptation) degrades performance by ~8-10%.
  • Removing Stage II (bypass) increases decoding time to ~30s (Kodak) and ~296s (DIV2K), proving the necessity of the one-step mechanism.
  • Removing Conditional Encoding leads to semantic drift (e.g., distorted facial features).

5. Significance

This paper represents a significant shift in Generative Image Compression. By leveraging Video Diffusion Models (VDMs) not just as generators but as temporal priors for detail refinement, the authors solve the trade-off between fidelity (faithfulness to source) and realism (perceptual quality).

The Next-Frame Decoding paradigm offers a principled way to inject conditions into diffusion models (via token concatenation and 3D attention) rather than simple channel concatenation, leading to better structural consistency. Furthermore, the One-Step Bypass technique demonstrates that high-quality generative compression can be made practical for real-time applications by collapsing multi-step diffusion into a single inference step, addressing the primary bottleneck of current diffusion-based codecs.