Technical Summary: Adaptive 1D Video Diffusion Autoencoder (One-DVA)

1. Problem Statement

Recent video generation models rely heavily on video autoencoders to compress pixel-space videos into latent representations. However, existing video autoencoders suffer from three critical limitations:

1. Fixed-Rate Compression: Current models use a fixed token count regardless of video complexity. This wastes tokens on simple videos and fails to optimize efficiency, as simple scenes require fewer tokens than those with complex textures and motion.
2. Inflexible CNN Architectures: Traditional Convolutional Neural Networks (CNNs) rely on fixed-size kernels and human-designed priors, making them ill-suited for processing variable-shaped inputs or decoding variable-length latents.
3. Deterministic Decoders: Standard decoders are deterministic and struggle to recover appropriate details from highly compressed latents. When token counts are low, lossy compression forces the decoder to infer missing details, often leading to reconstruction errors.

2. Methodology

The authors propose One-Dimensional Diffusion Video Autoencoder (One-DVA), a transformer-based framework designed for adaptive 1D encoding and diffusion-based decoding.

Architecture

• Encoder (Query-based Vision Transformer): The encoder utilizes a Vision Transformer (ViT) backbone to extract spatiotemporal features from input video frames. It employs learnable 1D queries that interact with these features via self-attention to extract essential visual content. This produces a hybrid latent representation consisting of:
  • Structural Latents: Derived from the ViT backbone, capturing spatial structure.
  • 1D Latent Sequence: Extracted via the query mechanism.
• Variable-Length Dropout: A "matryoshka" training strategy is applied where a variable-length dropout mechanism dynamically truncates the 1D latent sequence from the tail toward the head. The dropout ratio is sampled based on a motion score computed from pixel differences, allowing the model to adapt the latent size to the video's complexity.
• Decoder (Pixel-Space Diffusion Transformer): The decoder is a Diffusion Transformer (DiT) operating in pixel space. It treats the latent representations (structural and 1D) as conditional inputs and performs a diffusion process to reconstruct the video. This generative approach allows the decoder to learn the dataset distribution and compensate for omitted details.

Training Strategy

One-DVA employs a two-stage training strategy to ensure high reconstruction fidelity and generative capability:

1. Stage 1: Deterministic Pretraining: The encoder is optimized to extract critical features. The decoder receives pure random noise (bypassing the diffusion process) and the full latent input. This forces the encoder to capture all necessary information without relying on the decoder's generative priors.
2. Stage 2: Stochastic Post-Training: Diffusion timestep sampling and variable-length dropout are introduced. The model is trained with a composite loss function including diffusion loss (flow-matching), perceptual loss, KL divergence (to regularize the latent distribution), and REPA loss.

Adaptation for Downstream Generation

To support downstream latent diffusion models (LDM):

• Latent Space Alignment: A regularization term aligns the 1D latents with structural latents via cosine distance minimization and self-similarity maximization, injecting structural priors into the flexible 1D queries.
• Decoder Fine-tuning: The decoder is fine-tuned using latents sampled from the trained LDM (rather than encoded ground-truth latents). This bridges the training-inference gap, mitigating artifacts caused by prediction errors in the generative process.

3. Key Contributions

• Adaptive 1D Encoding: Introduction of a transformer-based encoder with query-based mechanisms and variable-length dropout, enabling dynamic compression ratios tailored to video content.
• Generative Decoding: Implementation of a pixel-space diffusion transformer as the decoder, shifting the reconstruction paradigm from deterministic mapping to conditional generation to recover lost details.
• Two-Stage Training & Regularization: A novel training pipeline that separates feature extraction optimization from generative decoding, coupled with specific alignment and fine-tuning techniques to optimize the latent space for downstream video generation.

4. Experimental Results

The model was evaluated on video reconstruction and generation tasks (class-to-video and text-to-video).

• Reconstruction Performance: At standard compression ratios (4×16×16), One-DVA achieves reconstruction metrics (PSNR, SSIM, rFVD) comparable to or better than state-of-the-art 3D-CNN VAEs (e.g., CogVideoX, HunyuanVideo, Wanx2.1/2.2).
  • PSNR: 36.48 (vs. 35.54 for HunyuanVideo).
  • rFVD: 56.96 (lower is better).
• Adaptive Compression: Experiments show that adaptive latent lengths (based on motion scores) outperform fixed-length baselines. Videos with high motion require longer latents to maintain quality, while simple videos can be compressed more aggressively.
• Generation Quality: When used as a backbone for Latent Diffusion Models, One-DVA achieves a class-to-video gFVD of 210.9, matching the performance of Hi-VAE + DiT.
  • Ablation studies confirm that the decoder fine-tuning step is crucial; omitting it results in a significant performance drop (gFVD 274.2).
  • Using only structural latents (0% 1D) yields reasonable results but is limited by a lack of high-frequency information.

5. Significance and Claims

The paper claims that One-DVA successfully unifies adaptive video tokenization and diffusion-based generative decoding within a single transformer framework. Its primary significance lies in:

1. Efficiency: By supporting variable-length encoding, it optimizes token usage, avoiding waste on simple content.
2. Flexibility: It overcomes the architectural rigidity of CNNs, allowing for inputs and outputs of arbitrary shapes via attention mechanisms.
3. Generative Reconstruction: It reframes video reconstruction as a generation subtask, enabling the model to recover details lost during aggressive compression by learning the data distribution.
4. Downstream Compatibility: The framework provides a latent space and decoder specifically regularized and fine-tuned to support high-quality downstream latent diffusion models for video generation.

The authors conclude that One-DVA achieves reconstruction quality comparable to advanced 3D-CNN VAEs while offering the flexibility and generative capabilities required for next-generation video synthesis.