Corruption-Aware Training of Latent Video Diffusion Models for Robust Text-to-Video Generation

1. Problem Statement

Latent Video Diffusion Models (LVDMs) have achieved state-of-the-art results in text-to-video (T2V) generation. However, they exhibit significant brittleness when conditioned on noisy or imperfect inputs (e.g., ambiguous text prompts or noisy multimodal embeddings).

• The Core Issue: In video generation, conditioning errors do not remain isolated; they accumulate and cascade across denoising timesteps. This leads to semantic drift (the video diverges from the prompt) and temporal incoherence (flickering, unnatural motion).
• Limitations of Existing Solutions: Standard corruption strategies used in image diffusion (e.g., adding independent Gaussian or Uniform noise to embeddings) fail in video settings. Static, isotropic noise disrupts the delicate temporal fidelity required for video, often causing catastrophic degradation rather than robustness.
• Gap: There is a lack of training frameworks that introduce structured, data-aligned noise specifically tailored to the temporal and spectral dynamics of video diffusion.

2. Methodology: CAT-LVDM

The authors propose CAT-LVDM, a corruption-aware training framework that injects structured, low-rank noise during the pretraining of LVDMs. Instead of arbitrary noise, the framework aligns perturbations with the intrinsic structure of the data (batch semantics or spectral dynamics).

The framework introduces two novel noise injection operators:

A. Batch-Centered Noise Injection (BCNI)

• Mechanism: BCNI perturbs each conditioning embedding ($z$) along its deviation from the batch mean ($\bar{z}$).
• Formula: $C_{BCNI}(z; \rho) = \rho \|z - \bar{z}\|_2 (2U(0, 1) - 1)$.
• Rationale: By scaling noise based on the distance from the batch mean, BCNI acts as a Mahalanobis-scaled regularizer. It increases conditional entropy along semantically meaningful axes (e.g., variations in stride or pose within a batch of "walking" videos) while leaving orthogonal directions untouched. This reinforces motion realism without introducing arbitrary noise.

B. Spectrum-Aware Contextual Noise (SACN)

• Mechanism: SACN injects noise selectively along the dominant spectral modes of the embedding. It utilizes Singular Value Decomposition (SVD) on the embedding matrix.
• Formula: $C_{SACN}(z; \rho) = \rho U (\xi \odot \sqrt{s}) V^\top$, where $\xi_j \sim \mathcal{N}(0, e^{-j/D})$.
• Rationale: This operator targets low-frequency, globally coherent semantics (e.g., the trajectory of a moving car) while preserving high-frequency details (texture). It ensures that noise is concentrated in the spectral directions that govern temporal consistency, effectively smoothing the score manifold.

Theoretical Foundation

The paper provides a rigorous theoretical analysis showing that these low-rank operators ($d \ll D$, where $d$ is the effective rank and $D$ is the ambient dimension) offer significant advantages over isotropic corruption:

• Complexity Gap: They achieve an $O(d)$ complexity vs. $O(D)$ for standard methods.
• Bounds: They tighten the 2-Wasserstein distance to the target distribution (scaling as $\rho\sqrt{d}$ vs. $\rho\sqrt{D}$), reduce score drift, and improve mixing times in the reverse diffusion process.
• Lipschitz Continuity: They enforce tighter Lipschitz constants on the score network, preventing error accumulation across timesteps.

3. Key Contributions

1. CAT-LVDM Framework: The first systematic corruption-aware training framework specifically designed for Latent Video Diffusion Models, moving beyond image-centric noise strategies.
2. Novel Operators: Introduction of BCNI and SACN, which align noise injection with batch semantics and spectral dynamics, respectively, to preserve temporal coherence.
3. Theoretical Guarantees: Formal proofs demonstrating that structured, low-rank corruption tightens entropy, Wasserstein, and generalization bounds, explaining why these methods outperform isotropic baselines.
4. Scalability: Demonstration that the CAT operators are model-agnostic, successfully transferring to autoregressive video generation (NOVA) and multimodal video understanding LLMs (PAVE).

4. Experimental Results

The authors trained 73 LVDM variants across four benchmarks (WebVid-2M, MSR-VTT, MSVD, UCF-101) using the DEMO architecture.

• Performance Gains:

  • BCNI reduced the Fréchet Video Distance (FVD) by 31.9% on WebVid-2M, MSR-VTT, and MSVD compared to clean baselines.
  • SACN improved FVD by 12.3% on UCF-101 (action-heavy dataset).
  • Efficiency: CAT-LVDM achieved these results training on 5x less data (2M videos) compared to large-scale baselines like DEMO (2.3B params, 10M videos) and LaVie (3B params, 35M videos).
  • Comparison: BCNI outperformed the 2.3B parameter DEMO model by 6% on MSR-VTT and the 3B parameter LaVie by 16% on UCF-101.

• Robustness Analysis:

  • Structured corruptions showed smoother degradation curves as noise levels increased compared to Gaussian or Uniform baselines, which collapsed rapidly.
  • BCNI excelled on caption-rich datasets (WebVid, MSR-VTT), while SACN was superior on class-labeled, motion-focused datasets (UCF-101).

• Generalization:

  • The framework successfully improved Autoregressive models (MAGVIT, CogVideo) and Multimodal Video LLMs (PAVE), achieving state-of-the-art CIDEr scores on the AVSD task.

5. Significance

• Paradigm Shift: The paper challenges the notion that noise injection must be isotropic. It establishes that structured, data-aligned noise is a superior regularizer for sequential generation tasks.
• Data Efficiency: It demonstrates that robustness can be achieved through better training objectives (structured corruption) rather than simply scaling up model size or dataset volume.
• Temporal Coherence: By explicitly addressing the "cascading error" problem in video diffusion, CAT-LVDM provides a new standard for generating temporally stable and semantically faithful videos under imperfect conditions.
• Broad Applicability: The success of the operator view across diffusion, autoregressive, and LLM architectures suggests a universal principle for robust generative modeling in multimodal settings.

In conclusion, CAT-LVDM sets a new framework for robust video diffusion, proving that aligning noise with the semantic and spectral structure of the data is the key to overcoming the brittleness of current text-to-video models.