Technical Summary: TeDiO – Temporal Diagonal Optimization for Training-Free Coherent Video Diffusion

1. Problem Statement

Despite rapid advancements in text-to-video diffusion transformers (e.g., Wan2.1, CogVideoX), current state-of-the-art models struggle with temporal coherence. While individual frames are often visually compelling, the generated videos frequently suffer from:

• Flickering and drifting: Textures and objects shift unpredictably between frames.
• Inconsistent motion: Global motion lacks continuity, leading to physically implausible dynamics.
• Artifacts: Issues such as subject duplication, merging objects, and "jittery" motion break the illusion of realism.

Existing solutions to these problems typically involve:

1. Retraining: Modifying architectures or training models with explicit motion-consistency supervision, which requires large-scale datasets and is computationally expensive.
2. External Supervision: Using auxiliary components like optical flow networks or multi-GPU pipelines during inference, which reduces practicality and generality.

The paper posits a fundamental question: Can temporal coherence be improved efficiently using only the diffusion model's internal signals, without retraining, external supervision, or architectural changes?

2. Key Observation

The authors identify a direct, measurable link between a model's internal temporal self-attention and the quality of generated motion:

• Incoherent Motion: Produces irregular, fragmented, and jagged patterns in the temporal diagonals of attention maps.
• Coherent Motion: Corresponds to smooth, clean, band-diagonal patterns in these maps, indicating stable frame-to-frame alignment.

This observation reveals that motion stability leaves a "footprint" inside the transformer's attention mechanism, offering a self-diagnostic signal that can be exploited during inference.

3. Methodology: TeDiO (Temporal Diagonal Optimization)

TeDiO is a training-free, inference-time method that enhances motion coherence by regularizing the temporal diagonals in self-attention maps. It operates as a plug-and-play enhancement for existing video diffusion transformers.

Core Mechanism

1. Reparameterization: During the sampling process, the queries ($Q$) and keys ($K$) from specific blocks of the frozen Diffusion Transformer (DiT) are reshaped to isolate temporal interactions. Instead of a flattened spatio-temporal axis, the method groups spatial positions and attention heads while keeping the temporal dimension ($F$) explicit.
2. Diagonal Analysis: For each spatial-head patch, a temporal attention map is computed. The method focuses on the main diagonal ($b=0$) and immediate neighbors ($b=\pm1$), which represent self-consistency and motion continuity between consecutive frames.
3. Variability Scoring: A diagonal variability score ($S(p)$) is calculated for each patch. This score measures the local fluctuations (gradients) along the diagonal bands. High scores indicate "jerky" temporal behavior (abrupt attention shifts), while low scores indicate stable motion.
4. Selective Optimization:
   • TeDiO identifies the top-$k$ patches with the highest variability scores (the most unstable regions).
   • It defines a loss function ($L_{TeDiO}$) based on the sum of these scores.
   • During the early diffusion timesteps (where global motion trajectories are shaped), the latent representation ($z_t$) is updated via a few gradient steps: $z_t \leftarrow z_t - \eta \nabla_{z_t} L_{TeDiO}$.
   • This update encourages the attention maps to adopt smoother, band-diagonal patterns.

Operational Characteristics

• Training-Free: No modification of model weights ($\theta$) is required.
• No External Supervision: Does not rely on optical flow estimators, discriminators, or other external models.
• Efficiency: Operates on a single GPU with negligible computational overhead (approx. 1.1x runtime compared to the baseline).
• Timing: Applied only during the first few diffusion steps ($\tau_1, \dots, \tau_\ell$) to seed coherence that propagates through the generation process.

4. Experimental Results

The method was evaluated on two state-of-the-art open-source models: Wan2.1-1.3B (Flow Matching) and CogVideoX-5B (Denoising).

Benchmarks

• VBench: Assesses perceptual and temporal quality on standard prompts.
• VideoJAM-Bench: Uses 128 challenging prompts designed to expose temporal instability and complex interactions.

Quantitative Findings

TeDiO consistently improved key metrics across both models and benchmarks:

• Motion Smoothness: Significant increases (e.g., Wan2.1 improved from 98.32 to 98.78 on VBench).
• Temporal Flickering: Reduced flickering in static prompts.
• Subject & Background Consistency: Higher similarity between consecutive frames.
• Perceptual Quality: Improved Aesthetic and Imaging Quality scores.
• Dynamic Degree: A slight decrease was observed, which the authors attribute to the removal of high-frequency temporal noise (jitter) rather than a loss of intentional motion. The resulting motion profiles align closely with real-world video datasets (WebVid, UCF-101).

Comparison with SOTA

Compared to FlowMo [42], a related inference-time optimization method:

• Performance: TeDiO achieved larger gains in motion smoothness (+0.62 vs +0.06) and subject consistency (+1.68 vs -0.69).
• Efficiency: TeDiO runs on 1 GPU with 1.1x runtime overhead, whereas FlowMo requires 2 GPUs and incurs 2.39x runtime overhead due to full backpropagation.

User Study

In a perceptual study with 10 participants using VideoJAM-Bench prompts, TeDiO-generated videos were strongly preferred over base models for their superior temporal coherence.

5. Key Contributions

1. Discovery of a Structural Link: The paper uncovers a quantifiable relationship between temporal attention diagonals and motion stability, providing a new perspective for analyzing video diffusion transformers.
2. TeDiO Framework: Introduces a training-free, inference-time approach that regularizes diagonal variability in temporal attention to optimize input latents. It requires no weight updates or external supervision.
3. Empirical Validation: Demonstrates consistent improvements in temporal smoothness, perceptual stability, and video realism across multiple diffusion backbones (Wan2.1, CogVideoX) and benchmarks.

6. Significance and Claims

The paper argues that significant gains in dynamic realism do not necessarily require scaling model sizes or training data. Instead, exploiting the temporal structure already encoded within diffusion transformers offers an efficient path forward.

TeDiO represents a shift toward inference-time optimization that is:

• Model-Agnostic: Applicable to various architectures (Flow Matching or Denoising) and scales.
• Practical: Eliminates the need for costly retraining or complex multi-GPU setups.
• Interpretable: Uses internal attention signals as a direct diagnostic and optimization tool.

Limitations

The authors modestly note that TeDiO:

• Does not correct semantic errors or prompt alignment issues inherent in the base model.
• May have diminishing returns for extremely long videos or complex scene transitions.
• Is less effective for intrinsically difficult motions (e.g., acrobatics) that require precise long-range coordination not well-represented in the underlying model.
• Requires access to intermediate attention maps, which may not be exposed in closed-source implementations.