Steering Video Diffusion Transformers with Massive Activations

1. Problem Statement

While Video Diffusion Transformers (DiTs) have achieved significant progress in text-to-video generation, leveraging their internal model signals to enhance quality without additional training or computational overhead remains an open challenge. Existing methods often rely on:

• Training-time objectives: Requiring extensive retraining or preference optimization.
• Inference-time heuristics: Using auxiliary models, external noise priors, or multiple forward passes, which increase computational cost.

The authors identify a gap in understanding how Massive Activations (MAs)—rare, high-magnitude spikes in hidden state dimensions—function specifically within the temporal context of video generation. Unlike image DiTs, video DiTs possess a temporal dimension and specific latent compression structures (chunking) that may create unique activation patterns, but these have not been systematically analyzed or utilized for guidance.

2. Key Observations

The authors conducted a systematic analysis of MAs across multiple video DiT architectures (Wan2.1, Wan2.2, CogVideoX) and discovered a distinct structured positional hierarchy that is unique to video generation:

1. First-Frame Dominance: Tokens corresponding to the first latent frame consistently exhibit the largest MA magnitudes across the entire sequence. These act as global temporal anchors.
2. Periodic Boundary Spikes: MA magnitudes show a periodic pattern, spiking significantly at latent-frame boundaries (the head and tail tokens of each temporal chunk). These boundaries correspond to the seams where temporal compression occurs in the VAE.
3. Temporal Decay: These structured patterns are most pronounced during early denoising steps and decay as the sampling process progresses.
4. Functional Impact:
   • Disrupting MAs at the first frame degrades visual quality.
   • Disrupting MAs at boundary tokens causes temporal discontinuities (flickering) between chunks.
   • Amplifying MAs at these specific positions improves both visual fidelity and temporal coherence.

3. Methodology: Structured Activation Steering (STAS)

Based on these observations, the authors propose STAS, a training-free, self-guidance-like method that selectively steers MA values at structurally significant token positions.

• Core Mechanism: STAS modifies the activation tensor during the inference process by amplifying MA dimensions at specific tokens toward a scaled global maximum reference magnitude.
• Target Selection ($\mathcal{S}$): The method targets two specific sets of tokens:
  1. First-Frame Tokens ($\mathcal{F}_0$): To reinforce global temporal anchors.
  2. Boundary Tokens ($\mathcal{B}(p)$): The head and tail $p\%$ of tokens within each latent frame chunk (e.g., 8%) to smooth transitions between compressed segments.
• Steering Rule ($g$): For a selected token $i$ and MA dimension $d$, the activation is updated as:
  $$\hat{D}_{\theta, m, i, d} = D_{\theta, m, i, d} + \mathbb{1}[t \geq t_K] \cdot \mathbb{1}[i \in \mathcal{S}] \cdot \mathbb{1}[d \in \mathcal{M}] \cdot (g(i, d) - D_{\theta, m, i, d})$$
  Where $g(i, d) = \alpha \cdot \max_j |D_{\theta, m, j, d}| \cdot \text{sign}(D_{\theta, m, i, d})$. This steers the value toward a scaled global maximum ($\alpha$) while preserving polarity.
• Timing: The steering is applied only during the early denoising steps ($t \geq t_K$), where the boundary signal is strongest.
• Efficiency: STAS operates within a single forward pass, requires no additional model parameters, and introduces negligible computational overhead (<0.1%). It is fully compatible with Classifier-Free Guidance (CFG).

4. Key Contributions

1. First Systematic Analysis of MAs in Video DiTs: Revealed a structured positional hierarchy (first-frame peaks and periodic boundary spikes) driven by the VAE's temporal compression, distinct from image DiTs.
2. STAS Algorithm: Proposed a novel, training-free method that exploits this structure to steer activations, improving temporal coherence and visual quality without extra training or inference cost.
3. Comprehensive Evaluation: Demonstrated consistent improvements across different model scales (1.3B to 5B) and architectures (Wan, CogVideoX), validated by both automated benchmarks and human preference studies.

5. Experimental Results

The method was evaluated on Wan2.1-1.3B, Wan2.2-5B, and CogVideoX-5B.

• Quantitative Improvements (VBench):
  • Total Score: Consistent gains across all models (e.g., Wan2.1-1.3B improved from 81.39 to 81.76).
  • Temporal Consistency: Significant improvements in Subject Consistency and Background Consistency.
  • Cross-Chunk Stability: STAS specifically reduced similarity drops at latent-frame boundaries (cross-chunk transitions), which are typically the weakest points in video generation.
  • Object-Attribute Binding: Improved scores on T2V-CompBench, indicating more stable object identities over time.
• Qualitative Improvements:
  • Mitigated artifacts like sudden appearance of objects (e.g., a teddy bear appearing out of nowhere) and structural changes in scenes.
  • Produced smoother transitions and more consistent motion trajectories.
• Efficiency: Inference time increased by only ~0.06 seconds per video (from 234.47s to 234.53s), confirming the negligible overhead.
• Compatibility: STAS can be combined with other training-free methods (e.g., FlowMo, CFG-Zero) to achieve additive performance gains.

6. Significance

This paper bridges the gap between understanding internal model dynamics and practical video generation enhancement. By identifying that Massive Activations are not just noise but carry structured temporal information, the authors provide a lightweight, plug-and-play solution to one of the most persistent problems in video generation: temporal incoherence.

STAS demonstrates that high-quality video generation can be improved by intelligently manipulating existing internal signals rather than relying on expensive retraining or heavy inference-time guidance. This approach offers a scalable path for enhancing future large-scale video diffusion models with minimal resource expenditure.