SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer

1. Problem Statement

Video Diffusion Transformers (DiTs) have achieved state-of-the-art results in video generation but suffer from quadratic computational complexity ($O(N^2)$) due to full attention mechanisms. This makes generating high-resolution, long-duration videos (e.g., 30k–100k tokens) computationally prohibitive.

Existing solutions face a trade-off:

• Training-free sparse attention: Methods like sliding-window or Top-K attention reduce latency but are limited to moderate sparsity (40–60%). They often fail to capture long-range dependencies, leading to artifacts like object duplication or temporal inconsistency.
• Training-based sparse attention: Methods that fine-tune sparse models can achieve high sparsity (80–95%) but require massive datasets (e.g., Koala-36M) and computational resources (hundreds of GPU hours).
• LoRA Limitations: Simply applying Low-Rank Adaptation (LoRA) to sparse attention models fails to fully recover the quality of dense models under extreme sparsity. Experiments show that LoRA-tuned sparse models still exhibit significant degradation in subject consistency and temporal coherence because LoRA alone cannot compensate for the severe information loss caused by restricting cross-token interactions.

2. Methodology: SALAD

The authors propose SALAD (High-Sparsity Attention paralleling with Linear Attention for Diffusion Transformer), a lightweight architecture that combines Sparse Attention with a parallel Linear Attention branch.

Core Architecture

• Hybrid Branch Design: SALAD introduces a linear attention branch running in parallel with the sparse attention branch. Both branches share the same Query ($Q$), Key ($K$), and Value ($V$) projections, which are initialized from the pre-trained model and fine-tuned using LoRA.
  • Sparse Branch: Handles the bulk of sequence modeling with high sparsity (e.g., 90%), ensuring efficiency.
  • Linear Branch: Uses ReLU-based linear attention to provide global token mixing ($O(N)$ complexity). It captures critical long-range dependencies and global context that the sparse branch misses.
• Parameter Efficiency: The linear branch shares weights with the sparse module, adding only ~4.99% additional parameters relative to the pre-trained model.

Key Innovation: Multi-level Static-Dynamic Scaling Strategy

The authors observed a rank imbalance: the linear branch output has a significantly lower effective rank than the sparse branch. Directly summing them causes "rank erosion," where the low-rank linear output suppresses the high-rank informative structures of the sparse branch. To solve this, SALAD employs a two-stage scaling mechanism:

1. Layer-wise Static Scaling (LWSS):
   • A zero-initialized projection layer is inserted after the linear branch in each Transformer block.
   • This allows the model to start from the pre-trained sparse state and gradually learn to incorporate the linear branch, preventing initial instability.
2. Input-Timestep-Aware Dynamic Scaling (ITADS):
   • A lightweight module computes a global scaling factor ($s \in (0, 1)$) based on the current hidden states and the denoising timestep ($t$).
   • This dynamically adjusts the contribution of the linear branch. For instance, the linear branch is more critical in early denoising steps (global structure) and less so in later steps (fine details), or vice versa depending on the input.
   • The scaling factor is constrained via a Sigmoid function to ensure the linear branch acts as a controlled auxiliary complement rather than dominating the output.

Post-Tuning Branch Dropping

To further optimize inference speed, the authors propose a branch-dropping strategy. Since the dynamic scaling factors indicate the necessity of the linear branch for specific timesteps and inputs, branches with scaling factors below a threshold $\tau_t$ are pruned during inference. This reduces computation without degrading quality.

3. Key Contributions

1. SALAD Architecture: A novel efficient attention module that enables ultra-high sparsity (up to 90%) by parallelizing a lightweight linear attention branch with sparse attention.
2. Multi-level Scaling Strategy: The introduction of Static-Dynamic Scaling to resolve the rank imbalance between sparse and linear branches, ensuring the linear branch supplements rather than suppresses the sparse output.
3. Extreme Efficiency:
   • Data Efficiency: Requires only 2,000 video samples (vs. millions for other methods).
   • Compute Efficiency: Requires <1,600 training steps and <30 GPU hours (batch size 8).
   • Inference Speedup: Achieves 1.52× to 2.03× speedup across different models and sequence lengths.
4. Quality Preservation: Maintains generation quality comparable to full-attention baselines, significantly outperforming LoRA-tuned sparse models and training-free methods.

4. Experimental Results

The method was evaluated on Wan2.1 video diffusion models (1.3B and 14B parameters) at 480p and 720p resolutions.

• Performance vs. Sparsity:
  • On Wan2.1-1.3B (30k tokens), SALAD achieves 90% sparsity with a 1.72× speedup, outperforming all baselines (SVG2, PARO, ST-SWA+LoRA) on VBench metrics (Subject Consistency, Background Consistency, Image Quality, Text Consistency).
  • On Wan2.1-1.3B (100k tokens), it achieves 90% sparsity with a 2.03× speedup.
  • On Wan2.1-14B, it achieves 88% sparsity with a 1.59× speedup while maintaining performance close to the full-attention model.
• Comparison with LoRA:
  • Standard LoRA tuning on sparse models fails to recover quality at high sparsity. SALAD outperforms LoRA variants even with fewer trainable parameters (165M vs. 189M+ for LoRA r=256).
• Ablation Studies:
  • Removing the dynamic scaling or using random initialization degrades performance, confirming the necessity of the multi-level scaling strategy.
  • Branch dropping (removing ~50% of linear branches based on thresholds) slightly improves or maintains quality while boosting efficiency.

5. Significance

SALAD addresses a critical bottleneck in video generation: the trade-off between computational efficiency and generation quality.

• Scalability: It makes high-resolution, long-duration video generation feasible on consumer-grade or limited enterprise hardware by drastically reducing inference latency.
• Accessibility: By requiring minimal data (2k samples) and compute (30 GPU hours) for fine-tuning, it democratizes the ability to adapt large video models to high-sparsity regimes, unlike previous methods that required massive pre-training on proprietary datasets.
• Theoretical Insight: The paper provides a deep analysis of the "rank imbalance" in hybrid attention, offering a principled solution (scaling) to integrate low-rank global context with high-rank local sparse modeling.

In summary, SALAD demonstrates that ultra-high sparsity in video diffusion is achievable without sacrificing quality, provided that a lightweight, dynamically scaled linear branch is used to recover the global context lost by sparse attention.