Technical Summary: Attention Sinks in Diffusion Transformers: A Causal Analysis

Problem Statement

In autoregressive large language models (LLMs), "attention sinks"—specific tokens or positions that accumulate disproportionate attention mass regardless of semantic importance—are widely interpreted as functionally critical structures. They are hypothesized to serve as stable anchors for key-value caching, prevent attention entropy collapse, and act as implicit registers for residual information. Consequently, disrupting these sinks in autoregressive models often degrades generation quality.

However, it remains unclear whether this intuition transfers to Diffusion Transformers (DiTs). Unlike autoregressive decoding, which proceeds token-by-token under causal masking, diffusion models perform non-causal, bidirectional attention across multiple denoising steps, refining all positions simultaneously. In this setting, attention does not obviously serve as a persistent memory anchor for preceding context. This architectural distinction raises a fundamental question: Are attention sinks functionally necessary for high-quality generation in text-to-image diffusion, or do they merely represent replaceable routing patterns? Without causal evidence, design choices for attention sparsification and compression in DiTs lack a grounded understanding of which high-mass recipients must be preserved.

Methodology

The authors present a systematic causal analysis of attention sinks in diffusion transformers, moving beyond observational studies to interventionist testing.

1. Dynamic Sink Definition

Contrary to autoregressive models where sinks often align with fixed positions (e.g., the beginning-of-sequence token), the authors define sinks dynamically. For each attention head, layer, and denoising timestep, they identify the top-$k$ key positions receiving the largest incoming attention mass. This definition acknowledges that dominant recipients in diffusion models vary across timesteps and layers, rather than being fixed to structural tokens like index-0.

2. Causal Interventions

The study employs training-free, paired interventions during inference to test the necessity of these sinks. The interventions are applied along two independent pathways:

• Score Path: Adding a log-bias ($\log \eta$) to the logits of sink tokens before the softmax operation, effectively scaling their attention probability. Setting $\eta=0$ (or adding a large negative bias) effectively zeros out the attention mass.
• Value Path: Replacing the value vectors of sink tokens with alternatives (zeros, mean values, or linear interpolations).

3. Experimental Setup

• Models: Primary experiments on Stable Diffusion 3 (SD3) (joint attention over image/text tokens) and corroboration on SDXL (U-Net with separate self/cross-attention).
• Prompts: 553 prompts from the GenEval benchmark, supplemented by 5,000 COCO captions for SDXL validation.
• Protocol: Strict seed-matched paired generation to isolate intervention effects from sampling variance. Results are reported with 95% bootstrap confidence intervals.
• Metrics:
  • Alignment: CLIP-T (text-image semantic alignment).
  • Preference: ImageReward and HPS-v2 (human preference proxies).
  • Perceptual/Distributional: LPIPS (perceptual distance) and FIDshift (distributional distance between baseline and intervention outputs).
  • Equivalence Margin: A practical equivalence margin of $|\Delta| < 0.002$ for CLIP-T is established based on empirical noise floors.

Key Results

1. Dynamics of Attention Sinks

• Non-Fixed Positions: The overlap between dynamically identified top-1 sinks and the fixed index-0 position is negligible (<0.2%), confirming that diffusion sinks are not anchored to structural tokens like in autoregressive LLMs.
• Phase Dependence: Attention concentration peaks during early denoising steps (high-noise regime) and diminishes toward later steps.
• Layer Distribution: Middle layers (e.g., Layer 12 in SD3) exhibit the strongest concentration, with maximum incoming mass reaching ~9.5% and lowest entropy.
• Modality: In SD3's joint attention, over 99.9% of dynamically identified sinks correspond to text-conditioning tokens rather than visual-latent tokens.

2. Causal Necessity for Semantic Alignment

• No Degradation at Standard Intensity: Removing dynamically identified sinks (top-1, single or multi-layer) does not degrade semantic alignment (CLIP-T) or preference metrics (ImageReward, HPS-v2). All 95% confidence intervals for metric changes include zero.
• Robustness to Strength: Even under stronger interventions (top-5 removal), CLIP-T remains robust. While HPS-v2 shows a small negative shift under top-5 removal, the magnitude is marginal (<1%) and lies near the boundary of practical equivalence, suggesting a metric-dependent trade-off rather than a fundamental quality collapse.
• Cross-Architecture Validity: Results hold for both SD3 (joint attention) and SDXL (cross-attention), with cross-attention inducing slightly larger perceptual drift but no alignment loss.

3. Perceptual and Distributional Shifts

• Sink-Specific Perturbation: While alignment is preserved, suppressing sinks induces significant perceptual shifts. The LPIPS distance between baseline and sink-removed outputs is approximately 6× larger than that of equal-budget random masking.
• Interpretation: This indicates that sinks carry structured, trajectory-level information that influences the visual realization (layout, style, viewpoint) but is not necessary for maintaining semantic alignment with the prompt. The intervention moves samples within the model's output manifold without collapsing the distribution.

4. Dose-Response and Boundary Conditions

• Flat Response Curves: Dose-response sweeps across suppression ratios ($\eta \in [0, 1]$) and value replacements show uniformly flat curves with no significant degradation in alignment metrics.
• Metric-Dependent Boundaries: Under very strong interventions ($k \ge 10$), HPS-v2 exhibits sink-specific degradation compared to random masking, while CLIP-T remains stable. This suggests sinks may contribute to preference-oriented dimensions not captured by simple alignment proxies.

Significance and Claims

The paper claims to provide the first causal evidence that attention sinks in diffusion transformers are not functionally necessary for semantic alignment under standard inference settings.

• Decoupling Mass from Necessity: The findings challenge the assumption that high incoming attention mass indicates functional necessity. In diffusion transformers, dominant recipients can be suppressed without degrading the alignment between text and image, even though they significantly alter the perceptual trajectory.
• Design Implications: For efficient diffusion attention design (e.g., sparsification, token pruning), this suggests that high-mass recipients do not need to be hard-coded as privileged tokens to be preserved. They can be treated as candidates for budgeted removal, provided the target metric (alignment vs. perceptual consistency) is monitored.
• Distinction from Autoregressive Intuition: The study explicitly cautions against transferring autoregressive sink intuitions to diffusion models. The "anchor" role of sinks in autoregressive generation does not appear to translate to the bidirectional, iterative refinement of diffusion.
• Limitations: The authors note that their claims are scoped to widely used proxy metrics (CLIP-T, HPS-v2) and do not address fine-grained compositional fidelity (which may require detector-based benchmarks) or end-to-end efficiency gains (speed/FLOPs), which remain engineering challenges.

In summary, the paper establishes that while attention sinks in diffusion transformers induce structured perceptual drift, they are dispensable for maintaining semantic alignment, offering a new theoretical basis for attention compression and acceleration in generative diffusion models.