Technical Summary: Breaking the Lock-in via Representation Modulation

Problem Statement

Recent text-to-image (T2I) models, particularly those leveraging large-scale Transformer backbones and flow-based objectives, have achieved remarkable text-image alignment and photorealism. However, these models frequently suffer from a "diversity collapse" issue: when generating multiple images from a fixed prompt, the outputs converge to overly similar compositions, layouts, and stylistic patterns. This homogeneity limits the utility of generative models for tasks requiring broad candidate exploration or synthetic dataset construction.

Existing methods to enhance diversity typically fall into two categories: batch-wise diversification (e.g., Particle Guidance, OSCAR) and individual trajectory manipulation (e.g., CADS). While effective to varying degrees, these approaches often incur significant computational and memory overheads due to auxiliary optimization, parallel candidate generation, or inter-sample interactions. Furthermore, the fundamental mechanistic causes of diversity collapse within the model's internal representations remain largely unexplored, with prior work focusing primarily on inference-time heuristics rather than root-cause analysis.

Methodology: DAVE

The authors propose DAVE (DC Attenuation for diVersity Enhancement), a training-free, representation-level intervention designed to break the "lock-in" of generative trajectories.

Root Cause Analysis

Through an analysis of intermediate Transformer features, the authors identify that the zero-frequency spatial average (DC component) of hidden representations converges rapidly across different random seeds during the early stages of generation.

• Observation: In the early denoising steps (high noise regime), the DC component exhibits high cross-seed cosine similarity and low variance, accounting for over 50% of the total energy in the representation.
• Mechanism: This phenomenon, termed "early DC drift," is attributed to the spectral bias of neural networks (which prioritize low-frequency signals) combined with the nature of flow-matching objectives under high uncertainty. The model predicts the text-conditioned expectation of the data, causing the DC component to align with a seed-invariant global mean. This early alignment establishes a structural anchor that restricts the ability of initial noise seeds to induce sufficient variation in global layout and coarse semantics.

The DAVE Intervention

To mitigate this lock-in, DAVE selectively attenuates the DC component during the early generative phase.

• Operation: For a hidden representation $h^{(\ell)}_t$ at Transformer block $\ell$ and timestep $t$, the DC component $\mu^{(\ell)}_t$ is computed as the spatial mean across visual tokens. The representation is then modified as:
  $$\hat{h}^{(\ell)}_t = \alpha \cdot \mu^{(\ell)}_t + (h^{(\ell)}_t - \mu^{(\ell)}_t)$$
  where $\alpha \in (0, 1)$ is an attenuation strength parameter.
• Constraints: The intervention is applied only to specific Transformer blocks ($\ell \in \mathcal{L}$) identified as having strong cross-seed DC convergence, and only during early timesteps ($t < \tau$).
• Implementation: DAVE requires no retraining, no changes to the sampling pipeline, and no additional memory for feature banks. It operates as a surgical modification of intermediate outputs before they are passed to subsequent blocks.

Key Contributions

1. Mechanistic Insight: The paper identifies the rapid convergence of the DC component in early generation steps as a primary bottleneck for seed-level variation, providing a theoretical and empirical explanation for trajectory lock-in.
2. DAVE Algorithm: A lightweight, training-free method that attenuates the DC component to unlock generative diversity without compromising semantic alignment or visual quality.
3. Efficiency: Unlike interaction-based methods that scale quadratically with batch size ($O(TB^2)$), DAVE maintains linear complexity ($O(TB)$) with negligible overhead, making it scalable for modern large models.
4. Robustness: The method is shown to be robust across different model architectures (Stable Diffusion 3/3.5, FLUX.1-dev, SANA1.5) and prompt complexities, with the target block pool $\mathcal{L}$ being largely dataset-agnostic.

Experimental Results

The authors evaluated DAVE on ImageNet and MS-COCO benchmarks using metrics for visual quality (FID, Precision, Density), diversity (Recall, Coverage, Vendi Score), and text-image alignment (CLIP Score).

• Diversity Gains: DAVE consistently improves diversity metrics (Recall, Vendi Score) across all tested models, outperforming the original sampler and competing favorably with state-of-the-art methods like CADS and SPARKE.
• Quality Preservation: Unlike some baselines that trade fidelity for diversity (e.g., CADS showing degraded CLIP scores on certain models), DAVE maintains competitive image quality and text alignment.
• In-Batch Performance: In explicit in-batch generation settings, DAVE achieves lower feature-level similarity and higher perceptual distance compared to methods like Particle Guidance and SPELL, while incurring significantly lower computational costs.
• Distilled Models: The method is compatible with distilled models (e.g., SD3.5-Large-Turbo), improving diversity even in streamlined sampling regimes.
• Ablation Studies:
  • Temporal Cutoff ($\tau$): Intervening in the first 15–20% of steps yields optimal diversity without quality loss.
  • Attenuation Strength ($\alpha$): Values between 0.2 and 0.5 balance structural variation and semantic coherence.
  • Block Selection: The method is robust to the specific choice of blocks within the identified high-convergence pool $\mathcal{L}$.

Significance and Claims

The paper claims that DAVE offers a practical and scalable framework for addressing the critical issue of limited generation diversity in flow-based T2I models. By targeting the root cause of trajectory lock-in—the early homogenization of the DC component—the method successfully broadens the generative scope of pre-trained models.

The authors emphasize that DAVE achieves a favorable diversity-fidelity trade-off without the computational burdens of existing solutions. It serves as a "surgical" intervention that preserves the sampling pipeline while enabling the stochasticity of initial noise to drive meaningful structural variations. The work suggests that understanding and modulating internal representation dynamics is a viable path to enhancing generative capabilities without the need for retraining or complex optimization procedures.