Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective

Technical Summary: Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition

Problem Statement
Diffusion models have become the leading paradigm for image generation, largely driven by the attention mechanism's ability to integrate global context and model long-range dependencies. However, this global connectivity creates a fundamental trade-off: while it enhances novelty and variety, it is prone to causing "spurious mixing" of incompatible features (e.g., blending materials between distinct objects). Distinguishing between beneficial context integration and harmful semantic leakage is non-trivial because they share the same underlying mechanism. Current analyses often treat attention merely as a token-wise retrieval mechanism, failing to capture the rich interaction dynamics encoded within the attention matrix itself. Consequently, there is a lack of principled methods to identify when attention settles into incoherent "metastable mixtures" and to control this behavior to navigate the fidelity–diversity trade-off.

Methodology
The authors propose a framework that characterizes the pre-softmax attention matrix $QK^\top$ as a dynamic associative memory encoding pairwise feature associations. Building on the insight that transformer self-attention approximates the update rule of a modern Hopfield network, the paper decomposes the attention matrix into two distinct components:

1. Symmetric Component ($M_{sym}$): Represents the energy landscape. It governs the stability of retrieved features, where lower energy corresponds to states strongly supported by the associative memory.
2. Skew-Symmetric Component ($M_{skew}$): Represents circulation dynamics. Since skew-symmetric matrices contribute zero quadratic energy for real-valued states, this component acts as a directional force that drives circulation, potentially perturbing metastable states.

Based on this decomposition, the authors derive Hopfield-style stability measures to quantify retrieval quality:

• Energy ($E_X$): A global measure of how strongly a state is supported by the symmetric interaction.
• Instability Fraction ($r_X$): A local measure identifying the fraction of spatial positions where the local field conflicts with the current state.
• Alignment Score ($Align_X$): A cosine similarity measure quantifying the global directional agreement between a retrieved feature and its induced local field.

To control the generation process, the authors propose a training-free intervention mechanism. They modulate the skew-symmetric component via a scalar control parameter $\alpha$ (circulation scale) and blend the resulting perturbed retrieval with the baseline retrieval using a parameter $\beta$. This allows for the injection of "circulation-driven drift" to break metastable mixtures without altering the underlying energy landscape. Furthermore, they introduce an adaptive control variant that modulates the injection strength based on the realized functional symmetry index ($\eta_M$) of the attention matrix for each sample, preventing over-perturbation of already high-quality generations.

Key Contributions

• Associative Memory Framework: The paper establishes a framework that encodes pairwise feature associations within the attention matrix, introducing a symmetric/skew-symmetric decomposition that disentangles energy-based stability from circulation-driven drift.
• Stability Measures: Leveraging the symmetric component, the authors derive Hopfield-style stability measures ($E_X$, $r_X$, $Align_X$) that quantify feature stability. They demonstrate a strong correlation between these internal measures and external fidelity–diversity trade-offs.
• Controllable Circulation Knob: The authors propose using the skew-symmetric component as a tunable knob for test-time intervention. By injecting directional drift, this method perturbs metastable mixtures to restore structural coherence, offering a mechanism to balance fidelity and diversity.

Results
The proposed method was evaluated on SDXL using 1,000 COCO2014 prompts. Key findings include:

• Correlation with Quality: The proposed stability measures show consistent correlations with external metrics. High alignment scores ($Align_X$) correlate with higher Aesthetic Scores and lower LPIPS diversity (indicating structural coherence), while low alignment correlates with higher diversity but increased structural incoherence and artifacts.
• Regime-Dependent Effects: The intervention exhibits state-dependent behavior. On "Unstable" (low-quality) subsets, circulation injection significantly improves aesthetic scores and ImageReward by breaking spurious mixtures. Conversely, on "Stable" (high-quality) subsets, excessive circulation can degrade metrics by disrupting coherent structures.
• Adaptive Control: An adaptive variant that modulates the perturbation based on the sample's realized symmetry index ($\eta_M$) successfully mitigates the degradation seen in high-quality samples under static, excessive perturbation. It preserves gains in low-quality samples while maintaining performance on high-quality ones.
• Comparison to Global Tempering: Unlike global attention temperature scaling, which indiscriminately sharpens or dampens interactions (often creating artifacts like duplicated limbs), the proposed circulation control selectively suppresses weakly supported mixture artifacts while preserving dominant structural support.

Significance and Claims
The paper claims to offer a principled lens for analyzing attention not merely as a token-mixing operator, but as an interaction matrix with distinct energy-supported and circulation-driven components. By re-framing spurious mixing as entrapment in metastable states, the work provides a diagnostic tool (stability measures) and a control mechanism (circulation modulation) to navigate the fidelity–diversity trade-off. The authors suggest this perspective may extend beyond diffusion models to other transformer architectures, such as large language models, offering a new way to study attention dynamics. The method is presented as a training-free, post-hoc intervention that improves the reliability and controllability of generative systems by addressing the root cause of semantic leakage in the attention mechanism.