HAM: A Training-Free Style Transfer Approach via Heterogeneous Attention Modulation for Diffusion Models

1. Problem Statement

Diffusion models have revolutionized image generation and style transfer. However, existing methods face a critical style-content trade-off:

• Training-based methods (e.g., using LoRA or ControlNet) are computationally expensive, require fine-tuning for specific styles, and lack robustness across diverse references.
• Training-free methods (e.g., StyleID, DiffArtist) typically rely on injecting style features into self-attention layers. Since self-attention encodes both spatial structure and semantic content, manipulating it often leads to identity distortion (loss of content structure) or insufficient style transfer.
• Current Gap: There is a lack of a training-free approach that can simultaneously preserve high-fidelity content identity and capture complex, diverse style references without requiring gradient optimization.

2. Methodology: HAM (Heterogeneous Attention Modulation)

The proposed HAM framework is a training-free approach that operates on pre-trained diffusion models (compatible with both SD2.1 and SD3.5). It utilizes a "Teacher-Student" paradigm where a Content Teacher and a Style Teacher guide a Student Generator. The method consists of three core modules:

A. Style-Infused Noise Initialization (SINI)

Instead of using pure content noise or a simple blend, HAM initializes the latent noise at timestep $T$ by fusing the inverted noise from the content and style images.

• Mechanism: It employs Adaptive Instance Normalization (AdaIN) to fuse the style noise into the content noise.
• Innovation: To prevent the style injection from overwhelming the content identity, a Content Residual Noise component is added.
• Formula: The initial noise $z_T^m$ is a weighted combination of the AdaIN-fused stylized noise and the original content noise, controlled by a hyperparameter $\gamma$. This ensures the generation trajectory starts with a balanced state of style and structure.

B. Global Attention Regulation (GAR)

This module operates on Self-Attention layers to manage the macroscopic balance between style and content.

• Goal: Preserve the spatial structure and identity of the content while introducing global style attributes.
• Mechanism:
  1. It extracts attention projections ($Q, K, V$) from both the Content Teacher and Style Teacher.
  2. It fuses these projections using AdaIN to create composite projections ($Q^{cs}, K^{cs}, V^{cs}$) that align the statistical distribution of style features with content features.
  3. These composite projections are then blended with the Student Generator's native projections using a weighting factor $\alpha$.
• Outcome: This ensures the self-attention mechanism retains the content's structural integrity while being globally modulated by the style.

C. Local Attention Transplantation (LAT)

This module operates on Cross-Attention layers to achieve precise, localized style injection without destroying content identity.

• Problem Solved: Previous methods often replace keys/values in self-attention, which destroys content. LAT shifts the injection to cross-attention.
• Mechanism:
  1. Style Injection: The Key ($K$) and Value ($V$) projections from the Style Teacher are directly transplanted into the Student Generator's cross-attention, replacing the native ones. This injects the style's semantic guidance.
  2. Content Protection: To prevent the content identity from degrading due to the new $K/V$, the Query ($Q$) is protected. The Query from the Content Teacher is weighted and fused with the Student's native Query using a factor $\beta$.
• Outcome: This "Heterogeneous" modulation (different sources for Q vs. K/V) effectively decouples style guidance (from K/V) and content preservation (from Q), allowing for high-fidelity stylization.

3. Key Contributions

1. Training-Free Framework: HAM achieves high-quality style transfer without any gradient optimization or fine-tuning of the diffusion model, making it efficient and adaptable.
2. Heterogeneous Attention Modulation: The novel combination of GAR (global self-attention regulation) and LAT (local cross-attention transplantation) successfully decouples style and content. GAR handles the global structure, while LAT handles precise style injection without compromising identity.
3. Architecture Agnostic: The method is validated on both DDIM-based (SD2.1) and DiT-based (SD3.5) architectures, demonstrating universal compatibility.
4. Noise Initialization Strategy: The introduction of SINI with a content residual component resolves the conflict between style intensity and content fidelity at the very start of the diffusion process.

4. Experimental Results

The authors evaluated HAM against state-of-the-art (SOTA) methods (e.g., StyleID, DiffArtist, CSGO, STAM) on MS-COCO and WikiArt datasets.

• Quantitative Performance:
  • Style Strength: HAM achieved the best CLIP-T score (0.223), indicating superior alignment with text-based style prompts, and competitive FID/ArtFID scores.
  • Content Preservation: HAM significantly outperformed all baselines in LPIPS (0.479) and DINO (0.728), proving it retains structural details and semantic coherence better than competitors.
  • Composite Metrics: It achieved the highest scores in DC (2.113) and CC (2.057), which balance style and content preservation.
• Qualitative Performance:
  • Visual comparisons show HAM preserves fine details (e.g., text, facial features, object boundaries) that other methods distort or lose.
  • It successfully transfers complex styles (e.g., Van Gogh, Cubism, Sketch) while maintaining the original image's identity, avoiding the "style leakage" or "content collapse" seen in other methods.
• Ablation Studies:
  • Removing any of the three modules (GAR, LAT, SINI) resulted in a drop in performance, confirming their synergistic necessity.
  • Hyperparameter tuning ($\alpha, \beta, \gamma$) showed that specific balances (e.g., $\alpha=0.75, \beta=0.25, \gamma=0.5$) yield the optimal trade-off.

5. Significance

• Solving the Trade-off: HAM provides a robust solution to the long-standing challenge of balancing style transfer intensity with content identity preservation in diffusion models.
• Efficiency: By being training-free, it lowers the barrier for applying style transfer to new styles, eliminating the need for per-style fine-tuning.
• Generalizability: The ability to work across different diffusion architectures (SD2.1 and SD3.5) suggests the method targets fundamental attention mechanisms rather than architecture-specific quirks, making it a versatile tool for future generative AI applications.
• Future Impact: This approach sets a new standard for zero-shot style transfer, enabling more reliable and high-fidelity image editing and generation tools for creative industries.