Technical Summary: Untwisting RoPE: Frequency Control for Shared Attention in DiTs

Problem Statement

In diffusion transformers (DiTs), such as the Multi-modal Diffusion Transformer (MMDiT) architecture used in models like Flux, positional information is encoded via Rotary Positional Embeddings (RoPE). While RoPE is essential for modeling spatial structure in standard image generation, its behavior in shared-attention settings—where a target image generation attends to tokens from a reference image—remains poorly understood.

The authors identify a critical failure mode in this setting: reference copying. Instead of transferring the stylistic attributes of a reference image to a target prompt (e.g., generating a giraffe in the style of a bull), the model collapses into reproducing the reference content almost verbatim. This occurs because the strong spatial bias introduced by RoPE forces queries in the target image to attend disproportionately to spatially aligned tokens in the reference image, overriding semantic similarity.

Existing attempts to mitigate this, such as restricting attention sharing to later denoising steps or using shifted positional indices (as in the concurrent work AlignedGen), offer only coarse control or introduce new artifacts like "ghost-cyclic inpainting" (where the model treats the reference as a continuation of the target).

Methodology

The paper proposes a frequency-aware modulation of RoPE to restore meaningful shared attention without retraining the model or altering its architecture.

1. Analysis of RoPE Frequency Structure

The authors provide a principled analysis showing that RoPE decomposes into frequency components with distinct positional sensitivities:

• High-frequency components: Correspond to lower-index dimensions in the embedding. These exhibit rapid angle changes with respect to positional displacement ($\Delta$), enforcing strong locality. They cause attention to drop off steeply as spatial distance increases.
• Low-frequency components: Correspond to higher-index dimensions. These vary slowly with displacement, contributing weak positional bias and preserving global spatial coherence.

In shared-attention scenarios, the high-frequency components dominate the attention computation, causing the model to lock onto spatially aligned reference tokens and copy their content.

2. Frequency-Aware Modulation

To counteract this, the authors introduce a method to selectively modulate the frequency bands of the reference image keys ($K_{ref}$) during the attention computation.

• Mechanism: Instead of disabling RoPE (which causes degenerate generations) or uniformly scaling attention, the method applies a smooth, polynomial interpolation of scaling factors across the RoPE spectrum.
• Scaling Strategy:
  • Attenuation: High-frequency components are scaled down (factor $s_{hf} \in (0, 1)$) to reduce the strong positional bias that drives copying.
  • Amplification: Low-frequency components are scaled up (factor $s_{lf} > 1$) to promote global, semantically guided attention.
• Timestep Scheduling: The modulation scales ($s_{hf}$ and $s_{lf}$) are linearly increased over the denoising timesteps. Early timesteps use lower modulation to establish global structure and broad style guidance, while later timesteps increase the scaling to allow for the transfer of fine-grained stylistic details and textures.

The final attention key for shared attention is formulated as:
K = [K'_{img}^{tar} \oplus K'_{txt}^{tar}] \oplus s_{FA}^t \cdot K_{img}^{ref}
where $s_{FA}^t$ is the frequency-aware modulation scale at timestep $t$, applied exclusively to the reference keys.

Key Contributions

1. Principled Analysis of RoPE in Shared Attention: The paper demonstrates that the "reference copying" phenomenon in DiTs is not a general failure of attention sharing but a specific consequence of RoPE's high-frequency components enforcing strict spatial locality.
2. Frequency Control Mechanism: The authors introduce a zero-shot, training-free method to modulate RoPE frequency bands. This allows for a controllable trade-off between positional alignment and semantic similarity.
3. Resolution of Artifacts: The method resolves the "ghost-cyclic inpainting" and content leakage issues observed in previous approaches (like shifted RoPE) by directly addressing the underlying frequency structure rather than just shifting coordinates.
4. Restoration of Style-Aligned Generation: The approach enables DiTs to generate diverse semantic content (e.g., different animals) while maintaining a consistent visual style from a reference, a capability previously limited to UNet-based models or requiring heavy fine-tuning.

Results

Evaluated on the Flux 1.-dev model, the method demonstrates:

• Effective Style Transfer: The model successfully transfers stylistic attributes (color, texture, brushwork) from references (including artworks and photos) to target prompts without duplicating the reference's content.
• Controllability: By adjusting the modulation scales, users can control the degree of style transfer versus content preservation. Higher $s_{hf}$ preserves reference structure (pose/shape) while transferring style; lower $s_{hf}$ allows for more structural variation.
• Superiority over Baselines:
  • Compared to StyleAligned (UNet-based), the method avoids content leakage and structural irregularities in DiTs.
  • Compared to AlignedGen (shifted RoPE), the method avoids periodic copying artifacts and ghost-cyclic inpainting, providing smoother style transfer even at high resolutions.
  • Compared to IP-Adapter and B-LoRA, the method requires no additional training or image encoders, offering a more generalizable zero-shot solution.

Significance

The paper claims that positional encodings are not neutral components but actively govern the balance between locality and semantic association in generative models. The findings explain why attention-sharing strategies developed for UNet-based models (which rely on convolutional inductive biases) fail when directly applied to DiTs (which rely exclusively on RoPE).

By revealing that high-frequency RoPE components drive content copying, the work provides a direct, interpretable handle for shaping attention flow. This suggests a broader direction for controlling generative models through targeted interventions in internal representations, enabling precise style-aligned generation without architectural changes or retraining.