Technical Summary: Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

Problem Statement

Modern text-to-image generation has shifted from U-Net architectures to Diffusion Transformers (DiTs), where text and image tokens interact via joint attention. Concurrently, text conditioning has evolved from isolated encoders (e.g., CLIP, T5) to Large Language Models (LLMs) that serialize user prompts into chat templates containing system, user, assistant, and delimiter tokens.

While the semantic content of the user prompt is clear, the role of the structural template tokens (the formatting residue like delimiters) remains poorly understood. In joint-attention DiTs, all conditioning tokens compete as keys for the image stream. It is an open question whether these structural tokens remain inert formatting residue, act as transparent conditioning, or acquire a deeper computational role. Furthermore, there is a broader gap in understanding the mechanistic organization of DiTs: specifically, which layers commit semantic content, which heads causally affect generation, and how text-image attention mediates conditioning across the denoising trajectory.

Methodology

The authors introduce a causal interpretability framework designed to trace where conditioning information is read, routed, and stored during denoising. This framework combines four specific techniques:

1. Token-level Attention Decomposition: Analyzing the Image-to-Text (I2T) attention mass to quantify how much attention image queries direct toward specific text token spans (semantic vs. structural).
2. Span-level Conditioning Interventions: Performing cross-prompt swaps and averaging structural tokens to test if they carry prompt-specific semantics.
3. Cross-trajectory Head Transplantation: Progressively swapping attention projections ($q, k, v$) between two different generation trajectories (e.g., "apple" vs. "banana") to identify which heads causally determine object identity.
4. Layer-wise Causal Masking: Masking attention flows (e.g., Register $\to$ Semantic or Register $\to$ Image) at specific layers to determine the direction of information flow.

The study primarily utilizes the Qwen-Image family (a dual-stream MMDiT) and evaluates across multiple benchmarks (GenEval, DPG-Bench, Qwen-Image-Bench) and languages (English and Chinese).

Key Findings

1. Structural Tokens as Dominant Attention Sinks

The analysis reveals that structural template tokens (e.g., delimiters like `