Technical Summary: Token-to-Token Alignment of Text Embeddings for Semantic Blending

Problem Statement

Modern text-to-image generative models rely on sequences of text tokens to define semantic conditioning. While expressive, the space of these token sequences lacks a consistent, accessible structure. Semantically similar images may correspond to token sequences with significant differences in wording, ordering, and concept placement, while similar token sequences may encode divergent semantics. This structural misalignment causes standard linear interpolation between prompt embeddings to yield incoherent, entangled intermediate states. Consequently, applications requiring smooth semantic transitions—such as continuous image editing, semantic blending, and concept synthesis—are hindered, as interpolation mixes unrelated attributes rather than following a meaningful semantic trajectory.

The authors argue that this limitation is not due to an absence of semantic structure within the embedding space, but rather stems from the misalignment between representations. Corresponding concepts are not consistently represented across different prompts, preventing the generative model from traversing a continuous semantic manifold.

Methodology: Token-to-Token Alignment

The proposed framework, Token-to-Token (T2T) alignment, establishes explicit semantic correspondence between tokens across different prompts. The method transforms unstructured prompts into a structured representation where semantically corresponding concepts map to consistent positions, enabling meaningful linear interpolation. The process consists of two primary stages followed by interpolation:

1. Concept-Level Structural Alignment

The first stage addresses linguistic variability by transforming input prompts ($p_A, p_B$) into a shared structured representation using a Large Language Model (LLM).

• Process: The LLM is instructed to jointly construct a pair of aligned scene descriptions ($J_A, J_B$) under explicit structural constraints.
• Schema: Descriptions are formatted as JSON strings with a fixed set of semantic fields (e.g., objects, attributes, lighting) and a consistent ordering.
• Goal: Corresponding elements are expressed using aligned textual phrasing within each field. This establishes coarse correspondence at the textual level, ensuring that semantic differences are captured as local substitutions within a shared context rather than independent rephrasing.

2. Embedding-Level Alignment

The second stage resolves discrepancies in the text encoder space, where structural alignment alone cannot guarantee token-wise correspondence due to tokenization differences (e.g., varying token counts for semantically similar words).

• Similarity Computation: A pairwise cosine similarity matrix $S$ is computed between token embeddings of the two aligned sequences.
• Constraints and Bias:
  • Field Masking: Matching is restricted to tokens within corresponding semantic fields to respect the scene decomposition.
  • Locality Bias: A Gaussian bias is applied over relative token positions within fields, assuming corresponding tokens appear in similar relative positions after structural alignment.
  • Temperature Scaling: A temperature parameter ($\tau$) is used to refine the similarity scores.
• Projection: A soft alignment matrix $A$ is derived via temperature-scaled softmax normalization. This matrix projects the semantic content of the source embeddings onto the target token structure, yielding aligned embedding sequences ($E_A, E_B$) of equal length with established token-wise correspondence.

3. Interpolation in Aligned Space

Once embeddings are aligned, simple linear interpolation is performed between corresponding tokens:
$$E_\alpha = \{(1 - \alpha)e^A_j + \alpha e^B_j\}_{j=1}^m, \quad \alpha \in [0, 1]$$
These interpolated embeddings serve as conditioning for the generative model (e.g., Flux2 or FIBO). For image-based tasks (editing and blending), the model is conditioned on both the input image(s) and the interpolated text embedding. In continuous blending, attention weights are modulated by $\alpha$ (using quadratic scaling) to balance the contribution of input images, preserving fidelity at endpoints while allowing semantic evolution in intermediate steps.

Key Contributions

1. Token-to-Token Alignment Framework: A two-stage method (structural and embedding-level) that enforces explicit semantic correspondence between tokens across arbitrary prompts, transforming the text embedding space into a linearly interpolable manifold.
2. Unified Control Mechanism: A demonstration that semantic control in text-to-image models can be achieved by organizing existing representations rather than modifying the generative model architecture or training new control parameters.
3. Application Versatility: The framework supports continuous synthesis (changing attributes), continuous editing (modifying images based on instructions), and continuous blending (morphing between two images) without task-specific training.

Results and Evaluation

The authors evaluated the method against state-of-the-art baselines across qualitative and quantitative metrics.

• Qualitative Performance:
  • Continuous Editing: Compared to methods like Kontinuous Kontext, SliderEdit, and GRAG, T2T produces smooth transitions with coherent intermediate states. Baselines often rely on appearance-based interpolation or introduce abrupt semantic changes.
  • Continuous Blending: Compared to DiffMorpher, FreeMorph, and VibeSpace, T2T avoids ghosting artifacts and structural discontinuities. While pixel/latent space methods struggle with geometric changes, T2T maintains structural consistency by interpolating in the aligned text space.
• Quantitative Performance:
  • Blending (Morph4Data & BlendBench): T2T achieved competitive smoothness (PPL) while significantly outperforming baselines in visual quality (MUSIQ) and distribution alignment (FID). It demonstrated better adherence to input images than VibeSpace and fewer artifacts than morphing-based methods.
  • Editing (PIE-Bench): T2T achieved strong semantic alignment (Norm CLIP-Dir) and improved continuity compared to baselines, balancing smoothness with faithful adherence to the intended edit direction.
• User Study: In pairwise comparisons involving 20 participants, T2T achieved higher win rates across visual quality, semantic smoothness, and overall preference for both editing and blending tasks.
• Ablation Studies: Experiments confirmed that both structural and embedding-level alignment are complementary. Direct interpolation fails; text-level alignment alone lacks fine-grained detail; embedding alignment alone is unstable without structural constraints. The full method yields the most coherent results.

Significance and Claims

The paper posits that the ability to perform smooth semantic transitions in generative models is fundamentally a problem of representation rather than manipulation. The text embedding space already implicitly encodes a continuous semantic structure, but this structure is entangled and inaccessible under standard prompt formulations due to misalignment.

By enforcing token-level correspondence, the authors claim to effectively "linearize" this space, turning interpolation from an unreliable heuristic into a principled operation. In this view, the generative model acts largely as a passive renderer of trajectories defined in the aligned semantic space. The significance lies in demonstrating that complex semantic control can be achieved by organizing existing representations, avoiding the need for model modification, fine-tuning, or the introduction of explicit control parameters.

The authors acknowledge limitations, noting that the approach relies on the quality of the initial structural alignment and that linear interpolation may not capture highly non-linear semantic transformations. They suggest future work could focus on improving alignment robustness and extending the formulation to temporally coherent settings like video generation.