💻 computer science

Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment

This paper addresses the failure of text-to-image diffusion models to faithfully render "one-and-only" objects by proposing an intermediate text representation-guided method that injects early-layer encoder states to recover suppressed concept information, achieving significant alignment improvements without additional training.

Original authors: Soyoun Won, Aryan Yazdan Parast, Basim Azam, Jean Honorio, Naveed Akhtar

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Soyoun Won, Aryan Yazdan Parast, Basim Azam, Jean Honorio, Naveed Akhtar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Stubborn Artist"

Imagine you have a very talented artist who has spent their entire life painting the same few subjects. They know the Earth is blue and green, the Mona Lisa is a painting on canvas, and Saturn has rings. These facts are so deeply ingrained in their memory that they act like a "default setting."

Now, you ask this artist to paint a purple Earth or a knitted Mona Lisa. Even though you gave clear instructions, the artist ignores you. They say, "No, I know what the Earth looks like," and they paint a blue one anyway. They are so focused on their "learned habits" that they can't hear your new idea.

In the world of AI, this is called Concept Association Bias. The AI models (like Stable Diffusion) are so good at generating realistic images that they refuse to break their own rules, even when you explicitly tell them to change something. This is especially hard with "One-and-Only" objects—things that only exist in one famous form, like the Eiffel Tower or the Sun.

The Discovery: The "Lost Draft"

The researchers discovered why this happens. When the AI reads your text prompt, it processes it through a series of layers, kind of like a factory assembly line.

Early Layers (The Draft): In the beginning, the AI reads the words and understands specific details. It knows "purple," "knitted," and "octagon."
Final Layer (The Summary): By the time the text reaches the very end of the processing line, the AI summarizes everything into a single, high-level "vibe." In this summary, the specific details (like "purple") get smoothed over and lost. The AI only remembers the big picture: "It's an Earth."

The researchers found that the middle layers of the text processor still hold onto those specific details that the final summary threw away. It's like the AI wrote a detailed draft but then threw it in the trash before starting the final painting, only keeping a vague note that said "Draw Earth."

The Solution: "IR-Guided Diffusion"

The team came up with a clever trick to fix this without needing to retrain the AI or teach it new things. They call it Intermediate Text Representation (IR) Guidance.

Think of the AI's painting process like a sculptor working with clay:

The Early Steps (Structure): The sculptor first blocks out the rough shape of the statue.
The Later Steps (Details): The sculptor then adds the fine details, like the texture of the skin or the folds in the clothes.

The researchers realized that if they want the AI to paint a "purple Earth," they need to remind the AI of the word "purple" while it is still blocking out the rough shape.

How they do it:

They take the "Lost Draft" (the intermediate text representation) that still remembers the word "purple."
They mix it back into the instructions the AI is following during the early stages of image creation.
Once the rough shape is set, they stop mixing in the draft and let the AI finish the details using its normal instructions.

It's like whispering a reminder to the sculptor: "Hey, don't forget, this is supposed to be purple!" right when they are shaping the clay. Once the shape is set, the reminder stops so the sculptor can focus on making it look realistic.

The Result: Breaking the Rules

The researchers tested this on a new challenge set they created called OAO-AttackBench. This was a list of impossible requests, like "A square Earth" or "A glass Saturn."

Before: The AI would ignore the request and draw a normal, round, rocky Earth.
After: The AI successfully drew a square-shaped Earth and a glass Saturn, while still keeping them recognizable as Earth and Saturn.

They found that this method:

Recovers the lost details: It forces the AI to listen to specific attributes it usually ignores.
Doesn't break the image: The pictures still look high-quality and realistic; they just follow the weird instructions better.
Needs no extra training: It works with the existing AI models immediately, like plugging in a new accessory.

Summary

The paper shows that AI models sometimes "forget" specific instructions because they get too focused on what they already know. By peeking at the AI's "middle thoughts" (intermediate text layers) and feeding those thoughts back into the early stages of image creation, the researchers can trick the AI into following unusual instructions—like painting a knitted Mona Lisa or a purple planet—without needing to teach the AI anything new.

Technical Summary: Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment

1. Problem Statement

Text-to-image (T2I) diffusion models frequently fail to faithfully render explicit textual descriptions, particularly when those descriptions contradict the model's strongly learned visual priors. This phenomenon is termed concept association bias. The bias is most severe for One-and-Only (OAO) entities—objects that exist in a single canonical form (e.g., the Earth, the Mona Lisa, the Eiffel Tower). For these entities, the model's ingrained visual identity often overrides explicit textual instructions (e.g., failing to generate a "grey Earth" or a "knitted Mona Lisa").

The authors hypothesize that this failure stems from the information-theoretic properties of text encoders. While simple concepts emerge in early layers of the encoder, the final text embedding (used for conditioning) discards specific concept-level information through excessive abstraction. Consequently, the mutual information between the input text and the final embedding is lower than that of intermediate hidden states, leading to the suppression of specific attributes during the denoising process.

2. Methodology: IR-Guided Diffusion

The paper proposes Intermediate Text Representation (IR)-guided diffusion, a training-free, optimization-free method that recovers suppressed concepts by injecting intermediate hidden states of the text encoder into the conditioning signal.

2.1 Theoretical Foundation

Using an information-theoretic perspective, the authors formalize the concept omission problem. They demonstrate that for a text prompt $Y$ describing an object $A$ with an attribute $b$ , the generated latent $x_t$ often contains negligible information about $b$ once $A$ is accounted for ( $I(b; x_t | A) \approx 0$ ).
By analyzing the text encoder as a Markov chain $Y \rightarrow c_h \rightarrow c_f$ (where $c_h$ is an intermediate representation and $c_f$ is the final embedding), they apply the Data Processing Inequality (DPI). This proves that the mutual information between the input text and the intermediate representation is greater than or equal to that of the final embedding:
$I(Y; c_h) \ge I(Y; c_f)$
The authors posit that conditioning the denoising process on $c_h$ (or a combination of $c_h$ and $c_f$ ) preserves the attribute-specific information lost in $c_f$ .

2.2 Injection Mechanism

The method constructs an IR-guided embedding $\tilde{c}$ by adding the intermediate representation to the final embedding:
$\tilde{c} = c_f + \lambda c_h$
where $\lambda \in [0, 1]$ controls the injection strength. To ensure compatibility, the intermediate representation $c_h$ undergoes the text encoder's final layer normalization to align its distribution with $c_f$ . This modified embedding $\tilde{c}$ is used to compute the score function $\nabla_{x_t} \log p_\theta(x_t | \tilde{c})$ .

The approach supports both single-encoder architectures (e.g., Stable Diffusion 2.1) and multi-encoder architectures (e.g., Stable Diffusion 3.0) by extracting intermediate states from all encoders, concatenating them, and adding the result to the concatenated final embeddings.

2.3 Adaptive Scheduling

Injecting intermediate representations throughout the entire denoising process can introduce artifacts. The authors observe that diffusion models establish global structure in early steps and refine details in later steps. Therefore, they employ an Adaptive Scheduler to determine the optimal transition point $t^*$ :

Early Steps ( $t > t^*$ ): The model is conditioned on $\tilde{c}$ (intermediate-guided) to establish global structure and incorporate suppressed attributes.
Later Steps ( $t \le t^*$ ): The model reverts to the standard final embedding $c_f$ for detail refinement.

The transition point $t^*$ is determined dynamically by monitoring the structural acceleration $\Delta D_t$ of the estimated clean image $\hat{x}_{0|t}$ . The scheduler stops the injection when the structural changes stabilize ( $\Delta D_t \le 0$ ).

3. Key Contributions

Information-Theoretic Analysis: The paper identifies that final text embeddings discard concept-level information present in intermediate layers, reducing the mutual information available to the denoising process.
IR-Guided Diffusion: A novel, training-free method that injects intermediate text representations into the conditioning signal during early denoising steps. It recovers suppressed concepts without requiring external models, predefined concept sets, or additional optimization.
OAO-AttackBench: A new benchmark comprising 504 counterfactual prompts designed to attack the core visual identity of OAO entities across three categories: celestial objects, landmarks, and artworks. This benchmark systematically evaluates concept association bias.

4. Experimental Results

The method was evaluated on four benchmarks: OAO-AttackBench, Whoops (commonsense-violating prompts), Gecko(R), and Gecko(S), using Stable Diffusion 2.1 and 3.0.

Text-to-Image Alignment: IR-guided diffusion achieved up to a 19.1 percentage-point improvement in VQAScore on OAO-AttackBench compared to the baseline. It consistently outperformed the baseline and alternative post-processing strategies (e.g., Mean-Std normalization, orthogonal projection) in capturing fine-grained attributes.
Fidelity and Human Preference: The method preserved generation fidelity, with Kernel Inception Distance (KID) scores comparable to or better than the baseline. Human preference scores (HPSv2 and HPSv2.1) also improved, indicating that the generated images were not only more faithful to the prompt but also visually plausible.
Generalization: On general datasets like Gecko, the method maintained comparable performance to the baseline, demonstrating that it does not degrade quality for standard prompts.
Qualitative Improvements: The method successfully overrode strong priors (e.g., generating a "glass Saturn" or "octagon-shaped Earth") while preserving the object's identity, whereas baselines often ignored the attribute or failed to associate it with the correct object.

5. Significance and Claims

The paper claims that IR-guided diffusion addresses a fundamental limitation in current T2I models: the inability to override deeply ingrained visual priors for OAO entities without retraining. By leveraging the "untapped internal resource" of the text encoder's intermediate states, the method offers a practical, inference-time solution that requires no additional training or external models.

The authors emphasize that their approach is distinct from prior work that aims to remove features or introduce diversity; instead, it solves the complementary problem of generating explicitly specified concepts that the model typically suppresses. The introduction of OAO-AttackBench provides a rigorous protocol for evaluating and addressing concept association bias in future research. The method is presented as a robust, adaptive strategy that improves alignment while maintaining the high-quality generation standards of state-of-the-art diffusion models.