💻 computer science

Orthogonal Negative Guidance in Attention Feature Space for Text-to-Image Generation

This paper proposes Orthogonal Negative Guidance, a training-free method for text-to-image generation that suppresses unwanted concepts by orthogonalizing negative-prompt attention features relative to positive ones, thereby achieving superior concept removal while preserving image quality and prompt alignment.

Original authors: Jungmin Ko, Jungwon Park, Jimyeong Kim, Changin Choi, Wonseok Lee, Wonjong Rhee

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jungmin Ko, Jungwon Park, Jimyeong Kim, Changin Choi, Wonseok Lee, Wonjong Rhee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef (the AI) who is incredibly talented at cooking dishes based on a recipe (the text prompt). You can make a "bowl of ramen" look delicious, but there's a problem: every time you make ramen, you automatically add a boiled egg, even if the customer didn't ask for it.

If you simply tell the chef, "No eggs!" (a negative prompt), the chef might get confused and accidentally put more eggs in, or they might ruin the whole dish trying to avoid the eggs. This is the current struggle with AI image generators: telling them what not to include is surprisingly hard.

This paper introduces a new, clever technique called Orthogonal Negative Guidance. Here is how it works, using simple analogies:

1. The Problem: The "Clumsy Eraser"

Current methods for removing unwanted things (like "eggs" from "ramen") are like using a giant, clumsy eraser on a drawing.

The "Prompt Negation" approach: You just say "No eggs." The AI often ignores you or gets confused.
The "Subtraction" approach: Some methods try to subtract the "egg" idea from the "ramen" idea. But imagine if the "egg" and the "ramen" were written in the same ink. If you try to erase the word "egg," you might accidentally erase parts of the word "ramen" too, leaving you with a blurry, broken picture.

2. The Solution: The "Smart Filter" (Orthogonal Negative Guidance)

The authors propose a method that acts like a smart filter or a specialized sieve. Instead of blindly erasing, it looks at the "ingredients" (mathematical features) the AI is using to build the image.

The Setup: The AI is building the image using two streams of information:
1. The Positive Stream: "Make a bowl of ramen."
2. The Negative Stream: "Do not make an egg."
The Magic Trick (Orthogonalization):
Imagine the "Ramen" stream is a vector (an arrow) pointing in a specific direction. The "No Egg" stream is another arrow.
- Sometimes, the "No Egg" arrow points in a direction that overlaps with the "Ramen" arrow. If you just subtract the "No Egg" arrow, you mess up the "Ramen."
- This new method calculates the "No Egg" arrow and rotates it so it is perfectly perpendicular (at a 90-degree angle) to the "Ramen" arrow.
- It then only removes the part of the "No Egg" arrow that sticks out sideways (the part that doesn't overlap with the ramen).

The Analogy: Think of the "Ramen" as a red light beam and the "Egg" as a blue light beam. They are shining on the same wall.

Old methods try to turn off the blue light, but in doing so, they dim the red light too.
This new method finds the part of the blue light that is not shining on the red light, and only blocks that specific part. The red light (the ramen) stays bright and clear, while the blue light (the egg) is completely blocked.

3. What This Achieves

Because this method is so precise, it can do things other methods can't:

Multi-Concept Suppression: You can tell the AI to remove both the "egg" and the "chopsticks" at the same time, and it will handle both without ruining the bowl of ramen.
Adjustable Strength: You can turn a "dial" to decide how much you want to suppress the egg. You can go from "maybe no egg" to "definitely no egg" without the image turning into garbage.
No Re-training: The best part is that this doesn't require re-teaching the AI. It's a "plug-and-play" trick applied while the image is being made.

4. The Results

The authors tested this on a benchmark called DCS-Bench (Diverse Concept Suppression Benchmark), which is like a test with 200 different scenarios (e.g., "a doctor" without a "stethoscope," or "a living room" without a "sofa").

The Score: In human tests, people preferred this new method over the next best option by nearly 19%.
The Quality: The images didn't get blurry or weird. The "ramen" still looked like ramen, just without the unwanted "egg."

Summary

In short, this paper solves the problem of "telling an AI what not to draw" by using a mathematical trick to separate the "bad" ideas from the "good" ideas before they get mixed together. It's like having a bouncer at a club who knows exactly how to stop the unwanted guest (the egg) from entering without accidentally kicking out the VIP (the ramen).

Technical Summary: Orthogonal Negative Guidance in Attention Feature Space for Text-to-Image Generation

Problem Statement

While Text-to-Image (T2I) models have advanced significantly in generating high-quality images, explicitly enforcing the absence of specific objects or attributes remains a fundamental challenge. Existing strategies suffer from notable limitations:

Prompt Negation: Simply inserting words like "no" or "without" often fails, sometimes causing the model to generate the negated concept even more frequently than without negation.
Generate-and-Edit Pipelines: These methods first generate an image and then use editing models to remove unwanted concepts. However, because editing models are trained to preserve spatial and structural integrity, they struggle to remove concepts that require significant structural reorganization of the scene.
Standard Negative Guidance: Techniques like Classifier-Free Guidance (CFG) and its variants (e.g., NAG, NASA) steer generation away from negative prompts via linear subtraction. However, they often subtract the entire negative signal, which can degrade the alignment with the positive prompt and overall image quality before the unwanted concept is sufficiently suppressed.

Methodology

The authors propose Orthogonal Negative Guidance, a training-free method designed for rectified flow T2I models based on Multimodal Diffusion Transformer (MM-DiT) architectures (e.g., FLUX-dev, FLUX-schnell). The method operates within the attention feature space to suppress unwanted concepts while preserving desired semantics.

Core Mechanism

The method intervenes in the image-to-text attention outputs ( $Z_{I2T}$ ) within MM-DiT blocks, which mediate cross-modal interactions. The process involves two parallel branches:

Positive Branch: Computes attention outputs based on the original generation prompt ( $P^+$ ).
Negative Branch: Computes attention outputs based on an auxiliary negative prompt ( $P^-$ ) specifying the concept to suppress. Crucially, this branch reuses the image-side features ( $Q_I, K_I, V_I$ ) from the positive branch to ensure spatial alignment, while using its own text-side features.

Orthogonal Subtraction

Instead of directly subtracting the negative attention output ( $Z_{I2T}^-$ ) from the positive output ( $Z_{I2T}^+$ ), the method isolates and removes only the component of the negative signal that is orthogonal to the positive signal. The modified attention output $\widehat{Z}_{I2T}^+$ is calculated as:

$\widehat{Z}_{I2T}^+ = Z_{I2T}^+ - \alpha \cdot \left[ Z_{I2T}^- - \mathrm{proj}_{Z_{I2T}^+}(Z_{I2T}^-) \right]$

Where:

$\alpha$ is the guidance scale.
$\mathrm{proj}_{Z_{I2T}^+}(\cdot)$ denotes the projection of the negative vector onto the positive vector along the hidden dimension.

By subtracting only the orthogonal component, the method suppresses information specific to the unwanted concept without interfering with the semantic directions aligned with the positive prompt. This procedure is applied to all MM-DiT blocks starting from a specific timestep $\tau$ .

Key Contributions

Orthogonal Negative Guidance: A novel, training-free intervention in the attention feature space that effectively suppresses unwanted concepts by removing only the orthogonal component of negative guidance signals, thereby preserving prompt alignment and image quality.
Diverse Concept Suppression Benchmark (DCS-Bench): The authors introduce a new benchmark comprising 200 suppression scenarios across six semantic categories (e.g., Place/Scene, Event/Action, Co-occurring Objects). Unlike previous benchmarks focused on perceptual quality (e.g., "blurry"), DCS-Bench evaluates the ability to suppress specific, naturally occurring concepts that models tend to hallucinate.
Comprehensive Evaluation: Extensive experiments on FLUX-dev, FLUX-schnell, and Stable Diffusion 3.5 Large demonstrate that the method outperforms existing negative guidance baselines (CFG, NASA, NAG, VSF) and generate-and-edit pipelines (Kontext, Qwen-Image-Edit, OmniGen2).

Experimental Results

Quantitative Performance

Using VLM-based metrics (Negative Concept Suppression, Prompt Alignment, Image Quality), the proposed method achieves superior trade-offs:

Suppression: On FLUX-dev, the method improves Negative Concept Suppression by at least 15% over all prior negative guidance methods.
Alignment & Quality: Unlike linear subtraction methods that degrade image quality or prompt alignment, Orthogonal Negative Guidance maintains high scores in both metrics, comparable to the original model.
Human Evaluation: In a study with 111 participants, the method achieved a preference score of 70.81%, outperforming the second-best baseline (NAG at 46.22%) by 18.78%.

Qualitative Observations

Multi-Concept Suppression: The method can suppress multiple concepts simultaneously by listing them in a single negative prompt.
Adjustable Suppression: By varying the guidance scale $\alpha$ , users can continuously control the strength of suppression, gradually removing concepts without destroying the scene.
Failure Cases: The method occasionally struggles with Place/Scene and Event/Action categories where the concept is deeply integrated into the scene structure (e.g., failing to fully remove "softbox lights" from a "professional photography studio").

Significance and Claims

The paper claims that Orthogonal Negative Guidance represents a significant step forward in controllable T2I generation by addressing the trade-off between concept suppression and semantic fidelity. The authors emphasize that their method:

Does not require retraining or additional parameters.
Seamlessly integrates into modern rectified-flow transformer architectures.
Provides a more explicit mechanism for concept removal compared to prompt negation or post-hoc editing.
Establishes a new standard for evaluation via DCS-Bench, highlighting that current methods are insufficient for diverse, real-world suppression scenarios.

The authors conclude that while the method is not a panacea for all structural editing tasks (where generate-and-edit pipelines may still be preferred for sequential editing), it substantially improves the ability to generate images that strictly adhere to user constraints regarding the absence of specific concepts.