🤖 AI

EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation

EmoStyle is a first-place winning framework for the AffectiveArt Challenge 2026 that bridges the control gap in emotional image generation by using an LLM to infer affective cues and style-specific LoRA experts to modulate a Z-Image-based generator, ensuring outputs align with prompts, artistic styles, and target emotions.

Original authors: Dexiang Hong, Yijie Guo, Weidong Chen, Xinyan Liu, Zixuan Zou, Zhendong Mao, Yongdong Zhang

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dexiang Hong, Yijie Guo, Weidong Chen, Xinyan Liu, Zixuan Zou, Zhendong Mao, Yongdong Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to paint a picture based on a single sentence, like "a sad robot in a rainy city." But here's the catch: you don't just need to draw a robot; you need to make it look like a specific art style (maybe "Impressionism" or "Ink Wash") and, most importantly, you have to make the viewer feel the sadness without the robot even crying.

This is the challenge the EmoStyle team tackled. They built a smart system that acts like a super-organized art director, a style-shifting chameleon, and a strict art critic all rolled into one. Their goal? To win the AffectiveArt Challenge 2026 (Track 1), and guess what? They took first place.

Here's how their magic trick works, broken down into three simple steps:

1. The "Mind Reader" (The LLM Reasoner)

Usually, if you just tell a computer "draw a sad robot," it might draw a robot that looks happy or confused because it doesn't know how to show sadness. It needs more clues.

In the training data, the computer had a cheat sheet with secret notes like "Valence: Low," "Arousal: High," and "Dominant Emotion: Grief." But at the test time (when they actually had to create art), those cheat sheets were gone! The computer only had the short sentence.

EmoStyle's solution: They hired an AI "Mind Reader" (an LLM reasoner). Before drawing a single pixel, this Mind Reader looks at the short sentence and the target art style, then guesses the missing secret notes. It figures out the exact emotional coordinates (how positive/negative and how calm/excited the image should be) and even decides if the picture should be wide or tall. It turns a vague idea into a detailed, structured plan.

2. The "Style Chameleon" and the "Emotion Injector"

Now the computer has a plan, but it needs to paint. Most AI painters try to learn everything at once, which can get messy. EmoStyle did something smarter:

The Style Chameleon (LoRA Experts): Imagine you have a different pair of glasses for every art style. If you want "Renaissance," you put on the Renaissance glasses; if you want "Ukiyo-e," you switch to those. The team trained a tiny, specialized "expert" (called a LoRA adapter) for each of the 8 art styles they used (like Ink Wash, Baroque, and Soviet Realism). When the computer needs to paint, it simply picks the right pair of glasses. This keeps the style super consistent.
The Emotion Injector (Affective Conditioning): But wait, how do you make the Renaissance robot look sad? You don't just add the word "sad" to the prompt again. Instead, they took the "Mind Reader's" emotional plan and turned it into a secret code (a vector). They injected this code directly into the painting machine's brain (the denoising blocks) using a technique called AdaLN-style modulation. Think of it like whispering a secret instruction directly into the artist's ear while they are painting, telling them exactly how to twist the brushstrokes to convey that specific feeling.

3. The "Strict Art Critic" (VLM-Guided Refinement)

Even with a great plan and the right glasses, the first try might still be a bit off. Maybe the robot looks too happy, or the colors are wrong.

So, the system doesn't just stop at one picture. It generates multiple candidates (like sketching five different versions). Then, it brings in a "Strict Art Critic" (a Vision-Language Model). This critic looks at every sketch and scores them on four things:

Does it match the prompt?
Does it look like the right art style?
Does it actually feel the right emotion?
Is it visually high quality?

The critic picks the winner. If none are good enough, it can even trigger a regeneration. This happens without needing to retrain the whole system; it's just a smart selection step at the end.

The Results: Did it Work?

The team tested their system on a dataset called EmoArt, which has over 132,664 paintings. They compared their full system against the base model and other famous AI painters.

The Score: Their system, USTC_PI_LAB_TEAM, scored an Overall Score of 0.80, beating the second-place team (EmoForge) who scored 0.78.
The Proof: They measured how close their images were to real art using a metric called FID (Frechet Inception Distance). Lower is better. Their full system dropped the FID from 75.83 (the base model) down to 58.63, meaning the images looked much more realistic and true to the style.
The "What If" Tests: They also ran experiments to see what happens if they remove parts of their system.
- If they removed the "Mind Reader" (the affective planning), the images got a bit worse at capturing specific details.
- If they removed the "Style Chameleon" (the specific LoRA experts), the style consistency dropped significantly.
- If they removed the "Strict Art Critic" (the refinement step), the quality of the final image went down.

What They Didn't Do

It's important to know what this system isn't. The authors explicitly ruled out just rewriting the prompt with more words or using the emotional notes as extra text in the description. They found that those methods were unstable and didn't give the computer enough precise control. Instead, they insisted on converting emotions into that secret "code" injected directly into the machine's layers.

They also didn't try to build a single giant model that learns everything. Instead, they kept the main brain frozen and just swapped out the tiny "expert" modules for each style.

The Bottom Line

The EmoStyle team showed that by breaking the problem down—first planning the emotion, then picking the right style expert, and finally letting a critic pick the best result—you can create art that doesn't just look right, but feels right. They proved that separating style from emotion and using a smart "Mind Reader" to fill in the blanks is a winning strategy, securing them the top spot in the 2026 challenge.

Technical Summary: EmoStyle

Problem Statement

The paper addresses the challenge of emotion-aware artistic image generation, a task requiring a model to simultaneously satisfy three constraints: semantic fidelity to an input text prompt, adherence to a specified artistic style, and the expression of a target emotional state.

The core difficulty lies in the control gap between the available test-time inputs and the fine-grained conditions required for generation. While training datasets (such as EmoArt) contain rich structured annotations—including visual attributes, valence-arousal scores, dominant emotions, and therapeutic-effect labels—these structured cues are absent at test time. The model receives only a compact input prompt and a target style bucket. Consequently, the generator must implicitly infer how to translate the target emotion into specific visual parameters (color, lighting, brushwork, composition, line, and layout) without explicit guidance, a task that standard prompt-conditioned diffusion models struggle to perform with high fidelity.

Methodology

The authors propose EmoStyle, a framework built upon the Z-Image diffusion model. The system operates through a four-stage pipeline designed to bridge the gap between compact prompts and structured generation states:

1. Prompt-to-Affect Planning (LLM Reasoner)

Instead of rewriting the prompt or generating free-form visual attributes, the system employs an LLM reasoner to infer missing affective and layout variables. Given the input caption ( $x_i$ ) and target style bucket ( $y_i$ ), the reasoner predicts a structured affective plan ( $P_i$ ) consisting of:

Affective Summary ( $F_i$ ): Continuous valence ( $v$ ) and arousal ( $\eta$ ) scores, a dominant emotion label ( $e$ ), and therapeutic-effect labels ( $H$ ).
Aspect Ratio Decision ( $\rho$ ): A predicted canvas orientation to guide resolution.

These predictions are derived from predefined vocabularies and normalized continuous ranges, ensuring the control signals remain compact and stable.

2. Bucket-Specific Style LoRA Experts

To handle the diversity of artistic styles, the framework adopts a Mixture-of-Experts (MoE) approach tailored to the available style buckets (e.g., inkwash, ukiyoe, gongbi, renaissance, abstract, etc.).

For each style bucket, a dedicated LoRA (Low-Rank Adaptation) adapter is trained on the corresponding subset of the EmoArt dataset while keeping the Z-Image backbone frozen.
During inference, the system deterministically selects the LoRA expert corresponding to the target style bucket. This provides bucket-specific priors for color, texture, brushwork, and composition, ensuring style consistency without complex routing mechanisms.

3. Affective Condition Modulation

The inferred affective plan is encoded into a compact affective condition vector ( $c_{i,aff}$ ) and injected into the denoising network.

Encoding: Continuous valence-arousal coordinates are embedded using sinusoidal positional encoding, while discrete emotion and therapeutic labels are mapped to learnable embeddings. These are combined with an interaction term to capture the relationship between continuous intensity and discrete emotion types.
Modulation: The affective vector is injected via AdaLN-style residual modulation. Rather than adding a new layer, the affective head predicts residual offsets ( $\Delta s, \Delta g$ ) to the native scale and gate parameters of the Z-Image blocks. This allows the inferred emotional cues to directly guide intermediate features (Attention and MLP layers) while the style LoRA handles the artistic appearance.

4. VLM-Guided Candidate Refinement

Recognizing that a single stochastic generation may fail to satisfy all constraints simultaneously, the framework employs a test-time refinement strategy.

Multiple candidate images are generated using different random seeds.
A Vision-Language Model (VLM) judge (Qwen3.5-397B-A17B) evaluates each candidate based on four criteria: prompt alignment, style consistency, affective-plan consistency, and no-reference visual quality.
The candidate with the highest aggregate score is selected as the final output, requiring no additional training.

Key Contributions

The paper outlines three primary contributions:

LLM-Based Affective Planner: A mechanism that infers missing structured affective (valence-arousal, emotion labels) and layout (aspect ratio) cues from compact prompts, converting them into actionable generation variables.
Decoupled Style-Affect Framework: A novel architecture on top of Z-Image that separates style selection (via bucket-specific LoRA experts) from affective modulation (via AdaLN-style residual injection). This allows the same affective cues to be rendered with distinct style-specific priors.
VLM-Guided Refinement: A training-free test-time strategy that ranks generated candidates based on multi-dimensional criteria (content, style, affect, quality) to mitigate generation failures.

Experimental Results

The framework was evaluated in Track 1 of the AffectiveArt Challenge 2026 (Emotion-Aware Artistic Image Generation) using the EmoArt dataset.

Challenge Performance: The USTC_PI_LAB_TEAM submission achieved 1st place with an overall score of 0.80, outperforming the second-place team (EmoForge, 0.78).
Quantitative Metrics: On the local validation set, the full EmoStyle framework reduced the FID (Frechet Inception Distance) from 75.84 (Base Z-Image) to 58.64, indicating significantly improved visual realism. The AAS (Attribute Alignment Score) increased from 0.73 to 0.79, demonstrating better alignment with target attributes.
Ablation Studies:
- Removing aspect-ratio prediction caused the most significant FID degradation (to 72.59), highlighting the importance of layout planning.
- Removing style-specific LoRA resulted in a notable drop in style alignment (from 0.75 to 0.71), confirming the necessity of specialized adapters.
- Removing candidate refinement degraded both FID and AAS, validating the utility of the VLM-guided selection.
Cross-Backbone Comparison: When applied to other backbones (Flux2, GPT Image 2, Nano Banana 2) using only the model-agnostic components (planning and refinement), EmoStyle still improved performance, though the full architecture on Z-Image yielded the best overall results.

Significance and Claims

The paper claims that EmoStyle effectively bridges the gap between unstructured prompts and the fine-grained control required for emotional artistic generation. By explicitly modeling the inference of affective states and decoupling style specialization from emotional modulation, the framework enables the generator to "decide not only what to depict, but also how the target emotion should be expressed."

The authors position the work as a solution to the "control gap" in test-time generation, demonstrating that structured reasoning (LLM planning) combined with parameter-efficient adaptation (LoRA) and test-time scaling (VLM refinement) can achieve state-of-the-art results in emotion-aware creative design without retraining the base diffusion backbone. The success in the AffectiveArt Challenge 2026 serves as empirical validation of this structured affective conditioning approach.