EmoCtrl: Controllable Emotional Image Content Generation

The paper introduces EmoCtrl, a novel framework for Controllable Emotional Image Content Generation that successfully balances faithful content adherence with expressive emotional control by leveraging a specialized dataset, multimodal enhancement modules, and emotion-driven preference optimization.

The Gist

Imagine you are an artist standing in front of a blank canvas. You have two very specific instructions from a client:

1. "Paint a dog." (This is the Content).
2. "Make the dog look happy." (This is the Emotion).

Most AI art generators today are like talented but emotionally tone-deaf painters. If you ask them to "paint a happy dog," they might paint a dog, but it could look bored, angry, or just plain weird. They are great at following the "what" (the dog) but terrible at capturing the "how it feels" (the happiness).

Other AI models are the opposite: they are great at painting "happiness" (maybe a bright sun, a smiling face), but if you ask for a specific dog, they might forget the dog entirely and just paint a generic happy scene.

Enter EmoCtrl.

Think of EmoCtrl as a new kind of "Emotional Director" for AI art. It's a system designed to solve a specific problem: How do we keep the story exactly as you told it, but change the mood of the movie?

Here is how it works, broken down into simple analogies:

1. The Problem: The "Lost in Translation" Gap

The paper explains that current AI models struggle to connect abstract words like "Fear" or "Joy" with concrete images like "a tree" or "a beach."

• Old AI: You say "Scary tree." It might just draw a normal tree and hope you feel scared.
• EmoCtrl: It understands that "Scary tree" needs twisted branches, dark shadows, and a stormy sky to actually feel scary, while still looking like a tree.

2. The Solution: The "Two-Brain" Approach

EmoCtrl uses a clever trick called Dual Enhancement. Imagine the AI has two brains working together:

• Brain A (The Writer): This part reads your prompt ("A dog on the floor") and the emotion ("Amusement"). It rewrites the prompt in its head, adding invisible emotional keywords. It's like a screenwriter who takes a simple line of dialogue and adds stage directions: "A playful dog, wagging its tail, with a goofy grin, running across a sunny floor."
• Brain B (The Painter): This part takes those rewritten instructions and paints the picture. But it has a special "Emotion Token" (a secret code) that tells the brushstrokes exactly how to feel. It knows that "Amusement" means bright colors and bouncy lines, while "Sadness" means muted colors and heavy, slow strokes.

By combining the Writer's smart descriptions with the Painter's emotional brushstrokes, the result is a picture that is both accurate to your request and full of feeling.

3. The Training: Learning from Human Feelings

To teach this system, the researchers didn't just give it pictures; they gave it a "feeling dictionary."

• They took thousands of images and labeled them not just with what they were (e.g., "Ocean"), but with how they made people feel (e.g., "Contentment").
• They then used a "Human Preference" system. Imagine a panel of judges tasting the AI's art. If the AI makes a "scary" ocean that looks like a sunny beach, the judges give it a low score. The AI learns from these scores to get better at matching the mood.

4. The Result: A Master of Mood Swings

The paper shows that EmoCtrl is a master of mood swings.

• Same Content, Different Vibes: You can give it the exact same prompt, "A city street," and ask for "Fear," "Joy," and "Sadness."
  • Fear: The city becomes dark, foggy, and ominous.
  • Joy: The city becomes bright, colorful, and bustling with happy people.
  • Sadness: The city becomes gray, rainy, and empty.
• The Magic: In all three cases, it still looks like a city street. The content didn't change; only the emotional atmosphere did.

Why Does This Matter?

This isn't just about making pretty pictures. It's about giving creators control.

• For Artists: You can generate a scene and then say, "Make it more dramatic," or "Make it more peaceful," without losing the original idea.
• For Storytellers: You can visualize a story where the setting stays the same, but the characters' emotions change the world around them.

In a nutshell:
EmoCtrl is like a mood-shifting lens for AI. It takes a clear, sharp image of "what you want" and overlays a perfect, vivid layer of "how you want it to feel," ensuring the final image is both faithful to your description and deeply moving to your heart.

Technical Summary

1. Problem Statement

The paper addresses the Controllable Emotional Image Content Generation (C-EICG) task. The core challenge is to generate images that simultaneously satisfy two often conflicting constraints:

1. Semantic Faithfulness: The image must accurately depict a specific content description (e.g., "a dog on the floor" or "ocean").
2. Emotional Expressiveness: The image must vividly convey a specific target emotion (selected from eight categories: amusement, awe, contentment, excitement, anger, disgust, fear, sadness).

Limitations of Existing Methods:

• Text-to-Image (T2I) Models (e.g., Stable Diffusion): Excellent at preserving content semantics but lack "emotional awareness," often producing neutral or flat images regardless of the prompt's emotional tone.
• Emotion-Driven Models: Focus on generating affective results but often distort the original content or fail to adhere to specific semantic descriptions.
• LLM-Enhanced Approaches: While Large Language Models (LLMs) understand abstract emotion concepts, they struggle to translate these abstract words into concrete, content-aligned visual cues (e.g., describing "fear" as a concept rather than visualizing it).
• Data Gap: Existing datasets (like EmoSet) provide emotion labels but lack detailed content annotations and affective prompts necessary for joint control.

2. Methodology: EmoCtrl Framework

The authors propose EmoCtrl, a framework designed to bridge the gap between abstract emotions and visual content. The method consists of three main components:

A. Data Construction (EmoSet+ and EmoEditSet+)

To support C-EICG, the authors constructed new datasets by augmenting existing emotion datasets (EmoSet and EmoEditSet) with:

• Content Annotations: Extracted neutral visual concepts (e.g., "dog," "building") using Multimodal Large Language Models (MLLMs) and LLMs, filtering out concepts with inherent emotional bias.
• Affective Prompts: Detailed captions that bridge the gap between the content and the target emotion (e.g., transforming "a dog" + "amusement" into "A playful dog runs across the grass carrying a colorful ball").
• Result: Two quadruplet datasets: {emotion, concept, caption, image} and {emotion, caption1, caption2, image}, totaling 158,222 samples.

B. Dual-Module Enhancement

EmoCtrl enhances affective expression through two complementary modules:

1. Textual Emotion Enhancement:

   • Mechanism: Introduces learnable emotion tokens ($v_t$) for each of the 8 emotion categories.
   • Process: An LLM (fine-tuned with LoRA) takes the input content and the target emotion token. It fuses them to generate an emotion-aware textual prompt that enriches the semantic description with affective details (e.g., adding "warm sunlight" for contentment).
   • Goal: To ground abstract emotions into concrete semantic concepts within the text prompt.

2. Visual Emotion Enhancement:

   • Mechanism: Introduces learnable visual emotion tokens ($v_v$) that map the same emotion categories into the diffusion model's conditioning space.
   • Process: These tokens are injected into the Stable Diffusion model via cross-attention. They encode perceptual cues (color, brightness, style) associated with the target emotion.
   • Fusion: A fusion module combines the LLM-generated textual features and the visual emotion tokens to create a unified condition for image generation.
   • Goal: To inject perceptual emotional cues directly into the image generation process, ensuring the visual style matches the emotion.

C. Emotion-driven Preference Optimization (EDPO)

To align the model with human preferences beyond simple supervised learning, the authors introduce a reinforcement learning stage using Group Relative Policy Optimization (GRPO).

• Reward Function: A composite reward is calculated for generated images based on:
  1. Emotion Reward ($R_{emo}$): Confidence of the target emotion category (using a pre-trained classifier).
  2. Polarity Reward ($R_{pol}$): Ensures the image matches the correct valence (positive vs. negative) to avoid sparse supervision.
  3. PickScore ($R_{pick}$): A general metric for human preference and text-image alignment.
• Optimization: The model is fine-tuned to maximize this composite reward, improving both emotion accuracy and intensity while maintaining content fidelity.

3. Key Contributions

1. Task Definition: Introduced C-EICG, a new task requiring simultaneous control of semantic content and target emotion.
2. Dataset Curation: Created EmoSet+ and EmoEditSet+, the first datasets providing triplet annotations linking emotion, content, and detailed affective prompts.
3. Framework Design: Proposed EmoCtrl, featuring:
   • Textual & Visual Emotion Enhancement: Using learnable tokens to bridge abstract emotions with semantic descriptions and perceptual cues.
   • Emotion-driven Preference Optimization: A novel reward mechanism specifically designed for emotional alignment.
4. Performance: Demonstrated superior results in balancing content fidelity and emotional expressiveness compared to state-of-the-art methods.

4. Experimental Results

The authors evaluated EmoCtrl on a test set of 608 images (76 content descriptions × 8 emotions) against baselines including Stable Diffusion, SDXL, PixArt-α, Textual Inversion, DreamBooth, LLM4GEN, OmniGen2, EmoGen, and EmotiCrafter.

Quantitative Metrics:

• Emotion Accuracy (Emo-A): EmoCtrl achieved 64.64%, significantly outperforming the next best (EmoGen at 45.23%).
• Content Alignment (CLIP-A): Achieved 83.06%, comparable to standard T2I models, proving it does not sacrifice content for emotion.
• Joint Accuracy (EC-A): Achieved 50.99%, more than double the second-best method (DreamBooth at 24.86%), indicating superior ability to satisfy both constraints simultaneously.
• Semantic Clarity (Sem-C): Highest score (0.673), confirming high-level recognizability.

User Study:

• In a study with 48 participants, EmoCtrl received 88.75% votes for emotion evocation and 86.77% for content fidelity, far surpassing all competitors.

Ablation Studies:

• Removing Visual Emotion Tokens ($v_v$) caused the largest performance drop, highlighting the necessity of perceptual cues.
• Removing Textual Emotion Tokens ($v_t$) reduced the model's ability to ground emotions in appropriate semantics.
• Removing EDPO weakened the alignment with human preference, showing the value of the reward-based optimization.

5. Significance and Applications

• Robustness: EmoCtrl generalizes well to creative applications, including Stylized C-EICG (controlling emotion alongside artistic styles like Matisse or Watercolor) and Multi-emotion C-EICG (mixing multiple emotions).
• Impact: The work advances the field of Affective Computing by moving beyond simple emotion classification to controllable generation. It provides a robust solution for applications in artistic creation, human-computer interaction, and therapeutic tools where specific emotional tones are required without losing the subject matter.
• Limitations: Currently text-only input; relies on the quality of the EmoSet dataset which may introduce biases; existing image-text metrics struggle to fully evaluate emotional nuance, necessitating human feedback.

In conclusion, EmoCtrl represents a significant step forward in generative AI, successfully decoupling content and emotion control to produce images that are both semantically accurate and emotionally resonant.