Cross-Modal Alignment for Emotion-Consistent Text-to-Image Generation (ChinaMM 2026)

In the digital age, we have grown accustomed to asking computers to paint pictures from words. Type "a golden retriever playing in a park," and a machine can produce a convincing image in seconds. Yet, for all their speed and visual fidelity, these systems often miss the most human element of the request: the feeling. A prompt describing a "lonely birthday party" might generate a room full of balloons and bright colors, technically correct but emotionally hollow, because the machine sees the objects but not the sorrow behind them. This gap between what we say and what the machine feels is the central challenge researchers are now trying to close. The goal is not merely to make images that look right, but to create visuals that resonate with the specific emotional intent of the text, whether that is joy, fear, or sadness.

A team of researchers at the Communication University of China has proposed a new way to bridge this divide. They call their approach the Text-to-Image Emotion Consistent Generation framework, a two-step process designed to teach machines how to translate abstract feelings into concrete visual details. Instead of relying on pre-programmed labels or external guides, the system attempts to understand the emotional core of a sentence first, then uses that understanding to guide the creation of the image. The researchers argue that current methods often fail because they treat emotion as an afterthought or a separate tag, rather than weaving it into the very fabric of the image generation process.

The process begins with a stage the researchers call "Emotion-Intelligence Resonance Generation." Imagine a human artist reading a prompt like "a happy dog on the steps." Before picking up a brush, the artist might think about what happiness looks like: warm sunlight, bright colors, perhaps a playful posture. The computer does something similar. First, it analyzes the text to identify the dominant emotion, such as joy or anger. Once the emotion is identified, the system consults a specialized library of visual cues—associations like "warm sunlight" for joy or "dark shadows" for fear. It then rewrites the original prompt, enriching it with these specific visual details. So, the simple request for a happy dog becomes a more detailed instruction for a dog standing in warm sunlight with colorful balloons nearby. This enriched description is then fed into the image generator, which produces a picture that already carries a stronger emotional tone than the original text alone could provide.

However, the researchers found that even with a better description, the final image sometimes lost its emotional punch during the drawing process. To fix this, they added a second stage called "Emotion-Guided Cross-Modal Enhancement." This step acts like a final polish, looking at the image the machine just created and asking, "Does this truly feel like the emotion we intended?" If the target was fear, the system might subtly adjust the lighting or add a specific texture to the shadows to deepen the sense of unease, without changing the main subject of the picture. This refinement happens by injecting the emotional meaning directly into the visual features of the image, ensuring the final result aligns with the feeling the user described.

To test if this two-stage method actually worked, the team created a new dataset of 3,000 text prompts, carefully balanced to cover six basic human emotions: joy, surprise, anger, disgust, fear, and sadness. They compared their system against several existing models, including powerful standard generators and other tools designed specifically for emotional editing. The results showed a clear improvement. When asked to judge whether the generated images matched the intended emotion, their method achieved an accuracy of 88.43% for broad positive or negative feelings, and 69.23% for distinguishing between the six specific emotions. These numbers were higher than the best-performing alternatives, which lagged behind by several percentage points.

The researchers also conducted a study with human volunteers to see how people reacted to the images. Participants were shown the original text prompts alongside four different images generated by various systems and asked to pick the one that best captured the emotion. The new framework was chosen most often, with 27.04% of the votes, edging out specialized emotional editing tools. Importantly, the system managed to do this without distorting the original meaning of the prompt. While some older methods would make an image very emotional but change the scene entirely—turning a sad park into a completely different location—the new approach kept the scene recognizable while shifting its mood.

There are still limits to what the system can do. The researchers noted that the emotion of "surprise" proved particularly difficult for all methods, including their own, to capture accurately. This is likely because surprise shares visual traits with both fear and joy, making it hard to distinguish. Furthermore, the system currently relies on established, slightly older image-generation backbones to ensure stability, meaning it may not yet produce the highest possible image quality compared to the very latest models. However, the core finding stands: by breaking the task into understanding the emotion first and then refining the visual details, computers can generate images that feel more authentic to the human experience. This work suggests that for machines to truly understand our creative requests, they must learn not just what we are describing, but how we feel about it.