EmoOmni: Bridging Emotional Understanding and Expression in Omni-Modal LLMs

This paper introduces EmoOmni, a unified framework that leverages an emotional Chain-of-Thought (E-CoT) to bridge the gap between fine-grained multimodal perception and accurate emotional expression in Omni-LLMs, accompanied by a new dataset and benchmark for systematic evaluation.

The Gist

The Big Problem: The "Robotic" Robot

Imagine you are talking to a very smart robot. It can see your face, hear your voice, and read your text. But here's the catch: it's emotionally tone-deaf.

If you say, "I'm fine," while crying and shaking, a standard robot might just say, "Okay, that's good!" because it only heard the words. It misses the tears, the shaky voice, and the fact that you are actually heartbroken.

Current "Omni-Models" (robots that see, hear, and speak) are great at processing data, but they often fail at emotional intelligence. They get confused when your words and your face don't match (like smiling while saying something sad), and even when they understand the emotion, their voice sounds flat and robotic, like a GPS reading a love letter.

The Solution: EmoOmni (The "Empathic Actor")

The researchers built EmoOmni, a new system designed to act less like a calculator and more like a human actor. They realized that to be emotionally intelligent, a robot needs to stop guessing and start thinking.

They broke the robot's brain down into three distinct roles, mimicking how humans process a conversation:

1. The Detective (Perception)

Instead of just looking at the screen, the "Detective" looks for clues.

• The Analogy: Imagine a detective at a crime scene. They don't just look at the body; they look at the muddy footprints, the broken window, and the torn letter.
• In the Paper: The system analyzes your voice (is it shaky?), your face (are your eyes sad?), and your words. It looks for conflicts. If you are smiling but your voice is trembling, the Detective notes: "Wait, the smile is fake. The real emotion is fear."

2. The Therapist (Reasoning - The "E-CoT")

This is the paper's biggest innovation. They call it the Emotional Chain-of-Thought (E-CoT).

• The Analogy: Imagine a playwright writing a script for a movie. Before the actor speaks a line, the playwright writes a note in the margin: "Character is actually furious but trying to be polite. Speak slowly, keep voice low, but clench fists."
• In the Paper: Instead of jumping straight to an answer, the model pauses and writes a "thinking script." It explicitly reasons:
  1. What is the user feeling? (Sadness masked by humor).
  2. What is their goal? (They want comfort, not a lecture).
  3. How should I respond? (I need to be warm, gentle, and slightly humorous to match their defense mechanism).
     This "thinking script" is then passed to the next part of the robot.

3. The Actor (Expression)

Finally, the "Actor" takes the script and performs it.

• The Analogy: A voice actor reading a line. If the script says "Say this with a trembling, hopeful voice," the actor doesn't just read the words; they act the trembling and hope.
• In the Paper: The system takes the "thinking script" and turns it into a specific set of instructions for the voice generator (like "speak softly," "add a slight pause," "sound warm"). This ensures the final voice matches the emotion perfectly.

How They Trained It (The "Movie School")

To teach this robot, they couldn't just use boring textbooks. They needed real human drama.

• The Pipeline (EmoOmniPipe): They took thousands of hours of movies and TV shows. Why? Because actors are professional at showing complex emotions.
• The Process: They used AI to watch these movies, cut out the emotional scenes, and label them: "Here, the character is angry but trying to hide it." They turned these scenes into a massive training dataset so the robot could learn from real human interactions.

The Results: Small Brain, Big Heart

The most impressive part of the paper is the performance.

• The Analogy: Imagine a 7-year-old child (the 7B parameter model) who has been taught how to think about emotions, competing against a 30-year-old genius (the 30B parameter model) who just memorized facts.
• The Outcome: The "child" (EmoOmni) performed just as well as the "genius" (Qwen3-Omni-30B) in understanding and expressing emotions. This proves that teaching a model how to reason about feelings is more important than just making the model bigger.

Summary

EmoOmni is a new kind of AI that doesn't just hear your words; it feels your mood.

1. It Detects hidden clues in your voice and face.
2. It Thinks through a step-by-step reasoning process (like a scriptwriter) to decide the best emotional response.
3. It Acts out that response with a voice that sounds genuinely human.

It bridges the gap between "I know what you said" and "I understand how you feel."

Technical Summary

1. Problem Statement

Current Omni-Modal Large Language Models (Omni-LLMs) struggle to achieve emotionally intelligent interaction in complex real-world scenarios. The primary challenges identified are:

• Superficial Understanding: Existing models often fail to interpret conflicting or implicit multimodal cues (e.g., a cheerful tone paired with a frowning face), leading to incorrect intent inference.
• The "Thinker-Talker" Disconnect: Most Omni-LLMs use an architecture where a "Thinker" (comprehension) passes hidden states to a "Talker" (speech generation). This implicit connection causes emotional details to be lost or diluted during transmission. The resulting speech may be semantically correct but emotionally misaligned (e.g., delivering reassurance without warmth).
• Data and Evaluation Scarcity: There is a lack of real-world, fine-grained annotated dialogue data and comprehensive benchmarks that evaluate not just task correctness, but the emotional intelligence of the interaction.

2. Methodology

The authors propose EmoOmni, a unified framework that mimics human affective cognition through a Perception–Reasoning–Expression causal chain.

A. Core Architecture: The Causal Chain

EmoOmni explicitly disentangles the generation process into three stages:

1. Perception: Fine-grained extraction of acoustic and visual signals.
2. Reasoning: Deliberate inference of user intent and emotional state.
3. Expression: Generation of speech that aligns with the reasoned strategy.

B. Key Innovation: Emotional Chain-of-Thought (E-CoT)

Instead of treating dialogue as a black-box mapping, EmoOmni introduces E-CoT, a structured reasoning trajectory comprising four components:

1. Multimodal Emotion Analysis: Describing observable cues (e.g., vocal tension, facial micro-expressions).
2. User Intent Recognition: Inferring underlying motivations (e.g., sarcasm, masking).
3. Response Strategy Planning: Deciding how to react emotionally and pragmatically (e.g., "mirror excitement," "offer comfort").
4. Response Content Generation: Generating the actual text.

Crucially, the Response Strategy is treated as an explicit high-level instruction for the Talker module, ensuring the final speech output preserves the emotional intent derived during reasoning.

C. EmoOmni-Talker (Instruction-Guided Speech)

The Talker module is an instruction-guided Text-to-Speech (TTS) system. It receives the textual response ($z_t$) and the acoustic instructions ($I_{emo}$) derived from the reasoning strategy ($z_s$). A lightweight language model maps the strategy to specific acoustic parameters (pitch, tone, volume), enabling the Talker to synthesize speech that is not just a reading of text, but a performance aligned with the context.

D. Training Strategy

• Two-Stage Training:
  1. Perceptual Grounding: Fine-tuning the Thinker on Multimodal Emotion Understanding (MEU) data to ensure accurate initial perception ($P(z_p|M)$).
  2. Joint Reasoning Tuning: Training the full causal chain (Perception $\to$ Intent $\to$ Strategy $\to$ Text) using Multimodal Emotional Dialogue (MED) data.
• Data Pipeline (EmoOmniPipe): A pipeline constructed to process raw movies and TV series into high-fidelity, annotated dialogue datasets. It involves chunking, denoising, speaker diarization, and fine-grained annotation using SOTA models.

E. Evaluation Benchmark (EmoOmniEval)

A multidimensional benchmark using LLM-as-a-Judge (Gemini 2.5 Pro) to assess:

• VS (Video-to-Speech): End-to-end relevance and emotional strategy.
• VT (Video-to-Text): Quality of emotion analysis and reasoning.
• IF (Instruction Following): How well the speech adheres to the generated emotional instructions.

3. Key Contributions

1. Framework: Proposed EmoOmni, the first Omni-LLM framework to explicitly model emotional dialogue as a Perception–Reasoning–Expression causal chain, separating "what to say" from "how to say it."
2. Methodology: Introduced E-CoT, which serves dual purposes: as a reasoning process for the Thinker and as explicit emotional instructions for the Talker, preventing the loss of emotional nuance.
3. Data & Benchmark: Constructed EmoOmniPipe (a real-world data processing pipeline) and EmoOmniEval (a comprehensive benchmark), addressing the scarcity of high-quality emotional dialogue data.
4. Performance: Demonstrated that explicit emotional reasoning can compensate for model scale.

4. Experimental Results

• Model Scale Efficiency: The EmoOmni-7B model achieves performance comparable to the Qwen3-Omni-30B-A3B-Thinking model (a 30B parameter model) under the same Talker conditions.
• Metric Superiority:
  • Outperforms all comparable parameter-scale models (e.g., Qwen2.5-Omni-7B, Intern-S1-9B) across all metrics (VS-RES, VS-RC, VT-EA).
  • Significantly narrows the "semantic–acoustic emotion gap" seen in end-to-end models.
• Ablation Studies:
  • Removing E-CoT components (especially Response Strategy) causes a drastic drop in performance (e.g., VT-RC drops from 1.84 to 1.36).
  • Removing the "Perceptual Grounding" stage (Stage 1 training) leads to significant degradation, confirming the necessity of learning "how to see" before "how to reason."
  • Using real-world movie/TV data is critical; synthetic or academic datasets alone fail to capture nuanced human dynamics.

5. Significance

• Paradigm Shift: Moves Omni-LLMs from implicit, hidden-state emotional control to explicit, instruction-driven emotional reasoning.
• Scalability: Proves that architectural improvements (E-CoT) and high-quality data can bridge the performance gap between small (7B) and large (30B) models in affective computing tasks.
• Human-Centric Interaction: By mimicking the human cognitive chain of perception, reasoning, and expression, EmoOmni enables more natural, empathetic, and socially aligned human-computer interactions, particularly in applications like virtual companionship and personalized education.

In conclusion, EmoOmni establishes a new standard for multimodal emotional dialogue by treating emotional intelligence as a structured reasoning process rather than a byproduct of semantic generation.