🤖 AI

Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTS

The paper proposes Self-EmoQ, a reinforcement learning-based framework that utilizes Plutchik's wheel of emotions to enable an LLM to proactively plan emotional states prior to text generation, thereby driving a high-quality, streaming emotional text-to-speech system that outperforms existing baselines in both emotion determination and response quality.

Original authors: Yue Zhao, Hongyan Li, Yong Chen, Luo Ji

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yue Zhao, Hongyan Li, Yong Chen, Luo Ji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are having a conversation with a very smart robot. Usually, these robots are great at understanding your words and replying with the right facts. But when it comes to feeling, they often stumble. They might wait until they've finished typing a whole paragraph to decide, "Oh, I should sound sad now," and then try to force that sadness into their voice. By then, it's too late; the tone doesn't match the words, and the conversation feels robotic and awkward.

The paper "Self-EmoQ" proposes a new way to fix this. Think of it as teaching the robot to plan its mood before it even opens its mouth.

Here is how the paper breaks it down, using some simple analogies:

1. The Problem: The "Too Late" Robot

In current systems, the robot acts like a musician who hears a song, waits for the whole song to finish, and then tries to guess what emotion the song was supposed to have.

The Issue: In a real-time, streaming conversation (where words and voice come out instantly), the robot needs to know the emotion before it starts speaking so the voice can match the words as they are being generated.
The Paper's Solution: Instead of reacting after the fact, the robot decides its emotion first, like a director telling an actor, "You are angry," before the scene starts. This allows the robot's voice to naturally sound angry from the very first word.

2. The Brain: A "Strategic Planner"

The authors built a special module called Self-EmoQ. Think of this module as a strategic chess player sitting inside the robot's brain.

How it works: Before the robot generates a single word of a reply, this "chess player" looks at the conversation history and asks, "What is the best emotional move to make right now to keep the conversation flowing naturally?"
The Tool: It uses a technique called Reinforcement Learning. Imagine the robot is playing a video game where the goal isn't just to win, but to keep the "emotional flow" of the game smooth. It gets points (rewards) for making good emotional choices and loses points for making awkward ones.

3. The Rulebook: Plutchik's "Wheel of Emotions"

How does the robot know what a "good" emotional move is? It doesn't just guess; it follows a specific rulebook based on a psychological theory called Plutchik's Wheel of Emotions.

The Analogy: Imagine emotions are colors on a color wheel.
- Adjacent colors (like Blue and Green) blend naturally.
- Opposite colors (like Red and Green) clash and look weird if you switch between them instantly.
The Paper's Claim: The robot uses this "color wheel" theory to score its decisions. If the user is sad, and the robot suddenly jumps to "Ecstatic Joy" (an opposite emotion), the rulebook says, "That's a bad move, you lose points." If the robot moves from "Sadness" to "Sympathy" (adjacent emotions), it gets points. This ensures the robot's mood changes make sense to humans.

4. The Result: A Seamless Conversation

The researchers tested this on four different conversation datasets (like daily chats and movie scripts).

The Outcome: The Self-EmoQ robot was better at picking the right emotion than robots that just "guessed" (prompting) or robots that were just memorized from past data (fine-tuning).
The Voice: Because the robot decided the emotion first, the voice synthesis (TTS) could stream the audio perfectly. The voice sounded naturally sad, angry, or happy as the words were being spoken, rather than trying to force the emotion in after the fact.

Summary

In short, Self-EmoQ is a system that teaches an AI to plan its feelings like a human does—deciding on a mood based on the context and the rules of human interaction before it speaks. This allows the AI to talk and sound emotionally consistent in real-time, making the conversation feel much more natural and less like a computer reading a script.

The paper confirms this works well in tests, showing the robot makes better emotional choices and speaks more expressively, but it does not claim this technology is ready for clinical therapy or specific medical uses yet; it is focused on improving the quality of conversational AI.

Technical Summary: Self-EmoQ

Problem Statement

Current conversational AI systems integrated with Text-to-Speech (TTS) often lack a mechanism for self-emotion determination that operates effectively in streaming scenarios. In industrial real-time applications, text generation and speech synthesis are often linked such that audio segments are produced as text tokens are generated. However, traditional Emotion Recognition in Conversation (ERC) modules can only identify emotional states after the entire textual response is completed. This latency prevents the immediate conditioning of the TTS module, which requires an emotion label at the very beginning of generation to modulate prosody and acoustic style.

While existing methods like Emotion Prediction in Conversation (EPC) or prompt-based strategies attempt to predict emotions beforehand, they often suffer from suboptimal planning due to a lack of parameter updates (in prompting) or are constrained by dataset supervision, limiting their ability to optimize long-term conversation quality and generalize to versatile situations. The core challenge is to treat emotion not merely as a static label to be recognized, but as a controllable decision variable that can be explicitly planned to drive coordinated emotional alignment across text and speech modalities in a streaming pipeline.

Methodology

The authors propose Self-EmoQ, a novel framework that integrates value-based Reinforcement Learning (RL) with Large Language Model (LLM) generation to determine self-emotion prior to response generation.

1. Formulation as an MDP

The emotional dialogue task is formalized as a Markov Decision Process (MDP) at the utterance level:

State ( $S$ ): The concatenation of dialogue history, background information, and the user's current query.
Action ( $A$ ): The selection of the system's self-emotion from a discrete set (based on Plutchik's taxonomy).
Reward ( $R$ ): A hybrid reward function combining:
- Imitation Reward: Based on ground-truth emotion labels in the training dataset.
- Theoretical Reward (Plutchik Score): A theory-driven score generated by GPT-4o. This score evaluates three dimensions: Emotion Alignment, Emotion Transition Plausibility (adhering to the topological structure of Plutchik's wheel, where adjacent transitions are natural and opposite ones are less plausible), and Emotion–Function Consistency (matching the behavioral function associated with the emotion).
Transition: The history is updated with the selected emotion, the generated text, and the subsequent user utterance.

2. Self-EmoQ Architecture

Planner Module: A plug-and-play module initialized from a pretrained LLM (Llama3.1-1B-Instruct). It is fine-tuned to output state-action values ( $Q$ -values) rather than text tokens directly.
Q-Value Estimation: Similar to StraQ*, the model estimates $Q(s, a)$ by averaging the log-probabilities of output tokens when the action (emotion) is presented as a multiple-choice option in the prompt.
Training: The module is trained using Deep Q-Network (DQN) principles. The loss is minimized via the Bellman Equation, optimizing for long-term cumulative discounted rewards.
Streaming Pipeline: Once trained, the module determines the optimal emotion ( $a^* = \arg\max_a Q(s, a)$ ) before the LLM generates the response text. This emotion label is then injected into the prompt for text generation and used to condition the Emo-TTS model (CosyVoice2), enabling the TTS to synthesize speech with the correct emotional prosody in a streaming fashion.

Key Contributions

Emotion-Planning Framework: The authors introduce a framework that determines self-emotion via value-based RL prior to text generation, enabling the driving of streaming TTS systems.
Theory-Driven Reward: They design a reward mechanism grounded in Plutchik's Wheel of Emotion. This allows the agent to learn not just to imitate dataset trajectories but to align with human emotional behaviors, transition patterns, and functional roles.
Streaming Implementation: The paper implements and verifies a full streaming pipeline for emotional language and speech generation, demonstrating that pre-determined emotional planning can maintain high quality in real-time deployment.

Experimental Results

The method was evaluated on four datasets: DailyDialog, EmoryNLP, MELD, and IEMOCAP.

Emotion Determination: Self-EmoQ outperformed prompting baselines (0-shot, ECoT, Plan-and-Solve, Metacognitive Prompting) and supervised baselines (SFT, FSM, EMDP) in ranking metrics (Recall@3/5, NDCG, MRR). It demonstrated superior ability to rank emotionally appropriate actions.
Response Generation: The framework achieved higher scores in BLEU-2, ROUGE-L, CIDEr, and Distinct-2, indicating that better emotional decision-making at the policy level translates to higher quality text responses.
Human Evaluation: In a study involving 10 evaluators, Self-EmoQ achieved statistically significant improvements over baselines in fluency, emotion intensity, acceptance, effectiveness, sensitivity, alignment, and overall satisfaction.
Downstream TTS: In streaming settings (Setting C), Self-EmoQ maintained high speech quality (measured by SpeechBERTScore and PESQ), whereas other methods relying on post-generation emotion recognition (Setting B) or lacking planning suffered significant quality drops when forced into a streaming paradigm.
Ablation Studies: Results confirmed that the RL approach without prior SFT, the use of the LLM for Q-value estimation (rather than an MLP head), and the inclusion of dialogue history and descriptions were all critical for performance. The Plutchik score weight ( $w$ ) was found to be dataset-dependent, balancing theory-driven consistency with dataset annotations.

Significance and Claims

The paper claims that Self-EmoQ addresses a critical gap in industrial conversational AI: the inability to integrate emotional cognition into streaming TTS pipelines. By treating emotion as a strategic decision variable optimized via RL, the framework enables agents to plan long-term emotional trajectories rather than reacting to the immediate context.

The authors emphasize that their approach moves beyond simple data-driven prediction (which is limited by dataset annotations) to strategic emotional decision-making. By leveraging Plutchik's theory, the system learns to generate transitions that are psychologically plausible and functionally consistent, resulting in more natural and empathetic interactions. The successful deployment in a streaming pipeline confirms that pre-planning emotions does not compromise, but rather enhances, the coherence and expressiveness of both text and speech generation in real-time applications.

The paper concludes with modest limitations, noting the computational overhead of using LLMs for reward scoring and the restriction of the action space to the discrete emotions defined by Plutchik's theory, which may limit the representation of fine-grained or mixed emotional states.