⚡ electrical engineering

Expressive Prompting: Improving Emotion Intensity and Speaker Consistency in Zero-Shot TTS

This paper proposes a two-stage prompt selection strategy that evaluates candidates based on prosodic features, audio quality, and text-emotion coherence before synthesis, and aligns them with input text during synthesis, to significantly improve speaker consistency and emotional intensity in zero-shot text-to-speech generation.

Original authors: Haoyu Wang, Chunyu Qiang, Tianrui Wang, Cheng Gong, Yu Jiang, Yuheng Lu, Chen Zhang, Longbiao Wang, Jianwu Dang

Published 2026-04-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haoyu Wang, Chunyu Qiang, Tianrui Wang, Cheng Gong, Yu Jiang, Yuheng Lu, Chen Zhang, Longbiao Wang, Jianwu Dang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director trying to film a movie scene where an actor needs to deliver a line with perfect sadness or explosive anger. You have a script (the text), but you need a "reference clip" (the prompt) to show the actor how to say it.

If you hand the actor a reference clip of them whispering calmly, they might whisper the sad line. If you hand them a clip of them screaming in joy, they might shout the sad line. The reference clip dictates the vibe, the voice, and the intensity of the performance.

This paper, titled "Expressive Prompting," tackles a major problem in AI voice generation: How do we pick the perfect reference clip to make the AI sound exactly how we want?

Here is the breakdown of their solution, ExpPro, using simple analogies.

The Problem: The "Random Guess" Game

Currently, most AI voice systems pick a reference clip at random or just by matching the words.

The Random Approach: Like asking a friend to "pick a random song" to set the mood for a party. You might get a sad ballad when you wanted a dance track.
The Text-Only Approach: Like looking at a song title ("Happy Birthday") and assuming the song sounds happy. But what if the singer is singing it in a funeral dirge? The words match, but the emotion is wrong.

The result? The AI sounds flat, the emotion is weak, or the voice sounds like a different person than the one you wanted.

The Solution: The "Two-Stage Casting Call" (ExpPro)

The authors propose a smart, two-step strategy to find the best reference clip without needing to retrain the AI (which is like teaching the actor a new skill from scratch). They call this ExpPro.

Stage 1: The "Audition" (Static Selection)

Before the AI even starts speaking, the system acts like a strict casting director holding a massive pile of reference clips. It filters them through three specific lenses:

The "Pitch" Check (The Volume Knob):
- The Analogy: Think of pitch as the "energy level" of a voice. A low, steady hum is usually calm or sad. A high, wiggly voice is usually happy or surprised.
- What they do: The system measures the average height (mean) and the wiggles (variance) of the voice in the clips. It groups them into clusters. If you want "Angry," it picks clips with high, jagged energy. If you want "Sad," it picks low, flat ones.
The "Vibe Check" (Perception & Text Match):
- The Analogy: Imagine a music critic listening to the clip. Is the audio clear and natural? Also, does the text in the clip actually match the emotion?
- What they do: They use an AI "critic" (ChatGPT) to read the text of the reference clip and ask, "Does this sentence actually sound like 'Angry'?" If the clip says "I am so happy" but sounds bored, it gets rejected.
The "Rehearsal" (Model Performance):
- The Analogy: This is the most crucial step. Just because a clip sounds good on its own doesn't mean the AI can copy it well. It's like a great singer who sounds amazing live but terrible when recorded on a cheap phone.
- What they do: The system actually tests the top clips by asking the AI to speak a few neutral sentences using that clip as a guide. It checks:
  - Did the AI understand the words? (Character Error Rate)
  - Did it keep the same voice? (Speaker Similarity)
  - Did it keep the same emotion? (Emotion Similarity)
- Only the clips that the AI can actually copy well are kept.

Stage 2: The "Final Match" (Dynamic Selection)

After the "Audition," you have a shortlist of the best 5 or 10 clips. Now, you need to pick the one that fits the specific sentence you want the AI to say right now.

The Analogy: You have a shortlist of 5 great actors. You are about to film a scene where the character says, "I can't believe you did that!"
- If the scene is a comedy, you pick the actor who is good at sarcastic delivery.
- If the scene is a tragedy, you pick the actor who is good at heartbreak.
What they do: The system compares the meaning of the new sentence with the meaning of the shortlisted clips. It picks the one that is semantically closest. If the new text is about "loss," it picks the "sad" clip from the shortlist, not the "happy" one, even if both passed the first stage.

Why This Matters

The paper proves that by using this two-step "Casting Call," the AI becomes:

More Emotional: The voices actually feel angry, sad, or happy, not just robotic.
More Consistent: The voice sounds like the same person throughout the speech.
Better Quality: The speech sounds clearer and more natural.

The Bottom Line

Think of ExpPro as a super-smart assistant that doesn't just grab a random voice sample. Instead, it:

Auditions thousands of clips to find the ones with the right energy and clarity.
Tests them to see which ones the AI can actually copy well.
Matches the final winner to the specific mood of your sentence.

The result? An AI that can tell a story with the same emotional depth and voice consistency as a human actor, simply by choosing the right "reference" to start with.

1. Problem Statement

Recent Large Language Model (LLM)-based Text-to-Speech (TTS) systems have achieved remarkable zero-shot voice cloning capabilities using input prompts. However, the quality of the generated speech is heavily dependent on the prompt selection. Existing methods suffer from two main limitations:

Random Selection: Often fails to provide rich emotional information or consistent speaker identity.
Text-Based Selection: Selects prompts based solely on text similarity, ignoring the disconnect between text content and the desired emotional intensity or speaker characteristics.

These limitations result in synthesized speech with weak emotional expression, unstable speaker identity, and poor perceptual quality. The paper addresses the question: Can we improve emotion intensity and speaker consistency in zero-shot TTS through better prompt selection without additional model training?

2. Methodology: ExpPro

The authors propose ExpPro, a novel two-stage prompt selection strategy designed to systematically screen and rank prompt candidates. It operates without modifying the underlying TTS model.

Stage 1: Static Selection (Pre-Synthesis)

This stage evaluates the intrinsic quality of prompt candidates and their specific performance within the target TTS model. It involves three key dimensions:

Pitch Analysis:
- Extracts Mean and Variance of the fundamental frequency (pitch).
- Uses K-Means clustering to group prompts based on pitch characteristics associated with specific emotions (e.g., high variance for "Happy/Surprised," low variance for "Sad/Comfort").
- Selects the top $m$ clusters that best match the target emotional state.
Perceptual and Textual Coherence:
- Perceptual Quality: Uses DNSMOS (Deep Neural Network for Subjective Mean Opinion Score) to evaluate audio clarity and naturalness.
- Textual Emotional Coherence: Uses ChatGPT as a semantic comparator to score the alignment between the prompt's text and its labeled emotion.
- Combines DNSMOS and ChatGPT scores to select the top $n\%$ of candidates.
Model-Specific Performance Evaluation:
- Synthesizes speech using the remaining candidates to evaluate how the specific TTS model handles them.
- Metrics include:
  - Character Error Rate (CER): For content accuracy.
  - Speaker Similarity: Using Resemblyzer and WavLM.
  - Emotion Similarity (ES): Using emotion2vec.
- Candidates are ranked based on a weighted sum of these metrics to retain the top $k$ prompts.

Stage 2: Dynamic Selection (During Synthesis)

For a specific synthesis task, the system analyzes the input text to be synthesized.
It calculates semantic similarity between the input text and the text of the $k$ pre-selected static candidates using stsb-distilroberta-base.
The prompt with the highest semantic relevance is chosen as the final prompt for generation.

3. Key Contributions

Two-Stage Strategy (ExpPro): A training-free framework combining static filtering (based on acoustic and semantic quality) and dynamic matching (based on text relevance) for LLM-based TTS.
Multi-Perspective Analysis: Unlike previous methods, ExpPro evaluates prompts based on:
- Intrinsic acoustic features (Pitch).
- Perceptual quality (DNSMOS).
- Text-emotion alignment (LLM-based scoring).
- Model-specific generation performance (CER, Speaker/Emotion similarity).
Generalizability: The strategy is flexible and applicable to any TTS model that utilizes prompt speech, demonstrated across different architectures.

4. Experimental Results

The method was validated using two state-of-the-art LM-based TTS models (CosyVoice and GPT-SoVITS) on two datasets (HE2D and ESD) covering five emotions (Happy, Sad, Angry, Surprised, Comfort).

Performance vs. Baselines: ExpPro significantly outperformed Random selection and Text-based (MiniLM) selection.
- Emotional Intensity: Measured by Strength Perception (SP), ExpPro showed consistent improvements (e.g., SP increased from ~0.76 to ~0.81 for CosyVoice on HE2D).
- Speaker Consistency: Improved speaker similarity scores (Resemblyzer/WavLM) were observed across all emotions.
- MOS (Mean Opinion Score): Subjective evaluations showed higher quality scores for ExpPro-generated speech.
Ablation Studies:
- Prompt Speech Quality: Removing the quality selection stage (using the "reverse" method) led to a significant drop in SP and emotional similarity, proving the necessity of filtering for acoustic quality.
- Model Performance: Selecting prompts based on how well a specific model generates them (rather than just the prompt's raw quality) was crucial. The "Top 3" prompts for one model were often the "Bottom 2" for another, highlighting the need for model-specific evaluation.
Parameter Sensitivity: Experiments showed that stricter selection criteria (smaller $m$ and $n$ ) generally yielded higher emotional impact, with optimal parameters identified as $m=3, n=15, k=5$ .

5. Significance and Conclusion

This paper demonstrates that prompt selection is a critical, yet under-explored, factor in zero-shot expressive TTS. By introducing a rigorous, multi-stage evaluation framework, ExpPro bridges the gap between the potential of LLM-based TTS and the practical delivery of high-quality, emotionally expressive speech.

Practical Impact: The method requires no additional training of the TTS model, making it immediately deployable for improving existing systems.
Future Work: The authors plan to extend this strategy to other dimensions and apply it to broader tasks like text-to-audio generation.

In summary, ExpPro provides a systematic solution to the "garbage in, garbage out" problem in prompt-based TTS, ensuring that the input prompts are acoustically robust, emotionally coherent, and optimally matched to both the target text and the synthesis model.