Emotion-Aware Prefix: Towards Explicit Emotion Control in Voice Conversion Models

This paper proposes an Emotion-Aware Prefix for a two-stage voice conversion backbone that significantly improves emotion conversion accuracy from 42.40% to 85.50% while preserving speaker identity, linguistic integrity, and speech quality through joint control of sequence modulation and acoustic realization.

The Gist

Imagine you have a magical recording studio where you can take a person's voice and make them say anything you want, but in the voice of someone else. This is called Voice Conversion.

Now, imagine you want to do something even harder: You want that new voice to not just sound like the target person, but also to feel a specific emotion—like pure joy, deep sadness, or intense anger—just like the original speaker did.

This is the problem the researchers at the University of Texas at Dallas tried to solve. Their paper, "Emotion-Aware Prefix," is like inventing a new set of "magic instructions" that tell the computer exactly how to inject emotion into a voice without breaking the voice itself.

Here is the breakdown using simple analogies:

1. The Problem: The "Good Actor, Bad Director"

Think of existing voice conversion models (like the one they improved, called VEVO) as a very talented actor.

• The Talent: The actor is great at mimicking a specific person's voice (their accent, pitch, and tone).
• The Flaw: If you ask the actor to "be happy," they might just smile a little. They don't fully become the character. They rely on vague hints from the recording, like "the speaker sounded energetic," but they miss the specific, high-intensity emotion you want. It's like asking a chef to "make it spicy," and they just add a pinch of salt instead of the hot sauce you wanted.

2. The Solution: The "Emotion-Aware Prefix"

The researchers added a special tool called the Emotion-Aware Prefix. Think of this as a Director's Script that is handed to the actor before they even start speaking.

• How it works: Before the computer generates the voice, it looks at a reference clip (a sample of the target emotion). It extracts the "feeling" from that clip and turns it into a digital "instruction tag."
• The "Deep-Prefix" Trick: Instead of just whispering this instruction once at the start, the researchers used a technique called Deep-Prefix Prompting. Imagine this as the Director standing next to the actor at every single step of the performance, constantly reminding them, "Remember, you are furious right now!" or "You are overjoyed!" This ensures the emotion stays consistent from the first word to the last.

3. The Two-Stage Factory

The system they improved works like a two-step factory:

• Stage 1: The Architect (Sequence Modulation)
  This stage decides the structure of the speech. It's like the architect drawing the blueprint. It decides: "Is this sentence going to be shouted? Is it going to be whispered? Where are the pauses?"

  • The Discovery: The researchers found that this is where the biggest change happens. If you tell the Architect to "build a house of sadness," the whole structure changes.

• Stage 2: The Builder (Acoustic Realization)
  This stage takes the blueprint and actually builds the house (the sound waves). It focuses on the texture and quality of the voice.

  • The Discovery: The Builder needs to know the emotion too, but if the Architect didn't get the memo, the Builder can't fix it. However, if both the Architect and the Builder are on the same page, the result is perfect.

4. The "Identity Shield" (Why the Voice Doesn't Break)

One of the biggest fears in voice conversion is that if you change the emotion too much, the voice will stop sounding like the target person. It might start sounding like a robot or a different person entirely.

The researchers found a secret ingredient: Acoustic Decoupling.

• The Analogy: Imagine you are painting a portrait.
  • Without Decoupling: You try to paint the emotion and the person's face on the same wet canvas at the same time. If you add too much "anger" (red paint), the face gets distorted and looks like a monster.
  • With Decoupling (Their Method): You first sketch the face (the identity) on a sturdy canvas. Then, you paint the emotion (the red paint) on a separate, transparent layer on top. You can change the emotion as much as you want, but the face underneath remains perfectly intact.

5. The Results: A Massive Leap

Before this new method, the computer could only get the emotion right about 42% of the time. It was guessing.
After adding the "Emotion-Aware Prefix" and the "Director's Script," the success rate jumped to 85.5%.

• What this means: The computer can now reliably make a voice sound genuinely happy, sad, or angry, while still sounding exactly like the person you asked it to mimic.

Summary

In short, the researchers built a smart instruction manual (the Prefix) that tells a voice-conversion AI exactly how to feel. They discovered that you need to control the "big picture" (the structure) and the "details" (the sound) at the same time, and you need to keep the person's identity separate from the emotion so the voice doesn't get lost.

It's the difference between a robot reading a script with a flat voice and a method actor who can truly feel the lines while sounding exactly like the character you chose.

Technical Summary

Here is a detailed technical summary of the paper "Emotion-Aware Prefix: Towards Explicit Emotion Control in Voice Conversion Models."

1. Problem Statement

Recent zero-shot voice conversion (VC) models have shown potential for emotion control but suffer from suboptimal and inconsistent performance. While existing models (like VEVO) can mimic general speaking styles, they lack the capability to explicitly shift a source utterance into a specific, high-intensity target emotion.

• Root Cause: Current models rely heavily on implicit cues (e.g., global energy or average pitch) provided by acoustic prompts during the dynamic modulation stage. They lack an explicit mechanism to control emotion, leading to a failure in generating distinct, high-fidelity emotional expressions without compromising speaker identity or linguistic content.

2. Methodology

The authors propose Emotion-Aware Prefix, a novel framework designed to integrate explicit emotion control into a two-stage voice conversion backbone (based on the VEVO architecture). The method consists of three core components:

A. Two-Stage Architecture

The framework follows the VEVO design, separating speech generation into:

1. Sequence Modulation Stage: An Autoregressive (AR) Transformer predicts discrete, style-rich audio tokens.
2. Acoustic Realization Stage: A Flow-Matching (FM) Transformer reconstructs the mel-spectrogram from the predicted tokens, conditioned on reference audio.

B. Emotion-Aware Prefix Encoder

To provide explicit emotion control, the authors introduce an encoder that generates a content-invariant style embedding:

• Temporal-Shuffle Transformer: The reference mel-spectrogram is temporally shuffled (random permutation) to disrupt phonetic/linguistic structures while preserving global prosodic and timbral statistics.
• Perceiver Layer: Compresses the variable-length shuffled features into a fixed-length style embedding ($s$) using learnable latent tokens.
• Emotion Fusion Layer: A pre-trained emotion encoder (Emotion2Vec+) extracts an emotion embedding from the reference. This is concatenated with the style embedding $s$ and projected to form the final Emotion-Aware Prefix ($E$).

C. Deep-Prefix Prompting

Instead of simply prepending the prefix $E$ to the input sequence, the authors employ a Deep-Prefix Prompting mechanism (inspired by P-Tuning-v2):

• The prefix $E$ is injected as the Key-Value (KV) cache at every layer of the AR Transformer.
• Independent projection matrices ($W_K, W_V$) map the prefix to the latent space of each layer.
• This ensures consistent emotion control throughout the entire generation process, rather than just at the input.

3. Key Contributions

1. Enhanced Emotion Controllability: The method significantly boosts Emotion Conversion Accuracy (ECA) from 42.40% to 85.50% while preserving speaker identity, linguistic integrity, and speech quality.
2. Hierarchical Sensitivity Analysis: The study reveals that sequence-level modulation is the primary driver for high-level prosodic intent, but joint control (combining sequence and acoustic stages) yields a non-additive, synergistic improvement in accuracy.
3. Acoustic Decoupling Importance: Comparative analysis demonstrates that acoustic decoupling (separating sequence modeling from acoustic realization) is critical for preserving speaker identity during emotion transfer. Models without this decoupling suffer from identity collapse when forced to generate strong emotions.

4. Experimental Results

The model was evaluated on the Emotion Speech Dataset (ESD) with 10 speakers and 5 emotions (Neutral, Happy, Sad, Angry, Surprised).

Objective Evaluation (Table 1)

• Emotion Conversion Accuracy (ECA): The proposed method achieved 85.50%, nearly doubling the baseline VEVO (42.40%).
• Speaker Identity: Maintained high Speaker Centroid Similarity (0.500) and low Equal Error Rate (4.50%), comparable to the baseline.
• Quality: Word Error Rate (WER) and perceptual quality scores (UTMOS, DMOS) remained stable or improved, indicating no degradation in intelligibility.

Subjective Evaluation

• Preference: In ABX tests, the proposed method was preferred over VEVO for Emotion Similarity (75.2% vs. 17.5%) and Speaker Similarity (58.7% vs. 16.8%).
• MOS: Slight improvement in Mean Opinion Score (4.018 vs. 3.878).

Ablation & Comparative Studies

• Deep-Prefix vs. Simple Prefix: Removing the Deep-Prompting mechanism (using simple input prepending) dropped ECA from 85.50% to 83.50%, proving the efficacy of layer-wise injection.
• Stage Isolation: Controlling emotion only at the Sequence stage improved ECA to 47.00% (vs. 12.50% for baseline), while controlling only the Acoustic stage yielded 34.50%. Joint control was essential for the highest performance.
• Acoustic Decoupling Test: Applying the method to GenVC (a single-stage model without acoustic decoupling) improved ECA (32.48% $\to$ 58.35%) but caused severe speaker identity degradation (EER jumped from 20.87% to 44.51%). This confirms that the two-stage, decoupled architecture is vital for identity preservation.

5. Significance

This paper establishes a new paradigm for explicit emotion control in generative voice conversion. By introducing the Emotion-Aware Prefix and Deep-Prompting, it solves the inconsistency issues of previous zero-shot models. Crucially, it provides a theoretical insight that acoustic decoupling is not just a design choice for quality, but a structural necessity for maintaining speaker identity when manipulating high-level emotional states. This makes the method highly applicable for immersive Human-Computer Interfaces, expressive dubbing, and speaker anonymization tasks requiring nuanced emotional expression.