Technical Summary: TRACE-EVC

Problem Statement

Traditional Emotional Voice Conversion (EVC) systems rely on explicit target conditions—such as discrete emotion labels, reference audio utterances, or absolute style descriptions—to guide generation. While effective, these paradigms define a fixed target state but fail to capture the direction or nature of the emotional transition relative to the source. In many practical scenarios, users may not have a specific target emotion in mind but instead wish to modify the source utterance relatively (e.g., "make the speech slightly calmer" or "sound noticeably more confident"). Existing methods lack the capability to interpret such relative, source-conditioned instructions, and there is a significant data gap regarding datasets that describe affective transformations rather than static utterance styles.

Methodology

The authors propose TRACE-EVC, a zero-shot framework for instruction-guided relative EVC, supported by the TRACE-Instruct dataset. The framework consists of two primary components:

1. TRACE-Instruct Dataset Construction

To address the lack of training data for relative instructions, the authors constructed TRACE-Instruct using paired emotional speech from the ESD and MEAD corpora.

• Pair Construction: Source-target pairs are formed across different emotions (inter-emotion) and intensity levels (intra-emotion).
• Affective Difference Calculation: A Speech Emotion Recognition (SER) model predicts Valence-Arousal-Dominance (VAD) coordinates for both source ($VAD_{src}$) and target ($VAD_{tgt}$) utterances. The relative affective change is computed as $\Delta VAD = VAD_{tgt} - VAD_{src}$.
• Instruction Generation: A Large Language Model (LLM, Qwen3) is prompted with the source style description, target style description, and the calculated $\Delta VAD$. The LLM generates natural language instructions that describe the transformation (e.g., "raise the liveliness") rather than the absolute target style.
• Filtering: Rule-based filtering removes explicit target emotion labels, numerical VAD values, and metadata to ensure instructions remain relative.

2. TRACE-EVC Framework

The model operates in a zero-shot manner, requiring no target labels or reference utterances at inference. It comprises two cascaded modules:

• Emo-Compass (Emotion Predictor):

  • Input: Source emotion embedding ($z_{src}$) and a text-encoded instruction embedding ($p$).
  • Representation: The emotion embedding $z$ concatenates an emotion2vec embedding (category-level), VAD coordinates (continuous affective state), and mean prosodic features (F0 and energy).
  • Mechanism: Unlike diffusion models that generate from noise, Emo-Compass models the conversion as a source-anchored rectified flow. It treats the instruction as a displacement vector in the affective space.
  • Training: A Transformer $D_\theta$ predicts the velocity $\hat{v}$ of the flow, trained to match the ground-truth displacement $v^* = z_{tgt} - z_{src}$. The loss function includes a velocity regression term and auxiliary guidance terms for direction, category, VAD/prosody reconstruction, and intensity ordering.
  • Inference: Starting from $z_{src}$, the model integrates the predicted velocity over $K$ Euler steps to generate the target embedding $\hat{z}_{tgt}$.

• Synthesis Module:

  • Based on the DurFlex-EVC backbone, this module performs zero-shot conversion using continuous embeddings.
  • It combines the source speaker embedding with the predicted target emotion embedding to create a target style representation ($m_{tgt}$).
  • A duration regulator and a conditional diffusion decoder generate the mel-spectrogram, which is converted to a waveform by a HiFi-GAN vocoder. This process preserves the source speaker identity and linguistic content while applying the instructed emotional shift.

Key Contributions

1. Instruction-Guided Relative EVC Task: The introduction of a new task paradigm where natural language instructions specify source-conditioned affective transformations (categorical shifts and intensity modifications) rather than fixed target states.
2. TRACE-Instruct Dataset: A dataset of 0.54M relative emotion instructions covering cross-emotion shifts, intra-emotion intensity modifications, and open-ended transitions, generated via LLMs guided by VAD differences.
3. TRACE-EVC Framework: A zero-shot system featuring Emo-Compass, which rethinks prompt-based EVC by anchoring the generative process at the source embedding and modeling the instruction as a rectified flow velocity. This enables flexible control without target labels or reference audio.
4. Empirical Validation: Comprehensive experiments demonstrating that the system accurately follows relative instructions while preserving speaker identity, linguistic content, and speech quality.

Experimental Results

Inter-Emotion Conversion (ESD Dataset)

• Performance: TRACE-EVC achieved the highest Emotion Similarity (EECS: 0.82), Speaker Similarity (SECS: 0.68), and Emotion Classification Accuracy (ACCcls: 93.00%) among compared baselines (including label-based SGEVC/DurFlex-EVC and reference-based ZEST/VEVO).
• Quality: It maintained competitive Word Error Rate (WER: 9.96%) and UTMOS scores (3.57), indicating high intelligibility and naturalness.
• Ablation: Removing Emo-Compass and simply concatenating embeddings resulted in a significant drop in EECS (0.65) and ACCcls (43.08%), validating the necessity of explicitly predicting the target embedding via flow.

Intra-Emotion Intensity Conversion (MEAD Dataset)

• Zero-Shot Capability: Evaluated on unseen speakers, TRACE-EVC achieved NISQA scores (3.147) close to ground truth (3.248), demonstrating high naturalness.
• Intensity Control: The system showed monotonic increases in mean arousal shifts corresponding to the strength of the instruction (e.g., "mild increase" vs. "strong increase"), confirming fine-grained control over emotional intensity.
• Speaker Preservation: SECS remained stable (0.706 for unseen speakers), indicating robust speaker identity preservation.

Instruction Following

• Subjective evaluation by 15 raters yielded high Instruction Following (IF) scores: 4.135 for inter-emotion and 4.296 for intra-emotion settings (on a 1–5 scale).

Component Analysis (Emo-Compass)

• Affective Cues: Ablation studies confirmed that while emotion2vec provides strong category-level accuracy, the inclusion of VAD and prosodic features is critical for continuous metrics. Removing VAD increased VAD L2 distance and intensity MSE; removing prosody further degraded intensity consistency.

Significance and Claims

The paper claims that TRACE-EVC successfully bridges the gap between natural language instructions and fine-grained emotional control in voice conversion. By shifting from target-oriented generation to source-conditioned relative transformation, the framework allows for more intuitive and flexible control (e.g., "calmer," "more expressive") without requiring users to specify a target emotion label or provide a reference recording. The work demonstrates that modeling emotional change as a displacement in a continuous affective space (via rectified flow) is an effective strategy for zero-shot EVC, achieving performance competitive with state-of-the-art systems on standard categorical tasks while enabling new capabilities in relative intensity and open-ended affective control.