Textless and Non-Parallel Speech-to-Speech Emotion Style Transfer

This paper proposes S2S-ZEST, a textless and non-parallel speech-to-speech framework that achieves zero-shot emotion style transfer by extracting semantic, speaker, and emotion representations to synthesize speech that preserves the source's content and identity while adopting the reference's emotional style.

The Gist

Imagine you have a friend who tells a funny joke, but they tell it in a completely flat, boring voice. Now, imagine you want to hear that exact same joke, but told with the wild, dramatic energy of a different friend who is known for being very excited.

That is essentially what this paper is about. The researchers have built a digital "magic trick" called S2S-ZEST that can take a recording of someone speaking (the Source) and rewrite the emotion of that speech to match a different recording (the Reference), all without changing who is speaking or what they are saying.

Here is a simple breakdown of how they did it, using some everyday analogies.

The Big Problem: The "Entangled" Mess

Usually, when computers try to change emotions in speech, they get confused. Think of a person's voice like a smoothie. The content (the words) is the fruit, the speaker (their identity) is the blender, and the emotion (happy, sad, angry) is the ice cream.

In older systems, the blender and the ice cream were stuck together. If you tried to change the ice cream (make it happy), the blender (the speaker's identity) would change too. You'd end up with a different person entirely, or the words would get garbled.

The Solution: The "De-Construction" Kit

The authors built a system that acts like a high-tech kitchen where they can separate the fruit, the blender, and the ice cream perfectly. They call this an Analysis-Synthesis Pipeline.

Here is how the process works in three simple steps:

1. The Analysis (Taking the Cake Apart)

First, the system listens to the audio and breaks it down into four distinct ingredients:

• The Script (Semantic Tokens): It converts the words into a digital code. Think of this as the recipe card.
• The Voiceprint (Speaker Embedding): It captures the unique sound of the person's voice, like a digital fingerprint.
• The Mood (Emotion Embedding): It figures out the feeling of the speech (e.g., is this angry? sad?).
• The Rhythm (Pitch and Duration): It maps out the melody (how high or low the voice goes) and the speed (how fast or slow they talk).

The Magic Trick: They use a special technique to make sure the "Voiceprint" doesn't accidentally steal the "Mood." They use a "tug-of-war" training method (called adversarial loss) where the computer is forced to forget the emotion when trying to identify the speaker, and forget the speaker when trying to identify the emotion. This keeps them separate.

2. The Transfer (Mixing New Ingredients)

Now comes the fun part. Let's say you have:

• Source: A robot reading a news report in a boring, neutral voice.
• Reference: A clip of a friend laughing hysterically.

The system takes the Recipe and Voiceprint from the Robot. But, it takes the Mood and the Rhythm from the Laughing Friend.

• It tells the Robot: "Keep your voice and your words, but start talking as fast and as high-pitched as the laughing friend."
• It also adjusts the duration (how long each word lasts) so the robot sounds like it's genuinely excited, not just speaking faster.

3. The Synthesis (Baking the New Cake)

Finally, the system uses a powerful engine called BigVGAN (think of this as a super-advanced 3D printer for sound) to rebuild the audio. It takes the Robot's voice, the Robot's words, and the Friend's excitement, and prints out a brand new audio file.

The result? You hear the Robot saying the news report, but it sounds like it's laughing and having a great time.

Why Is This a Big Deal?

• No Scripts Needed: Unlike older methods that required the speaker to read the exact same text in different emotions (which is hard to record), this works with any speech. You can take a random podcast clip and make it sound angry, happy, or sad.
• Zero-Shot Learning: It doesn't need to be taught specifically how to be "angry." It just looks at a reference clip of anger and copies that style instantly.
• Better Data: The researchers showed that this tool can be used to create fake data to help train other AI systems. For example, if a computer is bad at recognizing "fear," you can use this tool to take calm voices and turn them into fearful ones to teach the computer what fear sounds like.

The Results

The team tested this by asking humans to listen to the results. They found that:

1. The Emotion Stuck: The new speech sounded much more like the target emotion than previous methods.
2. The Voice Stayed: The speaker still sounded like themselves, not like the person who provided the reference emotion.
3. The Words Were Clear: The meaning of the speech didn't get lost in the shuffle.

In a Nutshell

This paper presents a tool that acts like a digital emotion chameleon. It allows you to take any voice, keep its identity and words intact, but dress it in the emotional clothes of anyone else, anywhere, anytime. It's a huge step forward for making computers sound more human and expressive.

Technical Summary

Here is a detailed technical summary of the paper "Textless and Non-Parallel Speech-to-Speech Emotion Style Transfer" by Soumya Dutta, Avni Jain, and Sriram Ganapathy.

1. Problem Statement

Speech-to-Speech (S2S) Emotion Style Transfer (EST) aims to modify the emotional expression of a source speech signal to match a reference speech signal while preserving the original speaker's identity and linguistic content.

• Challenges: Existing methods often rely on parallel datasets (same speaker, same text, different emotions), which are expensive and difficult to scale. Furthermore, many approaches suffer from entanglement, where speaker identity and emotional style are mixed, making it difficult to transfer emotions to unseen speakers or unseen emotional categories (Zero-Shot).
• Goal: The authors propose a textless (no transcription required) and non-parallel framework capable of Zero-Shot Emotion Style Transfer (ZEST), enabling the transfer of emotional attributes from a reference to a source without explicit supervised training on parallel data.

2. Methodology: S2S-ZEST Framework

The proposed framework, S2S-ZEST, follows an analysis-synthesis pipeline trained via an auto-encoding objective. It decomposes speech into disentangled factors and reconstructs them.

A. Analysis Module (Feature Extraction)

The input speech is processed to extract five key components:

1. Semantic Tokens: The authors use soft-HuBERT embeddings (continuous features) which are quantized into discrete tokens using k-means clustering. This preserves intelligibility while allowing control over speech duration.
2. Speaker Embeddings: Extracted using a pre-trained ECAPA-TDNN. To prevent emotion leakage into the speaker vector, an adversarial loss is applied to force the speaker embedding to be emotion-disentangled.
3. Emotion Embeddings: Extracted using a fine-tuned HuBERT-based emotion classifier. This produces both frame-level embeddings ($E$) and an utterance-level pooled embedding ($\bar{e}$). A speaker adversarial loss is used here to ensure the emotion embedding is speaker-disentangled.
4. Pitch Contour ($F_0$): A dedicated module reconstructs the pitch contour using a cross-attention mechanism. The content tokens act as queries, while the sum of speaker and emotion embeddings act as keys/values. This allows pitch generation to be conditioned on the target emotion and source speaker.
5. Duration Predictor: A module that predicts the duration of de-duplicated semantic tokens. It takes the source tokens, source speaker embedding, and the reference emotion embedding as input to adjust the speaking rate according to the target emotion.

B. Synthesis Module

• Architecture: The system uses BigVGAN, a universal vocoder known for high-fidelity waveform generation and generalization.
• Input: The synthesizer receives the token sequence, speaker embedding, utterance-level emotion embedding, and the reconstructed pitch contour.
• Training: The entire pipeline is trained using an auto-encoding loss (reconstructing the original speech from its own factors) to ensure efficient resynthesis.

C. Inference (Style Transfer)

During the transfer process:

1. Source: Content tokens and speaker embeddings are extracted from the source speech.
2. Reference: Emotion embeddings (frame-level and utterance-level) are extracted from the reference speech.
3. Synthesis: The duration predictor uses the source tokens and reference emotion to generate new token durations. The pitch reconstructor uses source content, source speaker, and reference emotion to generate a new pitch contour.
4. Output: BigVGAN synthesizes the final speech using the modified tokens, pitch, and the reference emotion embedding.

3. Key Contributions

1. Novel Architecture: Introduction of S2S-ZEST, a textless, non-parallel framework that combines soft-HuBERT tokenization with BigVGAN, replacing conventional HuBERT/HiFi-GAN combinations.
2. Disentanglement Mechanisms: Implementation of adversarial losses to effectively separate speaker identity from emotion in the embeddings, enabling transfer to unseen speakers.
3. Dynamic Control: Development of a duration predictor and a pitch reconstruction technique that explicitly condition on emotion and speaker factors, enhancing naturalness and rhythm transfer.
4. Data Augmentation Application: Demonstration of the framework's utility in augmenting data for Speech Emotion Recognition (SER) tasks, significantly improving model performance in low-resource settings.

4. Experimental Results

The framework was evaluated on the ESD (Emotional Speech Database), CREMA-D, and TIMIT datasets across various settings (Seen/Unseen speakers and emotions).

• Objective Metrics:

  • Emotion Similarity: S2S-ZEST achieved the highest emotion similarity scores (e.g., 0.69 in SSST, 0.64 in Unseen Target Emotion) compared to baselines like StarGANv2-EST and VEVO.
  • Content Preservation: It maintained low Word Error Rates (WER), outperforming baselines in 5 out of 7 test settings (Average WER: 7.79).
  • Speaker Preservation: While VEVO had slightly higher speaker similarity, S2S-ZEST showed robust generalization, especially with the "diverse" variant trained on augmented data.
  • Rhythm Transfer: The model showed strong correlation in word speaking rate (PCC) for same-text scenarios.

• Subjective Evaluation (MOS):

  • Human listeners rated S2S-ZEST-diverse significantly higher than baselines in emotion similarity and speech quality across all test settings.
  • The model successfully transferred emotions to unseen speakers and unseen emotion categories (e.g., Fear, Disgust) without degradation in quality.

• Ablation Studies:

  • Removing the duration predictor improved WER but significantly degraded emotion similarity and rhythm transfer, proving its necessity for style transfer.
  • Removing adversarial losses caused a drop in emotion accuracy and speaker similarity, confirming the critical role of disentanglement.
  • Soft-HuBERT + BigVGAN outperformed HuBERT-base + HiFi-GAN in both intelligibility and synthesis quality.

5. Significance and Impact

• Textless & Non-Parallel: The ability to perform high-quality emotion transfer without transcribed data or parallel recordings makes the system highly scalable and applicable to real-world scenarios where such data is unavailable.
• Zero-Shot Generalization: The framework successfully handles Unseen Speakers and Unseen Emotions, a significant leap over prior art that often fails in these conditions.
• Practical Application: The paper demonstrates a concrete use case in Speech Emotion Recognition (SER), showing that S2S-ZEST can generate synthetic data to boost SER model performance, particularly in low-resource regimes (up to 5.68% improvement in F1-score).
• Robustness: The system maintains stable emotion transfer even when the source speech is corrupted by additive noise (up to 10 dB SNR), though intelligibility (WER) naturally degrades.

In conclusion, S2S-ZEST represents a state-of-the-art approach to emotion style transfer, effectively balancing the preservation of speaker identity and content with the flexible transfer of emotional style, all within a textless, zero-shot framework.