💬 NLP

Integrating Feedback Loss from Bi-modal Sarcasm Detector for Sarcastic Speech Synthesis

This paper proposes a novel sarcastic speech synthesis framework that combines a two-stage transfer learning strategy with feedback loss from a bi-modal sarcasm detector to overcome data scarcity and improve the naturalness and expressiveness of generated sarcastic speech.

Original authors: Zhu Li, Yuqing Zhang, Xiyuan Gao, Devraj Raghuvanshi, Nagendra Kumar, Shekhar Nayak, Matt Coler

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhu Li, Yuqing Zhang, Xiyuan Gao, Devraj Raghuvanshi, Nagendra Kumar, Shekhar Nayak, Matt Coler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to tell a joke. But not just any joke—a sarcastic one. Sarcasm is tricky because it's like a "reverse signal": you say one thing, but you mean the exact opposite. To make it work, you need the right tone, the right pause, and a specific "twist" in your voice. If the robot sounds too serious or too flat, the joke falls flat, and nobody laughs.

This paper is about teaching a computer (specifically a Text-to-Speech system) how to master that tricky "sarcastic twist."

Here is how they did it, explained through simple analogies:

1. The Problem: The Robot is Too Literal

Most voice robots today are like very polite librarians. They read books out loud perfectly, but they sound neutral and boring. They struggle with sarcasm because sarcasm is subtle. It's like trying to teach someone to ride a bike by only showing them a picture of a bike; they need to feel the wobble and the balance. The researchers found that there wasn't enough "practice data" (recordings of people being sarcastic) to teach the robot properly.

2. The Solution: A "Sarcastic Coach" and a "Two-Step Dance"

To fix this, the team came up with a two-part strategy.

Part A: The "Sarcastic Coach" (Bi-modal Detector)

Usually, a voice robot just listens to the words. But sarcasm often hides in the tone and the context.

The Analogy: Imagine a coach watching a student practice a speech. If the student says, "Great job," but their voice sounds flat, the coach knows they aren't being sarcastic. If the student drags out the word "Great" with a sneer, the coach knows, "Ah, that's sarcasm!"
What they did: They built a special "detector" (the coach) that looks at both the text (the words) and the audio (the tone). They call this "bi-modal."
The Magic Trick: During the robot's training, this coach doesn't just watch; it gives feedback. If the robot tries to say something sarcastic but sounds too serious, the coach sends a "loss signal" (a gentle scolding) saying, "No, that doesn't sound sarcastic enough, try again!" This forces the robot to learn the subtle vocal cues of sarcasm.

Part B: The "Two-Step Dance" (Two-Stage Fine-Tuning)

You can't teach a robot to be a sarcastic comedian overnight. It needs to learn the basics first.

Step 1: The General Class (Conversational Speech): First, they taught the robot how to sound like a normal person having a casual conversation. They used clips from TV sitcoms (like Friends or The Big Bang Theory). This taught the robot how to sound natural, with pauses, laughter, and varying speeds, rather than just reading a script like a news anchor.
Step 2: The Specialized Class (Sarcastic Speech): Once the robot could sound like a real person, they gave it the "sarcastic" dataset. Now, using the "Sarcastic Coach" from Part A, they fine-tuned the robot specifically to master the art of saying the opposite of what it means.

3. The Results: Did it Work?

The researchers tested their new robot against the old, standard robot.

The "Coach" Test: When they asked their "Sarcastic Coach" to listen to the new robot's voice, the coach could identify the sarcasm much better than when listening to the old robot. This proved the new robot was actually sounding sarcastic, not just saying the words.
The Human Test: They asked real people to listen to the recordings.
- Naturalness: People preferred the new robot's voice.
- Sarcasm: When asked "Which one sounds more sarcastic?", 53% of people picked the new robot.
- The "Vibe": Listeners described the old robot's sarcasm as "monotonous" and "unconvincing," while the new robot captured the "nuanced expressiveness" of a real human joke.

4. What They Learned (and What They Didn't)

The paper claims that combining a "text-and-audio detector" with a "two-step training process" makes the robot much better at sarcasm.

However, the authors are honest about the limits:

They didn't test if the robot could tell the difference between a real sarcastic joke and a serious statement in a neutral context. They only tested it on things that were already labeled as sarcastic.
They couldn't perfectly separate which part of the success came from the "Coach" and which came from the "Two-Step Dance." Both helped, but they didn't measure them individually.

In a Nutshell

Think of this paper as teaching a robot to tell a joke. Instead of just giving it a script, they gave it a coach that listens to its tone and a training camp that starts with casual chatting and ends with advanced comedy. The result? The robot finally learned how to say "Oh, great" in a way that actually sounds like it means "Oh, terrible."

Technical Summary: Integrating Feedback Loss from Bi-modal Sarcasm Detector for Sarcastic Speech Synthesis

Problem Statement

Sarcastic speech synthesis remains a significant challenge in Text-to-Speech (TTS) research due to the nuanced prosody required to convey sarcasm (e.g., exaggerated intonation, unexpected pitch variations, and altered timing) and the scarcity of annotated sarcastic speech datasets. While deep learning has advanced general TTS capabilities, existing systems predominantly focus on neutral or reading-style speech, struggling to capture the dynamic and context-dependent nature of spontaneous sarcastic communication. Furthermore, the limited availability of dedicated sarcasm datasets restricts models' ability to learn and reproduce specific sarcastic features, and current data augmentation techniques have not fully overcome the difficulty of capturing the essence of sarcasm in synthesized audio.

Methodology

The authors propose a novel framework that integrates feedback from a bi-modal sarcasm detection model into the TTS training pipeline, combined with a two-stage fine-tuning strategy.

1. Bi-modal Sarcasm Detector with Feedback Loss

The core innovation involves training an independent bi-modal sarcasm detector that processes both audio (speech) and text inputs.

Architecture: The detector extracts mel spectrograms from speech, processes them through spectral and temporal modules with multi-head self-attention, and concatenates these features with BERT-encoded text embeddings. A fully connected layer predicts sarcasm labels.
Integration: This detector is integrated into the TTS training loop (based on FastSpeech 2). During training, the detector generates a "sarcasm embedding" from the target audio and text. This embedding is concatenated with the phoneme encoder output before the variance adaptor.
Loss Function: In addition to the standard Mean Absolute Error (MAE) between predicted and ground-truth mel spectrograms, the authors introduce a feedback loss. This loss is calculated as the cosine distance between the ground-truth audio+text embedding and the embedding extracted from the predicted mel spectrogram+text by the sarcasm detector. This forces the TTS model to generate audio that the detector recognizes as sarcastic, effectively making the synthesis model "sarcasm-aware."

2. Two-Stage Fine-Tuning Strategy

To address data scarcity, the authors employ a progressive fine-tuning approach:

Stage 1 (Conversational Adaptation): A pre-trained FastSpeech 2 model (trained on high-quality read speech like LibriTTS) is fine-tuned on a diverse conversational dataset (extracted from sitcoms like Friends and The Big Bang Theory). This stage adapts the model to natural dialogue patterns, varying intonations, and pacing.
Stage 2 (Sarcasm Specialization): The model is further fine-tuned on a labeled sarcastic dataset (MUStARD++). This stage specifically targets the acquisition of subtle intonational cues and pitch variations unique to sarcasm.

3. Data and Training

Datasets: The system utilizes LibriTTS for pre-training, a curated conversational dataset (6.17 hours) for Stage 1, and the MUStARD++ corpus (1,202 utterances) for Stage 2 and detector training.
Preprocessing: Audio is resampled to 22,050 Hz, silence is trimmed, and Demucs is used to isolate human voice from background noise and laughter.
Configuration: The TTS model uses 4 FFT blocks with 256-dimensional embeddings. The detector is trained for 50 epochs, while the TTS model undergoes 800k pre-training iterations and 100k fine-tuning iterations.

Key Contributions

Novel Integration of Sarcasm Detection Feedback: The paper introduces a method to incorporate a bi-modal sarcasm detector into the TTS pipeline via feedback loss. This allows the synthesis model to learn sarcasm nuances by optimizing for the detector's recognition of sarcastic features, enhancing the model's ability to convey sarcasm.
Comprehensive Input Modality Comparison: The study demonstrates that combining text and speech inputs for sarcasm detection significantly improves the F1-score compared to speech-only inputs, highlighting the necessity of textual semantic cues to identify sarcasm that may not be acoustically obvious.
Two-Stage Fine-Tuning Method: The authors propose a specific fine-tuning pipeline (Read $\to$ Conversational $\to$ Sarcastic) that effectively mitigates data scarcity, guiding the model to produce more accurate and appropriate sarcastic prosody.

Results

Sarcasm Detection Performance

Detector Accuracy: The proposed bi-modal detector outperformed the baseline MUStARD++ detector. On real data, the speech+text configuration achieved an F1-score of 71.2% (vs. 68.7% for the baseline).
Synthesized Speech Evaluation: When the detector was used to evaluate the synthesized speech, the proposed model achieved a higher F1-score (70.1%) compared to the baseline (68.8%) in the speech+text condition. This indicates that the synthesized prosody aligns more closely with the semantic intent of the text.

Subjective Evaluation

Mean Opinion Score (MOS): The proposed model achieved significantly higher MOS ratings than the baseline. Notably, 15% of the proposed model's utterances received the highest possible score (5/5), compared to only 5% for the baseline.
Preference Tests: In head-to-head comparisons, 53% of listeners rated the proposed model's output as having a stronger sarcasm tendency, and 49% preferred it overall. Listeners described the baseline output as "less convincing" and "monotonous," whereas the proposed model was perceived as more natural and expressive.

Significance and Claims

The paper claims to present the first comprehensive approach to sarcastic speech synthesis that bridges the gap between sarcasm detection and synthesis. By leveraging advances in multi-modal detection, the study offers a promising direction for generating emotionally intelligent and expressive speech.

The authors emphasize that their method improves the expressiveness and naturalness of synthesized sarcastic speech, successfully capturing unique prosodic and semantic features. They note that the use of sitcom-derived data ensures ecological validity, reflecting real-world sarcastic interactions.

However, the authors maintain a modest stance regarding their findings. They acknowledge limitations, specifically:

The inability to isolate the individual contributions of the feedback loss versus the two-stage fine-tuning.
The lack of a control condition with neutral, non-sarcastic texts, which may have primed listeners to expect sarcasm regardless of the actual prosodic cues.
The difficulty in disentangling whether listener judgments were based on the synthesized speech quality or the inherent sarcastic nature of the text.

Future work is suggested to expand datasets to include neutral texts, model fine-grained degrees of sarcasm, and refine evaluation metrics to better capture perceptual subtleties.