Modeling Sarcastic Speech: Semantic and Prosodic Cues in a Speech Synthesis Framework

Technical Summary: Modeling Sarcastic Speech via Semantic and Prosodic Cues

Problem Statement
Sarcasm is a pragmatic phenomenon where speakers convey meanings diverging from literal content, relying on a complex interplay between semantic incongruity and prosodic expression. While speakers often use prosodic strategies (e.g., exaggerated intonation, emphatic stress) to signal sarcasm, computational modeling of this phenomenon remains challenging. Existing approaches often struggle because:

1. Entanglement of Cues: In natural speech, semantic and prosodic cues are entangled, making it difficult to systematically manipulate them independently to study their specific contributions.
2. Limitations of Detection vs. Synthesis: Most prior work focuses on sarcasm detection (classification) rather than synthesis. Moving to synthesis is crucial for understanding how cues contribute to perception and for enabling interactive systems to actively deploy pragmatic phenomena.
3. Subtlety and Context: Sarcasm is more subtle and context-dependent than broad emotions (e.g., anger, joy), making it difficult to capture with handcrafted acoustic features or simple prosodic controls. Furthermore, high-quality, annotated sarcastic speech corpora are scarce, limiting purely data-driven approaches.

Methodology
The authors propose a Retrieval-Augmented LLM-enhanced sarcastic speech modeling framework that treats sarcasm as the integration of semantic interpretation and prosodic realization. The architecture, based on the VITS (Variational Inference with adversarial learning for Text-to-Speech) model, consists of three primary components:

1. Sarcasm-Aware Semantic Encoding (PEFT):

   • To capture high-level semantic features indicative of sarcastic intent, the authors employ Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA) on a LLaMA 3-8B model.
   • The model is fine-tuned on the News Headlines Sarcasm dataset (Misra & Arora, 2023) to learn discourse-level markers of sarcastic intent.
   • The output is a sequence of contextualized semantic embeddings ($E_s$) that serve as a controllable representation of sarcastic intent for the synthesis model.

2. Retrieval-Augmented Generation (RAG) for Prosody:

   • To provide realistic prosodic exemplars without manual selection, the system integrates a RAG module.
   • A database of sarcastic utterances is constructed from the MUStARD++ dataset. Each entry includes the utterance and its pre-computed semantic representation (using the same LoRA-adapted LLaMA 3 encoder).
   • For a given input text, the system computes cosine similarity between the input's semantic embedding and the database entries to retrieve the top-$K$ (set to 3) semantically aligned sarcastic utterances.
   • These retrieved utterances are encoded using WavLM to extract fixed-length prosodic embeddings ($E_w$), which capture characteristic intonation and emphasis patterns.

3. Integrated Speech Synthesis:

   • The synthesis model utilizes a cross-attention mechanism where phoneme embeddings act as queries and the sarcasm-aware semantic embeddings act as keys and values.
   • The retrieved prosodic embeddings are linearly projected and added to the cross-attention output to modulate the decoder.
   • This allows the generation of speech conditioned simultaneously on phonemes, sarcasm-aware semantics, and retrieved prosodic styles.

Experimental Setup and Conditions
The study evaluates the framework through a series of perceptual and objective experiments comparing four main conditions:

• Baseline: Standard VITS without sarcasm-specific guidance.
• Semantic-only: Speech conditioned on embeddings from BERT, pretrained LLaMA 3, or LoRA-fine-tuned LLaMA 3.
• Prosody-only: Speech guided by retrieved prosodic exemplars (RAG) with neutral semantic conditioning.
• Combined: Speech incorporating both LoRA-fine-tuned semantic embeddings and retrieved prosodic exemplars.

Key Results

• Semantic Modeling: Fine-tuning LLaMA 3 with LoRA significantly improved sarcasm detection performance on text (F1-score increased from ~65.5% for base LLaMA 3 to 72.5%), demonstrating that targeted adaptation effectively captures discourse-level sarcastic cues.
• Objective Metrics: The Combined condition achieved the best downstream sarcasm detection F1-score (62.5%) on synthesized audio, outperforming conditions using only semantic or only prosodic cues. Acoustic metrics (MCD, pitch, energy) remained stable across conditions, indicating that improvements were due to high-level cue manipulation rather than signal fidelity changes.
• Subjective Perception (MOS):
  • Naturalness (NMOS): The combined system maintained high naturalness (2.7 ± 0.1), comparable to the baseline and significantly better than using raw, unadapted LLaMA 3 embeddings (2.0 ± 0.1), which listeners found unnatural.
  • Sarcasm Expressivity (SMOS): The Combined and LoRA-only conditions achieved the highest sarcasm ratings (3.8 ± 0.2). While the combined system did not show a statistically massive additive boost over the LoRA-only condition in subjective ratings, it achieved the highest objective detection scores.
  • General-purpose embeddings (BERT, unadapted LLaMA 3) failed to improve sarcasm perception and, in the case of unadapted LLaMA 3, degraded naturalness.

Significance and Claims
The paper claims that its findings highlight the complementary roles of semantics and prosody in pragmatic interpretation. Specifically:

1. Complementarity: Semantic and prosodic cues jointly contribute to sarcasm recognition. While prosodic cues alone enhance recognition, the integration of sarcasm-aware semantics (via LoRA) and prosodic exemplars (via RAG) yields the most robust performance in both objective detection and subjective perception.
2. Mechanism of Sarcasm: The results support theoretical accounts that sarcasm arises from the interaction of meaning and delivery, rather than isolated linguistic or acoustic features.
3. Methodological Contribution: The study demonstrates that synthesis-based models can serve as effective tools for modeling speech behavior. By independently manipulating cues, the framework allows for the systematic exploration of how different factors contribute to pragmatic perception.
4. Pragmatic Modeling: The work illustrates how advances in LLMs (for semantic-pragmatic conditioning) and retrieval-based methods (for expressive prosodic grounding) can be combined to model subtle pragmatic phenomena that are difficult to capture with conventional emotional TTS systems.

The authors conclude that their framework provides a principled way to condition TTS on sarcasm-relevant cues, offering a flexible and cognitively grounded approach to expressive speech generation. They note that future work should focus on moving beyond isolated utterances toward discourse-aware detection and generation.