Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detection

1. Problem Statement

Sarcasm detection in speech remains a significant challenge in human-computer interaction due to two primary bottlenecks:

• Data Scarcity: High-quality, large-scale annotated datasets for sarcastic speech are rare. Existing datasets (e.g., MUStARD, MUStARD++) are often small, derived from scripted TV shows, and rely heavily on multimodal data (visual cues like facial expressions) which limits their applicability to audio-only scenarios (e.g., podcasts, radio, phone calls).
• Annotation Cost: Manually annotating sarcasm is labor-intensive and requires expert knowledge of prosody, context, and linguistic nuance, making the creation of large-scale datasets prohibitively expensive.
• Modality Limitations: Most existing detection systems rely on multimodal fusion. However, real-world applications often require detection based solely on speech (audio + text), where visual context is unavailable.

2. Methodology

The authors propose a hybrid LLM-assisted annotation pipeline to generate a large-scale, high-quality sarcastic speech dataset. The process consists of three main stages:

A. Automatic Data Collection & Processing

• Source: The team utilized the Overly Sarcastic Podcast (OSPod), a collection of naturalistic, spontaneous speech rich in sarcasm.
• Preprocessing: They employed the Emilia-Pipe framework to process 30 episodes of raw audio. This pipeline performed segmentation, speaker diarization, noise removal, and automatic transcription, resulting in 11,024 utterance-level segments aligned with transcripts.

B. LLM-Based Annotation

• Models: Two state-of-the-art Large Language Models (LLMs) were used as annotators: GPT-4o and LLaMA 3 (specifically the 70B variant).
• Prompt Engineering: The models were provided with structured prompts containing:
  • Raw text, context, timestamps, and speaker info.
  • Few-shot examples of sarcastic and non-sarcastic utterances.
  • Specific instructions to focus on linguistic cues (exaggeration, contradiction, absurdity) and implied meanings.
• Dual-Annotator Strategy: Both models independently labeled each segment. This allowed the authors to identify samples where the models agreed and samples where they disagreed.

C. Human Verification

• Conflict Resolution: Of the 11,024 segments, 2,884 showed disagreement between GPT-4o and LLaMA 3.
• Expert Review: Two PhD students specializing in phonetics manually reviewed these ambiguous cases. Crucially, unlike the LLMs which only had text, human reviewers analyzed prosodic cues (tone, hesitation, pitch) alongside the text to make final determinations.
• Final Dataset: The verified labels were merged to create the final dataset, PodSarc.

3. Key Contributions

1. Novel Annotation Pipeline: Demonstrated the feasibility of using LLMs to facilitate sarcasm annotation, significantly reducing reliance on human annotators while maintaining high quality through a "LLM-first, human-verification-on-disagreement" workflow.
2. PodSarc Dataset: Introduced PodSarc, a large-scale, bimodal (text + audio) sarcastic speech dataset containing 11,024 utterances (approx. 29.42 hours of speech). It features a balanced distribution of sarcastic (4,026) and non-sarcastic (6,998) utterances.
3. Validation of Approach: Validated the pipeline by comparing LLM-generated labels against gold-standard human labels on the MUStARD++ dataset, proving that LLMs can serve as effective proxies for initial dataset creation.

4. Experimental Results

The authors evaluated sarcasm detection performance using a collaborative gating architecture (multimodal feature fusion) across different datasets and label sources.

• Validation on MUStARD++:

  • Models trained on Human-Verified LLM labels (the proposed pipeline) achieved an F1 score of 67.4% (Audio+Text).
  • This was significantly higher than models trained on raw GPT-4o labels alone (61.8% F1), demonstrating that human verification on disagreements is critical for quality.
  • While slightly lower than models trained on original human gold labels (70.8% F1), the pipeline proved viable for creating functional detection models.

• Performance on PodSarc:

  • Multimodal (Audio + Text): The best performance was achieved using human-verified labels, reaching an F1 score of 73.63%.
  • LLM-Only Labels: Models trained solely on LLM-generated labels (without human verification of disagreements) still achieved a strong 71.47% F1, indicating LLMs provide a robust starting point.
  • Modality Comparison: Text-only models generally outperformed audio-only models, but the combination of both modalities yielded the highest accuracy, confirming the necessity of prosodic features for sarcasm detection.

• Dataset Diversity: Analysis showed PodSarc has comparable acoustic and semantic diversity to MUStARD++, covering a wide range of speakers and speech characteristics.

5. Significance and Future Work

• Scalability: This work provides a scalable solution to the "data bottleneck" in speech technology. It allows researchers to create large-scale, domain-specific datasets (like podcasts) without the prohibitive cost of full manual annotation.
• Real-World Applicability: By focusing on audio-only scenarios, PodSarc addresses a gap in current research, enabling the development of sarcasm detection systems for voice assistants, call centers, and radio analysis where visual data is unavailable.
• Future Directions: The authors suggest extending the pipeline to distinguish sarcasm from other non-literal speech (hyperbole, irony), incorporating newer language models for better accuracy, and expanding the dataset to include greater cultural and linguistic diversity.

In conclusion, the paper successfully demonstrates that a hybrid approach combining LLM efficiency with targeted human expertise can generate high-quality datasets that drive state-of-the-art performance in sarcasm detection.