Acoustic-Prosodic Evidence in Multimodal Sarcasm Detection: A Controlled and Interpretable Evaluation of PEFM-CMAE on the Complete MUStARD Corpus

Technical Summary: Acoustic-Prosodic Evidence in Multimodal Sarcasm Detection

Problem Statement
Spoken sarcasm often relies on vocal delivery (prosody) rather than lexical content alone to disambiguate intended meaning. While the importance of prosody is established in linguistic literature, the specific quantitative value of adding acoustic information to strong text-only models remains a subject of debate. Furthermore, it is unclear whether complex fusion mechanisms—specifically those incorporating speech-derived emotion and explicit cross-modal incongruity—provide measurable improvements over simpler, direct text-plus-audio fusion baselines. This study addresses these questions through a controlled evaluation on the complete MUStARD corpus (690 utterances), testing whether acoustic-prosodic features, emotion channels, and explicit incongruity modeling offer reliable gains beyond standard bimodal fusion.

Methodology
The study employs a controlled comparative framework using six distinct model configurations evaluated on identical stratified five-fold cross-validation partitions (repeated across three seeds).

1. Data and Encoders:

   • Corpus: The complete 690-utterance audio-bearing MUStARD corpus, balanced globally (345 sarcastic, 345 non-sarcastic) but unevenly distributed across four source shows (Friends, The Big Bang Theory, The Golden Girls, Sarcasmaholics Anonymous).
   • Text Encoder: Frozen RoBERTa-base, processing target utterances and up to three context turns (max 192 tokens), outputting 768-dimensional [CLS] vectors.
   • Acoustic Encoder: Frozen HuBERT-base, processing 16 kHz audio (capped at 8s), aggregated via mean-pooling to produce 768-dimensional vectors.
   • Emotion Stream: A frozen HuBERT-based speech-emotion recognizer generating a 4-class posterior vector (neutral, happy, angry, sad).

2. Model Architectures:

   • Baselines: Text-only (RoBERTa), Audio-only (HuBERT), and direct Text+Audio fusion.
   • PEFM-AE2: A three-stream model (Text, Audio, Emotion) projected into a shared space with a learned softmax modality gate.
   • PEFM-CMAE (Proposed): An extension of PEFM-AE2 that introduces an explicit Cross-Modal Incongruity block. This block computes an incongruity vector ($i$) from the projected text ($t$) and audio ($a$) representations using the absolute difference ($|t-a|$) and element-wise product ($t \odot a$). This vector is fed into both the modality gate and the final classifier.

3. Training Protocol:

   • Frozen encoders; only projection layers, the incongruity network, the modality gate, and the classifier were trained.
   • Primary metric: Macro-F1.
   • Statistical analysis included paired t-tests and a post-hoc Two One-Sided Tests (TOST) equivalence analysis with a practical bound of ±0.010 macro-F1.
   • Stress testing was performed via "Leave-One-Show-Out" evaluation to assess generalization.

Key Results

• Acoustic Complementarity: The addition of acoustic information significantly improved performance. Audio-only HuBERT (0.7291) outperformed Text-only RoBERTa (0.6909). Direct Text+Audio fusion (0.7479) significantly exceeded Text-only (p = 0.000021).
• PEFM-CMAE Performance: The proposed PEFM-CMAE achieved the highest mean macro-F1 of 0.7504.
  • It significantly outperformed Text-only (p = 0.000002) and the earlier PEFM-AE2 model (0.7325; p = 0.001062).
  • Crucially, the difference between PEFM-CMAE (0.7504) and the simpler direct Text+Audio fusion (0.7479) was not statistically significant (p = 0.688882). Furthermore, TOST analysis failed to establish equivalence, indicating the complex architecture did not demonstrably surpass the simple baseline.
• Modality Gating: The learned gate weights indicated a dominant reliance on text (mean weight ~0.73 in PEFM-CMAE) and a supplementary, non-dominant role for the emotion stream (mean weight ~0.033). The emotion channel did not provide a reliable aggregate performance gain over direct fusion.
• Generalization: Leave-one-show-out performance dropped materially (e.g., to 0.6713 on Friends and 0.6426 on The Big Bang Theory), highlighting a lack of robust generalization to unseen shows, likely due to confounding factors of speaker identity, script style, and label priors.

Key Contributions

1. Quantified Acoustic Value: Demonstrated that under a controlled protocol, HuBERT-derived acoustic features significantly improve sarcasm detection over text-only models.
2. Controlled Comparison of Fusion Mechanisms: Provided a rigorous, paired comparison showing that while explicit incongruity modeling improves upon the earlier gated formulation (PEFM-AE2), it does not yield a statistically significant advantage over direct text-plus-audio fusion.
3. Interpretability: Made the incongruity computation and modality allocation explicit, revealing that the emotion stream acts as a supplementary conditioner rather than a primary predictor.
4. Uncertainty and Generalization Analysis: Separated within-corpus performance from generalization evidence, explicitly reporting the failure of the model to generalize robustly to unseen shows and the limitations of the current dataset composition.

Significance and Claims
The paper modestly claims that acoustic-prosodic information provides reliable complementary evidence for sarcasm detection within the MUStARD corpus. However, it explicitly rejects the claim that a complex architecture involving explicit incongruity modeling and adaptive gating decisively surpasses simple direct fusion.

The study concludes that while the specific PEFM-CMAE design improves upon the earlier PEFM-AE2 gated formulation, the marginal gain over a simple text-plus-audio baseline is negligible and not statistically significant. The primary value of the work lies in its controlled, interpretable evaluation which isolates the contribution of acoustic features from the complexity of fusion mechanisms, and in its clear demonstration that robust cross-show generalization remains an unresolved challenge for current multimodal sarcasm detection systems.