A Baseline Multimodal Approach to Emotion Recognition in Conversations

Technical Summary: A Baseline Multimodal Approach to Emotion Recognition in Conversations

Problem Statement
Emotion recognition in conversations (ERC) is a critical task in affective computing, yet traditional unimodal approaches (relying solely on text or audio) often fail to capture the nuanced interplay between verbal and non-verbal cues. Human emotions are dynamic and context-dependent; for instance, sarcasm or suppressed emotions may be conveyed through vocal tone contradicting semantic content. While recent research advocates for multimodal integration, there is a need for accessible, transparent reference implementations that establish a baseline for future comparisons, particularly on the SemEval-2024 Task 3 dataset derived from the Friends sitcom.

Methodology
The authors present a lightweight, multimodal baseline system designed for transparency and reproducibility rather than state-of-the-art performance. The methodology consists of three primary components:

1. Dataset and Preprocessing: The study utilizes the SemEval-2024 Task 3 dataset, which contains dialogues from Friends annotated with cause-emotion pairs. Preprocessing involved aligning textual and audio data for each utterance and standardizing audio signals to a 16kHz sampling rate. The dataset was split into 80% training and 20% testing.
2. Model Architecture (Unimodal Baselines):
   • Text Modality: Four transformer-based models were fine-tuned: RoBERTa, DistilBERT, DeBERTa, and DistilRoBERTa (specifically a sentiment-finetuned variant). These models utilized pre-trained embeddings to capture semantic and contextual nuances.
   • Audio Modality: Three self-supervised speech representation models were employed: HuBERT, Wav2Vec2, and Wav2Vec2-large-robust. These models encode temporal and spectral features to extract paralinguistic cues such as tone and intonation.
3. Fusion Strategy: The system employs a late-fusion ensemble approach. Predictions from the best-performing text model (RoBERTa) and the best-performing audio model (Wav2Vec2) were combined to generate the final classification. This strategy leverages the complementary strengths of semantic understanding and vocal cues without requiring complex intermediate alignment.

Key Contributions

• Reference Implementation: The paper provides a documented, accessible baseline for ERC using the SemEval-2024 Task 3 dataset, explicitly avoiding novel architectural proposals in favor of a "straightforward" setup.
• Empirical Comparison: It reports empirical results comparing unimodal text models, unimodal audio models, and a multimodal ensemble under a limited training protocol (minimal hyperparameter tuning).
• Transparency: The authors clearly delineate the limitations of their study, framing the results as a reference point rather than a definitive benchmark. Code and materials are made publicly available to support future rigorous comparisons.

Results
The study evaluated models based on accuracy and execution time (wall-clock):

• Text Models: RoBERTa achieved the highest accuracy among text models at 50.68%, followed by DistilBERT (49.82%), DeBERTa (48.29%), and DistilRoBERTa (41.83%). RoBERTa also required the most execution time (74.6s), while DistilRoBERTa was the fastest (5.61s) but least accurate.
• Audio Models: Audio-only models performed significantly lower than text models. Wav2Vec2 achieved the highest audio accuracy at 35.43%, followed by Wav2Vec2-large-robust (32.54%) and HuBERT (31.08%).
• Ensemble Model: The late-fusion ensemble combining RoBERTa and Wav2Vec2 achieved an accuracy of 62.97%. This result significantly outperformed both individual unimodal baselines, demonstrating the efficacy of integrating textual and audio modalities.

Significance and Claims
The authors position this work as a baseline technical report derived from an undergraduate final-year project and an internal replication effort. They explicitly state that the system is not intended to be state-of-the-art. The significance of the paper lies in:

1. Establishing a Reference Point: Providing a clear, reproducible benchmark for the SemEval-2024 Task 3 dataset to facilitate future comparisons.
2. Validating Multimodal Fusion: Demonstrating that even with a simple late-fusion strategy and limited tuning, multimodal integration (62.97%) substantially improves upon unimodal performance (max 50.68% for text, 35.43% for audio).
3. Highlighting Limitations: The authors caution that their results are not directly comparable to official SemEval leaderboard submissions due to differences in data splits and training protocols. They note that the dataset's origin (Friends) may limit generalizability to other domains, languages, or interaction styles, and that the study excludes visual modalities.

In conclusion, the paper affirms the potential of multimodal systems for emotion recognition while maintaining a modest stance, urging future research to address dataset biases, incorporate additional modalities (e.g., visual, physiological), and develop more efficient methods for real-time application.