Technical Summary: Attention Isn't All You Need for Emotion Recognition

Problem Statement

This study addresses the challenge of multimodal emotion recognition using the EAV dataset, which contains synchronized EEG, audio, and video recordings from 42 participants engaged in conversational interactions. While the dataset offers a unique resource for multimodal research, it presents a significant constraint for deep learning: after standard data splitting, there are only approximately 280 training samples per subject.

The central research question investigates whether complex attention mechanisms (specifically factorized attention) improve performance on such small datasets, or if simpler approaches leveraging domain-appropriate features yield superior results. The authors hypothesize that architectural complexity may lead to overfitting and the degradation of pretrained features in low-data regimes.

Methodology

The authors systematically evaluated three categories of models across the three modalities (EEG, Audio, Vision):

1. M1: Baseline Transformers

The authors implemented standard transformer architectures using pretrained backbones from Hugging Face to establish reference performance:

• EEG: A custom architecture with 1D convolutional feature extraction followed by a 6-layer transformer encoder with 4 attention heads.
• Audio: Fine-tuning of the Audio Spectrogram Transformer (AST) with a custom 5-class classifier head, utilizing a two-stage training strategy (frozen backbone followed by unfrozen).
• Vision: A Vision Transformer (ViT) pretrained specifically on facial emotion recognition, processing 25 frames per video segment with averaged predictions.

2. M2: Factorized Attention Mechanisms

Motivated by the unique structures of each modality, the authors designed novel factorized attention models intended to outperform baselines:

• M2-EEG (Tri-Stream): A transformer branching into three parallel streams: spatial (channel interactions), temporal (time dynamics), and hemispheric asymmetry (left-right differences based on Davidson's model). It includes a learnable skip connection to preserve convolutional features.
• M2-EEG (EEGNet-Improved): An enhanced EEGNet incorporating squeeze-excitation blocks, multi-scale convolutions, and advanced regularization techniques.
• M2-Audio: A Temporal-Frequency Dual Attention mechanism applied to AST features. It reshapes patches into a 2D grid (time × frequency) and applies sequential temporal and frequency attention blocks, preserving ~88% of pretrained features via a skip connection.
• M2-Vision: A Factorized Space-Time Attention model (following ViViT Model 2) that decomposes full space-time attention into spatial attention within frames and temporal attention across frames to reduce computational complexity.

3. M3: Improved CNN Baselines

Following the underperformance of M2, the authors explored minimal modifications to baseline CNNs focusing on bug fixes and domain-specific feature engineering:

• Audio: Augmented the feature set with delta MFCCs (temporal derivatives) to capture short-term spectral dynamics, increasing input dimensionality from 180 to 220.
• EEG:
  • v1 (Bug Fix): Removed an erroneous nn.Softmax layer applied before CrossEntropyLoss in the original implementation.
  • v2 (Band Power): Replaced raw time-domain signals with frequency-domain features, including band power (delta, theta, alpha, beta, gamma), differential entropy, and explicit alpha asymmetry features (306 dimensions total), fed into a simple MLP.
• Vision:
  • v1 (SE Fix): Corrected the Squeeze-and-Excitation (SE) block reduction ratio from 1 (no bottleneck) to 16 to ensure effective channel reweighting.
  • v2 (Delta Features): Added temporal delta features to capture frame-to-frame expression dynamics.

Key Results

The experiments, conducted on NVIDIA RTX 6000 Ada GPUs with subject-dependent evaluation, yielded the following findings (Table 1 in the paper):

• M2 Underperformance: All factorized attention models (M2) consistently underperformed their respective baselines by 5 to 13 percentage points.

  • EEG Tri-Stream: 48.00% (vs. 60.00% paper baseline).
  • Audio Dual-Attention: 49.46% (vs. 62.70% paper baseline).
  • Vision ViViT-style: 69.54% (vs. 74.50% paper baseline).
  • The authors attribute this to overfitting, the destruction of pretrained features, and the inability of complex inductive biases to generalize with ~280 samples.

• M3 Success: Simple, domain-appropriate modifications achieved state-of-the-art results on the dataset:

  • EEG: Band power features achieved 67.62% (+7.62pp over the paper baseline).
  • Audio: Delta MFCCs achieved 65.56% (+3.66pp over the paper baseline).
  • Vision: Delta features achieved 72.68% (+1.28pp over the paper CNN).
  • Notably, the M1 Vision Transformer (75.30%) outperformed all M3 CNN modifications, demonstrating that domain-specific pretraining can be more impactful than architectural changes.

• Subject Variability: Significant individual differences were observed. Subject 17 consistently achieved the highest accuracy across modalities (up to 93.33% for vision), while Subjects 35 and 12 presented persistent challenges, likely due to subtle expressions or physiological differences.

Significance and Claims

The paper concludes that for small-scale emotion recognition tasks (approx. 280 training samples per subject), domain knowledge and proper implementation outperform architectural complexity.

The authors claim three primary contributions:

1. Demonstration of Failure: They show that factorized attention mechanisms, despite being theoretically motivated, consistently fail on small datasets, underperforming simpler baselines.
2. Validation of Simple Features: They demonstrate that simple, domain-appropriate modifications (specifically frequency-domain features for EEG and delta coefficients for audio/video) achieve the best results on this dataset.
3. Methodological Analysis: They provide an analysis of failure modes, identifying that the degradation of pretrained representations, insufficient training data for complex inductive biases, and implementation bugs (e.g., double softmax) are critical factors in low-sample regimes.

The practical lesson emphasized is that researchers should prioritize verifying baseline implementations for bugs, applying domain-specific feature engineering, and matching model capacity to available data before introducing complex attention mechanisms. The authors note that these factorized approaches may still be beneficial on larger emotion datasets where sufficient supervision exists to learn stable fusion.