An Adaptive Cross Attention Multimodal Transformer Architecture for Multilingual Speech Emotion Recognition

Technical Summary: Adaptive Cross-Attention Multimodal Transformer for Multilingual Speech Emotion Recognition

Problem Statement
Current Speech Emotion Recognition (SER) systems face three primary limitations. First, many high-performing systems are unimodal, discarding valuable lexical and visual evidence. Second, existing multimodal systems often rely on static fusion methods (e.g., concatenation or fixed-weight averaging) that fail to adapt to instance-dependent modality reliability, where the trustworthiness of acoustic, textual, or visual signals varies by utterance, speaker, or language. Third, the field remains predominantly English-centric, with multilingual and cross-lingual SER being under-explored, particularly regarding parameter-efficient adaptation. Furthermore, standard evaluation protocols often fail to distinguish between within-corpus performance and genuine cross-corpus or cross-lingual generalization.

Methodology: The ACAMT Architecture
The paper proposes the Adaptive Cross-Attention Multimodal Transformer (ACAMT), a framework designed to integrate acoustic, textual, and visual modalities through a dynamic, bidirectional mechanism.

• Feature Extraction:

  • Acoustic: Combines handcrafted features (40-dim MFCCs with deltas, 128-band log-mel spectrograms) with self-supervised embeddings from a pretrained Wav2Vec 2.0 (facebook/wav2vec2-large-xlsr-53) model. The acoustic stream utilizes a trainable weighted combination of all transformer layers.
  • Textual: Utilizes XLM-RoBERTa-base, a multilingual transformer encoder pretrained on over 100 languages, to process transcripts. For audio-only corpora, transcripts are generated via Whisper large-v3.
  • Visual: Employs a ResNet-18 backbone (pretrained on VGGFace2) followed by a bidirectional GRU to capture facial expression dynamics (onset–apex–offset).

• Bidirectional Cross-Attention Core:
  Unlike prior two-modality systems, ACAMT computes six directional cross-attention maps covering all pairs of the three modalities (Acoustic↔Text, Text↔Visual, Visual↔Acoustic).

  • Because cross-attention outputs retain the query sequence's length (e.g., $A \to T$ has acoustic length, $T \to A$ has textual length), the architecture concatenates the two directional outputs for each pair and projects them to a fixed dimension $d=256$.
  • These are then mean-pooled over time to create three fixed-length pair vectors: $z_{AT}$, $z_{TV}$, and $z_{VA}$.

• Masked Adaptive Gating:
  The three pair vectors are fused via a masked, per-instance adaptive gate.

  • Missing Modality Handling: If a modality is unavailable for a specific corpus or utterance (e.g., video for TESS or EMO-DB), the gate assigns $-\infty$ logits to any pair vector dependent on that missing modality. This ensures the unavailable pair receives exactly zero weight after the softmax operation, rather than relying on implicit redistribution.
  • Modality Dropout: During training, a randomly chosen available modality is masked with probability $p=0.15$ to ensure the gate learns to redistribute weights robustly even when a normally available modality is absent at test time.

• Training Strategy:
  The model employs a two-stage protocol:

  1. Pretraining: Joint training on the pooled six-corpus dataset to learn shared acoustic and fusion parameters.
  2. Fine-tuning: Low-Rank Adaptation (LoRA) is applied to the query and value projection matrices of the Wav2Vec 2.0, XLM-R, and cross-attention blocks (rank $r=8$, scaling $\alpha=16$). All other backbone parameters remain frozen, targeting a trainable parameter budget of roughly 0.5–1% of the full model.

Evaluation Protocol
The paper defines a rigorous, four-regime evaluation protocol to test generalization:

1. Within-Corpus: Speaker-independent 5-fold cross-validation (2-fold for TESS due to only 2 speakers).
2. Pooled-Multilingual: 5-fold CV over the harmonized six-corpus set.
3. Leave-One-Corpus-Out (LOCO): Training on five corpora and testing on the sixth to measure cross-corpus generalization.
4. Leave-One-Language-Out (LOLO): Training on two languages and testing on the third to measure cross-lingual transfer.

Metrics include Accuracy, Precision, Recall, F1, ROC-AUC, Matthews Correlation Coefficient (MCC), and Cohen's Kappa.

Results and Empirical Status
The paper explicitly states that full empirical training has not been completed. Due to a lack of GPU access and source dataset availability in the authoring environment at the time of writing, no measured accuracy figures or performance results are reported.

The manuscript presents the architecture, a corrected mathematical formulation, and a fully specified evaluation protocol. It positions the proposed method against prior work (e.g., MemoCMT, CNN-LSTM hybrids) in terms of architectural scope (tri-modal, bidirectional, missing-modality aware) but does not claim performance superiority, as the empirical validation is identified as the immediate next step.

Significance and Contributions
The authors position the significance of this work not in the invention of cross-attention or adaptive gating individually—both of which have been demonstrated in prior two-modality or unimodal settings—but in their joint instantiation for a tri-modal, multilingual context.

Key contributions include:

1. Tri-Modal Bidirectional Fusion: A mathematically consistent formulation for six-directional cross-attention that resolves dimensionality mismatches between variable-length sequences.
2. Explicit Missing-Modality Handling: A gating mechanism that guarantees the exclusion of unavailable modality pairs via masking, rather than assuming implicit redistribution.
3. Parameter-Efficient Multilingual Adaptation: A specific LoRA-based fine-tuning protocol designed to handle low-resource languages (German, Italian) alongside English without retraining massive backbones.
4. Comprehensive Evaluation Design: A four-regime protocol that separates within-corpus performance from cross-corpus and cross-lingual generalization, addressing a gap in current literature where evaluation often conflates these distinct capabilities.

The paper concludes by outlining future work, including extending language coverage to typologically distinct low-resource languages, model compression for on-device inference, fairness auditing, and extending the framework to continuous regression tasks.