Deep Temporal Modeling and Ensemble Fusion for Multimodal Emotion Recognition from Physiological Signals

Technical Summary: Deep Temporal Modeling and Ensemble Fusion for Multimodal Emotion Recognition

Problem Statement
The accurate, real-time detection of human emotional states and stress from physiological signals is critical for mental health monitoring, wearable computing, and affective computing. While traditional methods rely on subjective self-reports, recent advances in wearable sensors (e.g., EDA, ECG, respiration, accelerometry) enable data-driven inference. However, significant challenges remain in effectively modeling temporal dependencies, handling heterogeneous sensor modalities, and optimizing fusion strategies to ensure generalization across different users. Existing literature often evaluates single architectures or specific fusion strategies in isolation, lacking a unified, controlled comparison of leading temporal deep learning models (LSTM, TCN, Transformer) across unimodal, multimodal, and ensemble settings.

Methodology
The authors present a comprehensive evaluation framework using the WESAD dataset, a multimodal benchmark involving 15 participants under baseline, stress, and amusement conditions. The study employs a Leave-One-Subject-Out Cross-Validation (LOSO-CV) protocol to ensure subject-independent generalization.

• Data Preprocessing: Signals from the Empatica E4 (wrist: EDA, BVP, TEMP, ACC) and RespiBAN (chest: RES, ECG, ACC) were resampled to 4 Hz, segmented into non-overlapping 10-step windows, and normalized per subject to prevent data leakage.
• Model Architectures: Three distinct temporal deep learning architectures were implemented:
  • LSTM: A two-layer bi-directional LSTM designed to capture long-range temporal dependencies and gradual transitions in affective states.
  • TCN: A stack of 1D dilated causal convolutions with residual connections, optimized for parallel processing and multi-scale temporal feature extraction.
  • Transformer: A compact encoder utilizing multi-head self-attention to model dependencies over arbitrary time steps without assuming local structure, capable of focusing on salient temporal segments.
• Fusion Strategies:
  • Early Fusion (Sensor Level): Wrist and chest features were concatenated along the channel dimension before input to the models.
  • Late Fusion (Ensemble Level): An ensemble strategy was implemented by averaging the softmax probability outputs of the three independently trained models (LSTM, TCN, Transformer) to produce a final prediction.
• Ablation Studies: The authors conducted extensive ablation studies by training models on wrist-only and chest-only inputs to quantify the individual contribution of each modality.

Key Results
The experiments yielded the following performance metrics (reported as mean ± standard deviation across LOSO folds):

• Ensemble Performance: The late-fusion ensemble achieved the highest overall performance, reaching 98.91 ± 0.13% accuracy and a 98.56 ± 0.17% macro-F1 score. This approach demonstrated superior stability, with significantly lower variance across subjects compared to individual models.
• Architecture Comparison (Multimodal): In the multimodal setting, the Transformer architecture consistently achieved the highest accuracy among standalone models (99.02 ± 0.51%), attributed to its ability to model complex temporal dependencies via self-attention.
• Modality-Specific Performance:
  • Wrist-Only: The TCN model performed best in the wrist-only configuration (98.20 ± 0.41% accuracy), outperforming both LSTM and Transformer in this specific ablation setting.
  • Chest-Only: The LSTM achieved the strongest results for chest-only inputs, though overall performance was lower than wrist-only configurations.
• Modality Contribution: Multimodal fusion consistently outperformed unimodal configurations across all architectures. However, wrist-only models demonstrated competitive performance, suggesting feasibility for resource-constrained wearable applications. The "Amusement" class consistently yielded lower scores across all models, highlighting the difficulty of classifying this state compared to stress or baseline.

Key Contributions

1. Unified Benchmarking Framework: The paper establishes a systematic, architecture-agnostic comparison of LSTM, TCN, and Transformer models under unimodal, multimodal, and ensemble settings using a consistent LOSO-CV protocol on the WESAD dataset.
2. Dual-Fusion Strategy: The study demonstrates the efficacy of combining early sensor fusion (feature concatenation) with late model ensemble fusion (probability averaging), showing that this dual approach leverages both cross-sensor synergy and variance reduction.
3. Modality and Architecture Insights: The work clarifies that while Transformers excel in multimodal contexts, TCNs and LSTMs remain highly effective in sensor-limited scenarios (wrist-only), providing practical guidance for system design based on available hardware.
4. Robustness Analysis: By reporting standard deviations across 15 LOSO folds, the study provides a reliable estimate of inter-subject variability, demonstrating that ensemble methods significantly reduce variance and improve generalization.

Significance and Claims
The authors claim that their findings offer practical insights for developing robust, scalable emotion recognition systems. Specifically, the paper argues that:

• Ensemble methods are highly effective for physiological emotion recognition, providing superior accuracy and stability compared to single-model approaches.
• Sensor fusion is critical, but wrist-only signals (common in smartwatches) can support accurate recognition, making the technology viable for real-world, non-invasive deployment.
• There is no single "best" architecture for all scenarios; the optimal choice depends on the input modality (e.g., TCN for wrist-only, Transformer for multimodal).
• This work fills a gap in the literature by providing a controlled, comparative analysis of these three major temporal architectures, isolating the effects of architecture choice, sensor modality, and fusion design on affect recognition performance.

The paper concludes that the combination of sensor fusion and ensemble-based fusion is a promising direction for creating reliable systems for physiological stress and emotion detection.