Dynamic Stress Detection: A Study of Temporal Progression Modelling of Stress in Speech

1. Problem Statement

Current speech-based stress detection systems predominantly treat stress as a static feature, assigning a single label to an entire speech segment. This approach oversimplifies the nature of stress, which is inherently a temporally evolving phenomenon influenced by historical emotional states and prior stress levels.

• Limitations of Existing Work: Most datasets (e.g., StressID, MuSE) lack fine-grained, temporally dynamic stress annotations, forcing models to rely on coarse, static labels.
• Gap: There is a lack of frameworks that explicitly model the progression of stress over time using sequential dependencies and dynamically derived labels.

2. Methodology

The authors propose a novel framework that models stress as a dynamic construct using three core components: a Dynamic Labelling Strategy, Temporal Feature Extraction, and Sequential Architectures.

A. Dynamic Labelling Strategy (Stress Progression)

Since many datasets lack continuous stress labels, the authors derive proxy stress labels from temporally annotated emotional states using a distance-based relabelling strategy:

1. Binary VAD Encoding: Emotions are encoded using the Valence-Arousal-Dominance (VAD) framework, discretized into binary values (0 or 1). Stress is represented as a specific canonical VAD vector: (0, 1, 0) (Low Valence, High Arousal, Low Dominance).
2. Hamming Distance: For each time window $t$, the Hamming distance ($D_t$) is calculated between the current emotion's VAD encoding and the canonical stress encoding.
3. Temporal Weighting: A decaying weight $\delta_{t'} = e^{-\lambda(t-t')}$ is applied to previous windows to account for the fading influence of past states.
4. Weighted Distance Calculation: The total weighted distance ($\theta_{total}$) is computed by summing the weighted distances of the previous $n$ windows.
5. Label Assignment: If $\theta_{total}$ is below a threshold $T_{stress}$, the segment is labeled as "Stress"; otherwise, it retains its original emotional label. This creates a continuous stream of stress labels for training.

B. Data Preprocessing & Augmentation

• Segmentation: Continuous speech is divided into overlapping 10-second windows (5-second overlap).
• Augmentation: For short datasets (e.g., CREMA-D, RAVDESS), utterances from the same speaker with different emotional states are concatenated to simulate temporal progression and emotional transitions.
• Feature Extraction: The study compares three feature sets:
  • MFCC: Traditional handcrafted features.
  • Wav2Vec 2.0: Pretrained self-supervised embeddings.
  • HuBERT: Pretrained embeddings known for capturing higher-level semantics.

C. Model Architectures

Two sequence-based models are designed to capture dependencies between acoustic features and stress history. Both utilize a Cross-Attention mechanism to condition current predictions on past stress states.

1. Unidirectional LSTM: Two parallel LSTM layers (one for speech, one for stress history) feed into a cross-attention mechanism, followed by a fully connected classification layer.
2. Transformer Encoder: Speech features pass through a Transformer encoder, while the stress context is encoded via a BERT-based encoder. A cross-attention block fuses these representations to capture long-range dependencies.
3. Training Strategy: The models use Probabilistic Teacher Forcing (80% ground truth, 20% past predictions) to bridge the gap between training (where ground truth history is available) and inference (where it is not).

3. Key Contributions

1. Stress Progression Labelling Framework: A novel method to infer fine-grained, temporally dynamic stress labels from static emotion annotations using VAD-based distance metrics and temporal decay.
2. Temporal Stress Classification Models: Implementation of Unidirectional LSTM and Transformer Encoder architectures enhanced with cross-attention to explicitly model the interplay between acoustic cues and historical stress states.
3. Comprehensive Evaluation: Validation across multiple datasets (MuSE, StressID) and a custom real-world maritime dataset (10 professionals, 45-minute sessions with EEG ground truth), demonstrating the generalizability of the approach.

4. Results

The proposed dynamic models significantly outperform static baselines across all tested datasets:

• MuSE Dataset:
  • Transformer + HuBERT: Achieved 83% Accuracy and 81% F1-score.
  • Improvement: +5% accuracy over the best baseline (MUSER Acoustic Encoder).
• StressID Dataset:
  • Transformer + HuBERT: Achieved 78% Accuracy and 80% F1-score.
  • Improvement: +18% accuracy over the best baseline (Wav2Vec 2.0 Classifier).
• Custom Maritime Dataset:
  • Transformer + HuBERT: Achieved 81% Accuracy and 82% F1-score.
  • LSTM + Wav2Vec 2.0: Achieved 80% Accuracy.
• Ablation Findings:
  • Context Window ($n$): Optimal performance varied by dataset ($n=4$ for MuSE/Custom, $n=3$ for StressID), indicating that the optimal historical context depends on the task type (conversational vs. short-task).
  • Features: Pretrained models (Wav2Vec 2.0, HuBERT) consistently outperformed MFCCs. HuBERT paired best with Transformers, while Wav2Vec 2.0 paired best with LSTMs.

5. Significance

• Paradigm Shift: The study validates the hypothesis that stress is not instantaneous but evolves over time. Modeling this temporal progression yields significant accuracy gains.
• Real-World Applicability: The successful application to a custom maritime dataset (simulated emergencies with EEG ground truth) proves the framework's viability in high-stakes, non-intrusive occupational monitoring.
• Methodological Innovation: The distance-based relabelling strategy provides a scalable solution for leveraging existing emotion datasets to train stress detection models without requiring new, expensive stress-labeled data collection.
• Future Directions: The work suggests that future systems should adapt temporal window lengths to specific contexts and explore multimodal integration (e.g., combining speech with physiological signals) for even greater robustness.