Speech Emotion Recognition Using MFCC Features and LSTM-Based Deep Learning Model

1. Problem Statement

Speech Emotion Recognition (SER) aims to enable machines to detect human emotional states based solely on speech data. While human communication relies heavily on vocal cues (pitch, rhythm, intensity) to convey emotion, most computational systems focus only on the semantic content (words), ignoring the emotive context. This limitation hinders the effectiveness of intelligent systems in domains like virtual assistants, mental health monitoring, and customer service.

The core challenges in SER include:

• Speaker Variability: Differences in voice, accent, and speaking style.
• Contextual Variability: Changes in recording environments and situations.
• Acoustic Similarity: Certain emotions (e.g., fear vs. anger) share similar acoustic patterns, making them difficult to distinguish.
• Temporal Dynamics: Emotions evolve over time within an utterance, requiring models that can capture long-term dependencies rather than just static snapshots.

2. Methodology

The proposed system utilizes a Mel-Frequency Cepstral Coefficient (MFCC) feature extraction method combined with a Long Short-Term Memory (LSTM) neural network.

A. Dataset

• Source: Toronto Emotional Speech Set (TESS).
• Composition: 2,800 audio recordings by two professional female actors.
• Classes: Seven emotional categories: Angry, Disgust, Fear, Happy, Pleasant Surprise, Sad, and Neutral.
• Distribution: Balanced class distribution (200 words per actor per emotion).

B. Data Preprocessing

1. Standardization: Audio files were trimmed to a fixed duration of 3 seconds.
2. Sampling: Resampled to 22,050 Hz.
3. Visualization: Waveforms and spectrograms were generated for exploratory data analysis (EDA) to observe temporal and spectral differences between emotions.
4. Feature Extraction:
   • MFCCs: Extracted using n_fft=2048, hop_length=512, and 40 coefficients.
   • Input Format: Each utterance was converted into a sequence of 40-dimensional vectors, resulting in an input tensor of shape (N, t, 40), where N is the number of samples and t is the time steps.
5. Label Encoding: Emotion labels were converted to one-hot encoded categorical vectors.

C. Model Architecture

The study proposes a unidirectional LSTM architecture designed for simplicity and efficiency:

• Input Layer: Accepts the MFCC time-series sequence.
• LSTM Layer: Contains 128 hidden units. It utilizes gating mechanisms (forget, input, and output gates) to manage the cell state, allowing the model to retain relevant emotional patterns over time while discarding noise. This addresses the vanishing gradient problem common in standard RNNs.
• Dense Layers: Followed by Fully Connected (Dense) layers with ReLU activation to introduce non-linearity.
• Regularization: Dropout layers (rate=0.2) are applied to prevent overfitting.
• Output Layer: A Dense layer with Softmax activation to output probability distributions across the 7 emotion classes.
• Loss Function: Categorical Cross-Entropy.
• Optimizer: Adam (with adaptive learning rate).

D. Baseline Comparison

To validate the LSTM approach, a classical Support Vector Machine (SVM) with an RBF kernel was implemented as a baseline.

• SVM Input: Instead of the full time-series, the SVM used a static 40-dimensional feature vector created by averaging the MFCCs across the time dimension.

3. Key Contributions

• Temporal Modeling: Demonstrated that modeling the temporal dynamics of speech using LSTM yields higher accuracy than static feature averaging (SVM), proving that emotional cues unfold over time.
• Simplified Architecture: Proposed a lightweight, single-stream LSTM model that achieves state-of-the-art results without the computational complexity of hybrid CNN-LSTM models or multi-branch architectures.
• Benchmarking: Provided a rigorous comparison between a deep learning temporal model and a classical machine learning baseline on the TESS dataset.

4. Results

The model was trained for 100 epochs with a batch size of 512 on an 80/20 stratified train-test split.

• Performance Metrics:
  • SVM Baseline: Achieved 98.00% accuracy.
  • Proposed LSTM: Achieved 99.00% accuracy on the test set.
• Training Dynamics:
  • Training Accuracy: 99.56%
  • Validation Accuracy: 99.82%
  • Validation Loss: 0.0036 (indicating minimal overfitting).
• Confusion Matrix Analysis:
  • The matrix showed strong diagonal dominance, indicating near-perfect classification for most classes (e.g., 80/80 correct predictions for several emotions).
  • Misclassifications were rare and primarily occurred between emotionally similar classes (e.g., Fear vs. Sad), which is a known challenge in SER.

5. Significance and Applications

Significance:
The study confirms that LSTM-based architectures are highly effective for SER tasks, even when using a single feature type (MFCC). The 1% improvement over the SVM baseline highlights the value of capturing sequential dependencies in speech. The high accuracy and low computational cost suggest that deep learning models can be deployed efficiently.

Practical Applications:

• Virtual Assistants: Enhancing responsiveness by detecting user frustration or satisfaction.
• Mental Health Monitoring: Detecting signs of depression or anxiety through speech patterns.
• Customer Service Analytics: Automating sentiment analysis in call centers.
• Human-Robot Interaction: Enabling social robots to react appropriately to human emotional states.

Limitations & Future Work:

• Limitations: The model relies on clean, acted speech (TESS dataset) and may struggle with real-world noise, speaker variability (gender/age), and spontaneous speech. The architecture is limited to MFCCs and a single LSTM layer.
• Future Directions:
  • Testing on diverse, real-world datasets with noise and multiple speakers.
  • Incorporating additional features (pitch, energy, formants).
  • Exploring bidirectional LSTMs, attention mechanisms, or hybrid CNN-LSTM models.
  • Deploying the system in real-time end-to-end applications.