Benchmarking PyCaret AutoML Against BiLSTM for Fine-Grained Emotion Classification: A Comparative Study on 20-Class Emotion Detection

1. Problem Statement

The paper addresses the challenge of fine-grained emotion classification in Natural Language Processing (NLP). Unlike binary sentiment analysis (positive/negative), this task requires distinguishing between 20 distinct and nuanced emotional states (e.g., happiness, anxiety, nostalgia, guilt). The core problem investigated is the comparative efficacy of traditional Machine Learning (ML) approaches, which rely on handcrafted features, versus Deep Learning (DL) approaches, which learn representations directly from raw text, specifically for a large-scale multi-class problem (20 classes) with a dataset of ~80,000 samples.

2. Methodology

The study employs a systematic benchmarking framework comparing three ML classifiers and three DL architectures under identical experimental conditions.

• Dataset: The 20-Emotion Text Classification Dataset, containing 79,595 English sentences labeled with one of 20 emotion categories.

• Preprocessing:

  • Text cleaning (lowercasing, punctuation removal).
  • ML Pipeline: Stopword removal and tokenization. Features are extracted using TF-IDF (Term Frequency-Inverse Document Frequency) with a vocabulary size of 50,000.
  • DL Pipeline: Tokenization with a vocabulary of 30,000 and an embedding dimension of 128.
  • Data Split: 80% Training, 10% Validation, 10% Test (stratified).

• Models Evaluated:

  • Machine Learning (ML):
    1. Logistic Regression (LR): Linear classifier with L2 regularization.
    2. Multinomial Naive Bayes (MNB): Probabilistic classifier with Laplace smoothing.
    3. Support Vector Machine (LinearSVC): Linear SVM optimized with squared hinge loss.
    • Note: These were evaluated using an AutoML approach (referenced as PyCaret in the title, though the text details specific scikit-learn implementations).
  • Deep Learning (DL): Implemented in PyTorch on GPU (NVIDIA Tesla T4).
    1. BiLSTM: Bidirectional Long Short-Term Memory with two stacked layers (hidden size 256).
    2. GRU: Gated Recurrent Unit with two stacked bidirectional layers (hidden size 256).
    3. Transformer: A lightweight encoder-only model (4 attention heads, 2 layers, 512 feed-forward dimension).
  • Hyperparameters: All DL models used Adam optimizer (lr=0.001), dropout (0.3), batch size 64, and trained for 10 epochs.

3. Key Contributions

1. Systematic Benchmark: A direct comparison of ML and DL paradigms on a specific 20-class emotion detection task, a gap noted in existing literature.
2. Comprehensive Evaluation: Analysis of six distinct models (3 ML, 3 DL) focusing on accuracy, F1-scores, and computational efficiency (training time).
3. Trade-off Analysis: Detailed examination of the balance between model performance, training infrastructure requirements (CPU vs. GPU), and computational complexity.
4. Deployment: The best-performing ML and DL models were deployed as interactive web applications on Hugging Face Spaces.

4. Results

The experiments yielded the following performance metrics on the test set:

• Top Performer: The BiLSTM model achieved the highest overall performance with 89% accuracy and a 0.89 weighted F1-score.
• ML Performance: The SVM model was the strongest ML contender, achieving 88.11% accuracy, which is remarkably close to the best DL model.
• DL Variants: The GRU performed comparably to BiLSTM but with fewer parameters. The lightweight Transformer underperformed relative to recurrent models, likely due to the dataset size being insufficient to fully leverage self-attention mechanisms without pre-training.

5. Significance and Conclusion

• Contextual Advantage: The study confirms that sequence-based deep learning models (specifically BiLSTM) better capture contextual emotional cues and word order dependencies than bag-of-words TF-IDF features, resulting in a marginal but consistent performance gain (~0.9% over SVM).
• Competitiveness of Classical ML: Despite the rise of DL, traditional ML models (specifically SVM) remain highly competitive. They offer a computationally efficient alternative that requires no GPU acceleration and trains almost instantaneously, making them ideal for resource-constrained environments.
• Practical Implications: For fine-grained emotion detection, BiLSTM offers the best balance of accuracy and stability. However, if computational resources are limited, SVM provides a robust, lightweight alternative with negligible performance loss.
• Future Directions: The authors suggest that Transformer models could outperform recurrent models if trained on larger datasets or with pre-trained representations (e.g., BERT, RoBERTa), and recommend exploring data augmentation for underrepresented emotion classes.