A Comparison of Traditional Machine Learning Algorithms and LSTM-Based Deep Learning Models for Email Sentiment Analysis

Technical Summary: A Comparison of Traditional Machine Learning Algorithms and LSTM-Based Deep Learning Models for Email Sentiment Analysis

Problem Statement
The rapid expansion of electronic communication has increased the vulnerability of email systems to spam, phishing, and malware, necessitating robust automated classification systems. While Natural Language Processing (NLP) offers solutions, there is a need to determine the optimal balance between predictive precision and computational efficiency when applying machine learning (ML) versus deep learning (DL) architectures to email sentiment analysis. Specifically, this study addresses the classification of emails into "spam" and "non-spam" (ham) categories within the Indonesian language context, evaluating whether traditional discriminative models can outperform or match the capabilities of sequence-based deep learning models like Long Short-Term Memory (LSTM) when utilizing Word2Vec feature representations.

Methodology
The study utilized a dataset of 2,620 Indonesian email messages (1,363 spam, 1,258 ham) sourced from Kaggle. The experimental pipeline involved the following stages:

1. Preprocessing: Text data underwent a rigorous cleaning process including case folding (lowercasing), URL and email attribute removal, character filtering (retaining only Indonesian alphabetic letters), and whitespace trimming. Stopword removal was performed using the Sastrawi library to eliminate non-semantic tokens.
2. Feature Extraction:
   • Machine Learning Models: Cleaned text was tokenized and converted into dense vectors using Word2Vec. These word vectors were averaged to create a single document vector for each email, serving as input for classical classifiers.
   • Deep Learning Model: Tokens were mapped to integer IDs from a training-only vocabulary, padded or truncated to a fixed sequence length of 50. A trainable embedding layer (dimension 64) mapped these IDs to dense vectors.
3. Model Architectures:
   • Traditional ML: Three algorithms were tested: Support Vector Machine (SVM) with a linear kernel, Logistic Regression, and Naive Bayes.
   • Deep Learning: A single-layer bidirectional-off LSTM network was implemented. The architecture included an embedding layer, an LSTM layer (hidden size 64), sequence pooling (using the final timestep's hidden state), and a linear output layer with a Sigmoid activation for binary classification.
4. Training and Evaluation: The dataset was split into 80% training and 20% testing. The LSTM model was trained for 30 epochs using the Adam optimizer and Binary Cross-Entropy Loss. Performance was evaluated using Accuracy, Precision, Recall, F1-Score, AUC, and training time (TT).

Key Results
The experimental results highlighted distinct performance characteristics between the traditional and deep learning approaches:

• Support Vector Machine (SVM): The SVM model with a linear kernel achieved the highest overall performance, recording an accuracy of 98.74% and an F1-score of 98.74%. It demonstrated exceptional efficiency with a training time of only 0.908 seconds. The confusion matrix indicated flawless generalization on the test set, with zero misclassifications (0 False Positives and 0 False Negatives).
• Logistic Regression: This model served as a strong alternative with 97.53% accuracy but required significantly more computational time (2.695 seconds) compared to the SVM.
• Naive Bayes: This algorithm yielded the lowest performance among the traditional models, with 94.49% accuracy, suggesting it is less suited for the dense, continuous vector spaces generated by Word2Vec in this context.
• LSTM (Deep Learning): The LSTM model achieved an overall accuracy of 97% with a macro/weighted average F1-score of 0.97. It demonstrated superior recall (0.99) for the "Spam" class, effectively minimizing false negatives (only 4 instances of spam missed). However, it incurred a higher rate of false positives (10 legitimate emails flagged as spam) compared to the SVM. The training process required significantly more time and computational resources than the discriminative statistical models.

Significance and Claims
The paper concludes that for email detection tasks utilizing dense Word2Vec embeddings, traditional discriminative models, specifically the SVM with a linear kernel, offer the most optimal balance between predictive precision and processing speed.

While the study acknowledges that the LSTM model demonstrates exceptional recall capabilities in detecting spam-related sentiments and effectively captures long-range dependencies, it did not surpass the efficiency and precision of the SVM in this specific vector space configuration. The authors assert that traditional classifiers remain highly robust for dense vector spaces and provide a more practical solution for developing high-performance automated email filtering systems in professional and academic environments where processing speed and accuracy are critical. The findings suggest that for this specific domain and feature representation, the added complexity of deep learning does not yield a performance advantage sufficient to justify the increased computational cost over well-tuned traditional machine learning algorithms.