💬 NLP

Benchmarking Logistic Regression, SVM, and LightGBM Against BiLSTM with Attention for Sentiment Analysis on Indonesian Product Reviews

This paper benchmarks traditional machine learning algorithms (Logistic Regression, SVM, and LightGBM) against a BiLSTM with Attention model for Indonesian sentiment analysis, finding that the best-performing ML model (Logistic Regression) slightly outperforms the deep learning approach while offering greater computational efficiency.

Original authors: Razin Hafid Hamdi, Ivana Margareth Hutabarat, Hanna Gresia Sinaga, Luluk Muthoharoh, Ardika Satria, Martin C. T. Manullang

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Razin Hafid Hamdi, Ivana Margareth Hutabarat, Hanna Gresia Sinaga, Luluk Muthoharoh, Ardika Satria, Martin C. T. Manullang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive online store in Indonesia. Every day, thousands of customers leave reviews about your products. Some say, "This is amazing!" and others say, "This is terrible!" Reading every single one manually is like trying to drink from a firehose—it's impossible, slow, and you might get tired and make mistakes.

To solve this, the authors of this paper built two different "digital assistants" to read these reviews and tell you if they are happy or unhappy. They wanted to see which assistant was better: the Old-School Detective (Machine Learning) or the High-Tech Brain (Deep Learning).

Here is how they set up the race and what happened.

The Contestants

1. The Old-School Detective (Machine Learning)
This team used a smart toolkit called PyCaret. Think of this as a super-efficient filing cabinet. It takes the messy reviews, cleans them up, and turns them into a neat list of keywords (like "good," "bad," "fast," "slow").

The Tools: They tested three types of detectives:
- Logistic Regression: A straightforward, logical thinker that looks for simple patterns.
- SVM: A strict judge that draws a sharp line between "good" and "bad" reviews.
- LightGBM: A fast, powerful calculator that builds many small decision trees to find the answer.
The Strategy: They gave these tools 15,782 reviews to study and then tested them on a fresh batch of 3,946 reviews they hadn't seen before.

2. The High-Tech Brain (Deep Learning)
This team built a BiLSTM with Attention using PyTorch. Imagine this as a super-intelligent robot that doesn't just read words; it reads the whole sentence from start to finish and then from finish to start.

The "Attention" Superpower: This robot has a special ability called "Attention." It's like having a magnifying glass that instantly zooms in on the most important words in a sentence (like "not" or "love") and ignores the boring ones. It tries to understand the context and feelings behind the words, not just the words themselves.

The Race Conditions

The Track: They used a dataset of 19,728 Indonesian product reviews.
The Balance: The track was perfectly fair. Exactly half the reviews were positive, and half were negative. This meant neither assistant had an unfair advantage.
The Prep: Before the race, they cleaned the text. They removed links, hashtags, and converted slang (like changing "bgt" to "banget") so both assistants were reading the same clear language.

The Results: Who Won?

You might expect the High-Tech Brain to crush the Old-School Detective because it's more complex and "smarter." But the results were a surprise!

The Old-School Detective (Logistic Regression):
- Score: 97.26% accuracy.
- Speed: It was incredibly fast, finishing its training in about 12 seconds per round.
- Verdict: It was the winner by a tiny, tiny margin.
The High-Tech Brain (BiLSTM + Attention):
- Score: 97.24% accuracy.
- Speed: It took much longer to train and required a powerful computer (GPU) to run.
- Verdict: It came in a very close second, losing by just 0.02%.

What Does This Mean? (The "Aha!" Moment)

The authors found something interesting: Sometimes, a simple tool is better than a complex one.

Because the reviews were short and the difference between "happy" and "sad" was very clear (the data was "linearly separable"), the simple Logistic Regression model didn't need to overthink it. It just needed to count the right keywords, and it did that perfectly.

The High-Tech Brain was excellent at understanding context and didn't get confused by the balance of the data, but it was like using a Ferrari to drive to the corner store. It could do the job, but it was overkill and took much more energy.

The Bottom Line

If you have a huge pile of short product reviews in Indonesian and you need to sort them quickly:

Don't overcomplicate it. A simple, well-prepared Machine Learning model (like Logistic Regression) is just as good as a fancy Deep Learning model.
Save your resources. The simple model is faster, cheaper to run, and easier to explain.
The "Attention" mechanism in the complex model worked great to keep things balanced, but for this specific job, the simple model was already at the "ceiling" of performance.

In short: For this specific task, the simple, logical detective beat the high-tech robot by a hair's breadth, proving that you don't always need the most complex tool to get the job done.

1. Problem Statement

The rapid expansion of e-commerce in Indonesia has generated vast amounts of unstructured textual data in the form of product reviews. Manually analyzing these reviews is inefficient and prone to bias. While automated sentiment analysis using Natural Language Processing (NLP) is a standard solution, a significant challenge remains: Indonesian product reviews are characterized by colloquial language, slang (bahasa gaul), abbreviations, and irregular sentence structures.

The core research question addresses a common assumption in NLP: that Deep Learning (DL) architectures inherently outperform traditional Machine Learning (ML) methods on text data. This study seeks to empirically verify whether complex sequential models (like BiLSTM) are necessary for Indonesian sentiment analysis or if optimized traditional ML approaches can achieve comparable or superior results with greater efficiency.

2. Methodology

Dataset

Source: Publicly available Indonesian product reviews from Hugging Face Datasets.
Size: 19,728 samples, perfectly balanced between Positive and Negative classes.
Split: 80% Training (15,782 samples) and 20% Held-out Test (3,946 samples).

Preprocessing Pipeline

A rigorous preprocessing step was applied to handle the linguistic nuances of Indonesian:

Case Folding: Conversion to lowercase.
Cleansing: Removal of URLs, HTML tags, mentions, hashtags, punctuation, and numbers.
Slang Normalization: Mapping informal words to standard Indonesian forms using a curated dictionary (e.g., bgt $\to$ banget).
Tokenization & Filtering: Splitting by whitespace and removing tokens shorter than 2 characters.

Approaches Compared

A. Machine Learning (ML) Pipeline (PyCaret AutoML)

Feature Extraction: TF-IDF vectorization with $N=5000$ features, utilizing unigrams and bigrams ( $ngram\_range=(1,2)$ ).
Algorithms Evaluated:
1. Logistic Regression (LR): Linear estimator.
2. Support Vector Machine (SVM): Linear kernel.
3. LightGBM: Gradient boosting decision trees.
Validation: 10-fold Stratified Cross-Validation.

B. Deep Learning (DL) Pipeline (PyTorch)

Architecture: Bidirectional Long Short-Term Memory (BiLSTM) with an Attention mechanism.
Components:
- Embedding Layer (Vocab size: 13,600; Dim: 64).
- Two BiLSTM layers (Hidden size: 128 each).
- Attention Layer (Softmax-weighted sum of hidden states).
- Classifier Head (Linear $\to$ ReLU $\to$ Dropout $\to$ Output).
Training: Trained from scratch (no pre-trained embeddings) using PyTorch. Hyperparameters were optimized via Optuna (best trial: LR=0.0038, Batch=128).
Validation: Evaluated on the held-out test set.

3. Key Contributions

Rigorous Empirical Comparison: Provides a head-to-head benchmark of AutoML-driven ML algorithms against a custom-built DL architecture on a specific, challenging Indonesian dataset.
Challenge to DL Dominance: Demonstrates that for high-dimensional, balanced binary classification tasks, classical linear models (specifically Logistic Regression) can marginally outperform complex sequential DL models.
Reproducibility: The study offers a fully reproducible experimental pipeline, including code deployed on Hugging Face Spaces, facilitating further research in Indonesian NLP.

4. Results

Machine Learning Performance (10-Fold CV)

Logistic Regression: Achieved the highest performance with 97.26% Accuracy and 97.26% F1-Score.
SVM (Linear): Close second with 97.17% Accuracy.
LightGBM: 96.43% Accuracy.
Observation: All ML models showed AUC scores > 0.993, indicating near-perfect discriminative power.

Deep Learning Performance (Held-out Test)

BiLSTM + Attention: Achieved 97.24% Accuracy and 97.24% F1-Score on the test set.
Confusion Matrix: Highly symmetric predictions (1,919 True Positives vs. 1,918 True Negatives), indicating the Attention mechanism successfully prevented class bias.

Head-to-Head Comparison

Approach	Best Model	Accuracy	F1-Score
Machine Learning	Logistic Regression	97.26%	97.26%
Deep Learning	BiLSTM + Attention	97.24%	97.24%

5. Significance and Discussion

Linear Separability: The results suggest that the TF-IDF feature space (5,000 dimensions) renders the Indonesian sentiment data linearly separable. In such scenarios, the added complexity of recurrent neural networks and attention mechanisms yields diminishing returns.
Computational Efficiency: The ML approach offers a significant advantage in speed and resource usage. For instance, SVM training took only ~6.5 seconds per fold, whereas the DL model required GPU acceleration and multiple epochs to converge.
Practical Implications: For production systems requiring low latency and high interpretability, the PyCaret-based Logistic Regression pipeline is recommended over complex DL models for this specific task.
Future Directions: The authors suggest extending this work to multi-class sentiment analysis (including "Neutral") and testing pre-trained Transformer models (e.g., IndoBERT), which might perform better on more complex linguistic tasks or unbalanced datasets.

Conclusion:
The paper concludes that while Deep Learning remains powerful, it is not universally superior. For balanced, high-volume binary sentiment analysis of Indonesian product reviews, a well-engineered traditional ML pipeline using AutoML is not only competitive but slightly superior in accuracy and significantly more efficient.