💻 computer science

Sentiment Analysis of Indonesian Spotify Reviews Using Machine Learning and BiLSTM

This paper benchmarks classical machine learning and BiLSTM deep learning models on 70,155 cleaned Indonesian Spotify reviews, finding that while BiLSTM achieves the highest overall weighted F1-score, Decision Trees combined with SMOTE offer superior balanced performance across all three sentiment classes, particularly for the minority neutral category.

Original authors: Uliano Wilyam Purba, Andre Hadiman Rotua Parhusip, Sahid Maulana, Luluk Muthoharoh, Ardika Satria, Martin C. T. Manullang

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Uliano Wilyam Purba, Andre Hadiman Rotua Parhusip, Sahid Maulana, Luluk Muthoharoh, Ardika Satria, Martin C. T. Manullang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the owner of a massive music streaming app, Spotify, and you have a mountain of 100,000 letters from your Indonesian users. Some are love letters ("Positif"), some are angry rants ("Negatif"), and some are just shrugs or complaints that don't quite fit either ("Netral"). Your goal is to build a robot that can read these letters and sort them into the right piles automatically.

This paper is a report card on a "race" between two different types of robots trying to do this sorting job.

The Two Contenders

The "Classical" Team (Machine Learning): Think of these as three very smart, experienced librarians (Support Vector Machine, Multinomial Naive Bayes, and Decision Tree). They are good at looking at individual words and counting how often they appear. They are fast, efficient, and don't need a supercomputer to work.
The "Deep Learning" Team (BiLSTM): This is a single, highly sophisticated robot built with a neural network (Bidirectional Long Short-Term Memory). Imagine this robot as a detective who doesn't just look at words, but tries to understand the story and the flow of the sentence, reading from left to right and right to left to catch the full context.

The Setup: A Race with Unequal Tracks

The researchers set up the race, but there was a catch: The playing fields weren't exactly the same size.

The Cleaning Crew: Before the race, all the letters were cleaned up. Slang was fixed (changing "gak" to "tidak"), typos were removed, and unnecessary words were deleted. This was the same for everyone.
The Classical Team's Track: The three librarians got to read all 70,000+ cleaned letters. They had the full dataset to learn from.
The Deep Learning Robot's Track: The fancy robot was only allowed to read a smaller sample of 20,000 letters. Why? Because the researchers were working on a standard computer (CPU) without a powerful graphics card. Training the big robot on the full mountain of data would have taken forever, so they gave it a smaller, representative slice.

The Results: Who Won?

1. The Classical Team's Performance
Among the three librarians, the Decision Tree was the clear winner.

How it worked: It built a flowchart of rules to sort the letters.
The Score: It got a "Weighted F1-Score" of 0.727.
The Catch: It was pretty good at sorting all three piles (Positive, Negative, and Neutral) fairly evenly. It didn't ignore the "Neutral" letters.

2. The Deep Learning Robot's Performance
The BiLSTM robot was a powerhouse, but it had a blind spot.

The Score: It achieved a higher overall score of 0.807.
The Problem: It was amazing at spotting "Positive" and "Negative" letters, but it completely failed at the "Neutral" category. It got a score of 0.000 for Neutral. It basically ignored that pile entirely, likely because there were so few Neutral letters in its training slice compared to the others.

The Big Takeaway: The Trade-Off

The paper concludes with a simple analogy:

If you want the best overall accuracy for telling if a review is generally happy or sad, the Deep Learning Robot (BiLSTM) is the better choice. It's like a sports car that is incredibly fast on a straight road but can't handle a sharp turn.
If you need to be fair to all three categories (including the tricky, quiet "Neutral" ones), the Classical Decision Tree is the better choice. It's like a reliable SUV that isn't the fastest, but it handles every type of terrain well and doesn't leave any passengers behind.

Why Did This Happen?

The "Neutral" Problem: The "Neutral" reviews were the minority (only about 6% of the data). The Deep Learning robot, trained on a small sample without special help, learned to ignore the minority group to maximize its overall score. The Classical team used a technique called SMOTE (which is like artificially creating more "Neutral" examples for the robot to study), which helped them treat all groups fairly.
The Data Gap: The Deep Learning robot had fewer examples to learn from (14,000 vs 70,000). Even with fewer examples, it still beat the classical models on the "Positive/Negative" split, suggesting that understanding the sequence of words is very powerful for short Indonesian reviews.

The Bottom Line

The researchers didn't just stop at numbers; they put both robots online so anyone can test them. They found that while the fancy Deep Learning model is more powerful overall, the simpler Machine Learning model is more balanced and reliable when you need to catch every type of review, not just the loud ones.

In short: The fancy robot is faster and stronger, but the simple librarian is more fair to the quiet voices in the crowd.

Technical Summary: Sentiment Analysis of Indonesian Spotify Reviews Using Machine Learning and BiLSTM

Problem Statement
This study addresses the challenge of sentiment analysis on Indonesian text derived from user reviews of the Spotify application on the Google Play Store. The task is complicated by the informal nature of the language, which includes inconsistent spelling, heavy use of slang, and morphological variations. The authors aim to benchmark the performance of classical machine learning (ML) classifiers against a deep learning (DL) approach (Bidirectional Long Short-Term Memory, or BiLSTM) for a three-class sentiment classification task (Negative, Neutral, Positive). A specific constraint of the study is the evaluation environment: a CPU-only setting, which necessitates different training data scales for the two approaches.

Methodology
The research employs a dual-track experimental design where both tracks share an identical preprocessing pipeline but differ in model architecture and training data volume.

Dataset: The raw dataset consists of 100,000 scraped reviews. After cleaning (removing duplicates, empty reviews, and non-text elements), 70,155 samples remain. The labels are derived from star ratings: 1–2 stars (Negative), 3 stars (Neutral), and 4–5 stars (Positive). The dataset is heavily imbalanced, with Positive reviews comprising 68.4%, Negative 25.7%, and Neutral only 5.9%.
Preprocessing Pipeline: A unified pipeline is applied to all text, including:
1. Lowercasing.
2. Noise removal (URLs, mentions, punctuation, digits).
3. Slang normalization (mapping 124 informal forms to standard Indonesian).
4. Stopword removal (809 general and domain-specific terms).
5. Morphological stemming using Sastrawi.
  Result: The median review length post-processing is 4 tokens, with a 95th percentile of 17 tokens.
Machine Learning Track:
- Data: Trained on the full 70,155 cleaned samples.
- Features: TF-IDF vectorization (max 3,000 features).
- Classifiers: Support Vector Machine (SVM), Multinomial Naive Bayes (MNB), and Decision Tree (DT).
- Technique: Stratified 5-fold cross-validation. To address class imbalance, SMOTE (Synthetic Minority Over-sampling Technique) is applied within each training fold.
- Evaluation: Metrics include Accuracy, AUC, Precision, Recall, F1-Score (weighted and macro), and Cohen's Kappa.
Deep Learning Track:
- Data: Due to CPU constraints, the BiLSTM is trained on a stratified subset of 20,000 samples (14,219 for training, 2,510 for validation, 2,953 for testing).
- Architecture: A two-layer Bidirectional LSTM with 128-dimensional embeddings, 128 hidden units per direction, and a fully connected classification head. Total parameters: ~1.1 million.
- Training: Adam optimizer, learning rate $10^{-3}$ , batch size 64, unweighted Cross-Entropy loss. Early stopping is used (best checkpoint at epoch 3).
- Evaluation: Evaluated on a held-out test set using the same metrics as the ML track.

Key Contributions

Unified Preprocessing: The establishment of a shared preprocessing pipeline for both ML and DL tracks, ensuring that performance differences are attributable to model architecture rather than data cleaning variations.
Benchmarking: A comparative analysis of three classical Scikit-learn classifiers (SVM, MNB, DT) against a compact BiLSTM model on large-scale Indonesian app reviews.
Transparency and Reproducibility: The authors provide public Hugging Face Spaces for both the ML and DL models, allowing for direct inspection of model behavior.
Analysis of Imbalance: A detailed examination of how class imbalance affects model performance, specifically highlighting the failure of the unweighted BiLSTM on the minority "Neutral" class compared to the ML approach utilizing SMOTE.

Results

Machine Learning Performance:
- The Decision Tree emerged as the strongest classical model, achieving a weighted F1-score of 0.7269 and an accuracy of 0.7286 in 5-fold cross-validation.
- SVM followed with a weighted F1 of 0.6958, while Multinomial Naive Bayes performed significantly lower (0.4619).
- The Decision Tree maintained balanced performance across all three classes, with the lowest F1 score on the Neutral class (0.717) but still functional.
Deep Learning Performance:
- The BiLSTM achieved a higher weighted F1-score of 0.8069 and an accuracy of 0.8314 on its held-out test set.
- However, the Macro F1-score dropped to 0.5478.
- Class-wise Breakdown: The model performed exceptionally well on Positive reviews (F1 0.9028) and reasonably on Negative reviews (F1 0.7406). Crucially, it failed completely on the Neutral class, achieving an F1-score of 0.0000 (0 correct predictions out of 176 samples).
Comparative Analysis:
- While the BiLSTM outperformed the ML models on weighted metrics (overall performance), this advantage is tempered by the fact that it was trained on significantly fewer samples (14,219 vs. 70,155) and lacked class balancing techniques.
- The ML approach (Decision Tree + SMOTE) provided more robust and balanced recognition across all three classes, whereas the BiLSTM was biased toward the majority classes.

Significance and Claims
The paper claims that while deep learning models like BiLSTM can offer superior overall weighted performance for sentiment detection in short, informal Indonesian texts, they are highly susceptible to class imbalance issues when trained without specific mitigation strategies (like class weighting or oversampling).

The authors conclude that the choice of model depends on the specific application goal:

BiLSTM is preferable when the primary goal is accurate discrimination between the dominant Positive and Negative sentiments.
Machine Learning (specifically Decision Tree with SMOTE) is more suitable when balanced recognition across all three classes, including the minority Neutral class, is required.

The study acknowledges its limitations, noting the asymmetry in training data sizes between the two tracks and the absence of pre-trained Indonesian embeddings or transformer-based models. Future work is suggested to involve training the BiLSTM on the full corpus, implementing class balancing in the DL pipeline, and exploring transformer architectures.