Technical Summary: Sentiment Analysis of Mobile Legends App Reviews Using ML and LSTM

Problem Statement

The rapid growth of the mobile gaming industry, particularly in Southeast Asia, has generated vast amounts of user feedback on platforms like the Google Play Store. For developers of Mobile Legends: Bang Bang (MLBB), manually analyzing thousands of reviews daily to gauge user satisfaction is impractical. While sentiment analysis offers a scalable solution, applying it to Indonesian-language reviews presents unique challenges. These reviews frequently contain informal language, slang, abbreviations, code-switching between Indonesian and English, and complex morphological structures (e.g., affixation). Furthermore, existing research often focuses on maximizing accuracy without addressing practical deployment constraints or providing a fair, systematic comparison between automated Machine Learning (ML) frameworks and custom Deep Learning (DL) implementations under identical preprocessing conditions.

Methodology

This study employs a quantitative approach to classify sentiments (Positive, Negative, Neutral) from 10,000 MLBB app reviews. The research pipeline is divided into two parallel tracks: a classical Machine Learning approach and a Deep Learning approach, both utilizing a rigorous, Indonesian-specific preprocessing pipeline.

Data Preprocessing

A reproducible cleaning pipeline was implemented to handle the noise and linguistic variability of the dataset:

1. Case Folding: Conversion of all text to lowercase.
2. Text Cleaning: Removal of punctuation, emojis, URLs, mentions, hashtags, numbers, and special characters via regex.
3. Normalization: Mapping slang and non-standard words to standard Indonesian equivalents.
4. Tokenization: Splitting sentences into individual tokens.
5. Stopword Removal: Filtering common Indonesian and English stopwords using NLTK.
6. Stemming: Reducing affixed words to root forms using the Sastrawi library.
7. Truncation: Limiting sequences to a maximum of 100 tokens.

Feature Representation and Models

• Machine Learning (ML) Track: Preprocessed text was vectorized using Term Frequency-Inverse Document Frequency (TF-IDF). Three classifiers were evaluated using the PyCaret AutoML framework with 5-fold stratified cross-validation:
  • Naive Bayes (NB)
  • Logistic Regression (LR)
  • Random Forest (RF)
• Deep Learning (DL) Track: A custom Long Short-Term Memory (LSTM) network was implemented in PyTorch. The architecture includes:
  • An Embedding Layer (64-dimensional vectors).
  • A single LSTM layer (64 hidden units).
  • A Dropout layer (rate 0.3) to prevent overfitting.
  • A Fully Connected output layer with Softmax activation for 3 classes.
  • Training was conducted using the Adam optimizer, CrossEntropyLoss, a batch size of 32, and 20 epochs.

Evaluation

Models were evaluated on a held-out test set (20% of data) using Accuracy, Precision, Recall, and F1-Score. Both approaches were deployed as interactive applications using Gradio on Hugging Face Spaces to ensure accessibility.

Key Contributions

The paper outlines five primary contributions:

1. Preprocessing Pipeline: A reproducible Indonesian text cleaning pipeline combining case folding, regex noise removal, NLTK stopword filtering, and Sastrawi stemming.
2. ML Benchmark: A comparative analysis of three classifiers (LR, NB, RF) using PyCaret AutoML and TF-IDF features.
3. DL Model: A compact, custom PyTorch LSTM model with approximately 690K parameters, designed for efficiency.
4. Deployment: Publicly accessible interactive demos built with Gradio.
5. Open Source: Full availability of code, data, and notebooks on GitHub.

Results

The study compared the performance of the ML baselines against the LSTM model on the imbalanced dataset (67.47% Negative, 23.73% Positive, 8.80% Neutral).

• Machine Learning Performance: Among the classical models, Logistic Regression achieved the best performance with an accuracy of 77.40% and an F1-score of 73.11%. Random Forest followed closely, while Naive Bayes performed significantly lower (Accuracy: 43.02%, F1-score: 47.67%), struggling with the dataset's lexical complexity and class imbalance.
• Deep Learning Performance: The LSTM model outperformed all ML baselines, achieving an Accuracy of 92%, a Precision of 94%, a Recall of 96%, and a Weighted F1-Score of 91%.
• Analysis: The confusion matrix revealed that while the model excelled at classifying dominant Negative and Positive sentiments, it faced challenges with the minority Neutral class, likely due to data imbalance and the inherent ambiguity of neutral expressions.

Significance and Claims

The paper claims that the results highlight the effectiveness of deep learning in handling sequential text data, particularly for informal, user-generated content containing slang and code-switching. The study demonstrates that LSTM models can capture contextual relationships and sequential dependencies that traditional TF-IDF-based ML models miss.

However, the authors maintain a modest stance regarding the findings:

• They acknowledge that while LSTM outperformed classical models, the data imbalance remains a significant challenge affecting minority class performance.
• They note that traditional ML models remain competitive in terms of computational efficiency and are still effective when combined with appropriate feature engineering.
• The study does not claim to solve the sentiment analysis problem entirely but rather provides a systematic, fair comparison under consistent preprocessing settings, validating the superiority of sequential modeling for this specific domain while identifying areas for future improvement (e.g., data balancing techniques like SMOTE or advanced architectures like Bi-LSTM/IndoBERT).

The work serves as a practical demonstration of how to deploy lightweight, reproducible sentiment analysis pipelines for Indonesian mobile game reviews, bridging the gap between academic model comparison and real-world application constraints.