Benchmarking PyCaret AutoML Against IndoBERT Fine-Tuning for Sentiment Analysis on Indonesian IKN Twitter Data

1. Problem Statement

The relocation of Indonesia's capital to Ibu Kota Nusantara (IKN) has generated significant public discourse on social media, particularly Twitter/X. Understanding public sentiment toward this policy is crucial for policymakers and researchers. However, analyzing this data presents specific challenges:

• Language Complexity: The data consists of informal Indonesian text containing slang, abbreviations, and varying sentence structures.
• Resource Constraints: Traditional Machine Learning (ML) approaches often struggle with contextual nuances in low-resource languages compared to Deep Learning (DL) models.
• Methodological Gap: There is a need to empirically compare the efficiency and effectiveness of automated ML frameworks (AutoML) against state-of-the-art Transformer-based models specifically for Indonesian social media sentiment.

2. Dataset

• Source: Indonesian-language Twitter/X comments related to IKN.
• Size: 1,472 manually labeled samples.
• Classes: Binary classification (Positive: 692 samples, 47%; Negative: 780 samples, 53%).
• Characteristics: Text is short (mean length ~50 characters) and informal.
• Preprocessing: Involved lowercasing, removing missing values (NaN), and calculating text length features. The data was split into Training (70%), Validation (15%), and Test (15%) sets using stratified sampling.

3. Methodology

The study employed a dual-pipeline approach to benchmark performance:

A. Machine Learning Pipeline (AutoML)

• Framework: PyCaret (v3.x), an open-source AutoML library.
• Feature Extraction: Automatic TF-IDF vectorization from the text column.
• Algorithms Evaluated:
  1. Logistic Regression (LR)
  2. Naive Bayes (NB)
  3. Support Vector Machine (SVM) with a linear kernel.
• Process:
  • 10-fold stratified cross-validation for initial evaluation.
  • Hyperparameter tuning using Optuna (10 candidates) on the best-performing model.
  • Finalization of the model.
• Hardware: Intel Core i5 CPU (Python 3.9).

B. Deep Learning Pipeline (IndoBERT Fine-Tuning)

• Model: indobenchmark/indobert-base-p1 (111M parameters, 12 Transformer layers).
• Framework: PyTorch with Hugging Face Transformers.
• Architecture: Standard BERT classification head added to the [CLS] token.
• Hyperparameters:
  • Max Length: 128 tokens.
  • Batch Size: 16.
  • Epochs: 5.
  • Optimizer: AdamW (LR: $2 \times 10^{-5}$, weight decay: 0.01).
  • Scheduler: Linear warmup (10%) followed by linear decay.
  • Gradient Clipping: Max norm 1.0.
• Hardware: Same Intel Core i5 CPU (no GPU acceleration used).

4. Key Contributions

1. Dataset Creation: A curated, manually labeled dataset of 1,472 Indonesian tweets regarding IKN sentiment.
2. AutoML Benchmarking: A comprehensive evaluation of three classical ML algorithms (LR, NB, SVM) using PyCaret's automated pipeline.
3. Transformer Implementation: A full fine-tuning implementation of IndoBERT for binary sentiment analysis on this specific domain.
4. Comparative Analysis: A direct performance comparison answering whether Transformer-based DL outperforms classical ML for informal Indonesian text, highlighting the trade-offs between accuracy and computational cost.
5. Reproducibility: The study utilized AI-assisted coding (Claude) to generate reproducible notebooks for EDA, PyCaret, and PyTorch pipelines.

5. Results

Machine Learning Performance (PyCaret)

• Best Model: Logistic Regression.
• Metrics (10-Fold CV):
  • Accuracy: 77.57%
  • F1-Score: 77.17%
  • AUC: 0.8563
• Other Models: Naive Bayes achieved ~70.6% accuracy. The initial Linear SVM performed poorly (52.72%) due to the high lexical variation in informal text, though tuning improved it to ~71.2%.

Deep Learning Performance (IndoBERT)

• Best Model: IndoBERT (Fine-tuned).
• Metrics (Test Set):
  • Accuracy: 89.59%
  • F1-Score (Macro): 89.37%
  • Precision/Recall: High recall for Negative (96%) and Positive (83%) classes.
• Observations: The model showed signs of overfitting after the first epoch (validation loss increased while training loss decreased), a common issue when fine-tuning large models on medium-sized datasets without a GPU.

Comparative Benchmark

• Performance Gap: IndoBERT outperformed the best ML model (Logistic Regression) by 12.02 percentage points in accuracy.
• Efficiency Trade-off: While IndoBERT offered superior accuracy, it required significantly more computational resources and training time compared to the lightweight ML models.

6. Significance and Conclusion

• Superiority of Context: The study confirms that Transformer-based models (IndoBERT) are significantly superior to classical ML for Indonesian sentiment analysis. This is attributed to IndoBERT's ability to capture bidirectional context, handle slang/abbreviations via pre-training, and manage Out-of-Vocabulary (OOV) words better than TF-IDF bag-of-words approaches.
• Practical Implications:
  • High-Stakes Scenarios: For applications requiring maximum accuracy (e.g., policy monitoring), IndoBERT is the recommended choice despite higher resource costs.
  • Resource-Constrained Scenarios: Logistic Regression via PyCaret remains a viable, pragmatic alternative for rapid prototyping or environments with limited computational power, offering a respectable ~77% accuracy.
• Future Directions: The authors suggest exploring data augmentation to mitigate overfitting in DL models and comparing IndoBERT against lighter architectures like DistilBERT or LSTM to find a balance between performance and efficiency.

Final Verdict: For informal Indonesian text sentiment analysis, Deep Learning (IndoBERT) is the clear winner in terms of accuracy, but AutoML (PyCaret/Logistic Regression) offers a strong, efficient baseline for scenarios where computational resources are limited.