💬 NLP

Sentiment Analysis of AI Adoption in Indonesian Higher Education Using Machine Learning and Transformer-Based Models

This study evaluates Indonesian student sentiment regarding AI adoption in higher education by comparing traditional machine learning models against the Transformer-based DistilBERT, finding that while DistilBERT achieves superior accuracy (84.78%) by better capturing context, SVM remains a competitive and efficient alternative with 82.14% accuracy.

Original authors: Happy Syahrul Ramadhan, Ahmad Sahidin Akbar, Karin Yehezkiel Sinaga, Luluk Muthoharoh, Ardika Satria, Martin C. T. Manullang

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Happy Syahrul Ramadhan, Ahmad Sahidin Akbar, Karin Yehezkiel Sinaga, Luluk Muthoharoh, Ardika Satria, Martin C. T. Manullang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling university campus in Indonesia where students are buzzing with opinions about a new, powerful tool: Artificial Intelligence (AI). Some see it as a magic wand for learning; others worry it's a cheat sheet that will ruin their critical thinking. The researchers in this paper wanted to listen to this "buzz" and figure out exactly what the students are feeling. They didn't just ask a few people; they gathered over 2,000 written opinions and used two different types of "digital ears" to listen and sort them into Happy/Supportive or Worried/Negative.

Here is how they did it, explained simply:

The Two "Digital Ears" (The Methods)

The researchers tested two different ways to understand these student opinions, like comparing a fast, efficient librarian against a deep-thinking professor.

1. The Fast Librarian (Machine Learning with SVM)

How it works: This method uses a technique called TF-IDF. Imagine taking every word a student wrote and counting how often it appears. If a student writes "AI is great," the system notes that "great" is a strong word for them. It treats the text like a bag of words, looking for specific keywords that signal happiness or worry.
The Tool: They used a model called SVM (Support Vector Machine). Think of SVM as a very sharp, efficient sorter. It draws a line in the sand to separate "Positive" words from "Negative" words. It's fast, doesn't need much computing power, and is great at spotting patterns in simple word lists.
The Result: This "Librarian" was very good! It got about 82% of the opinions right. It was the best of the traditional methods, proving that sometimes you don't need a supercomputer to get a good read on a crowd.

2. The Deep-Thinking Professor (Transformer with DistilBERT)

How it works: This method uses DistilBERT, a type of AI based on "Transformers." Unlike the Librarian who just counts words, the Professor reads the whole sentence and understands the context. It knows that "AI is not great" is very different from "AI is great," even though they share the same words. It understands the nuance, the tone, and the hidden meaning.
The Tool: DistilBERT is a "lite" version of a massive AI brain. It's like a professor who has read the entire internet but is small enough to fit on a standard laptop.
The Result: This "Professor" was the champion, getting about 85% of the opinions right. It was slightly better than the Librarian because it understood the feelings behind the words, not just the words themselves.

The Experiment Setup

The Data: They collected 2,295 student comments. It was a fair fight: about half the students were negative (worried about cheating or losing skills), and half were positive (excited about learning efficiency).
The Test: They split the data into three piles:
- Training (80%): Teaching the models.
- Validation (10%): Checking their homework.
- Testing (10%): The final exam to see who actually learned.

The Showdown: Who Won?

The Contender	The Style	Final Score (Accuracy)	Speed
LightGBM	The Speedster	~50% (Missed the mark)	Super Fast
Random Forest	The Team Player	~59% (Okay, but not great)	Fast
SVM	The Efficient Librarian	82.14%	Fast
DistilBERT	The Deep Professor	84.78%	Slower (takes minutes to train)

The Winner: DistilBERT took the gold medal with the highest accuracy. It proved that understanding the context of a sentence gives you a better read on human emotion.
The Runner-Up: SVM came in second. While it didn't win, the paper notes it is still a "strong, low-cost alternative." If you need a quick answer and don't have a supercomputer, the Librarian (SVM) is still a very reliable choice.

The Takeaway

The paper concludes that while the "Deep Professor" (DistilBERT) is the most accurate at understanding how Indonesian students feel about AI in their schools, the "Efficient Librarian" (SVM) is still a very powerful tool that is much faster and cheaper to run.

Finally, the researchers didn't just stop at the numbers. They built a live web app (using a tool called Gradio) where anyone can type in a sentence, and the system will instantly tell them if the sentiment is positive or negative, complete with a confidence score. It's like having a digital mood ring for student opinions.

In short: To understand student feelings about AI, you can use a fast, keyword-based sorter (SVM) or a smart, context-aware reader (DistilBERT). The smart reader wins on accuracy, but the fast sorter is still a very useful tool.

1. Problem Statement

The rapid adoption of Artificial Intelligence (AI) in Indonesian higher education has generated a complex mix of student opinions, ranging from support for learning efficiency to concerns regarding academic integrity, over-dependence, and the erosion of critical thinking skills.

The Gap: While sentiment analysis is a standard tool for gauging public opinion, there is a lack of comparative studies evaluating the effectiveness of classical machine learning (ML) versus Transformer-based deep learning specifically for Indonesian-language text in the context of AI adoption in education.
The Objective: To systematically analyze Indonesian student opinions on AI and determine which modeling approach (TF-IDF-based ML vs. Transformer-based DL) offers superior performance for binary sentiment classification (Positive vs. Negative) in this specific domain.

2. Methodology

A. Dataset Construction

Source: The study utilized a hybrid dataset combining 1,154 raw student opinions on AI in higher education with 1,141 samples from a positive/negative lexical dictionary.
Total Size: 2,295 labeled samples.
Class Distribution: The dataset is nearly balanced:
- Negative: 1,158 samples (50.46%)
- Positive: 1,137 samples (49.54%)
Preprocessing: Text cleaning involved lowercasing, removing URLs/mentions/hashtags/punctuation, normalizing non-standard words, and handling repeated characters.

B. Experimental Workflow

The data was split into an 80:10:10 ratio for Training, Validation, and Testing sets. Two distinct pipelines were implemented:

1. Machine Learning Pipeline (Baseline)

Feature Extraction: Text was converted into TF-IDF vectors using unigrams and bigrams.
Models Evaluated:
- LightGBM: A boosting ensemble (configured with n_estimators=100, max_depth=10).
- Random Forest: A parallel ensemble (configured with n_estimators=100, max_depth=20).
- Support Vector Machine (SVM): A kernel-based classifier using an RBF kernel (C=1.0).
Selection: Models were selected based on validation accuracy, with SVM chosen as the best ML representative for final comparison.

2. Deep Learning Pipeline (Transformer)

Model: DistilBERT (distilbert-base-uncased), a lightweight, distilled version of BERT designed to retain performance while reducing computational cost.
Architecture: Six-layer encoder with self-attention mechanisms.
Configuration:
- Max sequence length: 128 tokens.
- Batch size: 64.
- Optimizer: AdamW with a learning rate of $1 \times 10^{-6}$ .
- Regularization: Dropout (0.5), Weight Decay (0.001).
- Training: 100 epochs with Early Stopping (patience=5) based on Validation F1-score.

C. Evaluation Metrics

Performance was assessed using Accuracy, Precision, Recall, and F1-score.

3. Key Results

A. Machine Learning Performance

LightGBM: Performed poorly (Test Accuracy: 49.67%, F1: 48.46%), indicating it struggles with sparse, high-dimensional TF-IDF features in this context.
Random Forest: Showed moderate improvement (Test Accuracy: 58.61%, F1: 56.48%) but remained inferior to SVM.
SVM: Emerged as the best classical ML model.
- Validation Accuracy: 87.25%
- Test Accuracy: 82.14%
- Test F1-Score: 82.14%
- Observation: SVM demonstrated robustness in handling high-dimensional text data, correctly classifying 377 out of 459 test samples.

B. Transformer (DistilBERT) Performance

Test Accuracy: 84.78%
Test F1-Score: 84.75% (Weighted)
Class Breakdown:
- Negative Class F1: 0.8548
- Positive Class F1: 0.8402
Observation: DistilBERT showed a slight bias toward higher recall on negative samples but maintained balanced precision across classes. The training loss (0.28) and validation loss (0.36) indicated good generalization without severe overfitting.

C. Comparative Analysis

Performance Gap: DistilBERT outperformed the best ML model (SVM) by 2.64% in accuracy and 2.61% in F1-score.
Efficiency Trade-off:
- SVM: Extremely fast training time (0.210s).
- DistilBERT: Significantly longer training time (22.8 minutes) due to GPU-intensive processing.
Conclusion on Models: While Transformers capture richer semantic context leading to higher accuracy, SVM remains a highly competitive, low-cost alternative for efficient sentiment classification.

4. Key Contributions

Context-Specific Analysis: Provides empirical evidence on how Indonesian university students perceive AI adoption, addressing a gap in local NLP literature.
Methodological Comparison: Offers a rigorous benchmark comparing traditional TF-IDF/SVM approaches against modern Transformer-based (DistilBERT) models for Indonesian educational text.
Dataset Creation: Curated and processed a balanced dataset of 2,295 Indonesian samples specifically focused on AI in higher education.
Practical Deployment: The best-performing model (DistilBERT) was deployed as an interactive web application using Gradio and Hugging Face Spaces, enabling real-time sentiment prediction with confidence scores.

5. Significance and Future Work

Significance: The study validates that while deep learning models (Transformers) offer superior accuracy by understanding context, classical models like SVM are still viable for resource-constrained environments. This helps educational institutions choose appropriate tools for policy-making based on student feedback.
Future Directions:
- Expanding the dataset size.
- Introducing a neutral class for three-way classification.
- Utilizing Indonesian-specific pre-trained models (e.g., IndoBERT) instead of the English-centric distilbert-base-uncased.
- Conducting aspect-based sentiment analysis to identify specific concerns (e.g., integrity vs. learning efficiency) rather than just general polarity.