💬 NLP

A Comparative Study of PyCaret AutoML and CNN-BiLSTM for Binary Hate Speech Detection in Indonesian Twitter

This study demonstrates that a CNN-BiLSTM model outperforms PyCaret AutoML's best conventional classifier (Random Forest) in detecting binary hate speech on Indonesian Twitter, achieving a 6.6% higher accuracy and 4.2% higher F1-score by effectively leveraging dense token representations and bidirectional context.

Original authors: Tanty Widiyastuti, Mayada, Adisty Syawalda Ariyanto, Luluk Muthoharoh, Ardika Satria, Martin Clinton Tosima Manullang

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tanty Widiyastuti, Mayada, Adisty Syawalda Ariyanto, Luluk Muthoharoh, Ardika Satria, Martin Clinton Tosima Manullang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to spot "toxic" tweets on Indonesian Twitter. The goal is to figure out if a short message is hate speech (bad) or just normal speech (okay).

This paper is like a race between two different coaches trying to train a student to pass this test.

The Two Coaches

Coach 1: The "Auto-Organizer" (PyCaret)
Think of this coach as a super-efficient project manager. They don't invent new ways to think; instead, they take a toolbox of classic, proven methods (like counting specific bad words or looking at word frequency) and quickly try out dozens of different strategies to see which one works best.

How they work: They look at the text like a list of ingredients. If the tweet contains a known "bad word" from a dictionary, or if certain words appear together often, they flag it.
The Result: This coach found that a method called Random Forest (which is like asking a committee of decision-makers to vote on whether a tweet is bad) was the best of the bunch. They got about 77% accuracy.

Coach 2: The "Context Detective" (CNN-BiLSTM)
This coach is a deep-learning expert. Instead of just counting words, they teach the computer to "read" the tweet like a human does, paying attention to the order of words and how they change meaning based on what comes before or after them.

How they work: Imagine a sentence is a puzzle. The "CNN" part looks for small, local patterns (like a specific angry phrase), while the "BiLSTM" part looks at the whole sentence from left to right and right to left to understand the context. For example, it helps the computer understand that a word might be bad in one sentence but harmless in another because of a "not" or a joke nearby.
The Result: This coach built a model that got 83.8% accuracy.

The Race Conditions

To make it a fair race, the authors made sure both coaches started with the exact same ingredients:

Same Data: They used the same 13,000+ tweets from a famous Indonesian dataset.
Same Cleaning: Both coaches cleaned the tweets the same way (removing links, fixing slang, lowercasing).
Same Goal: Both were trying to solve the exact same binary problem: Is this hate speech (Yes/No)?

The Winner

The Context Detective (CNN-BiLSTM) won the race.

They were 6.6% more accurate than the best Auto-Organizer.
They were better at catching the bad tweets without accidentally flagging too many good ones.

Why Did the Detective Win?

The paper explains that Indonesian tweets are very short and often rely on subtle context.

The Problem: Sometimes, a tweet looks harmless if you just count words, but it's actually hate speech because of how the words are arranged. Or, a bad word might be used in a way that isn't hateful.
The Solution: The "Auto-Organizer" was good at spotting obvious bad words, but it missed the subtle "vibe" or context. The "Context Detective" was better at understanding that short, messy sentences often need to be read as a whole story, not just a list of parts.

The Catch (The "Overfitting" Warning)

The paper also noticed something interesting about the Detective's training.

The Detective learned the training data very quickly and very well, almost too well.
It's like a student who memorized the practice test perfectly but might struggle if the real test has slightly different questions. The paper notes that the model started to "overfit" (memorize the noise) a bit.
The Takeaway: Even though the Detective was still the winner, the authors suggest that in the future, we need to teach the Detective to be more careful and not just memorize, so it performs even better on new, unseen tweets.

The Bottom Line

PyCaret (The Auto-Organizer) is great if you want a quick, reliable, and easy-to-understand baseline. It's a strong "good enough" solution.
CNN-BiLSTM (The Context Detective) is the better choice if you need the highest possible accuracy and are willing to deal with a more complex system that understands the nuance of short, messy text.

The paper concludes that while the "Auto-Organizer" is a fantastic tool for setting a standard, the "Context Detective" is currently the superior tool for this specific job of detecting hate speech on Indonesian Twitter.

Technical Summary: A Comparative Study of PyCaret AutoML and CNN-BiLSTM for Binary Hate Speech Detection in Indonesian Twitter

Problem Statement
The paper addresses the challenge of automatic hate speech detection on Indonesian Twitter. The task is characterized by short, noisy, and informal text where harmful intent often arises from context, negation, or pragmatic framing rather than explicit slurs alone. The authors aim to determine whether a sequence-aware deep learning approach (CNN-BiLSTM) offers a meaningful advantage over a strengthened conventional machine learning branch when both are evaluated under identical preprocessing and benchmark conditions. The study utilizes the hate speech (HS) label from the corpus of Ibrohim and Budi [1], reformulating the multi-label dataset into a controlled binary classification problem (Hate Speech vs. Non-Hate Speech).

Methodology
The study employs a controlled benchmark design where both the conventional and neural branches share an identical preprocessing pipeline to ensure performance differences reflect modeling capacity rather than data preparation inconsistencies.

Data and Preprocessing:
- Dataset: A subset of 13,130 tweets from the Ibrohim and Budi corpus, with a class distribution of approximately 58% non-hate speech and 42% hate speech.
- Preprocessing: Includes lowercasing, removal of URLs/usernames, symbol normalization, slang normalization using corpus-provided mappings, and stopword removal.
- Feature Engineering: A shared "abusive-word count" feature is derived from the corpus's abusive lexicon.
Conventional Branch (PyCaret AutoML):
- Representation: Tweets are vectorized using TF-IDF (unigrams and bigrams, max vocabulary 5,000) concatenated with the abusive-word count.
- Workflow: PyCaret AutoML is used not as a single classifier but as a benchmarking layer to standardize the comparison across multiple estimators.
- Models: The workflow evaluates and ranks linear SVM, Naive Bayes, and Random Forest. Random Forest emerges as the strongest conventional candidate.
Neural Branch (CNN-BiLSTM):
- Architecture: A compact architecture designed for short text.
  - Embedding: 100-dimensional trainable word embeddings.
  - CNN Layer: Extracts local phrase patterns (kernel size 3) to capture abusive fragments and intensifiers.
  - BiLSTM Layer: Encodes bidirectional context to interpret local cues within the surrounding text.
  - Output: Sigmoid activation for binary probability.
- Training: Optimized with binary cross-entropy loss, using a learning rate of $10^{-3}$ , batch size of 64, and dropout of 0.2.

Key Contributions
The paper makes four specific contributions:

Benchmark Reformulation: It converts the released multi-label corpus into a controlled binary HS benchmark.
Operationalized Conventional Baseline: Instead of a single weak baseline, it uses PyCaret AutoML to establish a strong, systematic conventional benchmark.
Compact Neural Evaluation: It evaluates a tailored CNN-BiLSTM architecture specifically for short Indonesian contextual text.
Comprehensive Interpretation: It analyzes the results through corpus statistics, learning curves, and confusion matrices to distinguish between representation quality and training stability.

Results
The study compares the models on a held-out 20% test split (2,626 tweets).

Performance Metrics:
- CNN-BiLSTM: Achieved 83.8% Accuracy, 79.8% Precision, 82.7% Recall, and 81.2% F1-score.
- PyCaret-Random Forest (Strongest Conventional): Achieved 77.2% Accuracy, 75.8% Precision, 78.3% Recall, and 77.0% F1-score.
- PyCaret-SVM: 72.3% Accuracy (weakest).
- PyCaret-Naive Bayes: 75.8% Accuracy.
Comparative Gain: The neural branch improved accuracy by 6.6 percentage points and F1-score by 4.2 percentage points over the strongest conventional model.
Error Analysis:
- The CNN-BiLSTM showed signs of overfitting (validation loss rising after early epochs), suggesting that gains could be further optimized via early stopping and regularization.
- The confusion matrix indicated the model is somewhat aggressive but balanced, with errors concentrated on borderline cases involving implied hostility or group references rather than explicit toxic tokens.
- An auxiliary task on "abusive language" (broader than hate speech) showed more stable learning curves, confirming that strict HS detection is a harder, more context-dependent boundary.

Significance and Claims
The paper concludes with a nuanced distinction between the utility of the two approaches:

PyCaret AutoML as a Framework: The study validates PyCaret as an effective framework for establishing strong, reproducible conventional benchmarks. It demonstrates that even with a robust lexical baseline (TF-IDF + abusive counts) and AutoML optimization, the conventional branch cannot fully capture the nuances of Indonesian hate speech.
CNN-BiLSTM as the Superior End Model: The results confirm that for this specific benchmark, the CNN-BiLSTM architecture provides a significant advantage. The authors attribute this to the model's ability to capture both local lexical patterns (via CNN) and short-range contextual interactions (via BiLSTM), which are critical for Indonesian tweets where intent often depends on framing.
Practical Implications: The paper posits that PyCaret is preferable for rapid screening, low-cost experimentation, and reproducible baselining. Conversely, CNN-BiLSTM is the recommended choice when the primary objective is maximizing predictive balance on the hate speech boundary and when the computational cost of deep learning is acceptable.

The authors explicitly state that these findings are specific to the benchmark configuration (one dataset, one preprocessing pipeline, one split) and do not claim universal superiority for all forms of online hate speech. They suggest future work could explore richer AutoML search spaces, stronger regularization, and transformer-based models like IndoBERT.