A Fusion of context-aware based BanglaBERT and Two-Layer Stacked LSTM Framework for Multi-Label Cyberbullying Detection

1. Problem Statement

Cyberbullying on social media is a growing threat to mental health, particularly in low-resource languages like Bangla. Existing detection systems face three primary challenges:

• Single-Label Limitation: Most current models assume a comment contains only one type of abuse (e.g., just "hate speech"). In reality, comments often contain overlapping categories (e.g., a threat combined with religious hate and spam), requiring multi-label classification.
• Language Scarcity: Bangla is a low-resource language with complex morphology. Robust, pre-trained transformer models specifically for Bangla multi-label tasks are scarce.
• Model Architecture Gaps: Traditional Machine Learning (ML) models lack contextual understanding, while standard Deep Learning (DL) models (like CNNs or RNNs) often struggle with long-term dependencies or fail to capture deep semantic meaning. Furthermore, many studies ignore class imbalance, leading to biased predictions.

2. Methodology

The authors propose a hybrid deep learning framework that fuses BanglaBERT-Large (a transformer-based model) with a Two-Layer Stacked LSTM to leverage both contextual semantics and sequential dependencies.

A. Dataset and Preprocessing

• Dataset: The study utilizes a publicly available dataset containing 12,557 Bangla social media comments (from Facebook and TikTok).
• Labels: Each comment is annotated for five binary abuse categories: Bully, Sexual, Religious, Threat, and Spam.
• Preprocessing: Includes cleaning (removing non-Bangla characters, noise), tokenization using the BanglaBERT tokenizer, and handling variable sequence lengths via padding/truncation (max length 64).
• Class Imbalance Handling: The dataset is highly imbalanced. The authors applied undersampling and oversampling techniques exclusively to the training split to balance class distributions while keeping validation and test sets original to prevent data leakage.

B. Model Architecture

The proposed architecture consists of three main stages:

1. Contextual Embedding (BanglaBERT-Large):

   • Acts as the feature extractor.
   • Uses 24 Transformer encoder layers with 16-head multi-head self-attention.
   • Generates rich, context-aware vector representations (dimension 1024) for each token, capturing semantic meaning and word relationships.

2. Sequential Modeling (Two-Layer Stacked LSTM):

   • The output embeddings from BanglaBERT are fed into a 2-layer unidirectional LSTM.
   • Layer 1: Captures local sequential patterns and short-term dependencies.
   • Layer 2: Captures global dependencies and long-range contextual flow across the sentence.
   • Dropout (0.3) is applied between layers to prevent overfitting.

3. Classification Head:

   • The final hidden state from the LSTM is passed through a Dense layer.
   • Activation: A Sigmoid activation function is used for each of the 5 output nodes (independent of each other), enabling multi-label prediction.
   • Loss Function: Binary Cross-Entropy (BCE) is used to optimize the model for multi-label classification.
   • Optimizer: Adafactor (identified as the best performer in experiments) with gradient clipping and a warmup scheduler.

C. Evaluation Strategy

• Cross-Validation: 5-fold cross-validation is used to ensure generalization.
• Metrics: Accuracy, Precision, Recall, F1-score, Hamming Loss, Matthews Correlation Coefficient (MCC), Cohen's Kappa, and AUC-ROC.
• Explainability: LIME (Local Interpretable Model-agnostic Explanations) is integrated to visualize which words contribute to specific label predictions, enhancing model trust.

3. Key Contributions

• Novel Hybrid Architecture: First to combine BanglaBERT-Large with a 2-layer Stacked LSTM specifically for multi-label cyberbullying detection in Bangla.
• State-of-the-Art Performance: Achieved the highest accuracy on the target dataset, outperforming previous single-label and hybrid models.
• Robustness to Imbalance: Systematically evaluated the impact of undersampling vs. oversampling, demonstrating that oversampling yields the best results for this specific dataset.
• Comprehensive Benchmarking: Compared against classical ML (RF, SVM), standalone Deep Learning (LSTM, GRU), and various Transformer models (mBERT, RoBERTa, DeBERTa, IndicBERT).
• Interpretability: Integrated XAI (LIME) to provide transparent insights into model decisions, addressing the "black box" nature of deep learning.

4. Experimental Results

The proposed BanglaBERT-LSTM model achieved superior performance compared to all baselines:

• Accuracy: 94.31% (with oversampling), outperforming the previous state-of-the-art by 0.67%.
• F1-Score: 88.37%.
• Hamming Loss: 0.0569% (indicating very low multi-label error).
• MCC & Kappa: Achieved 80.31% and 80.25% respectively, confirming strong correlation and agreement beyond chance.
• Comparison:
  • Outperformed standalone Transformers (e.g., IndicBERT at 94.00% accuracy).
  • Significantly outperformed classical ML models (Random Forest maxed at 88.95%).
  • Showed better generalization in 5-fold cross-validation (Average Accuracy: 93.42%).

5. Significance and Future Work

• Significance: This work establishes a new benchmark for low-resource language NLP, proving that hybrid architectures (Transformer + RNN) can effectively handle the complexity of multi-label abuse detection where single models fail. It addresses a critical gap in Bangla social media safety.
• Limitations: The current model is restricted to the Bangla language and does not handle code-mixed text (e.g., Banglish).
• Future Directions: The authors plan to extend the framework to support multilingual and code-mixed text detection to improve real-world applicability on diverse social media platforms.