Technical Summary: Multi-Stage Training for Abusive Comment Detection in Indic Languages

Problem Statement

The paper addresses the critical challenge of detecting abusive content on social media platforms, specifically within the context of Indic languages. While social media facilitates communication, it also hosts harmful content that requires automated moderation. The authors highlight that this task is inherently difficult due to the subjectivity of annotation, linguistic nuances (such as irony, humor, and spelling variations), and the scarcity of resources for Indian languages.

A specific dataset used for this study comprises 700,000 comments from the "Moj" app, covering over 15 Indian languages. The dataset presents significant challenges:

• Label Noise: The authors estimate that approximately 9–10% of the training data contains annotation errors (e.g., sarcastic comments marked as abusive or toxic comments marked as non-abusive).
• Linguistic Complexity: Comments often contain code-mixing (e.g., Hinglish) or are written in Latin script rather than native scripts.
• Resource Scarcity: Pre-trained language models for Indic languages are often trained on small corpora, limiting their effectiveness.

Methodology

The authors propose a multi-stage pipeline involving data preprocessing, model training, and post-processing to minimize false positives and handle label noise.

1. Data Preprocessing

• Cleaning and Transliteration: The raw dataset was cleaned to remove non-text entities (e.g., HTML tags). Crucially, comments written in Latin script (e.g., Hinglish) were transliterated into their assigned native Indic languages using the indic-trans library.
• Data Merging: An experiment was conducted where the original raw text and the cleaned/transliterated text were concatenated to provide the model with richer syntactic and semantic information.
• Dimensionality Reduction: Principal Component Analysis (PCA) was applied to reduce XLM-R embeddings from 768 dimensions to 200 to optimize performance.

2. Model Architecture and Training

The approach utilizes an ensemble of deep learning and gradient boosting models:

• Base Transformers: The authors fine-tuned XLM-R (a multilingual RoBERTa model) and Muril (a BERT variant specifically for Indic languages). XLM-R consistently outperformed Muril in cross-validation (CV) scores.
• Feature Engineering: Contextual embeddings from the transformer models were concatenated with explicit feedback metadata (number of likes and reports) from the dataset.
• Gradient Boosting: The pooled outputs (embeddings) from the transformer models were used as input features to train LightGBM (LGBM) classifiers.
• Language-Wise Modeling: Instead of a single global model, separate LGBM models were trained for each language within a 10-fold cross-validation setup. This yielded significant performance improvements over language-agnostic training.
• Pseudo-Labeling: To address label noise, the authors implemented an iterative pseudo-labeling technique. Predictions from the best-performing model were used as additional metadata to train subsequent models. This process was repeated until performance stagnated.

3. Post-Processing

The final stage involved applying language-specific thresholds to the model predictions. This simple adjustment allowed for fine-tuning the balance between precision and recall for each specific Indic language.

Key Contributions

The paper outlines three primary contributions:

1. Dataset Improvisation: Creation of a cleaned and transliterated version of the raw dataset, converting comments to their assigned native languages.
2. Robust Pipeline: Development of an effective, language-wise training pipeline that integrates transformer embeddings, metadata, and gradient boosting.
3. Noise Handling: Demonstration of a pipeline that effectively manages label noise through pseudo-labeling and language-specific thresholding, aiming to reduce false positives without compromising freedom of expression.

Results

The authors report a progression of scores (F1-score) through various stages of experimentation:

• Baseline: XLM-R fine-tuning achieved a CV score of 0.8804.
• Feature Augmentation: Adding metadata and concatenating raw/transliterated text improved the score to 0.8822.
• LGBM Integration: Using XLM-R embeddings with LGBM raised the score to 0.89481.
• Language-Wise Training: Training separate LGBM models per language boosted the score to 0.89801.
• Pseudo-Labeling: Iterative pseudo-labeling with language-wise LGBM achieved a single-model score of 0.89934.
• Final Ensemble: A weighted ensemble of high-scoring models, combined with language-wise post-processing thresholds, achieved a final score of 0.90000 on the leaderboard (LB).

Significance and Claims

The paper claims that its proposed pipeline successfully minimizes the false-positive rate, a crucial factor in preventing the suppression of legitimate speech while detecting abuse. The authors emphasize that their approach effectively handles the "noise" inherent in human-annotated datasets, a common issue in abusive content detection.

The study concludes that while innovative modeling procedures (such as pseudo-labeling and language-specific thresholds) have yielded strong results, there remains room for improvement. The authors modestly suggest that future work could focus on optimizing models with learning rate schedulers and developing more sophisticated methods to handle label noise directly. They do not claim to have solved the problem entirely but have provided a robust framework that significantly advances performance on Indic language abusive comment detection.