Challenger at MultiPRIDE: Is It Hate Speech or Reclaimed?

Technical Summary: Distinguishing Hate Speech from Reclaimed Language in MultiPRIDE

Problem Statement
The paper addresses a critical challenge in automated hate speech detection: differentiating between genuine hate speech and "reclaimed language." While hate speech targets individuals based on identity (religion, race, sexual orientation, etc.), members of marginalized groups (such as the LGBTQ+ community) often repurpose traditionally offensive terms to foster in-group solidarity through "semantic reclamation" or "mock impoliteness." Automated systems frequently misclassify these reclaimed expressions as hateful, leading to false positives that restrict self-expression. This task is complicated by the context-dependent nature of language, where meaning relies heavily on speaker identity and usage context, and is further exacerbated by extreme class imbalance in available datasets.

Methodology
The authors propose a language-agnostic pipeline designed to operate efficiently under limited computational resources while maintaining interpretability. The approach consists of four primary stages:

1. Data Preprocessing and Augmentation:

   • Cleaning: Raw text is cleaned using regular expressions to remove URLs, user mentions, and non-ASCII characters (including emojis). Crucially, the # symbol is stripped from hashtags to retain their semantic textual content.
   • Augmentation: To address severe class imbalance, the authors employ a back-translation strategy. Minority class samples are translated into pivot languages (French, German, Russian, Persian) via the Google Translate API and translated back to the source language. This generates synthetic positive samples to enrich the training distribution.

2. Feature Extraction and Label Cleaning:

   • Embeddings: Text samples are converted into 1024-dimensional dense semantic vectors using the state-of-the-art intfloat/e5-large-v2 Sentence Transformer model.
   • Noise Filtering: The authors utilize the Cleanlab framework (specifically Confident Learning) to identify and remove label noise. A Logistic Regression classifier is fitted to the embeddings to detect instances with a high probability of label errors. These noisy instances are filtered out before the final training phase to ensure a cleaner training signal.

3. Model Architecture:

   • The core classifier is a lightweight Multi-layer Perceptron (MLP) built on TensorFlow/Keras.
   • Structure: The MLP accepts the sentence embeddings and passes them through two dense hidden layers (16 and 8 units, respectively) using ReLU activation. The output layer uses a Sigmoid activation function for binary probability scoring.
   • Training: The model is trained using the Adam optimizer and Binary Crossentropy loss. To mitigate class imbalance, class weights are computed to penalize misclassifications of the minority class more severely.

4. Validation and Threshold Tuning:

   • Data is split using stratified sampling (80/20) to preserve label proportions.
   • Instead of a fixed decision boundary of 0.5, the system employs an adjusted threshold optimization method. It searches a range of decision boundaries (0.1 to 0.9) to select the threshold that maximizes the F1-score on the validation set.

Key Contributions
The paper outlines the following specific contributions to the MultiPRIDE shared task:

• Systematic text cleaning that preserves hashtag semantics while removing noise.
• Implementation of stratified sampling to ensure reliable evaluation on highly unbalanced data.
• Application of back-translation augmentation to balance the training distribution.
• Extraction of deep semantic features using the intfloat/e5-large-v2 model.
• Enhancement of training signal quality via Cleanlab for label noise filtering.
• Design of a lightweight, task-optimized Neural Network that balances performance with computational efficiency.

Results
The system was evaluated on the MultiPRIDE dataset, which includes English, Italian, and Spanish tweets. The evaluation metrics included Precision, Recall, and F1-score, with a focus on macro-averaged metrics.

• Italian: Achieved the strongest performance with a Macro-avg F1-score of 0.8815 (Class 0 F1: 0.9570; Class 1 F1: 0.8060).
• Spanish: Achieved a Macro-avg F1-score of 0.6773 (Class 0 F1: 0.8361; Class 1 F1: 0.5185).
• English: Showed the lowest performance with a Macro-avg F1-score of 0.6203 (Class 0 F1: 0.9318; Class 1 F1: 0.3089).

The authors note that the lower performance on the minority class (Reclaimed language) across all languages, particularly in English, is consistent with the challenges of extreme class imbalance in the dataset rather than model instability.

Significance and Claims
The paper claims that its methodology demonstrates that lightweight algorithms can effectively learn class relationships even with small training sets and limited computational resources. The authors position their approach as a "minimal alternative" that offers a balance between effectiveness and efficiency.

The significance of the work lies in its ability to operate in a multilingual environment without relying on heavy computational models, making it accessible for resource-constrained scenarios. The authors modestly conclude that while their system performs well, the variation in performance across languages (specifically the lower results for English) highlights the difficulty of the task. They suggest that future improvements could be achieved by utilizing larger embedding models, more advanced preprocessing, and complex learning approaches, but emphasize that their current approach provides a robust starting point for distinguishing hate speech from reclaimed language.