Technical Summary: Continual Learning with Multilingual Foundation Models for Reclaimed Slur Detection

Problem Statement

Automated content moderation systems face significant challenges in distinguishing between hate speech and reclaimed language, particularly within LGBTQ+ contexts. Reclaimed slurs are historically offensive terms repurposed by marginalized communities for self-empowerment and solidarity. When algorithms fail to recognize this context, they risk flagging harmless posts as toxic or missing actual hate speech. This paper addresses the MultiPride task at EVALITA 2026, which focuses on detecting reclaimed versus non-reclamatory usage of LGBTQ+-related slurs in English, Spanish, and Italian tweets.

The task is characterized by three primary methodological obstacles:

1. Data Scarcity: Limited availability of labeled training data.
2. Class Imbalance: A significant disparity between reclaimed (minority) and non-reclaimed (majority) instances, ranging from 9% (English) to 24% (Italian).
3. Cross-Linguistic Variation: Cultural and linguistic differences in how reclamation is expressed, meaning a model effective for English sarcasm may fail to grasp Italian cultural markers.

Methodology

The authors propose a four-stage multilingual framework designed to systematically address these challenges through data-driven model selection, augmentation, domain adaptation, and threshold calibration.

1. Data Analysis and Bias Detection

The framework begins with descriptive analytics and Chi-square tests to identify statistical relationships between language and label distributions. This analysis confirmed that labels are not randomly distributed across languages, justifying the need for language-specific strategies.

2. Foundation Model Selection

Eight multilingual embedding models were evaluated using a five-fold stratified cross-validation approach. The selection criteria prioritized macro-averaged F1 score stability over maximum absolute performance to ensure robustness against data distribution shifts.

• Selected Model: XLM-RoBERTa-large was chosen as the foundation model, achieving a macro-averaged F1 of 0.7553 ± 0.04, balancing performance and computational efficiency.

3. Inductive Transfer Learning with Data Augmentation (RUN 1)

To mitigate data scarcity and class imbalance, the framework employs:

• Back-Translation Augmentation: Using GPT-4o-mini, original tweets were translated into the other two target languages and back-translated. This "one-to-many" schema tripled the training corpus size while preserving semantic content and maintaining the original class distribution ratios (9–24%).
• Dynamic Epoch-Level Undersampling: A custom batch sampler enforces a 1:3 positive-to-negative ratio during training. For every positive instance, three negative instances are randomly selected without replacement. This stochastic approach prevents majority class collapse and overfitting while ensuring the model learns distinguishing features of the minority class.
• Hyperparameter Optimization: Utilizing Optuna with a Tree-structured Parzen Estimator (TPE) sampler and Median Pruner to efficiently search for optimal learning rates, batch sizes, and weight decay.

4. Domain Knowledge Integration (RUN 2)

A secondary pathway introduces domain-specific knowledge via Masked Language Modeling (MLM) pre-training.

• The foundation model undergoes MLM pre-training on the augmented dataset (15% token masking) to deepen its representation of LGBTQ+ discourse patterns.
• This adapted model is then fine-tuned using the same inductive transfer learning pipeline (augmentation, undersampling, and cross-validation) as RUN 1.

5. Language-Specific Threshold Refinement (RUN 3 & RUN 4)

Recognizing that model confidence scores vary significantly across languages, the framework moves away from a universal decision threshold (0.5).

• ROC Analysis: Receiver Operating Characteristic curves are analyzed independently for English, Spanish, and Italian validation sets.
• Optimization: Decision thresholds are tuned to maximize the macro-averaged F1 score for each language.
  • English: Higher threshold (~0.58) required due to higher average confidence scores.
  • Italian: Lower threshold (~0.42) required to capture subtle reclamation cues.
  • Spanish: Intermediate threshold (~0.50).
• Application: RUN 3 applies these thresholds to RUN 1 predictions, and RUN 4 applies them to RUN 2 predictions.

Key Results

The framework was evaluated on an RTX 4090 workstation using mixed precision training. The four experimental runs yielded the following insights:

• Performance Gains: The threshold-based optimization (RUN 3 and RUN 4) yielded an absolute 2–5% improvement in F1 scores without requiring model retraining.
• Model Selection: XLM-RoBERTa-large proved superior to dense retrieval embeddings (e.g., multilingual-e5-large, bge-m3) for this specific classification task.
• Augmentation Efficacy: Back-translation successfully expanded the dataset threefold while preserving the critical minority class distribution.
• MLM Impact: Domain adaptation via MLM (RUN 2) showed language-dependent results. While it offered marginal improvements for English, Spanish and Italian exhibited variable responses, suggesting that morphologically rich languages with limited pre-training data create different optimization landscapes requiring precise hyperparameter tuning.
• Error Analysis:
  • English: Errors often involved confusing sarcasm or hypothetical discourse with reclamation (e.g., questioning permission to use a slur).
  • Spanish: Errors occurred in activist contexts where slurs were reclaimed to oppose heteronormative structures.
  • Italian: The system struggled with false negatives, missing subtle cultural markers of in-group solidarity and intimate language usage.

Significance and Claims

The paper claims that a "one-size-fits-all" approach to multilingual sentiment analysis is insufficient for reclaimed slur detection. The primary contributions are:

1. Rejection of Universal Thresholds: The study demonstrates that optimal decision boundaries vary significantly across languages due to distributional differences in model confidence and linguistic variation. Adjusting thresholds per language is a computationally free method to significantly boost performance.
2. Robust Imbalance Handling: The combination of semantic-preserving back-translation and dynamic undersampling effectively addresses data scarcity and class imbalance without the side effects of aggressive resampling.
3. Pragmatic Limitations: The authors modestly conclude that while the framework improves statistical performance, text-only models struggle with pragmatic nuances. The system frequently confuses sarcasm with reclamation (English) or misses implicit solidarity (Italian).
4. Future Direction: The paper suggests that future accuracy improvements require moving beyond text to include non-linguistic cues (emojis, tone, social relationship signals) and culturally grounded discourse modeling, rather than relying solely on lexical features.

The methodology is fully reproducible, with code and experimental setups available via the provided GitHub repository.