💬 NLP

Multilingual Polarization Detection Using Transformer-Based Models with Class Weighting and Threshold Tuning

This paper presents a transformer-based approach using RoBERTa and AfroXLMR models with class weighting and threshold tuning to achieve competitive performance in detecting multilingual polarization, types, and manifestations for English and Swahili in SemEval-2026 Task 9.

Original authors: Aaron Bundi Anampiu

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aaron Bundi Anampiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, noisy town square where millions of people are shouting their opinions. Sometimes, these conversations are healthy debates, but often, they turn into shouting matches where people stop listening and start attacking each other based on their beliefs, race, religion, or gender. This is called polarization.

The paper you're reading is a report card from a team named "Aaron" who entered a global competition (SemEval-2026) to build a robot that can listen to this town square, specifically in English and Swahili, and figure out three things:

Is this conversation polarized (a shouting match) or not?
What is the fight about (politics, race, religion, etc.)?
How are they fighting (using stereotypes, insults, or dehumanizing language)?

Here is how they built their robot and what they found, explained simply:

1. The Tools: Choosing the Right "Ears"

The team didn't use a generic robot that speaks all languages equally. They realized that just like a person might understand a local dialect better than a foreigner, AI models need to be specialized.

For English: They gave the robot "RoBERTa," a model trained specifically to understand the nuances of English.
For Swahili: They gave the robot "AfroXLMR," a model specially trained on African languages.
The Analogy: Imagine trying to listen to a debate in a crowded room. If you use a generic hearing aid, you might miss the whispers. But if you use a hearing aid tuned specifically to the frequency of the room, you hear everything clearly. That's what they did with the models.

2. The Problem: The "Rare Guest" Issue

In the data they analyzed, most conversations were about common topics (like general politics), but some topics were very rare (like gender or sexual polarization).

The Analogy: Imagine a classroom where 95% of students are wearing red shirts and only 5% are wearing blue shirts. If you ask a student to guess the shirt color of a new person, they will almost always guess "red" because it's the most common. They will fail to spot the "blue" shirts.
The Fix: The team taught their robot to pay extra attention to the "blue shirts." They used a Class-Weighted Loss function. Think of this as a teacher who says, "If you miss a red shirt, it's a small mistake. But if you miss a blue shirt, you get a huge penalty." This forced the robot to learn the rare, difficult cases better.

3. The Fine-Tuning: Adjusting the Sensitivity

After training, the robot had to decide: "Is this text polarized or not?" Usually, robots use a 50/50 rule (if it's 50% likely, say yes). But the team found this rule was too rigid.

The Analogy: Think of a metal detector at an airport. If it's set too low, it ignores real weapons. If it's set too high, it screams at people for having keys. The team tuned the threshold for every single type of polarization. They adjusted the sensitivity for "stereotypes" differently than they did for "lack of empathy," ensuring the robot caught the right amount of trouble without crying wolf.

4. The Results: How Did They Do?

The robot performed very well, especially compared to other teams and older methods.

Binary Detection (Is it polarized?): The robot was about 79% accurate in both English and Swahili. It was slightly better at spotting polarization in Swahili, likely because the Swahili data had a more even mix of polarized and non-polarized texts.
Type & Manifestation (What and How?): This was harder. The robot got about 46–58% accurate. This is still a strong score because the task is incredibly difficult (like trying to guess the specific flavor of ice cream in a dark room).
The Ranking: For the Swahili task of identifying how people were fighting (manifestations), this team came in 2nd place out of 16 teams.

5. Where the Robot Stumbled (Error Analysis)

Even the smartest robots make mistakes. The team looked at the errors and found three main trouble spots:

Confusing "Strong" with "Bad": The robot sometimes thought a strong sports comment ("The Lakers will destroy the Celtics") was a hateful polarized attack. It confused intensity with polarization.
The "Theory of Mind" Gap: The robot struggled to detect dehumanization (treating people like objects) and lack of empathy. These require understanding deep human feelings and metaphors, which is like asking a calculator to understand a poem.
The Rare Labels: Even with their special "penalty" training, the robot still missed the very rare categories (like gender/sexual polarization) because there simply weren't enough examples to learn from.

Summary

The team built a specialized listening system that uses different "ears" for English and Swahili, teaches the system to care deeply about rare topics, and fine-tunes its sensitivity to catch the most trouble. While it isn't perfect yet—especially when it comes to understanding deep human cruelty or sarcasm—it is a significant step forward in helping us understand and moderate online fights.

Important Note: The authors explicitly state that this is a research tool. They warn that if used in the real world without human supervision, it could accidentally silence legitimate political voices or unfairly target marginalized groups. They believe humans must always be in the loop to make the final call.

Technical Summary: Aaron at SemEval-2026 Task 9

Problem Definition

This paper addresses SemEval-2026 Task 9, which focuses on detecting online polarization across multiple languages, cultures, and events. The task is divided into three specific subtasks for English and Swahili:

Binary Detection: Classifying text as polarized or non-polarized.
Type Classification: A multi-label classification identifying five polarization types (political/ideological, racial/ethnic, religious, gender/sexual, and other).
Manifestation Identification: A multi-label classification identifying six specific manifestations of polarization (stereotype, vilification, dehumanization, extreme language, lack of empathy, and invalidation).

The primary challenges identified are severe label imbalance (particularly for minority classes like gender/sexual polarization and specific manifestations) and the need for multilingual adaptability, specifically for African languages where general multilingual models may underperform.

Methodology

The proposed system employs a four-stage pipeline: preprocessing, language-specific model architecture, class-weighted training, and post-training threshold tuning.

1. Data Preprocessing

The authors adopted a minimal preprocessing strategy to preserve polarization signals. Original casing, emojis, and special characters were retained, while only control characters and null bytes were removed. Data was split 80/20 into training and validation sets using stratified sampling to maintain class distributions.

2. Model Architecture

The system utilizes transformer-based models selected specifically for each language:

English: RoBERTa-base (125M parameters) was chosen over BERT and XLM-RoBERTa. Ablation studies indicated that RoBERTa's dynamic masking and lack of next-sentence prediction yielded superior English-specific performance, whereas XLM-RoBERTa's multilingual pre-training diluted English representations.
Swahili: AfroXLMR-base (270M parameters) was selected. This model is specifically adapted for African languages through continued pre-training on diverse African corpora, outperforming general multilingual models like mBERT and XLM-RoBERTa on Swahili tasks.

Both models utilize a classification head consisting of a dropout layer (0.1) followed by a linear layer mapping 768-dimensional embeddings to the required output logits (2 for Subtask 1, 5 for Subtask 2, and 6 for Subtask 3).

3. Class-Weighted Loss Function

To address severe label imbalance, the authors implemented a class-weighted binary cross-entropy loss. The weight for each label $j$ ( $w_j$ ) is calculated as the ratio of negative to positive samples ( $n_{neg} / n_{pos}$ ). This increases the penalty for misclassifying minority classes. For instance, in English Subtask 2, weights ranged from 1.73 (political) to 41.95 (gender/sexual). Gradient clipping (max-norm 1.0) was applied to ensure training stability.

4. Threshold Tuning

Following training, a per-label threshold search was conducted on the validation set. Thresholds were scanned from 0.1 to 0.9 in steps of 0.05 to maximize the macro F1 score for each specific label. This approach improved macro F1 scores by 0.87 to 5.98 points compared to the standard 0.5 threshold.

Training Configuration

Models were fine-tuned using AdamW (learning rate $2 \times 10^{-5}$ , weight decay 0.01) with a linear warmup schedule. Training utilized mixed precision (FP16) on a single GPU. Batch sizes were 16 for Subtask 1 and 8 (with gradient accumulation) for Subtasks 2–3 to accommodate sequence lengths of 128 and 256 tokens, respectively. Early stopping was based on validation macro F1.

Key Contributions

The paper highlights three primary technical contributions:

Language-Specific Model Selection: Demonstrating that specialized models (RoBERTa for English, AfroXLMR for Swahili) outperform general multilingual baselines for polarization detection in these specific contexts.
Class-Weighted Loss Implementation: Successfully applying weighted loss functions to mitigate the effects of extreme label imbalance in multi-label polarization tasks.
Per-Label Threshold Tuning: Optimizing classification boundaries individually for each label, which proved critical for improving macro F1 scores in imbalanced datasets.

Experimental Results

The system was evaluated on the test set using Macro F1 as the primary metric.

Subtask 1 (Binary Detection): Achieved scores of 0.7901 (English) and 0.7910 (Swahili). Swahili performance slightly exceeded English, attributed to a more balanced class distribution in the Swahili dataset.
Subtask 2 (Type Classification): Achieved scores of 0.4615 (English) and 0.4808 (Swahili). Performance was hindered by extreme imbalance, particularly for gender/sexual and "other" categories.
Subtask 3 (Manifestation Identification): Achieved scores of 0.4791 (English) and 0.5830 (Swahili). The system ranked 2nd out of 16 teams for Swahili on this subtask.

Comparative Performance:
The system consistently outperformed both the POLAR Baseline and an mBERT fine-tuned baseline across all subtasks and languages. For example, in Swahili Subtask 3, the proposed system (0.5830) significantly outperformed the mBERT baseline (0.4380).

Ablation Studies:

Class Weighting: On English Subtask 2, removing class weighting dropped the macro F1 from 0.4721 to 0.4123, a decrease of 5.98 points.
Model Choice: RoBERTa-base outperformed BERT-base by 2.58% on English Subtask 1, while the larger XLM-RoBERTa underperformed, confirming the benefit of language-specific pre-training.

Error Analysis and Limitations

The authors identified specific failure modes:

False Positives: Neutral political headlines and hyperbolic sports comments were occasionally misclassified as polarized due to the model conflating strong language with polarization.
Multi-Label Ambiguity: In Subtask 2, the model struggled to detect racial undertones within political statements.
Rare Categories: The gender/sexual label (2.2% frequency) and dehumanization manifestation remained difficult to detect, even with class weighting.
Semantic Nuance: Dehumanization (31% of errors) and lack of empathy require nuanced semantic understanding and theory-of-mind reasoning that standard transformers struggle to capture.
Validation-Test Gap: A discrepancy of 2.29 to 10.58 points was observed between validation and test scores, attributed to threshold overfitting on a single validation split and potential domain shifts in the test data.

Significance and Claims

The paper claims that the proposed approach demonstrates competitive performance on the leaderboard, particularly highlighting the 2nd place ranking for Swahili in Subtask 3. The authors assert that their methods are effective for handling imbalanced multi-label polarization detection in both high-resource (English) and low-resource (Swahili) contexts.

The work underscores the necessity of:

Moving beyond generic multilingual models for African languages.
Employing specific techniques (class weighting and threshold tuning) to address the inherent skew in polarization datasets.

The authors conclude that while the system is effective, future work must address pragmatic reasoning, cultural modeling, and cross-lingual transfer to further improve detection of subtle manifestations like dehumanization and lack of empathy. They also emphasize the need for human oversight in real-world deployment to prevent the suppression of legitimate political expression.