💬 NLP

SemEval-2026 Task 9: Detecting Multilingual, Multicultural and Multievent Online Polarization

This paper presents SemEval-2026 Task 9, a large-scale shared task involving 67 teams that evaluated multilingual online polarization detection across 22 languages through three sub-tasks: identifying polarization presence, type, and manifestation.

Original authors: Usman Naseem, Robert Geislinger, Juan Ren, Sarah Kohail, Rudy Garrido Veliz, P Sam Sahil, Yiran Zhang, Marco Antonio Stranisci, Idris Abdulmumin, Özge Alaçam, Cengiz Acartürk, Aisha Jabr, Saba Anwar

Published 2026-04-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Usman Naseem, Robert Geislinger, Juan Ren, Sarah Kohail, Rudy Garrido Veliz, P Sam Sahil, Yiran Zhang, Marco Antonio Stranisci, Idris Abdulmumin, Özge Alaçam, Cengiz Acartürk, Aisha Jabr, Saba Anwar, Abinew Ali Ayele, Elena Tutubalina, Aung Kyaw Htet, Xintong Wang, Surendrabikram Thapa, Tanmoy Chakraborty, Dheeraj Kodati, Sahar Moradizeyveh, Firoj Alam, Ye Kyaw Thu, Shantipriya Parida, Ihsan Ayyub Qazi, Lilian Wanzare, Nelson Odhiambo Onyango, Clemencia Siro, Ibrahim Said Ahmad, Adem Chanie Ali, Martin Semmann, Chris Biemann, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling global town square. In this square, people from every corner of the world are shouting, debating, and sharing stories. Sometimes, these conversations are healthy and friendly. But often, they turn into a shouting match where two groups stop listening to each other and start hating one another. This is called polarization.

This paper is a report card on a massive international competition called SemEval-2026 Task 9, where computer scientists tried to build "digital detectives" to spot this toxic shouting match before it gets out of hand.

Here is the story of how they did it, broken down into simple parts.

1. The Mission: Building a Universal Translator for Anger

The organizers (a team of researchers from universities worldwide) realized that polarization isn't just an English problem. It happens in Hindi, Swahili, Chinese, Arabic, and dozens of other languages. It also happens for different reasons: sometimes it's about politics, sometimes religion, sometimes race, and sometimes just general identity.

To fix this, they built a massive library of data called POLAR.

The Library: It contains over 110,000 examples of online posts.
The Diversity: These posts are in 22 different languages, covering cultures from Africa to Asia to Europe.
The Labels: Every single post was carefully read and tagged by humans with three specific questions:
1. Is it polarized? (Yes/No)
2. What is it about? (Politics? Religion? Race?)
3. How is it being said? (Is it using insults? Is it dehumanizing people? Is it using extreme language?)

Think of this dataset as a giant "training manual" for computers, teaching them what anger and division look like in many different accents and dialects.

2. The Competition: The "Detective Olympics"

Once the manual was ready, they invited researchers from around the world to build AI systems (digital detectives) to solve three specific puzzles using this data:

Puzzle 1 (The Alarm Bell): Can your AI hear a post and say, "Hey, this is getting heated!"?
Puzzle 2 (The Detective's Notebook): If it is heated, what is the fight about? Is it a political argument or a religious one?
Puzzle 3 (The Rhetoric Radar): How is the person fighting? Are they calling names? Are they saying "You people are animals"? (This is the most dangerous part).

The Crowd: Over 1,000 participants from 28 countries joined the party. They submitted thousands of attempts to train their AI. In the end, 67 teams made it to the final round, submitting 73 different "system descriptions" (essentially, their secret recipes for success).

3. The Winners: Who Built the Best Detectives?

Three teams stood out as the champions, each using a different strategy:

Team UTokyo Tsuruoka Lab: They were the speedsters. They used a very large, smart AI model (Gemma) but taught it to be super efficient. Instead of reading a post and then thinking about it for a long time, they trained it to make a decision in one quick "glance." They won the most first-place spots in the first two puzzles.
Team NYCU-NLP: They were the team players. Instead of relying on one super-smart detective, they built a "committee" of three smaller, slightly less smart AIs. They let these three vote on the answer. If two agreed, they went with that. This "wisdom of the crowd" approach worked incredibly well.
Team SMASH: They were the specialists. They focused heavily on the third puzzle (identifying how people are fighting). They used a technique called "cross-validation," which is like testing a detective's skills on 5 different mock cases before letting them judge the real thing. They won the most first-place spots in the third puzzle.

4. The Big Lessons: What Did We Learn?

The competition revealed some interesting truths about the state of AI today:

One Size Does Not Fit All: An AI that is great at detecting angry political posts in English might be completely confused when it sees a heated religious argument in Swahili. The "cultural context" is huge.
The "Low-Resource" Problem: For languages with lots of data (like English or Chinese), the AI did well. But for languages with less data (like Khmer or Burmese), the AI struggled. It's like trying to teach someone to drive a car when they've only ever seen a bicycle.
The "Other" Category is Hard: The hardest thing for the AIs was identifying polarization that didn't fit into neat boxes like "Politics" or "Religion." When the fight was about something weird or new, the computers got lost.
The Best Tool is a Mix: The winners didn't just use one trick. They mixed "fine-tuning" (teaching the AI specific lessons), "ensembles" (using a team of AIs), and "threshold tuning" (adjusting how sensitive the alarm bell is).

5. Why Does This Matter?

Imagine if your town square had a security guard who could spot a fight before it turned into a riot. That is what this research aims to do. By building better tools to detect polarization early, we can:

Stop hate speech before it spreads.
Help social media platforms manage their communities better.
Protect democracy by keeping conversations civil.

The Bottom Line

This paper is a celebration of a global effort to teach computers how to understand human conflict. While the AI isn't perfect yet (it still gets confused by some cultures and languages), the fact that researchers from 28 countries came together to build a shared library of 110,000 examples is a huge step forward.

They didn't just build a better detector; they built a shared map of how the world argues, which will help researchers everywhere build better, fairer, and more understanding technology for the future.

1. Problem Definition

The paper addresses the critical challenge of detecting online polarization, defined as sharp divisions and antagonism between social, political, or identity groups that threaten democratic institutions and social cohesion. While previous research has focused on hate speech or toxicity, polarization detection requires a more nuanced understanding of rhetorical strategies and contextual divisions.

The specific challenges addressed in this task include:

Multilingualism: Covering 22 languages across 7 language families, including low-resource languages (e.g., Khmer, Burmese, Amharic).
Multiculturalism: Accounting for diverse cultural contexts where polarization manifests differently (e.g., religious vs. political divides).
Multievent: Handling polarization arising from various real-world events (elections, conflicts, migration, climate change).
Fine-grained Classification: Moving beyond simple binary detection to identify specific types of polarization and their rhetorical manifestations.

2. Methodology

2.1 The POLAR Dataset

The core contribution is the POLAR dataset, a large-scale resource comprising 110,656 annotated instances.

Scope: 22 languages (e.g., English, Arabic, Chinese, Hindi, Swahili) sourced from social media, news, and forums.
Annotation Strategy: A hybrid approach using crowd-sourced annotators (Mechanical Turk, Prolific) and trained community annotators for low-resource languages.
Labeling Schema: Each instance is multi-labeled with three dimensions:
1. Presence: Binary classification (Polarized vs. Not Polarized).
2. Type: Categorical classification (Political, Racial/Ethnic, Religious, Gender/Sexual, Other).
3. Manifestation: Identification of rhetorical tactics (Stereotyping, Vilification, Dehumanization, Extreme Language, Lack of Empathy, Invalidation).
Data Distribution: The dataset is split into Train (73k), Development (3.7k), and Test (33k) sets. Inter-annotator agreement (Fleiss's $\kappa$ ) varies by language, ranging from 0.10 (German) to 0.83 (Khmer).

2.2 Task Structure

The shared task was divided into three subtasks:

POLARDETECT (Subtask 1): Binary detection of polarization presence (All 22 languages).
POLARTYPE (Subtask 2): Classification of polarization type given a polarized text (All 22 languages).
POLARMANIFEST (Subtask 3): Identification of rhetorical manifestations (18 languages; Burmese, Italian, Polish, and Russian were excluded due to data constraints).

2.3 Baselines and Evaluation

Baseline: The organizers provided a baseline using LaBSE (Language-agnostic BERT Sentence Embedding), fine-tuned on training data for each language.
Metric: The primary evaluation metric was Average Macro F1-score across languages.

3. Key Contributions

Comprehensive Benchmark: The creation of the POLAR dataset, the first large-scale, fine-grained benchmark for multilingual, multicultural, and multievent polarization.
Granular Taxonomy: Moving beyond binary detection to a multi-label framework that captures the nature (Type) and mechanism (Manifestation) of polarization.
Community Engagement: The task attracted over 1,000 participants from 28 countries, resulting in 67 teams and 73 system description papers, making it the most popular SemEval-2026 task.
Cross-Lingual Insights: The paper provides extensive analysis on how polarization detection models perform across different language families and cultural contexts, highlighting the "cultural gap" in current NLP models.

4. Results and Analysis

4.1 Performance Overview

Subtask 1 (Detection): Generally high performance. Top systems achieved Macro F1 scores >0.8 in many languages. However, performance dropped significantly for low-resource languages (Khmer, Burmese) and some high-resource ones (Italian, German), suggesting the issue is contextual knowledge rather than just linguistic capability.
Subtask 2 (Type Classification): Performance was significantly lower than Subtask 1. Only 3 languages had scores >0.8, and 7 languages were below 0.6. The "Other" category proved particularly difficult.
Subtask 3 (Manifestation): The most challenging task. Only Urdu achieved a score >0.8. Five languages scored below 0.5, indicating a severe difficulty in identifying specific rhetorical tactics across cultures.

4.2 Top-Performing Systems

Three teams dominated the rankings:

UTokyo Tsuruoka Lab: Ranked 1st in 8 languages for Subtask 1 and 7 for Subtask 2.
- Method: Used Gemma-3 models with LoRA fine-tuning. Key innovation: A single-forward-pass inference strategy (predicting labels via one-token inference) rather than traditional multi-head classification or JSON generation, improving efficiency.
NYCU-NLP: Ranked 1st in 6 languages for Subtask 2 and 7 for Subtask 3.
- Method: Ensemble of small instruction-tuned LLMs (Gemma-3, Mistral, Phi-4) using stacking. They employed a heuristic for the "Other" category in Subtask 2 based on Subtask 1 predictions.
SMASH: Ranked 1st in 9 languages for Subtask 3.
- Method: Full fine-tuning of encoder-based models (mDeBERTa, XLM-R, mBERT) with 5-fold cross-validation. They utilized out-of-fold ensemble weight tuning and per-class threshold calibration to handle class imbalance.

4.3 Common Techniques

Model Families: Qwen (31%), LLaMA (20%), and Gemma (19%) were the most used.
Ensembling: Widely used to combine predictions from multiple models or seeds.
Fine-tuning: ~39% of teams used fine-tuning, with many employing Parameter-Efficient Fine-Tuning (PEFT/LoRA).
Loss Optimization: Asymmetric Loss (ASL) and Focal Loss were common to address class imbalance.
Threshold Tuning: Per-label and per-language threshold calibration was critical for improving Macro F1 scores.

5. Significance and Limitations

Significance

Research Impact: The task highlights that current NLP models struggle with cultural nuance. A model performing well in English may fail in Hindi or Swahili not due to language proficiency, but due to a lack of understanding of local sociopolitical contexts.
Practical Application: The dataset and findings are vital for developing tools for content moderation, peacebuilding, and early detection of hate speech escalation.
Open Science: The dataset is publicly available, fostering future research in cross-cultural NLP.

Limitations

Annotation Variability: Despite rigorous QA, crowd-sourced annotation for sensitive topics can introduce noise, especially in low-resource languages.
Annotator Well-being: The hostile nature of polarized content poses psychological risks to annotators.
Data Scarcity: Some languages still have limited data, constraining model generalizability.
Model Coverage: The baseline did not exhaustively test all leading multilingual models, particularly monolingual-specific ones.

Conclusion

SemEval-2026 Task 9 successfully established a new standard for polarization detection. It demonstrated that while Large Language Models (LLMs) are powerful, cultural adaptation and context-aware fine-tuning are essential for effective polarization detection across the globe. The results suggest that future research must move beyond purely linguistic modeling to incorporate sociopolitical context and cultural specificities.