Lingo_Research_Group at SemEval-2026 Task 9: Evaluating Prompt Variants for Polarization Detection

Technical Summary: Lingo_Research_Group at SemEval-2026 Task 9

Problem Definition
The paper addresses SemEval-2026 Task 9, a multilingual text classification challenge focused on polarization detection. The task is divided into three subtasks of increasing complexity:

1. Subtask 1 (Binary Detection): Determining whether a social media post exhibits attitude polarization (divisive in-group/out-group framing, hostility, or exclusion).
2. Subtask 2 (Target Classification): Identifying the specific targets of polarization (political, racial/ethnic, religious, gender/sexual, or other) in a multi-label setting.
3. Subtask 3 (Manifestation Identification): Detecting the rhetorical forms of polarization, such as stereotypes, vilification, dehumanization, extreme language, lack of empathy, or invalidation.

The challenge lies in the fact that polarization is often realized through framing, rhetoric, and cultural context rather than explicit hate speech, making it difficult to detect, particularly across 22 diverse languages with varying discourse patterns.

Methodology
The authors propose a prompt-based inference approach using Large Language Models (LLMs) without parameter fine-tuning. The core methodology involves:

• Prompt Engineering: Instead of a single prompt, the team designed and evaluated 12 distinct prompt variants. These variants systematically varied in terminology clarity, definition detail, reasoning guidance, and the inclusion of in-context examples. The prompts were structured as a conceptual gradient, moving from minimal context to detailed definitions and step-by-step reasoning instructions.
• Model Selection: Experiments were conducted using two multilingual LLMs: aya-101 and Gemma3-27B. Based on performance comparisons across subtasks and prompt configurations, Gemma3-27B was selected for the final submission due to its superior macro F1-scores.
• Inference Strategy: The system treats polarization analysis as an instruction-following problem. For Subtask 1, the model predicts a binary label. For Subtasks 2 and 3, it generates a list of binary predictions for each target or manifestation label. The team utilized a "conservative" prompting strategy, predicting polarization only when explicit and unambiguous evidence was present.
• Evaluation: The system was evaluated on held-out test sets across 22 languages using macro-averaged F1-scores to account for class imbalance. No external data or augmentation was used.

Key Results
The system achieved the following average macro F1-scores on the official test set:

• Subtask 1: 0.762 (Average Accuracy: 0.819)
• Subtask 2: 0.587 (Average Accuracy: 0.678)
• Subtask 3: 0.444 (Average Accuracy: 0.498)

Performance varied significantly by language and subtask. For instance, Chinese and Nepali achieved high F1-scores (up to 0.92) in Subtask 1, while Italian scored as low as 0.35. The authors observed a systematic degradation in performance as the task complexity increased from binary detection to fine-grained, multi-label sociolinguistic classification.

Analysis and Findings
The paper provides a detailed analysis of the results, highlighting several key trends:

• Task Complexity: There is a clear inverse relationship between task abstraction and performance. While prompt-based approaches are effective for coarse-grained detection (Subtask 1), they struggle with fine-grained, multi-label tasks (Subtasks 2 and 3) where labels are abstract (e.g., "lack of empathy") and lack clear lexical indicators.
• Recall Suppression: The conservative prompting strategy led to high accuracy but low macro F1 in Subtasks 2 and 3. The models frequently under-predicted positive labels, particularly in languages where polarization is conveyed indirectly through sarcasm, rhetorical questions, or cultural references (e.g., Hausa, Italian, Khmer).
• Language-Specific Nuances: Performance was not uniform across languages. Nepali and Chinese performed well in Subtasks 1 and 2 due to explicit group references, while Hindi performed best in Subtask 3, likely due to its prevalence of explicit rhetorical tactics (vilification, extreme language) that align well with the model's pretraining priors.
• The English Paradox: Despite extensive pretraining on English data, English underperformed relative to the median. The authors attribute this to the prevalence of irony, sarcasm, and ideological shorthand in English polarization, which conservative prompts often misclassify as non-polarized due to a lack of surface-level hostility.
• Translation Limitations: The paper notes that translation-based reasoning (encoding inputs in English before classification) can defuse emotionally charged expressions, failing to preserve the rhetorical intent necessary for manifestation identification.

Significance and Claims
The paper claims that prompt-based approaches using multilingual LLMs are a viable method for detecting coarse-grained polarization without the need for task-specific fine-tuning or large annotated datasets. However, the authors modestly conclude that these methods encounter increasing difficulties as the task moves toward fine-grained, multi-label sociolinguistic classification.

The work demonstrates that prompt design is critical; the articulation of task definitions, decision limits, and reasoning expectations heavily influences model performance. The study highlights that while LLMs can handle binary detection effectively, the "semantic equivalence" of translation does not guarantee the preservation of rhetorical intent, suggesting that future work must address the nuances of cross-lingual discourse and the limitations of conservative prompting in capturing implicit polarization.

Ethical Considerations
The authors emphasize that their system should not be used as a standalone moderation tool due to the risks of false positives (unfair banning of legal speech) and false negatives (tolerating dangerous rhetoric). They also acknowledge significant performance disparities across languages, which may stem from pretraining data representation, cultural variations, and annotation density.