mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection

Technical Summary: mdok-style at SemEval-2026 Task 9

Problem Definition
SemEval-2026 Task 9 addresses the critical challenge of detecting online polarization across 22 languages. The task is structured into three subtasks: (1) binary detection of whether a text expresses polarization; (2) classification of the polarization type based on social dimensions (e.g., political, racial/ethnic, religious); and (3) identification of rhetorical manifestations (e.g., dehumanization, lack of empathy, invalidation). The authors note that online polarization often precedes hate speech and social fragmentation, making early detection vital for safer digital spaces.

Methodology
The submitted system, named mdok-style, adapts the mdok (machine detector of KInIT) framework, originally developed for detecting machine-generated text at PAN@CLEF2025. The approach relies on fine-tuning mid-size Large Language Models (LLMs) using the QLoRA parameter-efficient fine-tuning (PEFT) technique with 4-bit quantization.

• Data Augmentation: To enhance robustness and increase training volume, the authors applied four augmentation techniques to the training data:
  1. Anonymization: Replacing emails, user mentions, and phone numbers with tags to reduce bias toward specific personal information.
  2. Case Variation: Converting duplicated texts to all lowercase and all uppercase to make detection invariant to visual style.
  3. Homoglyphication: Replacing characters with visually similar counterparts from other scripts to simulate obfuscation attacks and improve resilience against tokenization disruption.
• Model Selection:
  • For Subtask 1 (binary detection), Qwen3-32B was selected as the optimal base model due to its performance across 22 languages.
  • For Subtasks 2 and 3 (multi-label classification), the authors encountered difficulties fine-tuning Qwen3 models for multi-label scenarios. Consequently, they switched to Gemma-3-27B-pt, which supports over 140 languages.
• Training Strategy: The training sets for all 22 languages were combined to focus the model on the task rather than specific languages. Validation sets were constructed by sampling 100 texts per language and label. The models were fine-tuned for a single epoch using a constant learning rate of 2e-5, with checkpoint selection based on the best validation metric (AUC for Subtask 1, macro F1 for Subtasks 2 and 3).
• Alternative Approach (Appraisal Theory): The authors explored an alternative method using appraisal-theoretic emotion detection (based on the ISEAR corpus and AppraisePLM architecture) to capture cognitive signals of polarization. While this was not included in the final submitted system due to time constraints, it was evaluated on a hold-out portion of the training data.

Key Contributions

• Cross-Task Adaptation: Successfully transferring a machine-generated text detection framework (mdok) to the domain of polarization detection.
• Robust Augmentation: Demonstrating that specific text augmentations (anonymization, case variation, homoglyphs) improve model robustness against visual obfuscation and style variations common in social media.
• Multilingual Unification: Implementing a single system capable of handling 22 diverse languages, addressing the challenge of low-resource languages within a shared-task setting.
• Open Reproducibility: The authors have published the full source code and the IMGTB framework dependencies to ensure the detectors are replicable.

Results
The system's performance was evaluated using macro-averaged F1 scores per language:

• Subtask 1 (Detection): The system performed competitively, achieving macro F1 scores above 0.63 for all languages (ranging from 0.6293 for Khmer to 0.9237 for Chinese). It outperformed the official baseline in approximately two-thirds of the cases, with an average improvement of 3%.
• Subtask 2 (Type) & Subtask 3 (Manifestation): Performance was more variable. The system struggled with the multi-label nature of these tasks, particularly for languages with fewer samples or complex label distributions. For instance, Hausa achieved a 0.0 F1 in Subtask 3. However, the system achieved top-tier rankings (top 20%) in 14 out of 62 language-subtask combinations.
• Appraisal-Based Classifier: The alternative appraisal-based system showed promise in Subtask 1 but yielded random performance in Subtasks 2 and 3. However, AUC analysis suggested that appraisal dimensions could serve as effective polarization indicators worthy of further exploration.

Significance and Claims
The authors claim that their work demonstrates a single, unified system can effectively handle polarization detection across a wide range of languages, competing with other specialized systems. They acknowledge that while the approach is highly effective for binary detection (Subtask 1), the multi-label classification tasks (Subtasks 2 and 3) remain more challenging.

The paper modestly concludes that while the current system is competitive, future work could involve fine-tuning models individually for specific languages or language groups rather than a single monolithic model. Additionally, the authors suggest that the combination of fine-tuned detectors with appraisal-based signals represents a promising direction for future research, despite the current system's reliance on the former. The work highlights the potential of parameter-efficient fine-tuning and data augmentation in creating robust, multilingual NLP tools for social good.