💬 NLP

PSK at SemEval-2026 Task 9: Multilingual Polarization Detection Using Ensemble Gemma Models with Synthetic Data Augmentation

The PSK team achieved 2nd place overall in SemEval-2026 Task 9 with a mean macro-F1 of 0.811 by employing an ensemble of LoRA-fine-tuned Gemma 3 models augmented with GPT-4o-mini-generated synthetic data and per-language threshold tuning, while demonstrating the critical importance of generalization over architectures that performed well on development sets but failed on the test set.

Original authors: Srikar Kashyap Pulipaka

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Srikar Kashyap Pulipaka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of 22 different students (representing 22 different languages) how to spot a "heated argument" versus a "calm conversation" in social media posts. This was the challenge for the SemEval-2026 Task 9, and the team PSK (led by independent researcher Srikar Kashyap Pulipaka) built a system to solve it.

Here is how they did it, explained simply:

1. The Goal: Spotting the "Heat"

The task was to read social media posts in 22 languages (from English to Amharic) and decide: Is this post polarized (full of stereotypes, hate, or intolerance)? Or is it neutral?

The Challenge: Some languages had very few examples to learn from, and "polarization" looks different in every culture. It's like trying to teach someone to recognize a "storm" when they've only seen rain in one country.

2. The Teachers: The "Gemma" Models

Instead of using one giant teacher for everyone, the team hired two specific "tutors" for each language:

The 12B Tutor: A smart, efficient model (12 billion parameters).
The 27B Tutor: An even smarter, more powerful model (27 billion parameters).
They used a technique called LoRA, which is like giving these tutors a small, specialized "cheat sheet" to learn the specific language without having to rewrite their entire brains.

3. The Study Buddy: Synthetic Data

The team realized some students didn't have enough practice problems. So, they used an AI (GPT-4o-mini) to write fake practice problems (synthetic data).

Direct Generation: The AI wrote new, fake arguments from scratch.
Paraphrasing: The AI took real examples and rewrote them in different words (like a student rephrasing a homework question).
Contrastive Pairs: The AI created "twin" examples—one polarized and one calm—so the model could see the exact difference.
The Filter: They didn't just dump this fake data in. They ran it through a strict "quality control" filter to remove duplicates and nonsense, ensuring the students only studied high-quality material.

4. The Strategy: Tuning the "Volume Knob"

After training, the models had to make a final decision. Usually, AI says "Yes" if it's more than 50% sure. But the team found that for some languages, 50% was too high or too low.

The Fix: They treated the decision threshold like a volume knob. For some languages, they turned the knob down to 30% (to catch more arguments), and for others, they turned it up to 70% (to avoid false alarms). This simple tweak improved their scores by 2–4% without any extra training.

5. The "Ensemble": A Panel of Judges

For the final decision, they didn't just listen to one tutor. They used an Ensemble:

They let both the 12B and 27B tutors vote.
Sometimes they averaged the votes; other times, they gave more weight to the smarter tutor.
For each language, they picked the voting strategy that worked best on practice tests.

6. The Results: Who Won?

The Winner: The team finished 2nd overall out of 60 teams.
The Score: They achieved a score of 0.811 (a measure of accuracy). They took 1st place in 3 languages and top-3 in 8 others.
The Surprise: They tried using other famous AI models (XLM-RoBERTa and Qwen3). These models did great on practice tests but crashed on the real test, losing 30–50% of their accuracy.
- The Lesson: It's better to have a model that generalizes well (learns the concept of polarization) than one that just memorizes the practice answers. The Gemma models were the only ones that didn't "forget" when the test started.

7. The One Weak Spot: The Italian "Missing Chapter"

The system struggled the most with Italian. Why?

The training data for Italian was missing entire categories of topics (like "politics" or "other" issues).
However, the final test included many questions about those missing topics.
It's like studying for a math test only on addition, but the final exam includes a huge section on division. The model had never seen those specific types of arguments before, so it couldn't guess correctly.

Summary

The team built a system that uses specialized AI tutors, AI-generated practice problems, and custom-tuned decision rules to spot online arguments in 22 languages. Their secret sauce wasn't just using the biggest AI, but using the one that could actually learn the concept of polarization without memorizing the answers, and then fine-tuning the rules for each specific language.

Technical Summary: PSK at SemEval-2026 Task 9

Problem Definition
The paper addresses SemEval-2026 Task 9, a binary classification challenge focused on detecting attitude polarization in social media text across 22 typologically diverse languages. The task involves classifying text as either "polarized" (involving stereotyping, vilification, dehumanization, or intolerance) or "non-polarized." The dataset presents significant challenges, including varying dataset sizes (ranging from ~1,700 to ~7,000 samples per language), cultural variations in how polarization manifests, and severe class imbalances in specific languages (e.g., Khmer with a 10:1 ratio of polarized to non-polarized samples). The primary evaluation metric is the macro-averaged F1-score.

Methodology
The team's approach centers on fine-tuning separate models for each language using Google's Gemma 3 family, augmented by synthetic data and ensemble strategies.

Model Architecture: The system utilizes two model sizes: Gemma 3 12B (primary model) and Gemma 3 27B (ensemble model). Both are fine-tuned using Low-Rank Adaptation (LoRA) with a sequence classification head. The 12B model uses 8-bit quantization on a single A100 GPU, while the 27B model employs 4-bit NF4 quantization across two A100 GPUs. Training parameters include a learning rate of $5 \times 10^{-5}$ , 3 epochs, and batch sizes of 1–2 with gradient accumulation.
Synthetic Data Augmentation: To address data scarcity, the team generated approximately 1,000 synthetic samples per language using GPT-4o-mini via three strategies:
1. Direct Generation (50%): Native generation of samples across five culturally relevant topics (political, ethnic/racial, religious, social class, international relations).
2. Label-Preserving Paraphrasing (30%): Paraphrasing real training samples with a temperature of 0.7, filtered to maintain cosine similarity below 0.90 with the original.
3. Contrastive Pairs (20%): Minimal-edit pairs on the same topic, containing one polarized and one non-polarized instance to sharpen class boundaries.
- Quality Control: A multi-stage filtering pipeline was applied, including basic cleaning, regex-based label leakage detection, embedding-based deduplication (using paraphrase-multilingual-MiniLM-L12-v2 with a 0.90 threshold), and round-trip translation consistency checks.
Ensemble and Threshold Tuning:
- Threshold Optimization: Instead of using a default threshold of 0.5, the team searched for optimal thresholds ( $t \in \{0.3, \dots, 0.7\}$ ) on the development set to maximize macro-F1 for each language individually.
- Ensemble Strategies: Four strategies were evaluated per language: 12B only, 27B only, simple averaging, and weighted averaging ( $p = w \cdot p_{12B} + (1-w) \cdot p_{27B}$ ). The strategy yielding the highest development F1 was selected for the final submission.

Key Results

Performance: The final system achieved a mean macro-F1 of 0.811 across all 22 languages, ranking 2nd overall among 60 participating teams. The system secured 1st place in 3 languages (Amharic, Hindi, Swahili) and top-3 finishes in 8 languages.
Impact of Components:
- Threshold Tuning: Per-language threshold tuning on the development set yielded F1 improvements of 2% to 4% without retraining.
- Synthetic Data: An optimal synthetic data ratio of 30% provided modest gains (2–3%), with the most significant benefits observed in lower-resource languages.
- Ensembling: Weighted ensembles of the 12B and 27B models were the winning strategy for 14 out of 22 languages.
Generalization vs. Development Performance: A critical finding was the disparity between development and test performance across architectures. While XLM-RoBERTa and Qwen3 showed strong development set performance, they suffered massive F1 drops (30–50%) on the test set. For instance, XLM-RoBERTa dropped from 0.856 to 0.297 on Bengali. Only the Gemma models demonstrated reliable generalization from development to test.
Case Study (Italian): The system's weakest performance was in Italian (0.563 F1). Analysis revealed a "topic coverage gap": the training and development sets contained zero examples of "political" or "other" topics, yet these categories comprised 41% of the test set and 87% of its positive class. This highlighted a distribution shift where the model was evaluated on a positive sub-population it never observed during training.

Significance and Claims
The paper claims that robust generalization is a more critical factor for success than maximizing development set performance. The authors argue that while alternative architectures (XLM-RoBERTa, Qwen3) may appear superior during development, they fail to transfer effectively to unseen test distributions, whereas the Gemma family maintained consistent performance.

The study highlights the efficacy of per-language threshold tuning as a low-cost method to significantly boost performance by correcting probability miscalibration (e.g., Russian models being under-confident and Khmer models over-confident). Furthermore, the work demonstrates that synthetic data augmentation is most beneficial for low-resource languages, though it requires careful ratio management to avoid degradation in high-resource or sensitive languages.

Limitations
The authors acknowledge several limitations:

Compute Intensity: Training separate models for 22 languages requires significant GPU resources.
Synthetic Data Authenticity: The quality of synthetic data for low-resource languages (e.g., Khmer, Swahili) may be limited if the LLM cannot generate authentic text in those specific linguistic contexts.
Threshold Transferability: Thresholds optimized on a small development set may not fully transfer to the test distribution.
Heuristic Strategy Ratios: The 50/30/20 split for synthetic data strategies was chosen heuristically rather than through systematic ablation.

Conclusion
The PSK system successfully leveraged Gemma models, synthetic data augmentation, and ensemble methods to achieve a top-2 ranking in a challenging multilingual task. The primary takeaway is the necessity of prioritizing models that generalize well over those that merely overfit development metrics, alongside the utility of simple post-hoc calibration techniques like threshold tuning.