Technical Summary: PSK@EEUCA 2026

Problem Definition

This paper addresses the EEUCA 2026 Shared Task on understanding toxic behavior in gaming communities, specifically focusing on the classification of World of Tanks chat messages. The objective is to categorize messages into six distinct classes:

1. Non-toxic (Normal/positive communication)
2. Insults/Flaming (Personal attacks on skill)
3. Other Offensive (Inappropriate but not direct attacks)
4. Hate/Harassment (Identity-based abuse)
5. Threats (Violence/harm threats)
6. Extremism (Hate ideology/dehumanization)

The task presents significant challenges, including extreme class imbalance (81% Non-toxic vs. <0.2% for Threats and Extremism), short and informal text containing gaming-specific slang, multilingual content, and subtle distinctions between categories (e.g., skill-based criticism vs. identity-based hate).

Methodology

The authors explored a range of architectures and strategies, ultimately converging on a specific configuration for their best-performing system.

Model Architecture and Fine-Tuning

• Base Model: The final system utilizes Llama 3.1 8B.
• Efficiency: The model employs 4-bit NF4 quantization and LoRA (Low-Rank Adaptation) with rank=16 and alpha=64.
• Prompt Engineering: Structured class definitions were prepended to inputs to assist the model in distinguishing between similar toxic categories (e.g., distinguishing "Insults" from "Hate/Harassment").
• Alternative Architectures Tested: XLM-RoBERTa Large (full fine-tuning), Gemma 2B, and Gemma 12B (both with LoRA).

Synthetic Data Augmentation Strategy

A critical component of the methodology was the use of synthetic data augmentation to address class imbalance.

• Generation: Synthetic data was generated via LLM-based paraphrasing, specifically targeting minority classes (Classes 2–5). The generation process used a strict template to ensure the rewritten messages maintained the exact meaning and toxicity level while adopting natural gaming slang and abbreviations.
• Filtering: The synthetic pool underwent rigorous cleaning, including label-leakage regex filtering and embedding-based deduplication.
• Calibration: The authors found a narrow optimal ratio of 5% synthetic data.
  • <5%: Insufficient to improve test transfer.
  • >5% (7–15%): Led to overfitting on synthetic patterns and substantial degradation in test performance.
• Distribution: The 5% ratio (approx. 1,921 examples) was added only to the training partition; the validation set remained 100% real data.

Alternative Approaches (Underperforming)

Several strategies were explored but yielded inferior results:

• Hierarchical Classification: A two-stage approach (toxic/non-toxic binary, then multi-class) resulted in a large generalization gap (0.67 Val F1 → 0.47 Test F1).
• One-vs-Rest: Six binary classifiers with aggressive oversampling were too conservative.
• Ensemble Methods: Probability averaging and voting generally hurt performance as the single best model dominated all classes.
• Transfer Learning: Pre-training on Dota 2 toxicity data resulted in a "validation trap."

Key Results

The system achieved an F1-macro score of 0.6234 on the test set, placing 4th out of 35 participating teams.

• Performance Comparison:
  • Llama 8B + 5% Synthetic: 0.6234 Test F1 (Best)
  • Llama 8B (No Synthetic): 0.5971 Test F1
  • Gemma 12B: 0.52 Test F1 (Despite 0.66 Val F1)
  • XLM-RoBERTa Large: 0.30 Val F1
• Per-Class Performance: Performance correlated with class frequency. The model achieved high scores on Non-toxic (0.94) and Extremism (0.86), but struggled with "Other Offensive" (0.44) and "Hate/Harassment" (0.43).

The "Validation Trap" Phenomenon

A significant contribution of this paper is the identification of a "validation trap."

• Observation: Models achieving high validation F1 scores often did so by making conservative predictions that matched the training/validation distribution (heavily favoring the 81% Non-toxic class).
• Consequence: These models performed poorly on the test set, which appeared to have different annotation patterns or a shifted distribution.
• Evidence: The Gemma 12B model achieved 0.66 Val F1 but dropped to 0.52 Test F1. Similarly, the Two-stage approach dropped from 0.67 Val F1 to 0.47 Test F1.
• Resolution: The 5% synthetic data approach improved test performance by making predictions slightly more "aggressive" toward minority classes (specifically Classes 2 and 3), better aligning with the test distribution despite a slight drop in validation F1 (from 0.6554 to 0.6271).

Significance and Claims

The paper claims its primary significance lies in:

1. Demonstrating the efficacy of Llama 3.1 8B over both smaller (2B) and larger (12B) models for this specific task.
2. Identifying the critical sensitivity of synthetic data ratios, showing that a precise 5% augmentation is optimal, while deviations significantly degrade performance.
3. Highlighting the "validation trap," warning that high validation metrics in imbalanced, domain-specific tasks can be misleading if they reflect distribution matching rather than true generalization.
4. Establishing that ensemble methods are not always beneficial when a single model already dominates performance across classes.

The authors conclude that their approach successfully navigated the dataset's severe imbalance and annotation shifts to achieve a top-4 ranking, while noting limitations regarding the dataset-specific nature of the "validation trap" and the computational constraints that prevented exploration of larger models.