GemDetox at TextDetox CLEF 2025: Enhancing a Massively Multilingual Model for Text Detoxification on Low-resource Languages

Technical Summary: GemDetox at TextDetox CLEF 2025

Problem Statement
The paper addresses the urgent need for automated text detoxification on social media platforms, specifically within the context of the PAN 2025 Multilingual Text Detoxification Challenge. The task involves rewriting toxic single-sentence inputs into neutral paraphrases while preserving the original meaning across 15 typologically diverse languages. The challenge is constrained to "explicit toxicity" (obscene or offensive lexicon containing meaningful neutral content), excluding implicit toxicity such as sarcasm or coded hate speech. A critical difficulty lies in the disparity between high-resource languages (e.g., English, Spanish) and low-resource languages (e.g., Tatar, Hinglish, Amharic), where data scarcity often hinders model performance.

Methodology
The authors propose "GemDetox," a system built upon the 12B-parameter Gemma-3 multilingual transformer. The methodology is structured around three core components:

1. Data Construction and Augmentation:

   • Human-Authored Data: The system utilizes the organizer-provided ParaDetox-9 corpus, containing 3,600 toxic-neutral parallel pairs across nine languages.
   • Machine-Translated Extension: To cover the six languages absent from the original corpus (Italian, French, Hebrew, Hinglish, Japanese, Tatar), the authors translated all 3,600 original pairs using RLM-Hinglish and NLLB-200, generating 21,600 synthetic pairs (ParaDetox-MT6).
   • Synthetic Generation: Additional supervision was derived from a Multilingual Toxicity Dataset. The model generated neutralization candidates for toxic-only sentences, which were filtered based on Jaccard similarity thresholds (≥0.90 for machine-translated, ≥0.80 for synthetic) to ensure lexical divergence and semantic preservation.
   • Contextual Enrichment: During training and inference, inputs were enriched with three LaBSE-retrieved semantic neighbors and explicit toxic-span annotations extracted via a rule-based detector.

2. Model Architecture and Training:

   • Base Model: Gemma-3-12B-Instruct, quantized to 4-bit precision to fit within 24 GB VRAM.
   • Parameter-Efficient Fine-Tuning (PEFT): The authors employed Low-Rank Adaptation (LoRA) with a rank of 16, making only 0.55% (≈65M) of parameters trainable. Adapters were inserted into all language, attention, and MLP sub-modules.
   • Three-Phase Training Strategy:
     • Phase 1: Fine-tuning across all 15 languages to establish a baseline alignment.
     • Phase 2: Specialized fine-tuning on data for languages lacking parallel human annotations to prevent catastrophic interference.
     • Phase 3: Fine-tuning on high-quality synthetic data (filtered by Jaccard > 0.9) to further refine performance.
   • Prompting Strategy: The system utilizes a four-step Chain-of-Thought (CoT) prompting instruction (identify toxicity, retrieve semantics, rewrite neutrally, verify non-toxicity) combined with few-shot examples (three semantically closest neighbors) retrieved from the training corpus.

3. Inference:

   • The system samples three candidate rewrites per input. The final output is selected based on the highest joint score ($J$), calculated using Style Transfer Accuracy (STA), Content Preservation (SIM via LaBSE), and Fluency (xCOMET).

Key Results

• Automatic Evaluation: The GemDetox system ranked first overall on both high-resource and low-resource language groups. It achieved a joint score ($J$) of 0.685 on languages with human-annotated data and 0.643 on languages without.
  • Performance peaked on German ($J=0.798$) and French ($J=0.802$).
  • Ablation studies indicated that few-shot examples contributed a +0.081 increase in the joint score, while basic CoT prompting added +0.088.
• LLM-as-a-Judge Evaluation: When re-evaluated using a fine-tuned Llama-3.1-8B-Instruct as a judge (trained on human preferences), the system's ranking dropped to third place in both subgroups, with an average $J$ of 0.7669. This was significantly lower than the "Golden Annotation" score (0.8233) but competitive against other top teams.
• Statistical Analysis: An ANOVA analysis revealed that "language-resource status" was the strongest predictor of performance ($\eta^2 = 0.667, p < 0.01$), followed by genetic affiliation ($\eta^2 = 0.592$).

Significance and Claims
The paper claims that its primary contribution is demonstrating that parameter-efficient fine-tuning of a massively multilingual model, combined with strategic data augmentation and prompting, can effectively bridge the performance gap between high-resource and low-resource languages in detoxification tasks.

The authors modestly note that while their approach achieved strong automatic results, the gains from data augmentation were "especially pronounced for the low-resource languages for which they were designed." They acknowledge that the system's performance is heavily dependent on the base capabilities of the pre-trained Gemma-3 model, as the zero-shot mT0 baseline already achieved competitive scores (0.675).

The paper highlights that while the system successfully narrows the gap for low-resource languages, qualitative analysis revealed persistent challenges, including:

• Failure to detect compound toxicity in mixed-script languages like Hinglish.
• Generation of nonsensical outputs in very low-resource languages like Amharic.
• Ambiguity in toxicity definitions leading to over- or under-detoxification in politically charged contexts (e.g., Hebrew).

The authors conclude that robust detoxification requires bias-aware adaptations at pre-training, post-training, and evaluation stages, noting that structural biases in the foundation model's training corpus are irreversible and can only be mitigated downstream. They suggest future work should explore preference-based optimization (e.g., DPO, GRPO) and multi-stage reasoning pipelines to address creative obfuscation and emergent slang.