ToxSyn-PT: A Synthetic Fine-Grained Dataset of Minority-Targeted Toxic Language in Portuguese

Technical Summary: ToxSyn-PT

Problem Statement
The development of robust hate speech detection systems is currently hindered by a scarcity of large-scale, fine-grained training data, particularly for languages outside of English. Existing Portuguese corpora suffer from three primary limitations: (1) they predominantly rely on binary toxic/non-toxic labels rather than multi-label annotations for specific protected groups; (2) they lack positive counterexamples (benign or supportive text referencing minorities), making it difficult to distinguish genuine hate from mere discussion; and (3) they are confined to the social media domain, failing to capture the linguistic patterns of toxicity in other contexts such as news commentary or formal dialogue. Furthermore, existing datasets exhibit severe class imbalance and often omit neutral or supportive references to minority groups, preventing models from learning the distinction between the absence of hate and the presence of endorsement.

Methodology
To address these gaps, the authors introduce ToxSyn-PT, a large-scale synthetic dataset comprising 53,274 samples generated via a controllable, four-stage pipeline using Large Language Models (LLMs). The pipeline is designed to ensure balanced representation across nine protected minority groups (Black people, Jews, Muslims, Indigenous Brazilians, women, LGBTQIA+ individuals, elderly people, people with disabilities, and immigrants) and includes both toxic and non-toxic (positive/neutral) samples.

The generation process consists of:

1. Seed Dataset: A compact set of 160 human-written examples (40 toxic/non-toxic each for Black people and women) serving as in-context demonstrations.
2. Prompt-Based Expansion: Using GPT-4o Mini with few-shot prompting and 26 distinct discourse-type templates (14 non-toxic, 12 toxic) to generate 10,790 unique sentences across all nine target groups.
3. Paraphrase Augmentation: Each generated sentence is paraphrased twice (once via toxic transformation, once via non-toxic) using a new pool of rhetorical strategies (e.g., victim-blaming, positive negation). This stage also involves swapping minority targets and normalizing outputs, resulting in 20,799 additional instances.
4. Enrichment: A final generation round using the expanded corpus as the seed pool, introducing two new implicit hate discourse types ("Ambiguous Prejudice" and "Justification Prejudice") to increase linguistic complexity.
5. Neutral Samples: A separate augmentation module generates 3,212 neutral sentences unrelated to minority identity to improve generalization.

A human-annotated test set of 5,208 examples was created by blind annotators to validate the synthetic labels. The authors report that human annotators modified only ~6% of toxicity labels and <1% of target group labels, confirming the high fidelity of the pipeline.

Key Contributions

1. Dataset: ToxSyn-PT is the first publicly available Portuguese corpus explicitly designed for multi-label hate speech detection across nine protected groups. It uniquely includes non-toxic counterexamples (positive and neutral references to minorities) and discourse-type annotations (e.g., sarcasm, dehumanization, cultural appreciation).
2. Generation Pipeline: The authors present a controllable LLM-based framework capable of balancing class distributions, injecting low-frequency expressions, and applying safety constraints to generate diverse rhetorical strategies.
3. Evaluation: The paper provides a rigorous evaluation of toxicity detection in Portuguese, demonstrating that current models are highly domain-dependent.

Results
Experiments fine-tuning the BERTimbau model reveal a "catastrophic, mutual generalization failure" between general social media datasets (e.g., ToLD-BR, HateBR) and the minority-targeted ToxSyn dataset:

• In-Domain Performance: Models trained on social media data perform well on similar social media test sets (Macro F1: 0.60–0.95). The ToxSyn-trained model performs well on its own test set and generalizes effectively to the translated ToxiGen dataset (F1: 0.69).
• Out-of-Domain Failure: Models trained on general social media data fail to detect toxicity in minority-specific contexts (e.g., ToLD-BR model identifies only 10 of 2,472 toxic samples in ToxSyn). Conversely, the ToxSyn-trained model fails to generalize to general social media corpora (Toxic-Class Recall of 0.17 on ToLD-BR).
• Group-Specific Classification: On group-specific tasks, the ToxSyn-trained model achieves an average Macro Recall of 0.70 on the translated ToxiGen set and 0.64 on the Portuguese Merged set, demonstrating the ability to transfer knowledge to noisy, out-of-domain contexts.

Significance and Claims
The paper claims that ToxSyn-PT addresses a critical resource gap by enabling the training and evaluation of minority-aware toxicity detection models in Portuguese, a task previously impossible due to the lack of non-toxic counterexamples in public datasets.

A central finding of the work is that toxicity detection is not a monolithic problem; the rhetorical and contextual patterns of minority-targeted hate are fundamentally different from the explicit lexical cues of general social media insults. Consequently, the authors argue that summary metrics like Macro F1 can be dangerously deceptive, as they may mask a model's complete failure to detect specific target classes. The paper posits that future benchmarks must incorporate more granular metrics and diverse, minority-aware datasets to build truly robust classifiers. The authors release ToxSyn-PT to support reproducible research on synthetic data generation and to benchmark progress in hate speech detection for low- and mid-resource languages.