Toward Generalized Cross-Lingual Hateful Language Detection with Web-Scale Data and Ensemble LLM Annotations

This paper demonstrates that combining continued pre-training on web-scale unlabelled data with fine-tuning on synthetic annotations generated by an ensemble of large language models significantly improves hateful language detection performance, particularly for smaller models and low-resource languages.

The Gist

Imagine you are trying to build a spam filter for the internet, but instead of just spam, you're trying to catch hate speech in four different languages: English, German, Spanish, and Vietnamese.

The problem? There aren't enough "human teachers" to label every single bad comment on the internet. It's too expensive, too slow, and humans get tired or biased.

This paper is like a recipe for a new kind of filter that uses two clever tricks to solve this problem without needing a million human teachers.

The Two Main Tricks

Trick 1: The "Immersive Student" (Continued Pre-Training)

Imagine you have a smart student (a computer model called BERT) who already knows how to read and write in general. But they don't really understand the specific slang, tone, and drama of internet forums.

Usually, you'd just give them a textbook of "hate speech examples" and say, "Study this and pass the test."

The authors' idea: Before giving them the textbook, they send the student to a massive, unlabelled library of real internet conversations (called OpenWebSearch). They don't ask the student to find hate speech yet; they just let them read millions of forum posts, replies, and comments to get a feel for how people actually talk.

• The Result: When the student finally gets the textbook (the labeled data), they learn much faster and do a better job. It's like a musician who practices scales for months before learning a specific song; they play the song with much more soul and accuracy.
• The Win: This "immersion" helped the models get about 3% better at spotting hate speech, especially in languages where data is scarce (like Vietnamese).

Trick 2: The "Council of AI Judges" (LLM Ensembles)

Now, imagine you need to label millions of new comments as "Hate" or "Not Hate," but you don't have humans. You could ask one super-smart AI (a Large Language Model) to do it. But what if that AI is a bit biased or makes mistakes?

The authors' idea: Instead of asking one AI, they asked four different AIs (Mistral, Llama, Gemma, and Qwen) to read the same text and vote on whether it's hate speech.

• The Problem: If you just take the average of their votes, or let the majority win, you might still get it wrong if all four AIs share the same blind spot.

• The Solution: They used a "Meta-Judge" (a smaller AI called LightGBM) to listen to the four judges. This Meta-Judge learned which of the four judges was most reliable for which type of text. It's like a coach who knows that Judge A is great at spotting sarcasm, but Judge B is better at spotting direct insults. The coach combines their opinions intelligently.

• The Result: This "Council of Judges" created a massive dataset of synthetic labels.

  • For small models: It was a game-changer. A tiny model (Llama-1B) got 11% better just by studying these AI-generated labels. It was like a small student getting a private tutor made of four geniuses.
  • For big models: A huge model (Qwen-14B) didn't improve much. It was already so smart that the AI-generated labels didn't teach it anything new.

The Big Takeaways (In Plain English)

1. Reading the "Real World" helps: Letting AI models read billions of real, messy internet posts before training them makes them much better at understanding context, especially for languages that don't have many training examples.
2. Teamwork makes the dream work: Using a smart combination of multiple AI models to label data is better than relying on just one. It creates high-quality "fake" data that can train smaller, cheaper models to be very effective.
3. Size matters (but not in the way you think):
   • Small models benefit huge amounts from this AI-generated data. It's like giving a small car a turbocharger.
   • Big models are already so powerful that they don't need the turbo; they barely notice the extra data.
4. The "Silent Majority" Problem: The internet is mostly nice. When the AI judges labeled millions of posts, 97% were "Not Hate." This made it hard for the models to learn what hate actually looks like because there were so few examples. This remains a tricky hurdle.

The Bottom Line

The authors found that if you want to build a hate-speech detector for many languages (especially those with fewer resources), you shouldn't just rely on human labels. Instead, soak your model in real web data and use a team of AI judges to create training materials. This approach is the most cost-effective way to build robust, fair, and accurate detectors for the whole world.

Technical Summary

1. Problem Statement

The primary bottleneck in developing robust hate speech detectors is the scarcity of high-quality, labeled training data, particularly for low-resource languages. While large-scale web crawls (e.g., OpenWebSearch.eu) offer billions of unlabelled multilingual texts, leveraging them effectively remains an open challenge. Furthermore, existing approaches to automated annotation using Large Language Models (LLMs) are often limited to single languages, lack rigorous comparison with human-labeled baselines, and do not adequately address the biases inherent in individual LLMs.

The authors address two specific gaps:

1. Domain Adaptation: Can large-scale, unlabelled web data improve multilingual hate speech detection via continued pre-training of BERT models?
2. Synthetic Annotation: Can ensemble strategies of multiple open-source LLMs effectively replace or supplement human labeling to improve detection performance and cross-lingual generalization?

2. Methodology

The study employs an end-to-end pipeline involving two complementary tracks: BERT-based continued pre-training and LLM-based synthetic annotation.

A. Data Collection & Pre-processing

• Source: Data was harvested from OpenWebSearch.eu (OWS) and OpenWebIndex (OWI).
• Filtering: To ensure conversational relevance, the authors filtered for URLs containing keywords like "thread," "forum," or "post" and adhered to specific schema.org conversational types (e.g., DiscussionForumPosting).
• Scale: Approximately 17.5 million unlabelled texts were collected across four languages: English (6M), German (3M), Spanish (3M), and Vietnamese (0.5M).
• No Hate Filtering: Crucially, no hate-specific keywords were used to filter the unlabelled corpus; these texts served purely for domain adaptation.

B. Track 1: Continued Pre-Training (BERT)

The authors applied Domain-Adaptive Pre-Training (DAPT) to BERT-family models.

• Models: Four variants were trained:
  • OwsEng, OwsDeu, OwsSpa: Monolingual models pre-trained on respective language corpora.
  • Ows4L: A multilingual model pre-trained on a mixed corpus (43% EN, 32% DE, 17% ES, 6.7% VI).
• Fine-Tuning: These models were subsequently supervised fine-tuned on human-labeled datasets (16 benchmarks across 4 languages) and evaluated against standard baselines (BERT, HateBERT).

C. Track 2: Ensemble LLM Annotation

To generate synthetic labels for training smaller models, the authors utilized four open-source LLMs: Mistral-7B, Llama3.1-8B, Gemma2-9B, and Qwen2.5-14B.

• Annotation Process: The LLMs generated token-level probabilities for classifying OWS texts as "Hate" or "Neutral" using zero-shot prompting.
• Ensemble Strategies: Three aggregation methods were tested:
  1. Majority Voting (Vote): Hard labels based on a 2/4 vote threshold.
  2. Mean Averaging (Mean): Averaging probability scores across models.
  3. LightGBM Meta-Learner (LGB): A classifier trained on the 8-dimensional probability vectors (2 classes × 4 models) using human-labeled data as supervision. This learns to weight annotators based on reliability.
• Downstream Training: The resulting synthetic labels were used to fine-tune smaller models (Llama3.2-1B and Qwen2.5-14B) via LoRA (Low-Rank Adaptation).

3. Key Contributions

1. Large-Scale Web Data Utility: Demonstrated that continued pre-training on ~17.5M unlabelled web texts significantly boosts multilingual hate speech detection, particularly in low-resource settings.
2. Ensemble Annotation Framework: Proposed and evaluated a novel ensemble approach using four diverse LLMs, showing that a LightGBM meta-learner outperforms simple voting or averaging by correcting for shared model biases.
3. Cross-Lingual Generalization: Provided a comprehensive evaluation across 16 benchmarks in 4 languages (English, German, Spanish, Vietnamese), revealing how synthetic data and pre-training transfer across languages.
4. Capacity-Dependent Findings: Established that the value of synthetic data is inversely proportional to model capacity; it drastically improves small models but offers diminishing returns for large models.

4. Key Results

A. Continued Pre-Training (RQ1)

• Performance Gain: All OWS-pre-trained BERT models outperformed standard BERT and HateBERT baselines.
• Multilingual Advantage: The Ows4L (multilingual) model achieved the best overall average macro-F1 (77.0%), surpassing BERT (+1.5%) and HateBERT (+6.3%).
• Low-Resource Impact: Gains were most pronounced in low-resource scenarios. For example, Ows4L improved the Vietnamese (ViHSD) benchmark by +6.6% over HateBERT.
• Cross-Lingual Transfer: Monolingual pre-training (e.g., German-only) surprisingly boosted performance on English and Vietnamese tests, likely due to multilingual content within the German web crawl.

B. LLM Ensemble Annotations (RQ2)

• Small Model Boost: Fine-tuning Llama3.2-1B on synthetic labels (specifically the H+LGB configuration: Human + LightGBM synthetic data) yielded a massive +10.6% increase in pooled macro-F1 (reaching 65.4%).
• Large Model Saturation: Fine-tuning Qwen2.5-14B on the same synthetic data provided only a marginal +0.6% gain, indicating that large models already possess strong priors that are not easily improved by imbalanced synthetic data.
• Ensemble Strategy: The LightGBM strategy consistently outperformed Majority Voting and Mean Averaging. It was the only synthetic-only strategy to exceed human baselines in Spanish (zero-shot setting) and Vietnamese, proving its ability to learn language-agnostic calibration.
• Class Imbalance: A critical limitation identified was severe class imbalance in synthetic data (Neutral >97%), which constrained the detection of minority classes (Hate), especially for low-resource languages like Vietnamese.

5. Significance and Conclusion

This paper provides a roadmap for building robust, multilingual hate speech detectors without relying solely on expensive human annotation.

• For Small Models: The combination of web-scale unlabelled pre-training and LLM-ensemble synthetic annotation is highly effective, effectively "distilling" knowledge from large models into smaller, efficient ones.
• For Cross-Lingual Generalization: Multilingual pre-training (Ows4L) and sophisticated ensemble annotation (LightGBM) are superior to monolingual approaches, enabling better performance in languages with scarce labeled data.
• Future Directions: The authors highlight the need for better filtering to reduce class imbalance in synthetic datasets and the exploration of newer, more capable Small Language Models (SLMs) to further close the gap with large models.

Final Verdict: The study concludes that web-scale unlabelled data + LLM-ensemble annotations is the most valuable strategy for improving hate speech detection in low-resource languages and for smaller models, offering a scalable path toward generalized, cross-lingual safety systems.