💬 NLP

ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification

The paper introduces ArabicDialectSafety, a human-curated benchmark dataset of over 25,000 prompts across six Arabic varieties annotated for safety, which reveals that fine-tuned MARBERTv2 outperforms frontier LLMs in dialect-aware harm detection while highlighting persistent challenges in low-resource Maghrebi dialects.

Original authors: Wajdi Zaghouani, Md. Rafiul Biswas, Kholoud Khalil Aldous, Mabrouka Bessghaier

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wajdi Zaghouani, Md. Rafiul Biswas, Kholoud Khalil Aldous, Mabrouka Bessghaier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a giant, bustling library where millions of people are chatting, arguing, telling jokes, and sharing stories. This library is the internet, and the language being spoken is Arabic. But here's the twist: Arabic isn't just one single voice. It's like a choir where the conductor speaks a formal, polished version of the language (called Modern Standard Arabic) used for news and books, while the choir members speak dozens of different regional dialects—like Egyptian, Syrian, or Moroccan—each with their own slang, rhythm, and unique way of saying things.

Now, imagine the library has a very strict security guard whose job is to stop anyone from saying anything mean, dangerous, or harmful. The problem is, this guard was trained only on the formal, polished voice. When a kid from Morocco tries to whisper a mean joke in their local dialect, the guard might not understand the words or the tone, so they let the harmful comment slip right past. This paper is about building a new, super-smart security system that actually understands all these different voices, not just the formal one. It's like teaching the guard to speak every dialect fluently so they can spot trouble no matter how it's whispered.

The researchers behind this study, led by Wajdi Zaghouani and his team, decided to build a massive "training manual" for these security guards. They created a dataset called ARABICDIALECTSAFETY, which is essentially a collection of 24,053 different prompts (or questions/statements) written in six different Arabic varieties: Modern Standard Arabic, Syrian, Egyptian, Algerian, Palestinian, and Moroccan. They didn't just write these; they carefully crafted them to cover seven specific types of "harm," ranging from bullying and hate speech to self-harm and adult content. Think of it as a giant, diverse test bank where every question is labeled with exactly what dialect it's in and exactly what kind of trouble it represents.

Once they built this test bank, they put seven different "security guard" models to the test. Some were old-school, rule-based systems, while others were the latest, super-powerful AI models (Large Language Models) that people use every day. They asked a simple question: Can these models tell the difference between a safe sentence and an unsafe one, and can they do it equally well for all six dialects?

The results were quite revealing. The star of the show wasn't the biggest, flashiest AI model. Instead, a model called MARBERTv2, which had been specifically "fine-tuned" (like a student who studied the training manual intensely), crushed the competition. It got the binary task (Safe vs. Unsafe) right 95% of the time and the more detailed task (identifying exactly what kind of harm) right 90% of the time. This was a huge win, especially because the massive, general-purpose AI models that people usually talk about (the "frontier" models) actually struggled a bit more, scoring significantly lower.

The paper also tested a clever idea: what if we just tell the AI, "Hey, this is Egyptian Arabic!" before it reads the sentence? The researchers found that this trick only really helped the specialized models (like MARBERTv2) when the information was baked deep into the model's brain, rather than just whispered as a hint at the start of the sentence. It's like telling a chef, "This is Italian food," which helps if the chef already knows Italian cooking, but doesn't help much if the chef is just guessing based on the label.

Another interesting discovery was about the dialects themselves. The models were amazing at understanding Egyptian and Syrian Arabic, but they stumbled a bit more with Moroccan Arabic. It seems that even with a lot of data, the "Maghrebi" (North African) dialects are still a bit of a mystery to these AI systems, likely because they haven't seen enough examples of them in their training history.

Finally, the team asked the big AI models to actually answer these harmful prompts to see if they would accidentally generate bad content. Surprisingly, the models were mostly good at saying "no" or refusing to answer, with unsafe responses happening less than 5% of the time. However, the authors warn that this number might be even lower than it looks because human reviewers found that the automatic checkers sometimes missed subtle errors.

In short, this paper shows us that to keep the internet safe for Arabic speakers, we can't just use a one-size-fits-all approach. We need security systems that are specifically trained to understand the rich, messy, and beautiful variety of Arabic dialects. While the current AI models are getting better, there is still a long way to go to make sure a joke in Tunis is understood just as well as a joke in Cairo, and that no harmful content slips through the cracks just because it was spoken in a different accent. The authors note that many other dialects, including Gulf, Yemeni, Sudanese, Tunisian, and Libyan, were not covered in this study, meaning the findings may not yet apply to them.

Technical Summary: ARABICDIALECTSAFETY

Problem Statement
The rapid deployment of Large Language Models (LLMs) in Arabic-speaking communities has exposed critical gaps in safety mechanisms that fail to account for linguistic and cultural context. The core challenge is the linguistic diversity of Arabic, which exists primarily as Modern Standard Arabic (MSA) for formal settings and dozens of distinct regional dialects (e.g., Egyptian, Moroccan, Levantine) with unique vocabularies and grammatical structures. Current safety filters, often trained predominantly on MSA, fail to detect harmful content expressed in regional dialects. Furthermore, users can intentionally mix dialects to bypass safety filters, exploiting the gap between training data and real-world usage patterns. Existing Arabic safety benchmarks either focus exclusively on MSA, conflate dialects without explicit labeling, or are too small to support robust, dialect-stratified evaluation. Consequently, the field lacks a clear understanding of how classification performance degrades across dialects and whether LLMs exhibit dialect-specific safety blind spots.

Methodology
To address these gaps, the authors introduce ARABICDIALECTSAFETY, a human-curated dataset and evaluation framework.

Dataset Construction: The dataset comprises 25,071 prompts spanning six Arabic varieties: MSA, Syrian, Egyptian, Algerian, Palestinian, and Moroccan. After cleaning (removing null values, empty strings, and unannotated entries), 24,053 instances remain for evaluation. The dataset covers seven fine-grained harm categories: Self-Harming, Harming Others, Harassment, Fraud & Deception, Bullying, Hate Speech, and Adult Content. The construction involved a five-phase pipeline including annotator selection, training on ethical guidelines, prompt creation (both harmful and benign counterparts), cross-validation, and final release. Inter-annotator agreement was substantial (Cohen's $\kappa$ ranging from 0.72 to 0.91), with lower agreement observed in semantically overlapping categories like harassment and bullying.
Evaluation Framework: The authors propose a dual-task evaluation protocol:
1. Binary Classification: Safe vs. Unsafe detection.
2. Granular Classification: Multi-class harm categorization across the seven defined types.
  Both tasks are evaluated globally and per dialect.
Model Benchmarking: Seven systems were benchmarked, including:
- Supervised Classifiers: TF-IDF + Logistic Regression, multilingual-e5-base (frozen dense encoder), and fine-tuned MARBERTv2 (a BERT-based encoder pretrained on dialectal Arabic).
- Instruction-Tuned LLMs: FANAR-9B-Instruct, Jais-2-8B-Chat, Qwen2.5-7B-Instruct, and Llama3.1-8B-Instruct, evaluated in a few-shot setting.
- Dialect Conditioning: Models were tested with and without explicit dialect tags (e.g., [DIALECT]) to assess the impact of dialect signaling.
Response Generation Evaluation: Seven frontier LLMs (including GPT-4o, Gemini, Claude, and Arabic-specialized models) were queried with harmful dialectal prompts. Their responses were evaluated using an LLM-as-a-judge (GPT-5.2), validated against human annotations.

Key Results

Supervised Models Outperform LLMs: Fine-tuned MARBERTv2 achieved the strongest performance, reaching a Macro-F1 of 0.95 for binary classification and 0.90 for granular classification. This substantially outperformed prompted frontier LLMs, which scored between 0.70–0.74 (binary) and 0.58–0.69 (granular). Even a lightweight TF-IDF + LR baseline remained highly competitive (Macro-F1 0.93).
Dialect Conditioning Effects: Explicit dialect conditioning (adding dialect tags) improved performance only when integrated at the representation layer (as in MARBERTv2) rather than through prompt-level cues. For LLMs and other baselines, dialect cues yielded marginal or no improvements in binary detection, suggesting binary safety relies more on general harmful lexical patterns than dialect-specific variation. However, dialect cues provided small, consistent gains for fine-grained harm distinctions.
Dialectal Disparities: Performance varied significantly across dialects. MARBERTv2 achieved high F1 scores on Egyptian (0.96), Palestinian (0.97), and MSA (0.96) but dropped notably for Moroccan Arabic (0.85 in binary, 0.69 in granular). This suggests persistent challenges in evaluating low-resource Maghrebi varieties, likely due to limited representation in pretraining corpora.
LLM Safety: When acting as response generators, frontier LLMs generally refused or sanitized harmful dialectal prompts, with unsafe generation rates remaining below 5% for all models. The Arabic-specialized FANAR-v2 maintained a low unsafe rate (1.00%), indicating that Arabic adaptation does not inherently weaken safety alignment. However, human validation suggested these estimates might be conservative.
Error Analysis: Errors in granular classification concentrated between semantically adjacent categories (e.g., harassment vs. harming others, bullying vs. harming others), while categories with distinctive lexical signals (Adult Content, Hate Speech) were classified reliably.

Significance and Claims
The paper claims to be the first work to directly benchmark content safety classification across multiple Arabic dialects at scale. Its primary contributions are:

A Dialect-Aware Benchmark: Providing a large-scale, human-curated dataset with explicit dialect-level annotation, addressing the lack of resources for dialect-stratified safety evaluation.
Empirical Evidence of Dialect Bias: Demonstrating that current safety mechanisms and LLMs exhibit significant performance disparities across dialects, particularly disadvantaging Maghrebi varieties.
Methodological Insight: Establishing that supervised, fine-tuned models (specifically dialect-aware ones like MARBERTv2) currently offer a superior accuracy-cost trade-off for Arabic safety classification compared to few-shot prompting of frontier LLMs.
Safety Baseline: Providing a baseline showing that while frontier LLMs are generally robust against harmful dialectal prompts, the "unsafe" generation rates are likely under-estimated by automated judges, highlighting the need for continued, dialect-aware evaluation.

The authors conclude that future work must focus on dialect-equitable evaluation and refining annotation taxonomies to resolve semantic overlaps, rather than solely increasing model scale.

Technical Summary: ARABICDIALECTSAFETY

More like this