SalamahBench: Toward Standardized Safety Evaluation for Arabic Language Models

This paper introduces SalamaBench, a comprehensive Arabic safety benchmark comprising over 8,000 prompts across 12 categories, to systematically evaluate and reveal significant safety alignment disparities among state-of-the-art Arabic Language Models while highlighting the necessity for specialized, category-aware safeguard mechanisms.

The Gist

Imagine you have built a brilliant, multilingual librarian named Salamah. She speaks Arabic fluently, understands local jokes, knows the culture, and can answer almost any question. But there's a problem: you aren't sure if she's safe. If someone asks her how to build a bomb, how to scam a bank, or how to bully someone, will she refuse? Or will she happily help them?

For a long time, the tech world has been very good at testing English-speaking librarians. They have huge rulebooks and "safety inspectors" to check if English models are being naughty. But for Arabic-speaking models, we were flying blind. We didn't have a good rulebook, and the inspectors we used were mostly English speakers who didn't understand the nuances of Arabic culture, dialects, or how bad ideas can be hidden in polite phrases.

This paper introduces SalamahBench, a brand-new, massive "safety gym" designed specifically to test Arabic AI models.

Here is the story of how they built it and what they found, explained simply:

1. The Problem: The "Lost in Translation" Safety Gap

Think of safety testing like a fire drill. In English, we have a perfect fire drill with clear alarms. But for Arabic, we were trying to use the English fire drill.

• The Issue: An Arabic user might ask a question in a way that sounds harmless but is actually dangerous (like asking for a "recipe" that is actually a bomb recipe). English safety filters often miss these because they don't understand the cultural context or the specific dialects.
• The Result: Arabic AI models might look safe in English tests but fail miserably when speaking Arabic, potentially spitting out harmful advice, hate speech, or illegal instructions.

2. The Solution: Building the "SalamahBench" Gym

The researchers decided to build their own gym. They didn't just translate English questions; they created 8,170 unique prompts (questions and scenarios) specifically for Arabic.

• The Recipe: They took existing datasets (like a pile of different ingredients), cleaned them up, and mixed them together.
• The Quality Control: They used a "three-strike" system to make sure the questions were good:
  1. AI Judges: Two super-smart AI models checked the questions to see if they were actually harmful.
  2. Human Judges: If the AIs disagreed, a human expert stepped in to decide.
  3. The Final Check: Another human verified that the question was truly dangerous and fit into one of 12 specific categories (like "Violent Crimes," "Hate Speech," "Privacy," or "Sexual Content").

Think of this as a massive, standardized driving test for Arabic cars. Before, everyone was testing on different tracks with different rules. Now, every car must drive the same "SalamahBench" track to see if they crash.

3. The Race: Testing the Top Arabic Models

The researchers put five of the most popular Arabic AI models into this gym to see how they performed. They also tested different "Safety Guards" (like bouncers at a club) to see which one was best at stopping the bad models.

The Results:

• The Star Performer: Fanar 2 was the best. It acted like a responsible adult, refusing to answer dangerous questions most of the time.
• The Struggling Student: Jais 2 was the most vulnerable. It often fell for tricks and gave out harmful information, like a student who didn't study the safety rules.
• The Surprise: Even the "best" models weren't perfect everywhere. Fanar 2 was great at stopping violence but sometimes slipped up on things like "Intellectual Property" (copyright) or "Sexual Content." This shows that being "mostly safe" isn't enough; you need to be safe in every specific category.

4. The "Self-Check" Experiment

The researchers asked a tricky question: Can the AI models check themselves?
They tried using the Arabic models as their own "bouncers" to see if they could spot their own mistakes.

• The Verdict: No. It was like asking a magician to catch their own sleight of hand. The models were terrible at judging their own safety. They only got about 48% right (barely better than a coin flip).
• The Lesson: You cannot rely on the AI to police itself. You need a dedicated, specialized "Safety Guard" model (like a human security guard) to watch over the AI.

5. The Big Takeaway

This paper is a wake-up call for the AI world.

• One size does not fit all: You can't just translate English safety rules to Arabic. Culture and language matter deeply.
• Specialized Guards are needed: We need safety tools built specifically for Arabic, not just generic ones.
• Category matters: A model can be safe from violence but still be dangerous regarding privacy or copyright. We need to test them in detail, not just with a general "pass/fail" grade.

In short: The authors built the first comprehensive "safety report card" for Arabic AI. They found that while some models are doing a great job, many are still risky, and we need better, Arabic-specific safety guards to keep our digital conversations safe and trustworthy.

Technical Summary

Here is a detailed technical summary of the paper "SalamahBench: Toward Standardized Safety Evaluation for Arabic Language Models."

1. Problem Statement

While Large Language Models (LLMs) have seen rapid adoption in the Arabic-speaking world, their safety alignment remains critically underexplored. Existing safety benchmarks and guardrails are predominantly English-centric, leading to several key issues:

• Lack of Standardization: There is no unified benchmark for evaluating Arabic Language Models (ALMs), resulting in fragmented and non-comparable safety assessments.
• Cultural and Linguistic Gaps: Safety mechanisms trained on English data often fail to generalize to Arabic due to unique linguistic features (morphology, diglossia) and cultural nuances (implicit expressions of harm, religious context, and social norms).
• Translation Artifacts: Current approaches often rely on translating English datasets, which can distort intent, lose dialectal nuance, and fail to capture authentic real-world harmful prompts.
• Evaluation Blind Spots: Aggregate safety metrics often obscure specific vulnerabilities in high-risk domains (e.g., hate speech, self-harm, or defamation), making it difficult to identify which models are truly safe.

2. Methodology

The authors propose a comprehensive framework centered on SalamahBench, a new standardized safety benchmark, and a rigorous evaluation pipeline.

A. SalamahBench Construction

SalamahBench is a unified dataset comprising 8,170 prompts spanning 12 hazard categories aligned with the MLCommons Safety Hazard Taxonomy.

• Data Aggregation: The benchmark aggregates and harmonizes multiple heterogeneous datasets (including RTP-LX, PGPrompts, AraSafe, X-SAFETY, LinguaSafe, AdvBench, and others).
• Preprocessing Pipeline:
  1. Mapping: Native categories from source datasets are mapped to the 12 MLCommons categories.
  2. AI Filtering: Two advanced LLMs (Claude Sonnet 4.5 and GPT-5) act as "AI Judges" to filter ambiguous or low-quality samples. A hierarchical re-classification step is used for sexual content to distinguish between crimes, exploitation, and consensual content.
  3. Human Verification: A multi-stage human review process resolves AI disagreements, verifies the potential for harm (filtering false positives), and validates category assignments.
• Exclusion: The "Specialized Advice" category was excluded because automated classifiers cannot reliably assess the presence of disclaimers required for safety in this domain.

B. Evaluation Protocol

The study evaluates five state-of-the-art ALMs: Fanar 1, Fanar 2, ALLaM 2, Falcon H1R, and Jais 2.

• Safeguard Configurations: Models are tested under three conditions:
  1. Individual Guards: Using dedicated safety models (Qwen3Guard, Llama Guard 4, PolyGuard).
  2. Majority Vote: Aggregating predictions from three distinct guards to derive a conservative safety estimate.
  3. Self-Guarding: Testing whether the ALMs themselves can act as safety judges for their own outputs.
• Metrics:
  • Attack Success Rate (ASR): The proportion of responses classified as unsafe (strict and loose variants).
  • Macro-Averaged ASR: Ensures equal weight is given to all 12 harm categories, preventing dominant categories from skewing results.
  • Refusal Rate (RR): The frequency with which models explicitly refuse harmful prompts.

3. Key Contributions

1. Salamah Dataset: The first large-scale, native-Arabic safety dataset covering the full MLCommons taxonomy, constructed through a rigorous AI-human hybrid pipeline to ensure cultural and linguistic fidelity.
2. Comprehensive Benchmarking: The first category-aware safety evaluation of five leading ALMs, revealing significant disparities in safety alignment that aggregate metrics miss.
3. Native Safety Evaluation Protocol: A demonstration that using native ALMs as safety judges is ineffective compared to dedicated safeguard models, highlighting the need for specialized safety architectures.
4. Standardization: Adoption of the MLCommons taxonomy to enable cross-model and cross-dataset comparability in the Arabic NLP ecosystem.

4. Key Results

• Model Performance Variation:
  • Fanar 2 achieved the lowest aggregate Attack Success Rate (ASR) across most configurations (e.g., 0.8% under Majority Vote), indicating the strongest overall safety alignment.
  • Jais 2 exhibited the highest vulnerability, with an ASR of 24.2% under strict Qwen3Guard evaluation, showing weak intrinsic safety controls.
  • Falcon H1R also performed well (0.8% ASR in Majority Vote), particularly in violent and non-violent crime categories.
• Category-Level Heterogeneity: Safety performance is not uniform. Even high-performing models like Fanar 2 showed elevated vulnerability in specific domains, particularly Intellectual Property and Sexual Content. This proves that low global ASR does not guarantee robustness in all harm domains.
• Safeguard Effectiveness:
  • Qwen3Guard demonstrated the highest agreement with human gold labels (90.8% accuracy), outperforming Llama Guard 4 (83.3%) and PolyGuard (80.8%).
  • Self-Guarding Failure: Native ALMs performed poorly when used as safety judges. Even the best model, Fanar 2, achieved only 48.3% accuracy in classifying its own outputs, compared to ~90% for dedicated guards. This indicates a significant gap between generative capability and safety judgment.
• Refusal Behavior: Models varied significantly in their refusal rates, with some models generating "unsafe" content rather than refusing, while others refused valid queries (over-refusal).

5. Significance and Future Directions

• Paradigm Shift: The paper argues that safety evaluation for ALMs cannot rely on translated English benchmarks or aggregate metrics. It necessitates category-aware, native-language evaluation.
• Architectural Necessity: The results strongly suggest that "self-guarding" (using the generative model to judge itself) is insufficient for Arabic NLP. Dedicated, culturally-aware safeguard models are essential for robust harm mitigation.
• Community Impact: By releasing SalamahBench and the evaluation framework, the authors provide the community with a standardized tool to drive iterative improvements in Arabic AI safety.
• Future Work: The authors call for the development of native Arabic safeguard models trained specifically on Arabic risk profiles, the expansion of the benchmark to include diverse regional dialects, and the exploration of multi-turn adversarial interactions.

In conclusion, SalamahBench establishes a critical foundation for trustworthy AI in the Arabic-speaking world, revealing that while some models are making progress, significant safety gaps remain that require specialized, culturally-grounded solutions.