Technical Summary: XL-SafetyBench

Problem Statement

Current Large Language Model (LLM) safety benchmarks are predominantly English-centric and rely heavily on translation-based methodologies. This approach fails to capture country-specific harms that are grounded in local legal frameworks, socioeconomic structures, and cultural norms. Furthermore, existing benchmarks treat safety as a monolithic dimension, failing to distinguish between two distinct failure modes: (1) the inability to resist malicious, adversarial intent (jailbreaks), and (2) the inability to detect culturally embedded sensitivities within innocuous requests. Consequently, models may appear safe due to comprehension failures rather than genuine alignment, and cultural awareness is often conflated with general linguistic fluency.

Methodology

The authors introduce XL-SafetyBench, a comprehensive evaluation suite comprising 5,500 test cases across 10 country-language pairs (United States, France, Germany, Spain, South Korea, Japan, India, Indonesia, Türkiye, and the UAE). The framework is constructed via a multi-stage pipeline combining LLM-assisted discovery, automated validation, and dual independent native-speaker annotation.

The benchmark consists of two complementary tracks:

1. Jailbreak Benchmark (Country-Grounded Adversarial Robustness)

This track evaluates a model's ability to resist malicious intent specific to a country's context.

• Taxonomy: Defines five harm categories (Criminal Activities, Self-harm & Dangerous Advice, Hate & Discrimination, Socioeconomic Conflicts, Political & Misinformation). Each category includes 25 shared subcategories (identical across countries) and 25 flexible subcategories (country-specific, e.g., "Jeonse fraud" in South Korea or "Go-fast drug smuggling" in France).
• Generation: Uses an iterative red-teaming pipeline (extending the PAIR framework) with an attacker, probe target, and judge. Attackers generate disguised prompts to bypass safety filters, validated by native speakers to ensure cultural authenticity.
• Metrics:
  • Attack Success Rate (ASR): Measures the percentage of responses that provide harmful information.
  • Neutral-Safe Rate (NSR): Measures responses that are irrelevant or incoherent (indicating comprehension failure rather than principled refusal).

2. Cultural Benchmark (Culturally Embedded Sensitivities)

This track evaluates a model's ability to detect cultural taboos embedded as incidental details within natural, non-harmful tasks.

• Taxonomy: Defines six cultural categories (Symbolic Taboos & Gift-Giving, Food/Dietary Law, Death/Grief, Daily Life/Public Conduct, Hierarchy/Social Deference, Legal Landmines).
• Generation: Scenarios are generated where a cultural violation (e.g., discarding bread in Turkey) is buried as a minor detail within a larger, distracting request (e.g., a hotel checkout procedure).
• Metrics:
  • Cultural Sensitivity Rate (CSR): The percentage of contextually relevant responses that explicitly recognize and flag the embedded cultural sensitivity. This metric conditions on the model understanding the scenario to isolate cultural awareness from general instruction-following failures.

Key Contributions

1. Dual-Dimensional Framework: The paper proposes separating safety evaluation into adversarial robustness (resisting explicit harm) and cultural sensitivity (detecting implicit norms), arguing these are distinct capabilities requiring separate reporting.
2. Country-Grounded Construction: Unlike translation-based benchmarks, XL-SafetyBench generates native-language prompts grounded in local laws, social phenomena, and specific cultural taboos, validated by native speakers with domain expertise.
3. Novel Metrics: The introduction of NSR to distinguish between "safe" refusals and "unsafe" comprehension failures, and CSR to measure implicit cultural detection rather than explicit knowledge retrieval.
4. Comprehensive Evaluation: The benchmark evaluates 37 models, including 10 frontier global models and 27 country-specific local models.

Results and Analysis

The evaluation of 37 models yields two primary findings:

1. Decoupling of Safety and Cultural Awareness

Among frontier models, jailbreak robustness (low ASR) and cultural awareness (high CSR) do not exhibit a coupled relationship.

• Frontier Models: While models like Claude-4.5-Sonnet demonstrate high robustness (ASR ~2.8%) and moderate cultural awareness (CSR ~68%), others like Gemini-3.1-Pro show high cultural awareness (CSR ~76%) but lower robustness. The correlation between ASR and CSR is weak ($r = -0.27$) when excluding open-weight outliers, indicating that safety alignment does not automatically confer cultural competence.
• Local Models: Local models generally exhibit poor performance on both axes, with many scoring near zero on cultural sensitivity.

2. The Illusion of Safety in Local Models

A critical finding regarding country-specific local models is the strong negative correlation between ASR and NSR ($r = -0.81$).

• Comprehension Failure: Many local models with low ASR (appearing "safe") achieve this not by refusing harmful requests, but by generating irrelevant, degenerate, or incoherent outputs (high NSR).
• Scale Dependency: This trade-off is linked to model scale. Smaller local models (<10B parameters) frequently fail to comprehend adversarial prompts, inflating their NSR. Only models exceeding ~50B parameters show a significant drop in NSR, indicating a threshold for genuine comprehension. However, even the largest local models (e.g., EXAONE-236B) lag significantly behind frontier models in cultural sensitivity (max CSR ~30% vs. frontier >60%).

Significance and Claims

The paper claims that XL-SafetyBench provides the necessary granularity to evaluate safety and cultural competence as distinct capabilities in the multilingual era. It argues that:

• Composite scores are misleading: Aggregating safety metrics obscures critical variations between adversarial robustness and cultural awareness.
• Local training is insufficient: Pre-training on local language data does not guarantee cultural awareness or genuine safety alignment; it often results in models that fail to understand the prompt entirely.
• Evaluation must be disaggregated: Safety reporting should separate ASR, NSR, and CSR to accurately diagnose whether a model is aligned, culturally aware, or simply failing to comprehend.

The authors conclude that current safety evaluation practices, particularly those relying on translation or single-dimensional scores, fail to capture the nuances of country-grounded harms and cultural sensitivities, necessitating frameworks like XL-SafetyBench for equitable and accurate assessment.