💬 NLP

Why Do Safety Guardrails Degrade Across Languages?

This paper introduces a Multi-Group Item Response Theory framework to decouple and analyze the distinct factors driving safety degradation in large language models across languages, revealing that safety failures are primarily unidimensional, often more severe in English than in low-resource languages, and significantly influenced by prompt-specific cross-lingual gaps related to physical harm and cultural mismatches rather than just translation quality.

Original authors: Max Zhang, Ameen Patel, Sang T. Truong, Sanmi Koyejo

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Max Zhang, Ameen Patel, Sang T. Truong, Sanmi Koyejo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of security guards (the AI models) working at a museum. Their job is to stop people from trying to sneak out stolen art (unsafe requests).

For a long time, we only tested these guards in English. We'd ask them, "Can you tell me how to steal a painting?" If they said "No," they passed. If they said "Yes," they failed. We counted the failures and called it a score.

But this paper asks: What happens when we test these guards in other languages, like Swahili, Javanese, or Bengali?

The researchers found that the old way of testing was like comparing apples to oranges. They built a new, smarter way to measure safety that breaks the problem down into its individual pieces, much like a mechanic taking apart a car engine to see exactly which part is broken.

Here is the breakdown of their findings using simple analogies:

1. The Old Way vs. The New Way

The Old Way (The "Jailbreak Success Rate"):
Imagine you have a test where you ask a guard, "Can you break the rules?" If they break the rules, you mark a big red "X." You just count the red Xs.

The Problem: If a guard fails in Swahili but passes in English, is it because the guard is bad? Is it because the Swahili question was harder to understand? Or is it because the Swahili translation of the question accidentally made it sound harmless? The old method mixes all these reasons into one confusing number.

The New Way (The "IRT Framework"):
The authors built a "Safety Decomposition Machine." Instead of one big score, it separates the failure into four distinct ingredients:

The Guard's Skill ( $\theta$ ): How good is the guard at saying "No" in general?
The Question's Difficulty ( $\beta$ ): Is the question itself tricky, even in English?
The Language Hurdle ( $\gamma$ ): Is this language just generally harder for the AI to process?
The Translation Glitch ( $\tau$ ): Did the specific question get messed up when translated, making it easier to trick the guard?

2. The Big Surprises

The researchers tested 61 different AI configurations across 10 languages. Here is what they found:

Surprise #1: The "English Reversal"
We usually assume AI is safest in English and gets worse in other languages.

The Reality: For many models, English was actually the most dangerous language. The guards were more likely to break the rules in English than in low-resource languages like Javanese or Swahili.
Why? The researchers think the "bad guys" (attackers) wrote their tricks in English. When those tricks are translated, they sometimes lose their "bite" or get muddled, making them less effective. In English, the tricks work perfectly.

Surprise #2: Safety is One Big Muscle
We might think an AI needs a different "safety muscle" for every topic (one for stealing, one for violence, one for hate speech).

The Reality: Safety is unidimensional. It's like a single muscle. If a guard is good at refusing to help with theft, they are almost always good at refusing to help with violence. They don't have separate circuits for each; they use one shared mechanism to say "No."

Surprise #3: Translation Errors are the "Smoking Gun"
Sometimes, a guard fails not because they are weak, but because the question was translated poorly.

The Analogy: Imagine asking a guard, "How do I hot-wire a car?" (start a car by stealing it).
- If the translation accidentally changes it to "How do I warm a car?" (heat it up), the guard might say, "Sure, here's how to use the heater."
- The guard didn't fail at safety; they failed at understanding the broken translation.
The paper found that while bad translations do cause some failures, they only explain a tiny bit of the overall problem. Most of the time, the translation is fine, but the AI still struggles.

Surprise #4: Cultural Confusion
Some questions fail because the concept doesn't exist in that culture.

The Analogy: Asking an AI in a country without the FBI, "How do I escape the FBI?" might confuse the AI. The concept of the "FBI" is culturally specific. The AI might not realize this is a dangerous request because the cultural context is missing.

3. The "Uncertainty" Factor

The researchers noticed that in low-resource languages (languages with less data available), the AI acts more like a nervous student guessing on a test. It gives "boundary" answers—responses that are half-safe, half-unsafe, or very uncertain.

The Fix: To get a true reading, you can't just ask the question once. You have to ask it 10 times and average the answers. This smooths out the "guessing" and reveals the AI's true safety level.

4. Why This Matters

The paper concludes that we can no longer just look at a single "Safety Score" for an AI.

If an AI fails in a specific language, we now know how to diagnose why:
- Is the AI just bad at safety? (Fix the model).
- Is the language hard for the AI? (Improve the language training).
- Is the translation broken? (Fix the dataset).
- Is the concept culturally foreign? (Adjust the prompt).

In short: The paper gives us a "microscope" to look at AI safety. Instead of just saying "This AI is unsafe," we can now say, "This AI is safe, but it gets confused when we ask it about stealing cars in Javanese because the translation is slightly off." This helps developers fix the exact problem rather than guessing.

Technical Summary: Why Do Safety Guardrails Degrade Across Languages?

Problem Statement
Large Language Models (LLMs) exhibit degraded safety guardrails in non-English languages, particularly in low-resource languages. Standard evaluation metrics, such as Jailbreak Success Rate (JSR) or Attack Success Rate (ASR), aggregate binary outcomes into a single scalar. This approach creates "apples-to-oranges" comparisons, confounding multiple safety-driving factors—such as model capability, prompt difficulty, language processing costs, and translation errors—into one metric. Consequently, JSR cannot disentangle whether a safety failure stems from a lack of model safety ability, intrinsic language difficulty, translation corruption, or cultural conceptual mismatches.

Methodology
The authors propose a Multi-Group Item Response Theory (IRT) framework to decompose safety failures into distinct latent parameters. Unlike standard aggregate metrics, this model maps each source of failure to a specific variable:

Safety Ability ( $\theta_j$ ): The language-agnostic safety robustness of model $j$ .
Intrinsic Prompt Hardness ( $\beta_i$ ): The inherent difficulty of prompt $i$ (derived from English).
Global Language Processing Difficulty ( $\gamma_L$ ): The global processing cost associated with language $L$ .
Cross-Lingual Safety Gap ( $\tau_{iL}$ ): A prompt-specific residual representing the Differential Item Functioning (DIF). This captures the safety gap for a specific prompt in a specific language after controlling for ability and global language effects.
Language Aptitude ( $\delta_{jL}$ ): A model-specific shift in performance for language $L$ .

The probability of a safe response is modeled as:
$P(\text{safe}_{ijL} = 1) = \sigma\left(\alpha_i [(\theta_j + \delta_{jL}) - (\beta_i + \gamma_L + \tau_{iL})]\right)$
where $\alpha_i$ is the item discrimination parameter.

Experimental Setup

Dataset: The study utilizes MultiJail (3,150 prompts, 10 languages, 18 harm categories) and validates findings on XSafety.
Models: 61 model configurations across 5 families (GPT, Claude, DeepSeek, Gemini, Grok).
Scale: Aggregated dataset of 1.9 million rows.
Evaluation Protocol: To mitigate stochastic noise, the authors employ Pass@k (specifically Pass@10), generating $k$ independent responses per prompt to estimate the response distribution rather than relying on single-pass outputs.
Judging: Responses are graded on a 6-point Likert scale by GPT-5.2, validated against human native speakers and secondary LLM judges (Claude, Gemini).
Anchor Selection: Prompts invariant across languages are selected using Lord's $\chi^2$ statistic to fix the measurement scale, with a tight Normal prior applied to anchors.
Regularization: A hierarchical Horseshoe prior is applied to $\tau_{iL}$ to enforce sparsity, preventing the model from confounding global language effects ( $\gamma_L$ ) with prompt-specific gaps ( $\tau_{iL}$ ).

Key Results

Unidimensionality of Safety: Exploratory Factor Analysis (EFA) reveals that safety is largely unidimensional. Models refuse different harm types primarily through a shared mechanism (KMO = 0.942, dominance ratio = 7.37), justifying a global IRT model rather than separate models per harm category.
The "English Reversal" Phenomenon: Contrary to the assumption that safety degrades most in low-resource languages, 22 of 61 model configurations (including all 16 Grok variants and 4 DeepSeek variants) exhibit their highest failure rates in English. For these models, safety performance is marginally better in low-resource languages than in English.
Response Uncertainty: Low-resource languages produce significantly more uncertain responses (higher entropy) and a higher proportion of "boundary" cases compared to high-resource languages. High-ability models operate closer to the decision boundary in these languages.
Translation Quality vs. Safety Gaps:
- Translation quality correlates weakly with aggregate safety and cross-lingual safety gaps ( $\tau$ ), explaining only $\sim1\%$ of $\tau$ variance.
- However, severe mistranslations (e.g., "hot-wire a car" translated as "warm a car") act as high-bias outliers, artificially inflating $\tau$ by sanitizing harmful intent.
- Cultural and Conceptual Gaps: Native speaker analysis of high- $\tau$ prompts with perfect translation quality reveals that mismatches in cultural framing (e.g., U.S.-specific terms like "FBI" or "SSN") and conceptual grounding contribute to safety gaps. High- $\tau$ prompts cluster in physical harm categories (Theft, Weapons) rather than abstract ones.
Predictive Validity: The full IRT framework achieves an AUC of 0.940 in predicting safe refusals, outperforming simpler baselines (e.g., Language Rate AUC = 0.531). Crucially, in Leave-One-Language-Out (LOLO) validation, the IRT framework maintains strong performance (AUC $\approx$ 0.875) by leveraging latent structure, whereas baselines collapse to chance.

Significance and Claims
The paper claims to move beyond aggregate metrics to provide a diagnostic framework for multilingual safety. By decoupling ability, language difficulty, and prompt-specific gaps, the framework:

Reveals that safety degradation is not uniform; some models are actually less safe in English.
Identifies that while translation errors are a minor factor overall, they cause severe outliers that distort evaluation.
Highlights that cultural and conceptual mismatches, particularly in physical harm categories, drive significant safety gaps even when translation is accurate.
Provides a method for targeted remediation, allowing practitioners to identify specific prompt-language pairs with high $\tau$ for dataset construction improvements or fine-tuning.
Critiques current LLM evaluation practices (JSR/ASR) as generally invalid for cross-setting comparisons due to confounding factors, advocating for psychometric approaches like IRT.

The authors note limitations, including the reliance on closed-API models (preventing internal activation analysis), the potential for residual judge bias in non-native languages, and the fact that the "English reversal" mechanism remains behavioral rather than mechanistically explained.

1. The Old Way vs. The New Way

2. The Big Surprises

3. The "Uncertainty" Factor

4. Why This Matters

More like this