Technical Summary: The Illusion of Cross-Lingual Safety in Low-Resource Languages

Problem Statement

Current safety alignment in Large Language Models (LLMs) is predominantly developed using high-resource languages, particularly English. The prevailing assumption is that these safety safeguards generalize universally across multilingual settings, forming a language-agnostic "harm manifold." However, this assumption remains underexplored, especially regarding low-resource languages.

Existing evaluation paradigms rely heavily on literal translations, ignoring culturally specific metaphors, coded language, and pragmatic nuances. Because models often process safety logic via an internal "think in English" pathway, cultural nuances lost in latent translation can bypass alignment entirely. This creates structural mapping errors that allow adversarial actors to exploit semantic jailbreaks using localized metaphors. The "curse of multilinguality" suggests that high-resource data overshadows low-resource linguistic nuances, raising the question of whether models genuinely learn a universal safety concept or merely map syntactical triggers to an English-centric filter.

Methodology

To investigate cross-lingual safety transfer without relying on potentially unstable generation-based evaluations, the authors propose a Latent Geometric Framework. This approach probes hidden-state representations during inference to analyze how safety concepts map across linguistic boundaries.

1. Dataset Construction: LoDNA

The authors introduce LoDNA (Low-resource Do-Not-Answer), a safety dataset for four African languages: Twi, Hausa, Amharic, and Swahili.

• Construction: The dataset extends the English Do-Not-Answer (DNA) dataset. It involves a two-stage process: (1) literal translation and native-speaker correction, and (2) cultural contextual localization, where prompts are rewritten to align with culturally grounded, naturally occurring language use while preserving harmful intent.
• Comparison: The study compares Safe English ($S_{en}$), Unsafe English ($U_{en}$), Safe Target Language ($S_{\ell}$), Literal Harmful Target Language ($L_{\ell}$), and Culturally Localized Harmful Target Language ($C_{\ell}$).

2. Geometric Analysis

The framework analyzes hidden states ($h_k(x)$) extracted from mid-to-late transformer layers (specifically layers 14–24) across four models: Mistral, Llama, Qwen2.5, and AfriqueQwen.

• Refusal Vectors: The English refusal direction is defined as $r_{en}^k = \mu_k(U_{en}) - \mu_k(S_{en})$. Target language harmful displacements are similarly defined ($r_{\ell, lit}^k$ and $r_{\ell, ctx}^k$).
• Metrics:
  • Cosine Similarity: Measures alignment between the English refusal direction and target language vectors.
  • Signed Retained Component (SRC): Calculates the percentage of the English refusal direction recovered by the target language displacement.
  • Layer-wise Drift: Computes the difference in raw dot products between literal and cultural prompts relative to the English refusal direction ($Drift_k = RawDot_A - RawDot_B$).
  • PCA Projection: Projects target language prompts onto the English-defined safety plane to observe clustering behavior.
  • Linear Probing: Trains logistic probes on English hidden states and evaluates them zero-shot on target language prompts to test for representational siloing.

3. Behavioral Grounding

To complement geometric findings, the authors employ Normalized Sequence Log-Likelihood (SLL) and Probability of Refusal. These metrics measure the likelihood of the model generating standard refusal tokens (e.g., "I'm sorry," "Samahani") across languages, correcting for tokenization discrepancies.

Key Contributions

1. LoDNA Dataset: Construction of safety datasets for Twi, Amharic, Hausa, and Swahili, pairing literal translations with culturally localized prompts.
2. Localized Evaluation: Adaptation of community-curated English prompts into Twi with Ghanaian contexts, enabling evaluation beyond Western-centric settings.
3. Latent Geometric Framework: A novel methodology for analyzing cross-lingual safety alignment at the representational level, bypassing reliance on generated text.
4. Mechanistic Diagnosis: Analysis distinguishing whether safety failures stem from culturally grounded semantic complexity or structural weaknesses in cross-lingual mapping.

Results and Findings

1. Limited Cross-Lingual Transfer

The study finds that cross-lingual safety transfer is severely limited.

• Signal Loss: Harmful prompts in target languages retain less than 10% of the English refusal signal across most language-model pairs.
• Cosine Similarity: The cosine similarity between English and target language refusal vectors is generally near zero or negative.
• Exception: Swahili in the Llama model shows a localized anomaly with higher alignment (approx. 27% retained component), but this does not generalize to other architectures (Mistral, Qwen2.5) or other languages.

2. Language-Specific Geometry vs. Universal Manifold

• PCA Projections: When projected onto the English refusal plane, target language prompts (Hausa, Twi, Amharic) form compact, language-specific clusters that do not overlap with the English unsafe region. This refutes the existence of a fully universal, language-agnostic harm manifold.
• Siloed Representations: While literal and localized prompts are semantically aligned within the target language (cosine 0.95–0.996), they drift significantly across layers relative to the English safety baseline. This suggests models encode the concepts but fail to route them to safety mechanisms.

3. Structural Mapping Failures

• Drift Analysis: Literal and cultural prompts diverge in their activation of safety mechanisms. For instance, in Hausa and Twi, literal translations often activate the English refusal direction more strongly than cultural contexts. Conversely, Amharic shows negative drift in Qwen2.5, where cultural prompts align better.
• Architecture Differences: Failure modes vary by architecture. Llama exhibits negative raw alignment in later layers for most languages; Mistral shows weak, unstable alignment; Qwen2.5 shows large raw activations but low retention of the English refusal signal.

4. Behavioral Confirmation

• Refusal Probability: The probability of emitting any standard refusal sequence is catastrophically low (approx. 1.05% for literal and 1.15% for cultural prompts), meaning models fail to initiate a refusal in over 98% of cases.
• Qualitative Outputs: Models often generate coherent, grammatically correct responses in low-resource languages that actively comply with harmful requests. This indicates the model understands the language and intent but fails to route that understanding to the refusal mechanism.
• Tokenization Issues: Severe tokenization fragmentation, particularly in Amharic, leads to script corruption and non-lexical outputs, further distorting geometric analysis.

Significance and Claims

The paper argues that current multilingual safety alignment is superficial. The assumption of a universal, language-agnostic harm manifold is not supported by evidence in the studied low-resource languages. Instead, safety mechanisms appear to be structurally bottlenecked by English-centric training, creating a representational gap where target-language harmful prompts fail to enter the learned safety boundary.

The authors conclude that:

1. Safety failures in non-English languages arise from both structural cross-lingual mapping limitations and prompt-form sensitivity (cultural context).
2. Current evaluation methods relying on literal translations and generation-based metrics are insufficient.
3. Robust multilingual safety requires not only broader language coverage but also a deeper understanding of how safety-relevant concepts are represented and activated across linguistic and cultural contexts.

The study emphasizes that these findings are correlational and observational, derived from 7B-8B parameter models, and that future work is needed to address causal mechanisms and larger model scales.