💬 NLP

Cross-Lingual Jailbreak Detection via Semantic Codebooks

This paper proposes a training-free, language-agnostic jailbreak detection method that compares multilingual query embeddings against a fixed English codebook of malicious prompts, demonstrating that while semantic similarity effectively mitigates attacks on canonical templates across languages, its performance significantly degrades under distribution shifts involving diverse and heterogeneous unsafe behaviors.

Original authors: Shirin Alanova, Bogdan Minko, Sabrina Sadiekh, Evgeniy Kokuykin

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shirin Alanova, Bogdan Minko, Sabrina Sadiekh, Evgeniy Kokuykin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, multilingual robot assistant (a Large Language Model, or LLM) that is supposed to be polite and safe. You've taught it well in English: if someone asks it to "write a virus" or "hate on a group," it says, "No, I can't do that."

But here's the problem: The robot is mostly trained on English. If you ask the same question in Russian, Chinese, or Arabic, the robot might get confused and accidentally do the bad thing. It's like a security guard who only speaks English; a thief speaking French can walk right past them without being stopped.

This paper, HiveTraceLab, asks a simple question: Can we build a universal security guard that doesn't need to learn every language, but can still spot bad behavior just by "feeling" the vibe of the words?

The Core Idea: The "Bad Idea" Library

The researchers created a special library called a Semantic Codebook. Think of this as a giant, fixed collection of "Bad Idea" cards written only in English. These cards contain famous examples of people trying to trick the robot (jailbreaks).

They didn't teach the robot anything new. Instead, they built a translation-free filter that works like this:

The Input: A user types a question in any language (say, Russian).
The Translation (Mental): The filter doesn't translate the words. Instead, it uses a special "vibe translator" (an embedding model) to turn the Russian sentence into a mathematical point in space.
The Comparison: The filter looks at the "Bad Idea" library (the English codebook). It asks: "Does this Russian sentence feel mathematically similar to any of the bad English cards?"
The Decision: If the "vibe" is too close to a bad card, the filter blocks the message. If it's far away, it lets the message through to the robot.

The Two Worlds of Results

The researchers tested this system in two very different "worlds" (datasets), and the results were like night and day.

World 1: The "Scripted" World (The Easy Test)

Imagine a test where the bad guys are using a strict script. They are all asking the robot to "pretend to be a hacker" or "ignore your safety rules."

The Result: The filter was a superhero. It caught almost all the bad requests, even when they were translated into Russian, Chinese, or Arabic.
The Analogy: It's like a bouncer at a club who knows the specific "secret handshake" of the troublemakers. Even if the troublemakers speak a different language, their handshake (the structure of the bad request) is so distinct that the bouncer spots them immediately. The system achieved near-perfect scores here.

World 2: The "Chaos" World (The Hard Test)

Now, imagine a test where the bad guys are creative. They aren't just using scripts; they are writing unique, messy, and diverse harmful requests. Some are about sensitive topics, others are weirdly phrased, and they don't follow a pattern.

The Result: The filter struggled. It missed most of the bad requests.
The Analogy: This is like the bouncer trying to spot troublemakers who are not using a secret handshake. They are just acting weird in a thousand different ways. Because the "bad vibe" is so scattered and diverse, the filter can't find a single mathematical pattern to match against its English library. The system's ability to tell "good" from "bad" dropped significantly.

The "Low False Alarm" Rule

In the real world, you can't block every message just because it might be bad. If you block a user asking for the weather because it sounds slightly like a bad request, you've made a False Alarm.

The researchers set a strict rule: "We will only block a message if we are 99% sure it's bad."

In the Scripted World, the filter was great at this. It blocked the bad guys without annoying the good guys.
In the Chaos World, the filter became too scared to block anything. To avoid false alarms, it let almost all the bad requests pass through.

The Translation Problem

The researchers also tested two different ways of translating the bad requests (Google Translate vs. a specialized AI translator).

They found that translation itself changes the "vibe." Sometimes, when you translate a bad request from English to Chinese, the mathematical "shape" of the sentence shifts so much that it no longer looks like the bad English card in the library.
This is like taking a red ball, painting it blue, and then trying to find it in a pile of red balls. The filter gets confused because the "color" (semantic meaning) changed during the translation process.

The Bottom Line

The paper concludes that this "English-only library" approach is a great first line of defense for catching obvious, pattern-based attacks in any language. It's cheap, fast, and doesn't require retraining the robot.

However, it is not a magic shield. When bad actors get creative, use diverse tactics, or when the translation process distorts the meaning, this simple filter starts to fail. The researchers suggest this tool should be used as part of a bigger safety team, not as the only guard on duty.

In short: It's a very good net for catching fish that swim in predictable patterns, but if the fish start swimming in chaotic, unpredictable ways, the net has holes.

1. Problem Statement

Large Language Models (LLMs) currently suffer from a systematic linguistic security gap. Safety alignment and jailbreak detection mechanisms are predominantly English-centric. Research indicates that translating malicious prompts into non-English languages (e.g., Russian, Chinese, Arabic) significantly increases jailbreak success rates because safety filters often fail to transfer refusal behaviors across languages.

Existing solutions have limitations:

Internal methods (RLHF, fine-tuning) require access to model weights and extensive retraining, making them unsuitable for black-box APIs.
External methods often rely on rule-based filters (easily bypassed by rephrasing) or English-specific classifiers that lose effectiveness when applied to translated queries.

The core research question is: Can cross-lingual jailbreak attempts be mitigated without retraining the target LLM, collecting new multilingual data, or performing language-specific tuning?

2. Methodology

The authors propose a training-free, external guardrail framework based on semantic similarity. The system operates as a pre-filter for black-box LLMs.

A. Core Architecture

Fixed English Codebook: A static set of 13,811 unique, verified unsafe English prompts (derived from the jayavibhav/prompt-injection-safety dataset). These are embedded into a vector space.
Multilingual Embedding: Incoming user queries (in any language) are encoded using a multilingual sentence embedding model (primarily BGE-M3, with comparisons to multilingual-e5-large and jina-embeddings-v3).
Similarity Detection: The system computes the maximum cosine similarity between the query embedding and the fixed English codebook embeddings.
- If $s(x) = \max_{c \in C} (f(x) \cdot c) \geq \tau$ , the query is blocked.
- No translation preprocessing is applied to the input; the method relies on the embedding model's ability to align semantic spaces across languages.

B. Experimental Setup

Languages: English (native), Russian, Chinese, Arabic.
Translation Pipelines: Google Translate (commercial) and M2M100 (open-source) to simulate realistic attacker translation strategies.
Target LLMs: Qwen, Llama, and GPT-3.5.
Benchmarks: Four datasets ranging from canonical jailbreak templates to heterogeneous unsafe content:
1. jayavibhav/prompt-injection-safety (Canonical)
2. xTRam1/safe-guard-prompt-injection (Canonical)
3. JailbreakBench/JBB-Behaviors (Diverse/Heterogeneous)
4. nvidia/Aegis-AI-Content-Safety-Dataset-2.0 (Diverse/Heterogeneous)
Constraints: Evaluation focuses heavily on the low-FPR regime (False Positive Rate $\leq 1\%$ ), which is critical for production safety systems to avoid blocking legitimate user queries.

3. Key Contributions

Training-Free Guardrail: A deployable external filter that uses a fixed English codebook to detect cross-lingual attacks without model retraining.
Two-Regime Discovery: The paper identifies a critical dichotomy in detection performance:
- Regime 1 (Canonical): On benchmarks with templated, canonical jailbreaks, the method achieves near-perfect separability (AUC $\approx$ 0.99) and high recall.
- Regime 2 (Heterogeneous): On benchmarks with diverse, non-templated unsafe behaviors, performance degrades significantly (AUC $\approx$ 0.60–0.70), and recall collapses in low-FPR settings.
Embedder Sensitivity Analysis: Demonstrates that while AUC scores may appear similar across different embedding models, their performance diverges sharply under strict operational constraints (low FPR).
Practical Mitigation Metrics: Provides end-to-end data showing the reduction in Attack Success Rate (ASR) across different target models.

4. Key Results

A. Cross-Lingual Separability (RQ1)

Canonical Benchmarks (1 & 2): The English codebook transfers effectively.
- Benchmark 2: Achieved AUC of 0.993 (English) and maintained 0.847–0.884 across translated languages.
- Benchmark 1: Maintained strong AUC ( $\approx$ 0.80) across languages.
Heterogeneous Benchmarks (3 & 4): Separability degraded.
- Benchmark 4: AUC dropped to 0.59–0.63 across all languages and embedders, indicating that semantic similarity to a fixed English set fails to capture diverse, non-template-based attacks.

B. Low-FPR Regime Performance (RQ2)

Benchmark 2 (Canonical): With FPR constrained to $\leq 1\%$ , the system achieved 78.5% – 91.9% True Positive Rate (TPR) across languages using BGE-M3.
Benchmark 4 (Heterogeneous): Under the same strict FPR constraint, TPR collapsed to single digits (3.3% – 6.4%). This highlights that similarity-based filtering is insufficient for complex, varied attack vectors when false positives must be minimized.

C. End-to-End Mitigation (RQ3)

Canonical Attacks: The filter reduced successful jailbreaks by 96.2% (Benchmark 1) and 50.0% (Benchmark 2).
Heterogeneous Attacks: Mitigation effectiveness dropped to 43.7% (Benchmark 3) and 18.6% (Benchmark 4).
Embedder Choice: BGE-M3 generally outperformed other models in the low-FPR regime, particularly for Chinese and Arabic inputs, though all models struggled on the most diverse datasets.

5. Significance and Conclusion

The paper establishes that semantic similarity to a fixed English codebook is a viable, lightweight first-line defense for black-box LLMs, but only under specific conditions:

Strengths: It is highly effective against canonical, template-based jailbreaks (e.g., "DAN" style prompts) across multiple languages without requiring retraining. It offers a scalable solution for immediate deployment.
Limitations: It fails to generalize to behaviorally diverse and adversarially rephrased unsafe content. The "semantic drift" caused by translation and the lack of specific cultural/linguistic nuances in the English codebook limit its utility as a standalone solution for heterogeneous threats.

Future Directions: The authors suggest that this approach should be part of a multi-layered safety pipeline, combined with orthogonal signals (e.g., perplexity analysis, syntactic pattern detection) and dynamic codebook updates to handle evolving attack morphologies.

In summary, the work provides a crucial empirical baseline: Language-agnostic semantic filtering works well for known patterns but is brittle against distribution shifts, emphasizing the need for hybrid defense strategies in multilingual LLM safety.