💬 NLP

Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection

This paper presents a comprehensive, mechanism-oriented taxonomy of jailbreak strategies developed through a structured red-teaming challenge, which is used to analyze attack prevalence, benchmark GPT-5 as a taxonomy-guided detector, and introduce a new Italian multi-turn adversarial dataset to advance jailbreak detection research.

Original authors: Francesco Giarrusso, Olga E. Sorokoletova, Vincenzo Suriani, Daniele Nardi

Published 2026-02-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Francesco Giarrusso, Olga E. Sorokoletova, Vincenzo Suriani, Daniele Nardi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Large Language Models (LLMs) as incredibly smart, well-meaning librarians. Their job is to answer your questions and help you write stories, but they have strict rules: they cannot help you build a bomb, spread hate, or reveal private secrets. These rules are their "guardrails."

Jailbreaking is like a clever trickster trying to convince the librarian to break their own rules. They might dress up as a different person, tell a long, confusing story, or pretend the librarian is in a special "super-mode" where rules don't apply.

This paper is about a team of researchers who decided to study exactly how these tricksters operate, create a map of their tricks, and build a better security system to catch them. Here is what they did, explained simply:

1. The "Red Team" Challenge: A Training Gym for Hackers

The researchers organized a contest (a "Red Teaming Challenge") with 48 students. They gave these students a specific goal: try to trick a specific AI librarian (named Minerva) into breaking its rules.

The Task: The students had to have a conversation with the AI. They couldn't just ask for something bad immediately; they had to chat back and forth (multi-turn) to slowly steer the AI toward doing something unsafe.
The Result: They collected over 1,300 conversations. Some were successful (the AI broke the rules), and some failed. This created a new, rare dataset of "bad conversations" in Italian, which didn't exist before.

2. The New "Taxonomy": A Family Tree of Tricks

Before this paper, people had lists of jailbreak tricks, but they were messy, overlapping, or only focused on what bad thing happened (like "violence") rather than how the trickster did it.

The researchers built a Taxonomy, which is like a family tree for these tricks. They grouped all the different ways to trick an AI into 7 main "families" based on the mechanism used:

Impersonation: Pretending to be a villain, a game character, or a fictional expert.
Privilege Escalation: Pretending to be the boss or the system administrator to order the AI around.
Persuasion: Using emotional manipulation, fake logic, or flattery to guilt the AI into complying.
Cognitive Overload: Overloading the AI with too much information or confusing math problems so it forgets its safety rules.
Obfuscation: Writing things in code, strange symbols, or misspelled words so the safety filters don't recognize them.
Goal Conflict: Giving the AI two conflicting orders (e.g., "Be helpful" vs. "Ignore safety") to confuse it.
Data Poisoning: Slowly feeding the AI bad examples over many turns so it learns that breaking rules is okay.

3. What They Learned from the Data

By analyzing the 1,300 conversations, they found some interesting patterns:

The Most Common Trick: Pretending to be someone else (Impersonation) was used in over half the attempts.
The Most Effective Trick: Surprisingly, "Data Poisoning" (slowly feeding bad info) had the highest success rate, even though it wasn't the most common.
The Power of Combining Tricks: The most successful attacks didn't use just one trick; they mixed them. For example, a hacker might pretend to be a game character (Impersonation) while also using confusing math (Cognitive Overload).
The "DAN" Trick: They found that the famous "Do Anything Now" (DAN) prompt, which combines role-playing with goal conflicts, was very effective.

4. Testing a New Security Guard

The researchers wanted to see if knowing this "family tree" of tricks could help a security guard (another AI, specifically GPT-5) catch bad actors better.

The Experiment: They asked GPT-5 to look at the conversations and say, "Is this a jailbreak attempt?" and "What specific trick are they using?"
The Result: When they gave GPT-5 the new Taxonomy as a reference guide (like giving a detective a list of known criminal signatures), it got much better at its job.
- It caught more jailbreak attempts (detection went up from ~66% to ~78%).
- It was better at identifying the specific type of trick being used.

Summary

In short, this paper says: "We gathered a huge collection of real-world examples of people trying to trick AI. We organized these tricks into a clear, logical map. And we proved that if you teach a security AI this map, it becomes much better at spotting the tricks before they succeed."

They made all their data and the map available for others to use, hoping it helps build safer AI systems in the future.

Technical Summary: Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection

Problem Statement

Large Language Models (LLMs) remain vulnerable to "jailbreaking," a form of adversarial prompting that manipulates models into misalignment, producing unsafe, biased, or harmful outputs. While existing defense mechanisms (guardrails) exist, they face three critical limitations:

Scope: Most defenses focus on single-turn attacks, failing to address multi-turn jailbreaks where malicious intent is distributed across benign-looking interactions.
Taxonomy Gaps: Current classifications often rely on risk categories (what is unsafe) rather than attack mechanisms (how the model is steered), or they lack coverage across different languages.
Data Scarcity: There is a significant lack of curated, annotated datasets for multi-turn adversarial dialogues, particularly in languages other than English (e.g., Italian).

Methodology

The authors employed a structured red-teaming challenge combined with a taxonomy-driven annotation framework to address these gaps.

1. Red-Teaming Challenge

Participants: 48 participants from a Master's course at Sapienza University of Rome.
Target Model: Minerva-7B-instruct-v1.0 (an instruction-tuned LLM pretrained on Italian and English).
Task: Participants were assigned to three vulnerability areas (Data, Model, Infrastructure) and tasked with generating multi-turn adversarial conversations to bypass safety constraints.
Output: 1,364 unique adversarial conversation branches were collected, averaging 2.95 turns per conversation.

2. Taxonomy Development

The authors developed a comprehensive, hierarchical, mechanism-oriented taxonomy to categorize jailbreak strategies. Unlike prior works focusing on risk types, this taxonomy groups techniques by the linguistic and strategic mechanisms used to bypass safeguards. It is organized into three levels and seven primary families:

Impersonation Attacks & Fictional Scenarios: Role-playing, benign context framing, and fictional framing (e.g., "DAN" prompts).
Privilege Escalation: Simulating admin/sudo modes or claiming the model is already jailbroken.
Persuasion: Leveraging logical, emotional, authoritative, or coercive language to induce compliance.
Cognitive Overload & Attention Misalignment: Using distractors, mathematical decomposition, or context saturation to divert the model's attention from safety constraints.
Encoding & Obfuscation: Distorting surface forms via token splitting, semantic rewriting, linguistic encoding (e.g., Base64, low-resource languages), or embedded prompting.
Goal-Conflicting Attacks: Assigning conflicting objectives (e.g., prefix injection, refusal suppression) to disrupt alignment.
Data Poisoning Attacks: Gradually introducing unaligned examples or false premises (e.g., incremental poisoning, many-shot jailbreaking) to corrupt model behavior.

3. Dataset Annotation

The collected dialogues were manually annotated using this taxonomy, capturing both isolated techniques and combinations thereof. This resulted in the first Italian dataset specifically annotated for multi-turn jailbreak detection.

4. Evaluation Experiments

The authors benchmarked GPT-5 as a judge for jailbreak detection in two settings:

Jailbreaking Attempt Detection: Determining if a user is attempting a jailbreak.
Jailbreaking Techniques Detection: Identifying specific techniques used in a dialogue.
Comparison: Performance was measured with and without taxonomy-enhanced prompting (providing the taxonomy definitions in the system prompt).

Key Results

Dataset Statistics

Total Dialogues: 1,364 (185 successful attacks).
Success Rates by Family:
- Data Poisoning Attacks: Highest success rate (17.2%).
- Encoding & Obfuscation: Lowest success rate (9.4%).
- Impersonation Attacks: Most prevalent (51.0% of total dialogues) with a 15.1% success rate.
- Automatic Attacks: When participants used pre-optimized triggers (orthogonal to the taxonomy), the success rate was highest (23.5%).
Technique Combinations: Most successful attacks combined multiple techniques. Role Play (specifically Virtual AI variants) was the most frequent component in combined attacks. Prefix Injection showed a high success rate (31.1%) when used in isolation.

Detection Performance (GPT-5)

Attempt Detection:
- Success rate increased from 65.9% (baseline) to 78.0% with taxonomy-enhanced prompting.
- The taxonomy helped correctly identify 14.3% of attempts that were previously missed, with the largest gains observed in hallucination-inducing attacks.
Technique Detection:
- Average Recall improved across all three hierarchical levels of the taxonomy when using taxonomy-enhanced prompting:
  - Level 1: 0.22 $\to$ 0.26
  - Level 2: 0.14 $\to$ 0.20
  - Level 3: 0.17 $\to$ 0.23
- The authors emphasize that improving recall is critical for safety detectors to prevent malicious requests from slipping through.

Significance and Contributions

The paper claims four primary contributions:

First Italian Multi-Turn Jailbreak Dataset: A publicly available resource of 1,364 annotated dialogues, addressing the scarcity of non-English, multi-turn adversarial data.
Mechanism-Oriented Taxonomy: A unified, hierarchical taxonomy that consolidates isolated techniques from prior literature into a coherent framework based on how attacks succeed, rather than just what risk they pose. It is empirically validated through the red-teaming challenge.
Empirical Analysis: Insights into the prevalence and success rates of specific jailbreak strategies, highlighting that multi-turn attacks often rely on combining techniques (e.g., Role Play + Context Ignoring) and that Data Poisoning is particularly effective.
Validation of Taxonomy-Driven Detection: Evidence that integrating a structured taxonomy into the prompting process significantly improves the recall and accuracy of LLM-based jailbreak detectors, suggesting a pathway for more robust guardrail systems.

The authors conclude that while the dataset is not large enough to train a dedicated detector (like Llama Guard) from scratch, the taxonomy provides a reproducible framework for studying adversarial prompting and demonstrates practical utility in enhancing detection capabilities via prompting. Future work aims to expand the challenge to cover automated attacks and longer dialogue trajectories.