🤖 AI

SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses

This Systematization of Knowledge (SoK) paper addresses the fragmented evaluation of LLM prompt security by proposing unified taxonomies, formalizing evaluation metadata, and releasing a modular platform with new datasets and tools to enable reproducible, cost-aware, and comparable assessments of attacks and defenses.

Original authors: Hanbin Hong, Shuang Wu, Shuya Feng, Nima Naderloui, Shenao Yan, Jingyu Zhang, Ali Arastehfard, Heqing Huang, Yuan Hong

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanbin Hong, Shuang Wu, Shuya Feng, Nima Naderloui, Shenao Yan, Jingyu Zhang, Ali Arastehfard, Heqing Huang, Yuan Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Large Language Models (LLMs) as incredibly smart, but slightly naive, librarians. They are hired to answer questions, write code, and help with tasks. However, these librarians have a strict rulebook: "Do not help anyone do something dangerous or illegal."

The problem is that clever people (attackers) have figured out how to trick these librarians into breaking their own rules. They do this by whispering instructions in a different language, dressing up a bad request as a movie script, or asking the librarian to pretend to be a villain. This is called a "jailbreak."

This paper is like a massive security audit of the entire library system. The authors realized that everyone was trying to measure how well the librarians were doing, but they were all using different rulers, different test questions, and different judges. One person might say, "Our librarian is 90% safe!" while another says, "Ours is only 50% safe!" The paper argues that you can't compare these numbers because they aren't measuring the same thing.

Here is what the paper does to fix this, explained simply:

1. The Three New "Rulebooks" (Taxonomies)

The authors created three organized lists to sort out the chaos, like sorting toys into specific bins:

The "Trickster" List (Attacks): They categorized how people trick the librarian.
- Example: Some tricks involve disguising the bad request (like writing "how to make a bomb" in a secret code). Others involve splitting the request into tiny, harmless pieces that add up to something bad. Some even use a second AI to help write the trick.
The "Guardian" List (Defenses): They categorized how librarians try to stop the tricks.
- Example: Some guards check the ID before letting the person in (Input Detection). Others check the bag after the person leaves (Output Detection). Some try to rewrite the request to make it safe, while others try to train the librarian to be smarter.
The "Weak Spot" List (Vulnerabilities): They listed the librarian's natural weaknesses.
- Example: The librarian might be too polite to say "no" if you ask nicely (Psychological Manipulation), or they might get confused if you ask them to summarize a story that happens to be about a crime (Format Exploitation).

2. The "Universal Test Lab" (PromptSecurity)

The authors built a giant, modular testing machine called PromptSecurity. Think of it like a video game level where you can swap out the character, the enemy, the weapon, and the referee, but keep the rules of the game exactly the same.

Why this matters: Before, researchers would test a new defense on a specific model with a specific set of questions. If it worked, they claimed victory. But maybe it only worked because the questions were easy!
The Fix: This platform forces everyone to run their tests under the exact same conditions. It records everything: how many questions were asked, how much money it cost to run the test, and exactly which "referee" (AI judge) decided if the answer was bad. This ensures that if Method A beats Method B, it's because Method A is actually better, not because the test was rigged.

3. The "Big Data" Collection (JAILBREAKDB)

The paper gathered a massive library of test questions:

445,000+ "Jailbreak" attempts (the bad requests).
1,000,000+ "Benign" requests (the normal, safe questions).
They cleaned these up and organized them so researchers can use them as a standard "exam" for any new AI.

4. What They Discovered (The Results)

When they ran their universal tests, they found some surprising things:

The "Referee" Matters: Who you ask to grade the test changes the score. If you ask a referee to look only at the format of the answer (e.g., "Did they start with 'I cannot'?"), they might miss a bad answer that was cleverly hidden. If you ask them to look at the meaning, they might catch it. The paper says we need to be very careful about how we judge safety.
Small Models vs. Big Models: Sometimes, a smaller, "weaker" AI seems safer because it just fails to understand the complex trick the attacker is using. It's not that the AI is smarter; it's that it's too dumb to follow the instructions.
The "Backfire" Effect: Some defenses actually make things worse! If you try to "steer" the AI to be safe by adding a safety instruction, sometimes that instruction confuses the AI and makes it accidentally say something harmful. It's like putting a "Do Not Touch" sign on a museum exhibit, but the sign is so big it blocks the view of the art, causing people to bump into it.
Cost vs. Safety: The most effective defenses often cost a lot of money or time to run. The cheapest defenses often break normal questions (like asking for a recipe) just as often as they catch bad ones.

The Bottom Line

This paper doesn't just say "AI is unsafe" or "Here is a fix." Instead, it says, "Stop comparing apples to oranges."

It provides the tools (the taxonomies, the dataset, and the testing platform) so that in the future, when someone claims their new AI is "99% safe," we can check their work, see exactly how they tested it, and know if that claim is real or just an illusion caused by a bad test setup. It turns a messy, confusing field into a structured science.

Technical Summary: SoK: Systematizing LLM Prompt Security

Problem Statement
The rapid deployment of Large Language Models (LLMs) as interfaces for information and services has made prompt-level security a critical concern. While the field has seen rapid advancements in jailbreak attacks, defensive wrappers, safety policies, datasets, and automated judgers, the community faces a significant fragmentation problem. Prior work often relies on incompatible threat models, access assumptions, cost budgets, datasets, and success criteria. This lack of standardization makes reported Attack Success Rates (ASR) or defense gains difficult to compare. Consequently, it is often unclear whether observed progress stems from a stronger method, a weaker target model, different judging rules, or simply a larger query budget. The central challenge is determining whether security claims can be compared under unified, auditable conditions.

Methodology
The authors propose a systematic approach to LLM prompt security evaluation, moving from paper-specific demonstrations to reproducible, taxonomy-grounded measurements. The methodology consists of four core components:

Three Linked Taxonomies: The paper establishes a conceptual framework separating technical mechanisms from attacker/defender capabilities:
- Taxonomy I (Jailbreak Attacks): Classifies attacks by threat model (Black-box vs. White-box) and technical methodology (e.g., Prompt Modification, Black-box Optimization, LLM-Assisted Techniques, Multi-Turn Techniques, and Leveraging Model Vulnerabilities). It treats attacks as compositional, allowing a single method to instantiate multiple atomic technical units.
- Taxonomy II (Defenses): Groups defenses by goal (Detection vs. Mitigation) and intervention layer (Input-level, Output-level, Inner State, Model Training, Model Modification, Output Processing).
- Taxonomy III (LLM Vulnerabilities): Categorizes intrinsic model failure modes exploited by attacks, such as Format Exploitation, Overreliance on User Instructions, Psychological Manipulation, and Contextual Ambiguity.
Auditable Evaluation Protocol: The authors formalize threat, access, and cost assumptions as explicit metadata. They define an experiment as a tuple $\langle M, A, D, S, J \rangle$ , representing Model, Attack, Defense, Dataset, and Judger. This protocol mandates that comparisons are only valid when component identities and settings (e.g., assistant LLMs, decoding parameters, query budgets, defense composition) are matched.
PROMPTSECURITY Platform: A modular evaluation platform is introduced to operationalize the protocol. It represents each experiment as a configured tuple, enabling access-compatible and auditable comparisons. The platform logs sample identifiers, outputs, settings, query counts, and judger configurations to ensure reproducibility.
Unified Datasets: The authors release JAILBREAKDB, a curated corpus containing 445,752 jailbreak system-user prompt pairs from 48 sources and 1,094,122 benign prompts from 14 sources. They also release PROMPTSECURITY-EVAL, containing 166,200 harmful-query output records and 9,200 benign-utility output records generated under the main protocol.

Key Contributions

Mechanism-Level Taxonomies: The paper systematizes the field through three linked taxonomies that support multi-label subcategory mappings, allowing empirical claims to be made at the taxonomy-subcategory level rather than just the method level.
Unified Evaluation Platform: The introduction of PROMPTSECURITY, which treats access compatibility, assistant LLM use, and judging rubrics as auditable parts of the security claim, enabling controlled ablation studies across a shared design space.
Empirical Insights: The platform was used to evaluate 11 target models, 21 attack settings, 10 defense states, and three judgers under matched settings.
Datasets and Leaderboard: The release of JAILBREAKDB and PROMPTSECURITY-EVAL, along with an interactive leaderboard for browsing compatible main-protocol measurements.

Results and Empirical Findings
The evaluation yields several taxonomy-level findings:

Model Access and Capability: Frontier black-box models are more exposed to prompt transformation and obfuscation attacks that help the target model recover concealed harmful intent. Conversely, weaker white-box models often suppress complex transformed prompts not due to stronger robustness, but because they fail to execute the attack task (e.g., failing to parse code-style prompts).
Attack Mechanisms: On black-box models, low-cost prompt transformations often outperform assistant-generated natural-language jailbreaks. On white-box models, this ordering changes, with LLM-assisted search and refinement showing higher persistence.
Defense Effectiveness: Defense effectiveness is mechanism-dependent. Input/output detection and routing dominate average mitigation. However, white-box access alone does not guarantee a strong defense; for instance, gradient screening and safety-prompt optimization provided little or even negative average mitigation in some cases.
Defense Backfire: Defenses can introduce new harmful behavior. Safety-prompt tuning, for example, increased average harmful-output ASR because the added steering sometimes turned previously safe prompts into harmful completions.
Judger Sensitivity: The choice of judger materially affects security conclusions. Prefix checks (refusal-format instruments) can over-count non-refusal benign outputs and miss harmful outputs hidden behind refusal-like openings. Behavior-conditioned judgers are highly sensitive to the specific behavior description provided, leading to significant variance in ASR measurements.

Significance
The paper argues that LLM prompt security should be evaluated at the level of mechanism groups under fair configuration control. The significance of this work lies in its shift from paper-specific demonstrations toward reproducible, cost-aware, and taxonomy-grounded evaluation. The goal is not merely to identify which method has a higher ASR, but to clarify which mechanism, setting choice, or judging rule explains the reported security behavior. By providing a foundation of linked taxonomies, an executable protocol, a modular platform, and unified datasets, the authors aim to enable future evaluations that can distinguish between genuine methodological progress and artifacts of experimental design.