💬 NLP

STACK: Adversarial Attacks on LLM Safeguard Pipelines

This paper introduces the STACK adversarial attack procedure, which successfully compromises a novel few-shot-prompted LLM safeguard pipeline with a 71% attack success rate, thereby highlighting critical vulnerabilities in current frontier AI defense systems and proposing specific mitigations.

Original authors: Ian R. McKenzie, Oskar J. Hollinsworth, Tom Tseng, Xander Davies, Stephen Casper, Aaron D. Tucker, Robert Kirk, Adam Gleave

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ian R. McKenzie, Oskar J. Hollinsworth, Tom Tseng, Xander Davies, Stephen Casper, Aaron D. Tucker, Robert Kirk, Adam Gleave

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a very powerful, helpful robot assistant. You want it to be smart, but you are terrified it might accidentally (or on purpose) give someone instructions on how to build a bomb or create a dangerous virus.

To stop this, the robot's creators don't just rely on one "stop" sign. Instead, they build a security pipeline, like a multi-layered fortress. This paper, titled STACK, investigates how well these fortresses actually hold up against clever hackers.

Here is the breakdown of the paper's story, using simple analogies:

1. The Fortress: "Defense in Depth"

Think of the AI safety system as a castle with three guards standing in a line:

Guard 1 (Input Classifier): Checks the letter you send before it reaches the robot. If the letter looks suspicious, they stop it immediately.
The Robot (The Model): If the letter passes Guard 1, the robot reads it and writes a reply.
Guard 2 (Output Classifier): Checks the robot's reply before it gets to you. If the reply contains dangerous instructions, this guard snatches the letter back.

The idea is that for a bad guy to succeed, they have to trick both guards and the robot. This is called "Defense in Depth" (or the "Swiss Cheese" model), where even if one layer has a hole, the others should catch the threat.

2. The Problem: The Guards Are Sleepy

The researchers built a test version of this fortress using open-source tools (like a public blueprint). They found that the "guards" (the safety classifiers) were often too easy to trick.

They tested standard tricks (like rephrasing a question or adding random typos) and found that many existing safety guards failed to stop them, especially when the request was truly dangerous (like asking for chemical weapon recipes).
However, they also found that a simple trick—giving the guard a few examples of what to look for (called "few-shot prompting")—made the guard much sharper. This simple guard was better than the fancy, pre-trained ones.

3. The New Attack: STACK (Staged AttaCK)

This is the main discovery of the paper. The researchers realized that while the fortress is strong, the guards are looking at things separately. They don't talk to each other during the attack.

They invented a new hacking method called STACK (Staged AttaCK). Instead of trying to break the whole castle at once, they broke it one layer at a time, like a thief picking locks in a specific order:

Step 1: Trick the Robot. First, they used a persuasion technique to convince the robot to answer the dangerous question, ignoring its own internal safety rules.
Step 2: Trick Guard 1. They found a "magic phrase" (a universal jailbreak) that, when added to the front of the message, made Guard 1 think the message was harmless.
Step 3: Trick Guard 2. This was the hardest part. They needed the robot to write a reply that included a "magic phrase" at the end that would trick Guard 2 into thinking the reply was safe.

The Analogy: Imagine you want to sneak a forbidden note into a school.

You first write the note in a way the teacher (the robot) will read and answer.
You then wrap the note in a wrapper that the security camera at the door (Guard 1) thinks is just a lunchbox.
Finally, you add a specific stamp to the back of the note that the security camera at the exit (Guard 2) thinks is a permission slip.

By combining these three specific tricks, the researchers successfully bypassed the entire fortress.

4. The Results: How Well Did It Work?

The Fortress Held Up (Sort Of): When they used standard attacks, the fortress worked great. The "few-shot" guard stopped 100% of the attacks on the most dangerous dataset.
STACK Broke the Fortress: When they used their new STACK method, they bypassed the fortress 71% of the time.
The "Black Box" Surprise: Even when the hackers couldn't see the guards' internal logic (a "black box" scenario) and had to guess which guard stopped them, they still succeeded 33% of the time. This is scary because it means you don't need to be an insider to break the system; you just need to be clever.

5. The Real-World Test

The researchers didn't just test this on their own toy model. They tried their method on the real, top-tier AI models from big companies (like Anthropic's Opus 4 and OpenAI's GPT-5).

They found that these real-world systems also leak information (like how long a request takes to process or specific error messages) that tells a hacker which guard stopped them.
Using this information, they successfully bypassed these high-security systems to get dangerous answers. The companies acknowledged the issues and patched them.

6. The Takeaway

The paper concludes with a few simple lessons for the builders of these AI fortresses:

Don't reveal your secrets: If your system tells a hacker "The Input Guard stopped you" vs. "The Output Guard stopped you," you are giving them a map. Make all refusals look the same.
Don't let hackers peek: If you release a "sibling" model (a slightly smaller version of your safety guard) to the public, hackers can train their attacks on that and then use them against your main system.
Defense in Depth is good, but not perfect: Having layers of security is better than having none, but if the layers don't communicate or if the guards are too predictable, a clever, staged attack can slip right through.

In short: The paper shows that while AI safety layers are a good idea, they are currently vulnerable to a specific type of "divide and conquer" attack. If you attack the layers one by one, you can defeat the whole system.

Technical Summary: STACK: Adversarial Attacks on LLM Safeguard Pipelines

Problem Statement

Frontier AI developers are increasingly relying on "defense-in-depth" pipelines—layered systems of input classifiers, generative models, and output classifiers—to prevent the catastrophic misuse of Large Language Models (LLMs). While companies like Anthropic and OpenAI have deployed such pipelines for models like Opus 4 and GPT-5, the security of these multi-stage defenses remains unclear. Prior work has largely evaluated these pipelines using naive, non-adaptive attacks designed for single-model systems, potentially overestimating their robustness. There is a critical gap in understanding whether staged attacks can systematically bypass sequential safeguards, particularly in realistic black-box threat models where attackers cannot directly access individual components but can infer which stage blocked a query.

Methodology

Defense Pipeline Construction

The authors constructed an open-source defense-in-depth pipeline to serve as a testbed. This pipeline wraps a target generative model ( $m$ ) with two classifiers:

Input Classifier ( $f_q$ ): Evaluates the user query.
Output Classifier ( $f_r$ ): Evaluates the model's response.
The pipeline rejects any query or response scoring above a specific threshold. The authors evaluated various open-weight safeguard models (e.g., Llama Guard, ShieldGemma) and introduced a novel few-shot-prompted classifier using the Gemma 2 base model, which outperformed specialized guard models like ShieldGemma.

The STACK Attack

The paper introduces STaged AttaCK (STACK), a methodology designed to defeat defense-in-depth pipelines by breaking them component-by-component. The attack proceeds in three stages:

Jailbreak the Generative Model: Use a non-adaptive attack (e.g., Persuasive Adversarial Prompts, or PAP) to transform a harmful query into a form the target model will answer.
Jailbreak the Input Classifier: Optimize a universal adversarial string (Input Classifier Jailbreak, ICJ) that, when prepended to the transformed query, bypasses the input filter.
Jailbreak the Output Classifier: Optimize a universal adversarial string (Output Classifier Jailbreak, OCJ) that the model is instructed to repeat in its output. This is achieved by constructing a prompt that forces the model to output the OCJ string before answering the user's query.

The final attack payload concatenates the ICJ, a repetition instruction, and the transformed query.

Threat Models

The study evaluates STACK under two primary threat models:

Black-Box Semi-Separable: The attacker has black-box access to the full pipeline but can infer which component (input classifier, model, or output classifier) rejected a query (e.g., via distinct refusal messages or timing). This reflects current API behaviors (e.g., Claude Opus 4).
Transfer (White-Box Proxy to Black-Box Target): The attacker has full white-box access to a proxy pipeline (different model/classifier configuration) to train the attack, then transfers the resulting jailbreaks to the target pipeline without direct access to its internal weights.

Key Contributions

Open-Source Defense Pipeline: The authors created a state-of-the-art open-source pipeline. They found that a simple few-shot-prompted classifier (Gemma2) significantly outperformed existing open-weight safeguard models (like ShieldGemma) against baseline attacks, reducing the Attack Success Rate (ASR) to 0% on the ClearHarm dataset for standard attacks.
The STACK Procedure: They developed a concrete implementation of the staged attack methodology.
- Black-Box STACK: Achieved a 71% ASR on the ClearHarm dataset against the robust Gemma2 pipeline, demonstrating that even strong defenses can be bypassed if components are attacked sequentially.
- Transfer-STACK: Achieved a 33% ASR in a zero-shot transfer setting, proving that attacks can be designed against a proxy and successfully transferred to a target pipeline without direct interaction.
Evaluation of Frontier Models: The authors applied the STACK methodology to Anthropic's Opus 4 and OpenAI's GPT-5. They identified vulnerabilities in both systems, validating that the component-wise attack approach is effective against frontier deployments.

Results

Baseline Attacks: Standard black-box attacks (PAP, ReNeLLM, Best-of-N) failed against the few-shot-prompted Gemma2 pipeline on the ClearHarm dataset (0% ASR), confirming the efficacy of the defense-in-depth principle against non-adaptive threats.
STACK Performance:
- Against the Gemma2 pipeline, STACK bypassed the safeguards, achieving a 71% ASR.
- The attack cost approximately 507 H100 GPU-hours for the black-box version and 15 H100 GPU-hours for the transfer version.
- The transfer attack demonstrated that attackers do not need direct access to the target pipeline's components to succeed, provided they can train on a sufficiently similar proxy.
Frontier Model Vulnerabilities: The authors successfully elicited harmful responses from Opus 4 and GPT-5 using variations of the component-wise attack (e.g., using ROT13 encoding to bypass Opus 4's output classifier and image-based inputs for GPT-5's input classifier). Following disclosure, both companies acknowledged the findings and deployed mitigations.

Significance and Claims

The paper claims that while defense-in-depth is a valid and necessary strategy, existing open-weight safeguard models are insufficient when faced with adaptive, staged attacks. The authors emphasize that:

Defense-in-depth works against naive attacks: The few-shot-prompted classifier successfully blocked all baseline attacks, validating the layered approach.
Staged attacks are a critical threat: The STACK procedure demonstrates that if an attacker can identify the blocking component (semi-separable access) or train on a proxy, they can bypass even the strongest current pipelines.
Mitigation is possible: The authors suggest specific countermeasures, such as:
- Hiding component identity: Ensuring refusal responses look identical regardless of whether the input classifier, model, or output classifier triggered the block.
- Eliminating side-channels: Running queries through all pipeline stages even if an early stage rejects them to prevent timing-based inference.
- Controlling access: Avoiding the release of open-weight models that are too similar to the proprietary safeguards used in production, thereby reducing the feasibility of transfer attacks.

The paper concludes that while defense-in-depth is essential, the security of these pipelines is currently fragile against sophisticated, staged adversarial strategies, necessitating more robust classifier designs and stricter access controls.