💻 computer science

The Great Pretender: A Stochasticity Problem in LLM Jailbreak

This paper reveals that the instability of Attack Success Rate (ASR) in LLM jailbreaks stems from stochasticity during both evaluation and generation, leading to systematically inflated metrics, and proposes the CAS-eval and CAS-gen frameworks to accurately measure this variance and recover performance losses.

Original authors: Jean-Philippe Monteuuis, Cong Chen, Jonathan Petit

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jean-Philippe Monteuuis, Cong Chen, Jonathan Petit

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a very strict club (the AI model). Your job is to stop anyone from bringing in dangerous items (harmful prompts). Now, imagine a group of clever tricksters (the researchers) trying to find a way to sneak in.

For a long time, the security industry has been measuring how good these tricksters are using a single score: The Attack Success Rate (ASR). If a trickster manages to get past the guard even once out of ten tries, the old system says, "Great! They are 100% successful!"

The paper "The Great Pretender" argues that this way of measuring is a massive lie. It's like saying a magician is a master of illusion just because they pulled a rabbit out of a hat once, even if they failed the next nine times. The paper claims that the current success rates are "inflated" because they ignore randomness (stochasticity).

Here is the breakdown of the paper's findings using simple analogies:

1. The Two Sources of "Luck"

The paper says the current success scores are unreliable because of two types of randomness that everyone has been ignoring:

Generation Luck (The Dice Roll): When a trickster tries to create a jailbreak, they often generate many different versions of the same trick. It's like rolling a die 10 times to see if you get a "6." If you get a "6" on the first try, you stop and say, "I won!" But the next time you roll, you might not get a "6" again. The old papers often just report the result of that one lucky roll.
Evaluation Luck (The Flaky Judge): After the trickster tries their trick, a "Judge" (another AI) decides if it worked. But this Judge is also a bit drunk or tired; sometimes it says "Safe," and sometimes it says "Unsafe" for the exact same trick, just because of a random mood swing. If the Judge happens to be in a "lenient" mood during the test, the trickster looks like a genius. If the Judge is strict, the trickster looks like a failure.

2. The "Great Pretender" Problem

The authors found that many famous jailbreak methods (like "Best-of-N" or "Crescendo") are pretending to be stronger than they actually are.

The Scenario: A paper claims a method has an 80% success rate against a top-tier AI.
The Reality: When the authors tested it properly, that same method only worked 50% of the time when you asked it to succeed consistently.
The Analogy: It's like a basketball player who makes a shot 8 out of 10 times if you only count the shots they took when the wind was blowing perfectly in their favor. But if you ask them to make the shot 10 times in a row under normal conditions, they might only make 5. The old papers were counting the "windy day" shots as proof of mastery.

3. The Solution: The "Consistency" Test

To fix this, the authors propose a new way to measure success called CAS (Consistency for Attack Success).

Instead of asking, "Did this trick work once?", they ask, "Did this trick work every single time we tried it?"

They introduce two new tools:

CAS-gen (The Strict Generator): Before a trick is even added to the "Hall of Fame," it must prove it works consistently. It has to sneak past the guard 5 or 10 times in a row. If it fails even once, it's thrown out. This filters out the "lucky" tricks.
CAS-eval (The Strict Judge): When testing a trick, the Judge doesn't just look at it once. The Judge looks at the same trick 10 times. If the Judge says "Unsafe" 9 times but "Safe" 1 time (due to randomness), the trick is not counted as a success. It must pass the Judge's scrutiny every single time.

4. The Shocking Results

When the authors applied these strict rules, the numbers dropped dramatically:

The Drop: Some attacks that looked like they had an 80% success rate dropped to 50% or lower when tested for consistency. That is a 30-point drop just by removing the "luck."
The Temperature Effect: They found that if you turn up the "temperature" (randomness) of the Judge, the success rates look artificially high. It's like if the security guard was told to flip a coin to decide who gets in; you'd get a lot of "successes" by pure chance.
The Fix: By using their new "Consistency" framework, they could actually improve the attacks. By forcing the attackers to be consistent during the creation phase, they found tricks that were genuinely robust, recovering that lost 30% performance.

The Bottom Line

The paper concludes that the current way of ranking AI jailbreaks is broken. It's like grading a student based on a single lucky guess rather than their actual understanding.

The authors are calling for a new standard: Don't just tell us how many times you got lucky; tell us how many times you succeeded consistently. Until we do that, the "Attack Success Rates" we read in research papers are just "The Great Pretender"—looking strong on the surface but failing the real test.

Technical Summary: The Great Pretender: A Stochasticity Problem in LLM Jailbreak

1. Problem Statement

The current standard for evaluating Large Language Model (LLM) safety, the Attack Success Rate (ASR), is fundamentally flawed because it treats ASR as a fixed property of an attack. The authors argue that ASR is actually a random variable heavily influenced by two independent sources of stochasticity that are largely ignored in existing literature:

Generation Stochasticity: Most jailbreak attacks (e.g., Best-of-N, Crescendo) are stochastic by design. They generate multiple candidate prompts and select one based on a single successful trial. A single run produces a different set of candidates depending on the random seed, leading to variable ASR outcomes.
Evaluation Stochasticity: Automated judges (e.g., Llama-Guard) used to classify responses as "harmful" or "safe" are themselves stochastic when operating at temperature $T > 0$ . The same model response may be labeled harmful in one evaluation and benign in another.

Consequently, reported ASR numbers are systematically inflated and incomparable across papers. A prompt that succeeds 1 out of 10 attempts is often treated identically to one that succeeds 10 out of 10, masking the fact that the former may be a "lucky draw" rather than a genuine vulnerability.

2. Methodology

The paper proposes a systematic empirical study to quantify the impact of these stochastic factors.

Experimental Setup

Attacks: Four black-box attacks were evaluated: Best-of-N (BoN), PAIR, TAP, and Crescendo.
Target Models: Five instruction-tuned models ranging from 1B to 70B parameters across three families (Meta Llama-3, Google Gemma-3, IBM Granite).
Judges: Two LLM-based judges (Llama-Guard-3-1B and Llama-Guard-3-8B).
Parameters: The study varies six stochasticity parameters across the generation and evaluation pipelines:
- Temperatures: Target model generation ( $T_{gen}$ ), target model evaluation ( $T_{eval}$ ), judge generation ( $\theta_{gen}$ ), and judge evaluation ( $\theta_{eval}$ ).
- Consistency Thresholds: The number of consecutive successful evaluations required ( $k_{gen}$ for generation, $k_{eval}$ for evaluation).
- Seeds: Multiple independent random seeds ( $S=5$ ) to measure variance.

Proposed Frameworks

To address the instability of ASR, the authors introduce Consistency for Attack Success (CAS) and two associated frameworks:

CAS Metric: A binary indicator per prompt. A jailbreak is counted as successful only if all $k$ independent evaluations return a harmful verdict.
- Standard ASR corresponds to $k=1$ .
- The dataset-level metric is $ASR(k_{eval})$ , the fraction of prompts where $CAS(r, k_{eval}) = 1$ .
CAS-gen (Attack Generation Framework):
- Instead of accepting a jailbreak prompt after a single successful trial ( $k_{gen}=1$ ), the attack must elicit a harmful response consistently across $k_{gen}$ independent runs.
- This filters out "chance successes" at the source, ensuring the attack dataset contains only robust prompts.
CAS-eval (Attack Evaluation Framework):
- A fixed jailbreak prompt is evaluated $k_{eval}$ times. It is classified as a consistent jailbreak only if all $k_{eval}$ judge calls return a harmful verdict.
- This suppresses false positives caused by judge non-determinism.

3. Key Results

The evaluation reveals that stochasticity parameters can shift reported ASR by massive margins (up to 54 percentage points), rendering cross-paper comparisons invalid without standardization.

Impact of Evaluation Threshold ( $k_{eval}$ ): Moving from $k_{eval}=1$ to $k_{eval}=10$ reduces ASR by 12 to 24 percentage points across all configurations. This indicates that standard single-shot evaluation significantly overstates attack success.
Impact of Judge Temperature ( $\theta_{eval}$ ): This is the most volatile parameter. Raising $\theta_{eval}$ from 0.0 to 1.0 can collapse ASR by up to 54 percentage points (e.g., Crescendo dropping from 86% to 32%). At $\theta_{eval}=0$ , ASR becomes perfectly stable.
Impact of Generation Budget ( $k_{gen}$ ): Increasing $k_{gen}$ from 1 to 10 lifts reported ASR by 12 to 30 percentage points. A loose filter ( $k_{gen}=1$ ) admits borderline prompts that degrade sharply under repeated evaluation.
Impact of Target Model Temperature ( $T_{eval}$ ): Shifting $T_{eval}$ from 0.0 to 1.0 changes ASR by at most 18 percentage points with no systematic direction; it does not affect relative attack rankings.
Impact of Attack Seed: Seed-to-seed spread is modest (up to 20 percentage points) but can be higher for specific configurations (e.g., 50 points for BoN with LG-8B at $k_{eval}=10$ ).

4. Contributions and Significance

The paper makes three primary contributions:

Systematic Empirical Study: The first comprehensive analysis of both generation and evaluation stochasticity across multiple attacks, models, and judges, identifying specific parameters that drive ASR variance.
New Metric and Frameworks: The introduction of CAS (Consistency for Attack Success) and the CAS-gen/CAS-eval frameworks. These tools allow researchers to filter out lucky draws and judge noise, producing more reliable and reproducible ASR estimates.
Minimum Reporting Standard: The authors propose a checklist for future research to ensure comparability. At a minimum, papers must disclose:
- $k_{gen}$ (generation consistency threshold)
- $k_{eval}$ (evaluation consistency threshold)
- $\theta_{eval}$ (judge evaluation temperature)
- (For BoN attacks) $\theta_{gen}$ (judge search temperature)

Significance:
The paper argues that the current practice of reporting ASR as a point estimate without controlling for stochasticity is "fundamentally flawed." By ignoring these variables, the community creates a "Great Pretender" scenario where high ASR scores are often artifacts of random seeds and judge noise rather than genuine vulnerabilities. The proposed frameworks and reporting standards aim to restore statistical rigor to LLM safety benchmarking, ensuring that reported success rates reflect genuine, reproducible attack effectiveness. The authors note that their findings are based on open-weight models and that extending this analysis to proprietary, heavily RLHF-aligned models remains an open empirical question due to API restrictions and lack of transparency.