🤖 AI

Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Success

This position paper argues that current jailbreak evaluations relying on single-configuration success rates are insufficient for characterizing threats and proposes adopting distributional metrics like Variant Sensitivity Measure (VSM) and Union Coverage (UC) to better capture the full scope of attack performance across parameter variations.

Original authors: Carsten Maple, Abhishek Kumar, Riya Tapwal

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Carsten Maple, Abhishek Kumar, Riya Tapwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard trying to figure out how easy it is to break into a high-tech building.

In the world of AI safety, researchers often test "jailbreak" attacks—methods designed to trick AI models into saying things they shouldn't. The current standard for reporting these tests is like a thief saying: "I found one specific key that opens the front door 80% of the time. Therefore, this building is 80% vulnerable."

This paper argues that this way of reporting is misleading and incomplete. It's like saying a building is safe because you only tried one key on one door, ignoring the fact that there are 36 other keys that might open different doors, or that the "best" key only works on a very specific type of lock.

Here is the breakdown of the paper's argument using simple analogies:

The Problem: The "Best Key" Fallacy

Most AI safety papers report a single number: the Attack Success Rate (ASR). They pick the single best combination of settings (like the best "key") and say, "Look, this attack works 80% of the time!"

The authors say this is like a thief showing you one key that opens the front door and claiming, "This is how secure the building is." But what if:

That key only works on the front door, but there are 10 other keys that open the back, side, and basement doors?
The "80% success" was a lucky fluke that only happens with one very specific setting, while the other 99 settings barely work at all?

If defenders (the security guards) only look at that one "best" number, they might patch the front door and think they are safe, while the back and side doors remain wide open.

The Solution: Two New Measuring Cups

The authors propose two new ways to measure the danger, which they call VSM and UC.

1. VSM (Variant Sensitivity Measure): "How Lucky Was That Key?"

Imagine you have a bag of 100 different keys.

Scenario A: 95 of them open the door easily. One is a dud. If you report the "best" key, you are telling the truth about the average experience.
Scenario B: Only 1 key works perfectly, and the other 99 are useless. If you report the "best" key, you are lying about the average experience.

VSM measures the gap between the "best" result and the "average" result.

Low VSM: The attack is consistent. The headline number is reliable.
High VSM: The headline number is a fluke. The attack only works if you get incredibly lucky with the settings. The paper found that for some attacks, the "best" result was five times better than the average result, meaning the headline number was wildly optimistic.

2. UC (Union Coverage): "How Many Doors Can Be Opened?"

This is the most important part for security.
Imagine you have 36 different keys.

Key A opens 60% of the doors.
Key B opens a different 40% of the doors.
Key C opens the remaining 20%.

If you only look at the "best" key (Key A), you think 60% of the building is vulnerable. But if you try all the keys together, you realize 100% of the building is vulnerable.

UC measures the total percentage of doors that can be opened if an attacker tries every variation of the attack. The paper found that for some attacks, the "Union Coverage" was 33% higher than the "Best Single Configuration." This means defenders who only look at the best number are missing a huge chunk of the danger.

Real-World Examples from the Paper

The authors tested these ideas on two famous "attack families" (PAIR and Bijection Learning) using three different AI models.

The PAIR Attack: They tried different "personas" (like pretending to be an authority figure vs. a logical thinker).
- The Headline: "Authority Endorsement" worked 69% of the time.
- The Reality: When they combined all three personas, they could break into 88% of the cases. The single number missed 19% of the vulnerable prompts.
The Bijection Attack: This one uses different "encoding" tricks (like changing the language or spacing of the text).
- The Headline: The best single setting worked 81% of the time.
- The Reality: When they tried all 36 possible combinations, 100% of the prompts were broken. The "best" number completely missed the fact that every single test case was vulnerable if you tried enough variations.

Why This Matters

The paper isn't saying the current numbers are "wrong" mathematically; they are just incomplete.

For Defenders: If you only patch the "best" attack, you leave the other 30% of the building unlocked. You need to know the "Union Coverage" to know how much of the building is actually at risk.
For Researchers: If you compare two attacks, one with a "Best" score of 80% (but very inconsistent) and another with a "Best" score of 60% (but very consistent), you can't just say the first one is "better." You need to know if the first one is just a lucky fluke.

The Bottom Line

The authors are asking the AI research community to stop just shouting the "Best Case" number. Instead, they should report:

The Best Case (The headline).
The Average Case (How typical is that success?).
The Total Coverage (How many different prompts can be broken if you try all variations?).

Until researchers start reporting this full picture, we are like security guards who only check the front door and assume the rest of the building is safe.

Technical Summary: Single-Configuration Attack Success Rate Is Not Enough

Problem Statement

Current literature on Large Language Model (LLM) jailbreak attacks predominantly relies on reporting a single Attack Success Rate (ASR) derived from a limited, often arbitrarily selected, configuration of parameters. This practice assumes that a single "best-case" configuration adequately represents the entire attack family. However, modern jailbreak attacks are parameterized families involving multiple internal variables, such as system prompt templates, conversation rounds, cipher dispersion, teaching shots, and optimization seeds.

The paper argues that reporting only the best single configuration is fundamentally insufficient for characterizing the threat landscape because:

Incomplete Threat Characterization: Different variants of the same attack family often succeed on different subsets of prompts. A single configuration may miss a significant portion of the total attack surface.
Misleading Comparisons: Comparing attacks based solely on headline ASR numbers is flawed if the underlying variant spaces differ in sensitivity. An attack with a high headline ASR but high variance (fragile to parameter changes) is not directly comparable to one with a lower headline ASR but high consistency.
Defender Blind Spots: Defenders focusing only on the reported best configuration may fail to patch vulnerabilities exposed by other viable variants of the same attack family.

Methodology and Proposed Metrics

To address these gaps, the authors propose a shift from single-point reporting to distributional reporting. They introduce two new metrics to complement the standard Best ASR:

1. Variant Sensitivity Measure (VSM)

VSM quantifies how representative the best-reported ASR is of the mean performance across the tested variant space. It is defined as:
$VSM = \frac{\text{Best ASR} - \text{Mean ASR}}{\text{Best ASR}}$

Range: 0 to nearly 1.
Interpretation: A VSM near 0 indicates all variants perform similarly (the headline number is representative). A VSM near 1 indicates the headline number reflects an extreme outlier, and typical performance is significantly lower.

2. Union Coverage (UC)

UC measures the total fraction of prompts that elicit unsafe responses when taking the union of all tested configurations. It is defined as:
$UC = \frac{|\bigcup_{v \in V} S(v)|}{|P|}$
Where $V$ is the variant space, $S(v)$ is the set of prompts successfully jailbroken by variant $v$ , and $P$ is the full set of evaluated prompts.

Interpretation: UC captures the total attack surface accessible to an adversary willing to try multiple variants. The gap between Best ASR and UC indicates how much of the threat is missed by selecting a single configuration.

Reporting Convention

The authors propose that future papers should report:

ASR per variant across a clearly enumerated variant space.
Best ASR and Mean ASR.
The calculated VSM.
The Union Coverage (UC) across the full variant space.

Empirical Results

The authors validate their framework using two attack families (PAIR and Bijection Learning) across three open-source target models (Mistral-7B, Qwen3-0.6B, and Llama-3.1-8B) using a stratified subset of 100 prompts from the HarmBench dataset.

Case Study 1: PAIR (Prompt Automatic Iterative Refinement)

Setup: Tested 3 system prompt templates crossed with 3 conversation depths (9 configurations total) per model.
Findings:
- Mistral-7B: The best single configuration (Authority Endorsement, $K=3$ ) achieved 69% ASR. However, the Union Coverage across all 9 configurations was 88%. The VSM was 0.425, indicating the headline number significantly overstates typical performance.
- Qwen3-0.6B: Best ASR was 75%, while Union Coverage reached 93%. VSM was 0.312.
- Conclusion: Templates are complementary; different templates jailbreak different prompts. Relying on the best template misses 19–24 percentage points of the attack surface.

Case Study 2: Bijection Learning

Setup: Tested 6 dispersion levels crossed with 6 teaching-shot counts (36 configurations total) per model.
Findings:
- Extreme Sensitivity: VSM values were consistently high (0.80–0.86) across all models, indicating the best configuration is a severe outlier compared to the mean.
- Mistral-7B: Best ASR was 81%, but Union Coverage reached 100%. Every evaluated prompt was jailbroken by at least one of the 36 configurations, despite the best single variant covering only 81%.
- Llama-3.1-8B: Best ASR was 17%, but Union Coverage was 43%.
- Conclusion: Single-configuration reporting simultaneously understates the total vulnerability (UC) and overstates the consistency of the attack (high VSM).

Key Contributions

Identification of Reporting Failure: The paper identifies that current evaluation practices fail to reveal whether reported ASR values are representative, fragile, or incomplete regarding prompt coverage.
Proposal of Distributional Standards: It argues for a new standard requiring the reporting of variant spaces, per-variant ASR, VSM, and UC.
Empirical Evidence: Through PAIR and Bijection case studies, it demonstrates that variant choice drastically alters both measured ASR and prompt-level coverage, and that different variants expose disjoint sets of vulnerabilities.
Defender and Researcher Utility: It explains how these metrics distinguish between peak single-configuration performance and mean performance, aiding defenders in prioritizing patches and researchers in making fair cross-paper comparisons.

Significance and Claims

The paper claims that single-configuration ASR reporting is not "wrong" but incomplete. It does not argue that existing numbers are incorrect, but that they lack the necessary context (distributional and prompt-overlap information) to be interpreted correctly.

The significance of this work lies in its potential to:

Improve Defense: By highlighting the "Union–Best gap," defenders can prioritize patches for prompts vulnerable to any variant, rather than just the best-known variant.
Standardize Evaluation: By introducing VSM and UC, the community can move away from headline-driven comparisons toward a more robust characterization of jailbreak threats.
Reveal Hidden Threats: The metrics expose that an attack's total reachable attack surface can be substantially larger (18–33 percentage points higher in the experiments) than what is suggested by its best single configuration.

The authors conclude that distributional reporting, including VSM and UC, should become the new minimum standard for evaluating parameterized jailbreak attacks, as it requires only fuller reporting of data already generated during standard experimentation.