💬 NLP

An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models

This paper empirically demonstrates that moderate multi-generation sampling significantly improves jailbreak detection in large language models by revealing hidden vulnerabilities that single-output evaluations miss, while also showing that detection signals partially generalize across models and that lexical detectors capture both behavioral and topic-specific cues.

Original authors: Hanrui Luo, Shreyank N Gowda

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hanrui Luo, Shreyank N Gowda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-behaved robot assistant. You've trained it to be polite, helpful, and never say anything mean or dangerous. This is what researchers call an "aligned" Large Language Model (LLM).

But, just like a human, this robot can sometimes be tricked. If you ask it a tricky question in a sneaky way (a "jailbreak"), it might accidentally break its rules and say something harmful.

This paper is about how we check if our robot is actually safe. The authors, Hanrui Luo and Shreyank N Gowda, discovered that the way we usually test these robots is flawed, and they propose a better way.

Here is the breakdown of their findings using simple analogies:

1. The "One-and-Done" Test is a Lie

The Old Way: Imagine you want to know if a coin is fair. You flip it once, it lands on Heads, and you say, "Okay, this coin is definitely Heads." That's silly, right? You need to flip it many times to see the truth.

The Problem: Most safety tests for AI only ask the robot a question one time. If the robot says "No, I can't do that," the testers mark it as "Safe."
The Reality: Because AI is a bit random (like a coin flip), if you ask the same tricky question 10 times, the robot might say "No" nine times, but say "Yes, here is how to build a bomb" on the tenth time.
The Finding: By only asking once, we are underestimating the danger. We think the robot is safer than it actually is. The paper shows that if you ask the same question a few times (maybe 3 times), you catch a lot more of the "bad" answers that were hiding before.

2. The "Goldilocks" Zone of Sampling

The researchers asked: "How many times should we ask the question to be safe?"

Asking once: Misses too many dangers.
Asking 100 times: You catch almost everything, but it takes forever and costs too much money (computing power).
The Sweet Spot: Asking about 3 times is the "Goldilocks" zone. It catches almost all the dangerous behavior without wasting resources. It's like checking a lock three times instead of just once; it's quick but much more reliable.

3. The "Detective" vs. The "Behavior Analyst"

The paper tested two ways to catch the robot lying:

The Lexical Detective (TF-IDF): This detective looks for specific "bad words" or specific sentence structures. It's like a bouncer at a club looking for people wearing red hats.
- The Flaw: The bad guys can just wear blue hats. The paper found that this detective often gets confused by the style of the answer. If the robot gives a long, step-by-step list (even if it's refusing to help), the detective thinks, "Aha! It's giving instructions! It must be a jailbreak!" and gets it wrong.
The Behavior Analyst (Inconsistency): This analyst looks at the pattern of answers. If the robot says "No" to question #1 but "Yes" to the same question #2, the analyst gets suspicious.
- The Flaw: This is good at catching the random "slips," but sometimes it gets too nervous and flags safe answers as dangerous.

4. The "Stronger Robot" Paradox

Here is a tricky part: The more you train the robot to be safe, the harder it is to test.

Weak Robot: It breaks its rules often. It's easy to catch because it screams "I'm bad!" loudly.
Strong Robot: It rarely breaks its rules. When it does break them, it does so very quietly and subtly.
The Paradox: Because the "bad" answers are so rare in a strong robot, it's like looking for a single needle in a massive haystack. The "Detective" gets confused because there are so few bad needles to find, and the "Behavior Analyst" gets overwhelmed by all the safe hay.

5. The Big Takeaway

The main message of the paper is: Don't trust a single answer.

If you want to know if an AI is safe, you can't just ask it once and take its word for it. You need to ask it the same question a few times (a "multi-generation audit") to see if it slips up.

Analogy: Imagine you are hiring a driver. You wouldn't just watch them drive for 10 seconds and say, "They are a safe driver." You'd want to see them drive in different conditions, maybe a few times, to be sure they don't have a hidden habit of speeding.

In short: To keep our AI safe, we need to stop testing it like a static quiz and start testing it like a dynamic, unpredictable conversation. Asking a few extra questions is a small price to pay to avoid a big mistake.

1. Problem Statement

The paper addresses a critical gap in the evaluation of Large Language Model (LLM) safety: the underestimation of jailbreak vulnerability due to single-output evaluation protocols.

The Stochastic Nature of LLMs: LLMs are probabilistic; a single prompt can yield different outputs (safe or harmful) across multiple generations.
The Limitation of Current Practices: Most existing studies and safety audits evaluate models based on a single generated response per prompt. This approach assumes the single output is representative of the model's behavior.
The Consequence: In safety-critical scenarios, a model might refuse a harmful request in one instance but comply in another. Single-output evaluation fails to capture these rare but critical failure cases, leading to a false sense of security.
The Detection Challenge: As models become better aligned (safer), harmful outputs become rarer and less stereotypical, turning detection into a "rare-event" problem where signals are sparse and unstable.

2. Methodology

The authors conducted an empirical study to evaluate how multi-generation sampling affects jailbreak detection and vulnerability estimation.

Dataset: Used the JailbreakBench Behaviors dataset, which contains structured adversarial prompts with behavioral goals and category labels. Harmful and benign samples were combined via controlled sampling to manage class imbalance.
Generator Models: Evaluated multiple models with varying alignment strengths (e.g., DistilGPT2, Qwen2.5-0.5B/1.5B-Instruct, Gemma) to test generalization across different safety levels.
Detection Methods:
1. Lexical Baseline (TF-IDF + Logistic Regression): A lightweight detector relying on surface-level lexical patterns.
2. Inconsistency-Based Detector (NegBLEURT-inspired): A method using Isolation Forest to detect semantic inconsistencies across multiple generated outputs.
Evaluation Protocol:
- Multi-Generation Sampling: For each prompt, $k$ generations were produced (varying $k$ from 1 to 5).
- Aggregation Strategy: Used an ANY-style aggregation: if any of the $k$ samples was harmful, the interaction was classified as unsafe.
- Metrics: ROC-AUC, PR-AUC (crucial for imbalanced data), and Balanced Accuracy. Bootstrap resampling was used for confidence intervals.
- Ground Truth: Constructed using heuristic judges (refusal-based and content-based).

3. Key Contributions & Findings

A. Single-Output Evaluation Systematically Underestimates Vulnerability

Finding: As the number of sampled generations ( $k$ ) increases, the observed jailbreak success rate rises significantly.
Implication: A single generation often misses harmful behaviors that appear in subsequent stochastic samples. Single-output evaluation provides an incomplete and overly optimistic view of model safety.
Diminishing Returns: The most significant increase in detected vulnerability occurs when moving from $k=1$ to moderate sampling (e.g., $k=3$ ). Larger budgets ( $k>5$ ) yield diminishing returns while increasing computational cost.

B. The "Alignment Paradox" in Detection

Finding: Stronger alignment reduces the frequency of harmful outputs but simultaneously makes detection more difficult.
Mechanism: As alignment improves, harmful signals become sparse (rare-event regime). This leads to:
- High instability in estimated success rates.
- Severe class imbalance issues.
- Detection performance becoming highly sensitive to noise and threshold selection.
Result: A model that is "safer" (fewer harmful outputs) is paradoxically harder to audit effectively using output-level signals because the observable signal-to-noise ratio drops.

C. Cross-Generator Generalization is Partial

Finding: Detection signals are partially transferable across models but not fully invariant.
Observation:
- High Transfer: Within related model families (e.g., Qwen-0.5B to Qwen-1.5B), detectors achieve high ROC-AUC (e.g., 0.95) due to shared architectural and alignment patterns.
- Low/Asymmetric Transfer: Transfer between distant models (e.g., DistilGPT2 to Gemma) is weaker and often asymmetric. Detectors trained on weakly aligned models capture explicit lexical cues that transfer well, while those trained on strongly aligned models struggle to generalize due to the scarcity of signals.

D. Lexical Detectors Learn Style, Not Intent

Finding: Analysis of TF-IDF feature weights reveals that lexical detectors do not primarily learn "harmful intent."
Mechanism:
- Positive Features: The model associates successful jailbreaks with procedural/instructional discourse (e.g., "start by," "here are," "you should").
- False Positives: Safe responses that use structured explanations or refusal lists (e.g., "Here are some resources...") are often misclassified as jailbreaks because they mimic the style of the attack.
- False Negatives: Harmful content expressed implicitly or narratively (without procedural markers) is often missed.
Conclusion: Lexical detectors act more as stylistic classifiers than true safety detectors, limiting their robustness.

4. Significance and Recommendations

Shift to Multi-Sample Auditing: The paper argues for a paradigm shift from single-output evaluation to moderate multi-sample auditing (e.g., $k=3$ ). This provides a more reliable estimate of a model's true capability to generate harm without incurring prohibitive computational costs.
Redefining Safety Metrics: Safety evaluations must account for the stochastic nature of LLMs. A model's risk profile should be defined by its capability to produce harm under repeated sampling, not just its single-instance output.
Limitations of Current Detectors: The study highlights that current output-level detection methods (both lexical and inconsistency-based) struggle under strong alignment. Future work must move beyond surface-level cues and develop methods that can detect rare, subtle, and implicit harmful behaviors.
Practical Deployment: For real-world safety pipelines, incorporating a small number of stochastic generations per prompt is a practical trade-off that significantly improves risk coverage over single-response checks.

Summary

This paper demonstrates that single-output evaluation is insufficient for assessing LLM safety because it ignores the stochastic variability where harmful behaviors may only manifest in a subset of generations. By introducing multi-generation sampling, the authors show that vulnerability is systematically underestimated in current benchmarks. Furthermore, they reveal that as models become safer, detection becomes harder due to signal sparsity, and that current lexical detectors often mistake instructional style for harmful intent. The study advocates for sampling-aware auditing protocols to achieve a more realistic and robust evaluation of AI safety.