💻 computer science

LLM-Safety Evaluations Lack Robustness

This paper argues that current LLM safety research is hindered by significant noise and inconsistencies across the evaluation pipeline, and it proposes systematic guidelines to improve the robustness, fairness, and comparability of future attack and defense assessments.

Original authors: Tim Beyer, Sophie Xhonneux, Simon Geisler, Gauthier Gidel, Leo Schwinn, Stephan Günnemann

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tim Beyer, Sophie Xhonneux, Simon Geisler, Gauthier Gidel, Leo Schwinn, Stephan Günnemann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge which of two new security guards is better at stopping intruders. You set up a test where you send a series of "tricky questions" to the guards to see if they accidentally let a bad guy in.

This paper argues that the current way we test Large Language Models (LLMs) for safety is like using a broken, inconsistent, and poorly designed test for those security guards. Because the test itself is flawed, we can't truly tell who is the better guard, and progress in making AI safer is getting stuck.

Here is a breakdown of the paper's main points using simple analogies:

1. The Test Questions Are Too Small and Repetitive (Datasets)

Imagine you are testing a guard's ability to spot a thief. Instead of showing them 1,000 different types of thieves, you only show them 50 pictures, and 40 of them look exactly the same.

The Problem: The paper says current safety tests use very small lists of "bad prompts" (often just 100–500). Because the list is so small, the results are full of "noise" (random luck). If you test the same guard on a slightly different list of 50 pictures, the score might change wildly, making it impossible to know if they are actually safe.
The "Subsampling" Issue: Sometimes researchers take a huge list of questions but only pick a tiny, random handful to test. It's like a teacher grading a student based on only three questions out of a 100-question exam. This makes it hard to compare different students fairly because everyone is taking a different, tiny version of the test.
The "English-Only" Blind Spot: Most tests are only in English. But if a guard only speaks English, they might fail a test in Spanish. The paper notes that safety knowledge often changes depending on the language, so testing only in English gives a false sense of security.

2. The Rules of the Game Are Confusing (Algorithms)

Imagine two guards are being tested, but one is allowed to use a flashlight, while the other is forced to work in the dark. Or, one guard is tested with a stopwatch that runs fast, and the other with one that runs slow.

Hidden Settings: The paper points out that tiny, invisible details in how the tests are run change the results drastically. For example, how the computer handles "spaces" between words or what kind of "chat template" is used can change a guard's success rate by 14% or more.
The "Target" Trap: Many attacks try to force the AI to say a specific phrase like "Sure, here is how..." to prove it broke. But if the AI is trained to say "Of course!" instead, the attack fails not because the AI is safe, but because the attacker was using the wrong "key." This makes the AI look safer than it really is.
Unfair Comparisons: Some researchers test their attack method with unlimited computer power, while others test with very little. Comparing them is like comparing a professional athlete running a race with a sprinter who has a head start.

3. The Judges Are Biased and Inconsistent (Evaluation)

After the guard answers the tricky questions, a "Judge" has to decide: "Did they fail?"

The Fragmented Jury: There is no single "Supreme Court" for AI safety. Some researchers use one AI to judge the answers, others use a different AI, and some use humans. These "judges" often disagree with each other. One judge might say a response is safe, while another says it's dangerous, even for the exact same answer.
The "Greedy" Mistake: Most tests force the AI to give the single most likely answer (like a robot that always picks the first thing that comes to mind). But in the real world, AI is like a person who might say different things depending on their mood or how many times you ask. By only testing the "most likely" answer, we miss the times the AI might accidentally say something dangerous when it's "thinking" differently.
The "Over-Refusal" Blind Spot: A safe guard shouldn't just stop bad guys; they shouldn't stop good people either. If a guard refuses to let a harmless person in because they look suspicious, that's a problem called "over-refusal." The paper says most tests ignore this. They only check if the guard stops bad guys, not if they are being too grumpy with good guys.

The "Opposing View" (Why things are the way they are)

The paper also listens to the other side. Some researchers argue:

Small tests are cheaper: Big tests cost a lot of money and time. Small tests let researchers try new ideas quickly.
Perfection is impossible: Language is messy. We will never have a perfect "Judge" that understands every nuance of human conversation. We just have to keep improving the best tools we have.
Progress happens anyway: Even with messy tests, the field is still moving forward. New ideas are being found even if the scoreboard isn't perfect.

The Solution: A Better Rulebook

The authors aren't saying we should stop testing. They are saying we need to fix the rulebook so everyone plays by the same rules. They suggest:

Use bigger, better question lists so the results aren't just luck.
Standardize the settings so everyone uses the same "flashlight" and "stopwatch."
Use multiple judges and have humans double-check the results to catch biases.
Test both sides: Check if the AI stops bad guys and if it lets good guys in.

In short: The paper claims that right now, we are trying to measure how safe AI is with a ruler that keeps changing its own length. Until we fix the ruler, we can't be sure if we are actually making progress.

Technical Summary: LLM-Safety Evaluations Lack Robustness

Problem Statement

The paper argues that current research into Large Language Model (LLM) safety and alignment is significantly hindered by unreliable evaluation pipelines. Despite rapid advancements in model capabilities, the field suffers from "intertwined sources of noise" including small datasets, methodological inconsistencies, and flawed evaluation setups. These issues prevent fair comparisons between attack and defense strategies, obscure genuine progress, and slow down the development of robust safety measures. The authors draw parallels to historical struggles in computer vision adversarial robustness, where faulty evaluations produced misleading feedback that delayed genuine solutions.

Methodology

The authors conduct a systematic analysis of the LLM safety evaluation pipeline, decomposing it into three primary stages: Dataset Curation, Optimization Strategies (Algorithms), and Response Evaluation.

Dataset Analysis: The paper examines the scale, diversity, and composition of existing safety and overrefusal datasets (e.g., AdvBench, XSTest, HarmBench). It applies statistical modeling (Clopper-Pearson intervals) to demonstrate the uncertainty introduced by small sample sizes (typically 100–500 prompts).
Algorithmic Case Studies: The authors investigate implementation details that introduce bias, specifically focusing on:
- Tokenization and Whitespace: Analyzing how incorrect handling of whitespace tokens (e.g., SentencePiece underline tokens) affects optimization.
- Chat Templates and System Prompts: Comparing Attack Success Rates (ASR) across different template configurations (Meta vs. HuggingFace defaults).
- Quantization and Filtering: Evaluating the impact of precision (BF16 vs. Int8) and special token filtering on attack efficacy.
Evaluation Ecosystem Analysis: The study compares different "judge" models (e.g., HarmBench's Llama-2-13B classifier vs. StrongREJECT's Gemma-2B judge) and generation strategies (greedy decoding vs. distributional sampling).
Statistical Validation: The authors perform z-tests on public results from recent high-profile papers (e.g., Guan et al., 2024) to determine if reported improvements are statistically significant given the dataset sizes used.

Key Contributions and Findings

1. Dataset Limitations

Small Sample Sizes: Current datasets are often too small (100–500 prompts) to produce reliable statistical results. The authors demonstrate that with $n=150$ and 50% success, the 95% confidence interval for the success probability is wide ( $[0.417, 0.583]$ ), making it impossible to distinguish between methods with small performance differences.
Fragmentation and Subsampling: The field relies on fragmented datasets, and researchers frequently subsample them due to compute constraints. This leads to inconsistent evaluation environments where reproducing results requires re-implementing baselines on specific subsets, increasing engineering costs.
Lack of Diversity: Datasets are predominantly English-only, single-turn, and repetitive. This creates blind spots, allowing naive attacks (e.g., past-tense or translation attacks) to succeed where they would fail in more diverse, multi-turn, or multi-lingual real-world scenarios.
Ambiguity: Many datasets contain ambiguous prompts (e.g., legal in some jurisdictions but not others) that do not align with a universal definition of harm, introducing noise when comparing models from different developers.

2. Algorithmic Inconsistencies

Implementation Sensitivity: Minor implementation details drastically alter Attack Success Rates (ASR). For instance, the authors found that incorrect handling of whitespace tokens reduced GCG attack success by 28% against Llama-2. Similarly, changing chat templates or quantization levels caused ASR variations between 71% and 85% for the same attack on Llama-3.1.
Budget Inconsistency: Attacks are rarely compared under controlled compute budgets. Some methods maximize ASR by using excessive compute, while others are evaluated at suboptimal operating points, leading to misleading comparisons.
Biased Optimization Targets: Most attacks optimize for rigid target strings (e.g., "Sure, here..."). If a model's natural affirmative response differs from this template, the attack appears less effective than it truly is, or defenses trained on these specific targets fail against adaptive attacks with different objectives.

3. Evaluation Flaws

Greedy Decoding Bias: The dominant use of greedy decoding fails to capture the distributional nature of LLM outputs. Models that appear robust under greedy generation may produce harmful content frequently under realistic sampling strategies (e.g., temperature 1).
Judge Fragmentation and Bias: The ecosystem of automated judges is fragmented. The authors' large-scale study (5.2 million generations) revealed that different judges (HarmBench vs. StrongREJECT) can disagree on ASR by up to 25% for specific attacks and 24% across models.
Lack of Human Verification: Automated judges often have high false positive rates (FPR) for specific models or defenses, leading to over-optimistic results. The authors show that relying solely on automated judges can lead to statistically insignificant claims being reported as highly significant.
Ignored Trade-offs: Defenses rarely report the safety-overrefusal trade-off comprehensively. Many defenses improve safety metrics while significantly degrading the model's ability to answer benign prompts, a critical failure mode often overlooked.

Results and Evidence

Statistical Significance: Re-analysis of Guan et al. (2024) showed that while their internal results were robust, the public dataset results alone were not statistically significant at $\alpha=0.05$ for overrefusal improvements, highlighting the risk of relying on small public benchmarks.
Implementation Impact: Experiments demonstrated that standardizing chat templates and token filtering could change ASR by up to 14%. Incorrect whitespace handling led to a 28% drop in ASR.
Judge Discrepancy: Comparing two popular judges on 5.2 million generations revealed that one judge (StrongREJECT) consistently reported lower ASRs than the other (HarmBench), but the magnitude of the difference varied wildly depending on the specific attack and victim model, complicating cross-paper comparisons.

Significance and Claims

The paper posits that the field of LLM safety is currently unable to generate easily comparable results or make measurable progress due to these evaluation flaws. The authors argue that addressing these issues is not merely a technical refinement but a prerequisite for scientific rigor in the field.

Core Recommendations:

Datasets: Prioritize larger, high-quality benchmarks and avoid ad-hoc subsampling. If subsampling is necessary, use consistent, predefined splits (e.g., a dev split alongside the full benchmark).
Implementation: Standardize and document critical details such as quantization, tokenization, chat templates, and whitespace handling.
Fair Comparisons: Control attacks for effort (compute budget) or success rate. Defenses must consistently report both safety and overrefusal metrics.
Evaluation: Shift from greedy generation to distributional evaluation. Use multiple judge models and include manual verification to detect judge-specific biases.
Incentives: Reviewers and conference organizers must codify these reproducibility requirements into mandatory guidelines to ensure authors adhere to best practices.

The paper concludes that while imperfect benchmarks have historically allowed progress in other ML domains, the unique complexity and high stakes of LLM safety require a more rigorous, transparent, and standardized evaluation pipeline to ensure that safety claims are robust and reliable.